dalli 5.0.5 → 5.0.6
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +115 -0
- data/lib/dalli/client.rb +23 -7
- data/lib/dalli/options.rb +1 -1
- data/lib/dalli/pipelined_deleter.rb +17 -10
- data/lib/dalli/protocol/base.rb +14 -1
- data/lib/dalli/protocol/connection_manager.rb +34 -10
- data/lib/dalli/protocol/key_regularizer.rb +8 -9
- data/lib/dalli/protocol/meta.rb +28 -71
- data/lib/dalli/protocol/request_formatter.rb +73 -26
- data/lib/dalli/protocol/response_buffer.rb +21 -22
- data/lib/dalli/protocol/response_processor.rb +41 -15
- data/lib/dalli/socket.rb +47 -32
- data/lib/dalli/version.rb +1 -1
- metadata +2 -3
- data/lib/dalli/pid_cache.rb +0 -40
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 54f211dbcca14abc18e7e9ffc7e9fe5c992c1e95afab2b6a64e46b6ec32efa8a
|
|
4
|
+
data.tar.gz: f655e7c63cc622876ae47400afd59139eea3b8e19f6585af223fafccad94e659
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: de1470d516e61e2b970c0cfaa0de3afe572124081b300ce643405813c3ba232be08da32fbc3748fbc86b39e2aaea0d6e5803d34cd9722f50c3fc47b325970c67
|
|
7
|
+
data.tar.gz: 1835dc76b3cfebe33a4a8eca38d37dcfa8af84495e0d5af523402ddd4ffa1e3a2a8c808ce4c6e2605325f8d6eded6faa4a574022680528016f091e330e987a3e
|
data/CHANGELOG.md
CHANGED
|
@@ -4,6 +4,121 @@ Dalli Changelog
|
|
|
4
4
|
Unreleased
|
|
5
5
|
==========
|
|
6
6
|
|
|
7
|
+
5.0.6
|
|
8
|
+
==========
|
|
9
|
+
|
|
10
|
+
Performance:
|
|
11
|
+
|
|
12
|
+
- Skip the cas-return flag on quiet `meta_set` requests (#1131)
|
|
13
|
+
- In quiet mode memcached suppresses the `ms` response entirely, so the CAS requested by the `c` flag can never be read; sending it only added two bytes to every request
|
|
14
|
+
- Applies to the bulk-write paths, where quiet sets are emitted: `Dalli::Client#multi` blocks and the pipelined setter
|
|
15
|
+
- Extracted from #1130; thanks to Jianbin Chen for this contribution
|
|
16
|
+
|
|
17
|
+
- Reduce allocations in `KeyRegularizer` and multi-key request paths (#1120)
|
|
18
|
+
- Decomposed `KeyRegularizer#encode` into separate `needs_encoding?` and `encode` calls so the common happy path avoids allocating an intermediate array for the two-element return value
|
|
19
|
+
- Refactored `multi_get`/`multi_set`/`multi_delete` command generation into `RequestFormatter` to share its key-encoding helpers
|
|
20
|
+
- Thanks to Jean Boussier for this contribution
|
|
21
|
+
|
|
22
|
+
- Reduce allocations in `ResponseBuffer` pipelined getk parsing (#1117)
|
|
23
|
+
- `process_single_getk_response` was building a fresh array to return results alongside the updated offset; refactored to store the offset as the last element of the existing tokens array and pop it, saving one allocation per response
|
|
24
|
+
- Also skips trailing nils in the token array
|
|
25
|
+
- Thanks to Jean Boussier for this contribution
|
|
26
|
+
|
|
27
|
+
- Enable frozen string literals in `RequestFormatter` (#1118)
|
|
28
|
+
- Frozen string literals had been inadvertently disabled; re-enabling reduces allocations by ~300,000 objects in a 10,000-iteration `get_multi_cas` benchmark (562 MB → 550 MB total allocated)
|
|
29
|
+
- Thanks to Jean Boussier for this contribution
|
|
30
|
+
|
|
31
|
+
- Reduce allocations in `ResponseProcessor#value_from_tokens` (#1113)
|
|
32
|
+
- `token[1..].to_i` was allocating a new string for every token parsed; replaced with in-place `slice!` followed by a token reset to avoid poisoning subsequent token comparisons
|
|
33
|
+
- Saves 4 allocations per entry in `get_multi_cas` workloads (a hotspot for IdentityCache)
|
|
34
|
+
- Thanks to Jean Boussier for this contribution
|
|
35
|
+
|
|
36
|
+
- Reduce allocations in common operation paths (#1111)
|
|
37
|
+
- Use `Symbol#name` over `Symbol#to_s` to return a frozen string without allocation
|
|
38
|
+
- Skip trace attribute hash construction when OpenTelemetry instrumentation is disabled
|
|
39
|
+
- Use argument forwarding (`...`) in `Client#perform` and `Threadsafe#request` to avoid splat array allocation
|
|
40
|
+
- Use `match?` in `KeyRegularizer#encode` to avoid `MatchData` object allocation
|
|
41
|
+
- Reduces objects allocated by ~26% and memory by ~6% for a simple `get` workload
|
|
42
|
+
- Thanks to Jean Boussier for this contribution
|
|
43
|
+
|
|
44
|
+
- Fix pathological memory behavior in `ResponseBuffer` (#1114)
|
|
45
|
+
- `compact_if_needed` was intended to reclaim memory by slicing off consumed bytes, but `buffer.byteslice(@offset..)` on an unfrozen string causes Ruby to allocate a hidden third string as the copy-on-write owner rather than freeing the original
|
|
46
|
+
- Redesigns buffer management to pass reusable buffer objects directly to `read`/`read_nonblock`, avoiding reallocation on each response read
|
|
47
|
+
- Reduces allocations from ~2.38 GB to ~649 MB in a `get_multi_cas` benchmark over 10,000 iterations
|
|
48
|
+
- Accompanied by new unit tests for `ResponseBuffer` (#1115)
|
|
49
|
+
- Thanks to Jean Boussier for this contribution
|
|
50
|
+
|
|
51
|
+
Features:
|
|
52
|
+
|
|
53
|
+
- `delete_multi` now returns the number of keys found and deleted (#1126)
|
|
54
|
+
- Previously the return value was unspecified; callers (e.g. Rails, see rails/rails#58071) had no way to tell how many keys were actually removed
|
|
55
|
+
- The count is derived from the meta protocol's quiet-mode delete responses with no extra round-trips: successful deletes are suppressed while misses report `NF`, so any response received before the terminator is a key that was not deleted
|
|
56
|
+
- The single-server fast path now shares the pipelined path's bounded retry on transient (`RetryableNetworkError`) network errors, so both paths behave consistently; the returned count is best-effort and may under-report if a network error triggers a retry, since keys deleted before the error are not recounted
|
|
57
|
+
- Thanks to Iliana Hadzhiatanasova for this contribution
|
|
58
|
+
|
|
59
|
+
Bug Fixes:
|
|
60
|
+
|
|
61
|
+
- Raise instead of returning a truncated value when the peer closes mid-response (#1135)
|
|
62
|
+
- `IO#read(count)` on a blocking socket accumulates across TCP chunks and hands back a shorter buffer (or `nil`) in only one case: the stream hit EOF. That short buffer was passed through as the response body, so a memcached restart, proxy drop, or load balancer timeout partway through a response could surface a truncated but still decodable value to the caller, indistinguishable from a real one
|
|
63
|
+
- A short read is now treated as the premature EOF it is, raising and closing the dirty socket so the request is retried on a fresh connection
|
|
64
|
+
- CRuby only; the JRuby path already used `Socket#readfull`, which enforces the same contract
|
|
65
|
+
- Extracted from #1130; thanks to Ian Ker-Seymer for the original fix and Jianbin Chen for the port
|
|
66
|
+
|
|
67
|
+
- Tear down the connection when a non-`StandardError` aborts a request (#1136)
|
|
68
|
+
- `Async::Stop` and `Thread#kill` descend from `Exception` rather than `StandardError`, so the rescue clauses in `Protocol::Base#request` never saw them; a scheduler cancelling a fiber parked on a response read skipped `close` entirely, leaving the connection marked as having a request in progress with partial response bytes still unread on the wire, and returning that half-used client to the pool under `connection_pool`
|
|
69
|
+
- `Protocol::Base#request` now closes in an `ensure` unless the request ran to completion, and `ConnectionManager#close` performs its state cleanup in an `ensure` so a second cancellation landing inside `@sock.close` cannot leave the socket non-nil with the request still marked in progress
|
|
70
|
+
- `Dalli::DalliError` and `Dalli::MarshalError` now close the connection at the point of failure rather than at the start of the next request; those paths already left the request in progress and `ConnectionManager#confirm_ready!` closed on the next call, so this changes when the close happens rather than adding one
|
|
71
|
+
- Extracted from #1130; thanks to Dan Mayer for the original fix and Jianbin Chen for the port
|
|
72
|
+
|
|
73
|
+
- Fix `ResponseBuffer` compaction logic (#1119)
|
|
74
|
+
- `COMPACT_THRESHOLD` was removed in #1116 as apparently unused, but the constant was referenced by the compaction guard; its absence silently disabled buffer compaction
|
|
75
|
+
- Restored the constant, corrected the compaction condition, and improved the buffer-shrinking implementation to use `String#bytesplice` (backed by `memmove`) for true in-place compaction
|
|
76
|
+
- Adds targeted tests covering the compaction threshold and shrink behavior
|
|
77
|
+
- Thanks to Jean Boussier for this contribution
|
|
78
|
+
|
|
79
|
+
Maintenance:
|
|
80
|
+
|
|
81
|
+
- Scope `StrictWarnings` to Dalli's own source (#1134)
|
|
82
|
+
- The test suite runs under `-w` and prepends a hook to `Warning.singleton_class` that turns warnings into failures, but that hook is global: a warning emitted while loading any third-party gem aborted the whole suite before a single test ran
|
|
83
|
+
- `json` 2.21.2's pure-Ruby generator (used on JRuby, where the C extension is unavailable) warns `method redefined; discarding old to_hash` at require time, which took the `jruby-10` CI job red with no change to Dalli
|
|
84
|
+
- Warnings are now attributed to a source file and only raise for `lib/` and `test/`; attribution prefers the location Ruby embeds in the message, since the stack at that point describes the require chain rather than the offending code
|
|
85
|
+
- Portable attribution of `Kernel#warn` callers also required walking the stack rather than indexing it (Ruby 3.3/3.4 push an `<internal:warning>` frame that 4.0 does not), skipping RubyGems' `Kernel#warn` shim (active on JRuby but not CRuby), and resolving relative backtrace paths
|
|
86
|
+
|
|
87
|
+
- Make raw and namespace fast path tests actually use those options (#1129)
|
|
88
|
+
- Followup to #1127: the `raw` and `namespace` variants passed those options to the helper that starts memcached, which configures the client the tests then discarded, so neither option was ever exercised
|
|
89
|
+
- Passes the options to the client under test and adds assertions that fail if they are absent
|
|
90
|
+
- Thanks to Iliana Hadzhiatanasova for this contribution
|
|
91
|
+
|
|
92
|
+
- Benchmark `set_multi` and add a `delete_multi` target (#1132)
|
|
93
|
+
- Enables the two `set_multi` reports that were commented out pending the arrival of `set_multi`, resolving the accompanying TODO
|
|
94
|
+
- Adds a `delete_multi` target comparing the pipelined path against N single deletes
|
|
95
|
+
- Extracted from #1130; thanks to Jianbin Chen for this contribution
|
|
96
|
+
|
|
97
|
+
- Bump CI memcached to 1.6.41 and run benchmarks on pull requests (#1133)
|
|
98
|
+
- The tests workflow moves from 1.6.40 to 1.6.41; the benchmarks and profile workflows had drifted back on 1.6.23
|
|
99
|
+
- Extracted from #1130; thanks to Jianbin Chen for this contribution
|
|
100
|
+
|
|
101
|
+
- Disable RuboCop metrics cops (#1128)
|
|
102
|
+
- Thanks to Jean Boussier for this contribution
|
|
103
|
+
|
|
104
|
+
- Remove `PIDCache` module (#1125)
|
|
105
|
+
- `Process.pid` is cached natively by Ruby 3.3+ (via https://bugs.ruby-lang.org/issues/19443), making the manual cache unnecessary now that Dalli requires Ruby 3.3+
|
|
106
|
+
- Thanks to Jean Boussier for this contribution
|
|
107
|
+
|
|
108
|
+
- Remove unused `COMPACT_THRESHOLD` constant from `ResponseBuffer` (#1116)
|
|
109
|
+
- Followup cleanup after the buffer management redesign in #1114
|
|
110
|
+
- Note: subsequently found to be in use; restored and corrected in #1119
|
|
111
|
+
- Thanks to Jean Boussier for this contribution
|
|
112
|
+
|
|
113
|
+
- Use `String#byteindex` instead of `String#index` when searching for the response terminator in `getk_response_from_buffer` (#1112)
|
|
114
|
+
- The result feeds directly into `byteslice`; `byteindex` makes the intent explicit, though both return the same value since the buffer encoding is always `BINARY`
|
|
115
|
+
- Thanks to Jean Boussier for this contribution
|
|
116
|
+
|
|
117
|
+
- Make single-server fast path tests actually exercise the fast path (#1127)
|
|
118
|
+
- The batch-operation tests built clients through a helper that registers two address aliases for the same memcached process, so every client had a 2-server ring and the tests always ran through the pipelined path instead of the single-server fast path
|
|
119
|
+
- Adds a `single_server_client` test helper that builds a client with a single address, and uses it in the affected `get_multi`, `set_multi`, and `delete_multi` tests
|
|
120
|
+
- Thanks to Iliana Hadzhiatanasova for this contribution
|
|
121
|
+
|
|
7
122
|
5.0.5
|
|
8
123
|
==========
|
|
9
124
|
|
data/lib/dalli/client.rb
CHANGED
|
@@ -364,12 +364,15 @@ module Dalli
|
|
|
364
364
|
# it batches requests by server and uses quiet mode.
|
|
365
365
|
#
|
|
366
366
|
# @param keys [Array<String>] keys to delete
|
|
367
|
-
# @return [
|
|
367
|
+
# @return [Integer] the number of keys that were found and deleted. This is
|
|
368
|
+
# best-effort: on a network error the operation is retried, and keys
|
|
369
|
+
# deleted before the error are not recounted, so the result may
|
|
370
|
+
# under-report the number actually removed when a failure occurs.
|
|
368
371
|
#
|
|
369
372
|
# Example:
|
|
370
373
|
# client.delete_multi(['key1', 'key2', 'key3'])
|
|
371
374
|
def delete_multi(keys)
|
|
372
|
-
return if keys.empty?
|
|
375
|
+
return 0 if keys.empty?
|
|
373
376
|
|
|
374
377
|
Instrumentation.trace('delete_multi', multi_trace_attrs('delete_multi', keys.size, keys)) do
|
|
375
378
|
if ring.servers.size == 1
|
|
@@ -569,11 +572,18 @@ module Dalli
|
|
|
569
572
|
|
|
570
573
|
def single_server_delete_multi(keys)
|
|
571
574
|
validated_keys = keys.map { |k| @key_manager.validate_key(k.to_s) }
|
|
572
|
-
return unless (server = single_server)
|
|
575
|
+
return 0 unless (server = single_server)
|
|
573
576
|
|
|
574
577
|
server.request(:delete_multi_req, validated_keys)
|
|
578
|
+
rescue Dalli::RetryableNetworkError => e
|
|
579
|
+
# Mirror the pipelined path: retry transient errors so a momentary blip
|
|
580
|
+
# still yields a real count. Bounded by the server's socket_max_failures,
|
|
581
|
+
# after which a hard NetworkError is raised and handled below.
|
|
582
|
+
Dalli.logger.debug { e.inspect }
|
|
583
|
+
Dalli.logger.debug { 'retrying single-server delete_multi because of network error' }
|
|
584
|
+
retry
|
|
575
585
|
rescue Dalli::NetworkError
|
|
576
|
-
|
|
586
|
+
0
|
|
577
587
|
end
|
|
578
588
|
|
|
579
589
|
def get_multi_attributes(keys)
|
|
@@ -651,7 +661,7 @@ module Dalli
|
|
|
651
661
|
# operation times out.
|
|
652
662
|
##
|
|
653
663
|
# rubocop:disable Naming/MethodParameterName
|
|
654
|
-
def perform(op, key,
|
|
664
|
+
def perform(op, key, ...)
|
|
655
665
|
# rubocop:enable Naming/MethodParameterName
|
|
656
666
|
return yield if block_given?
|
|
657
667
|
|
|
@@ -659,8 +669,14 @@ module Dalli
|
|
|
659
669
|
key = @key_manager.validate_key(key)
|
|
660
670
|
|
|
661
671
|
server = ring.server_for_key(key)
|
|
662
|
-
|
|
663
|
-
|
|
672
|
+
|
|
673
|
+
if Instrumentation.enabled?
|
|
674
|
+
op_name = op.name
|
|
675
|
+
Instrumentation.trace(op_name, trace_attrs(op_name, key, server)) do
|
|
676
|
+
server.request(op, key, ...)
|
|
677
|
+
end
|
|
678
|
+
else
|
|
679
|
+
server.request(op, key, ...)
|
|
664
680
|
end
|
|
665
681
|
rescue RetryableNetworkError => e
|
|
666
682
|
Dalli.logger.debug { e.inspect }
|
data/lib/dalli/options.rb
CHANGED
|
@@ -16,16 +16,19 @@ module Dalli
|
|
|
16
16
|
# Deletes multiple keys from memcached.
|
|
17
17
|
#
|
|
18
18
|
# @param keys [Array<String>] keys to delete
|
|
19
|
-
# @return [
|
|
19
|
+
# @return [Integer] the number of keys that were deleted. This is
|
|
20
|
+
# best-effort: on a network error the operation is retried, and keys
|
|
21
|
+
# deleted before the error are not recounted, so the result may
|
|
22
|
+
# under-report the number actually removed when a failure occurs.
|
|
20
23
|
##
|
|
21
24
|
def process(keys)
|
|
22
|
-
return if keys.empty?
|
|
25
|
+
return 0 if keys.empty?
|
|
23
26
|
|
|
24
27
|
@ring.lock do
|
|
25
|
-
|
|
26
|
-
finish_requests(
|
|
28
|
+
groups = setup_requests(keys)
|
|
29
|
+
finish_requests(groups)
|
|
27
30
|
end
|
|
28
|
-
rescue
|
|
31
|
+
rescue Dalli::RetryableNetworkError => e
|
|
29
32
|
Dalli.logger.debug { e.inspect }
|
|
30
33
|
Dalli.logger.debug { 'retrying pipelined deletes because of network error' }
|
|
31
34
|
retry
|
|
@@ -36,7 +39,7 @@ module Dalli
|
|
|
36
39
|
def setup_requests(keys)
|
|
37
40
|
groups = groups_for_keys(keys)
|
|
38
41
|
make_delete_requests(groups)
|
|
39
|
-
groups
|
|
42
|
+
groups
|
|
40
43
|
end
|
|
41
44
|
|
|
42
45
|
##
|
|
@@ -45,24 +48,28 @@ module Dalli
|
|
|
45
48
|
##
|
|
46
49
|
def make_delete_requests(groups)
|
|
47
50
|
groups.each do |server, keys_for_server|
|
|
48
|
-
keys_for_server.
|
|
51
|
+
keys_for_server.select! do |key|
|
|
49
52
|
server.request(:pipelined_delete, key)
|
|
53
|
+
true
|
|
50
54
|
rescue DalliError, NetworkError => e
|
|
51
55
|
Dalli.logger.debug { e.inspect }
|
|
52
56
|
Dalli.logger.debug { "unable to delete key #{key} for server #{server.name}" }
|
|
57
|
+
false
|
|
53
58
|
end
|
|
54
59
|
end
|
|
55
60
|
end
|
|
56
61
|
|
|
57
62
|
##
|
|
58
63
|
# Sends noop to each server to flush responses and ensure all deletes complete.
|
|
64
|
+
# Returns the total successful deletes across servers.
|
|
59
65
|
##
|
|
60
|
-
def finish_requests(
|
|
61
|
-
|
|
62
|
-
server.request(:
|
|
66
|
+
def finish_requests(groups)
|
|
67
|
+
groups.sum do |server, keys_for_server|
|
|
68
|
+
server.request(:finish_pipelined_delete, keys_for_server.size)
|
|
63
69
|
rescue DalliError, NetworkError => e
|
|
64
70
|
Dalli.logger.debug { e.inspect }
|
|
65
71
|
Dalli.logger.debug { "unable to complete pipelined delete on server #{server.name}" }
|
|
72
|
+
0
|
|
66
73
|
end
|
|
67
74
|
end
|
|
68
75
|
|
data/lib/dalli/protocol/base.rb
CHANGED
|
@@ -40,12 +40,14 @@ module Dalli
|
|
|
40
40
|
verify_state(opkey)
|
|
41
41
|
|
|
42
42
|
begin
|
|
43
|
+
request_completed = false
|
|
43
44
|
@connection_manager.start_request!
|
|
44
45
|
response = send(opkey, *args)
|
|
45
46
|
|
|
46
47
|
# pipelined_get/pipelined_get_interleaved emit query but don't read the response(s)
|
|
47
48
|
@connection_manager.finish_request! unless %i[pipelined_get pipelined_get_interleaved].include?(opkey)
|
|
48
49
|
|
|
50
|
+
request_completed = true
|
|
49
51
|
response
|
|
50
52
|
rescue Dalli::MarshalError => e
|
|
51
53
|
log_marshal_err(args.first, e)
|
|
@@ -56,6 +58,18 @@ module Dalli
|
|
|
56
58
|
log_unexpected_err(e)
|
|
57
59
|
close
|
|
58
60
|
raise
|
|
61
|
+
ensure
|
|
62
|
+
# If the begin block didn't complete -- any exception, including a
|
|
63
|
+
# non-StandardError such as Async::Stop or Thread#kill, which the
|
|
64
|
+
# rescue clauses above never see -- tear down the connection so a
|
|
65
|
+
# half-used client isn't returned to the pool.
|
|
66
|
+
#
|
|
67
|
+
# The local flag is deliberate: reading $ERROR_INFO here would see
|
|
68
|
+
# the *outer* exception when `request` is called from inside a
|
|
69
|
+
# rescue clause (a common cache-fallback shape), and would then
|
|
70
|
+
# falsely tear down the pipelined_get happy path, which leaves
|
|
71
|
+
# @request_in_progress true on purpose until the caller drains.
|
|
72
|
+
close unless request_completed
|
|
59
73
|
end
|
|
60
74
|
end
|
|
61
75
|
|
|
@@ -95,7 +109,6 @@ module Dalli
|
|
|
95
109
|
# When a block is given, yields (key, value, cas) for each response,
|
|
96
110
|
# avoiding intermediate Hash allocation. Returns nil.
|
|
97
111
|
# Without a block, returns a Hash of { key => [value, cas] }.
|
|
98
|
-
# rubocop:disable Metrics/AbcSize, Metrics/CyclomaticComplexity, Metrics/PerceivedComplexity
|
|
99
112
|
def pipeline_next_responses(&block)
|
|
100
113
|
reconnect_on_pipeline_complete!
|
|
101
114
|
values = nil
|
|
@@ -4,8 +4,6 @@ require 'English'
|
|
|
4
4
|
require 'socket'
|
|
5
5
|
require 'timeout'
|
|
6
6
|
|
|
7
|
-
require 'dalli/pid_cache'
|
|
8
|
-
|
|
9
7
|
module Dalli
|
|
10
8
|
module Protocol
|
|
11
9
|
##
|
|
@@ -54,7 +52,7 @@ module Dalli
|
|
|
54
52
|
|
|
55
53
|
@sock = memcached_socket
|
|
56
54
|
@sock.sync = false # Enable buffered I/O for better performance
|
|
57
|
-
@pid =
|
|
55
|
+
@pid = Process.pid
|
|
58
56
|
@request_in_progress = false
|
|
59
57
|
rescue SystemCallError, *TIMEOUT_ERRORS, EOFError, SocketError => e
|
|
60
58
|
# SocketError = DNS resolution failure
|
|
@@ -117,10 +115,16 @@ module Dalli
|
|
|
117
115
|
@sock.close
|
|
118
116
|
rescue StandardError
|
|
119
117
|
nil
|
|
118
|
+
ensure
|
|
119
|
+
# A non-StandardError (e.g. a second Async::Stop fired into the
|
|
120
|
+
# fiber while it is already inside this cleanup) can escape
|
|
121
|
+
# @sock.close; run the state cleanup unconditionally so the client
|
|
122
|
+
# isn't returned to the pool with a half-closed socket and
|
|
123
|
+
# @request_in_progress == true.
|
|
124
|
+
@sock = nil
|
|
125
|
+
@pid = nil
|
|
126
|
+
abort_request!
|
|
120
127
|
end
|
|
121
|
-
@sock = nil
|
|
122
|
-
@pid = nil
|
|
123
|
-
abort_request!
|
|
124
128
|
end
|
|
125
129
|
|
|
126
130
|
def connected?
|
|
@@ -165,12 +169,16 @@ module Dalli
|
|
|
165
169
|
end
|
|
166
170
|
else
|
|
167
171
|
def read(count)
|
|
168
|
-
|
|
172
|
+
read_bytes(count)
|
|
169
173
|
rescue SystemCallError, *TIMEOUT_ERRORS, *SSL_ERRORS, EOFError => e
|
|
170
174
|
error_on_request!(e)
|
|
171
175
|
end
|
|
172
176
|
end
|
|
173
177
|
|
|
178
|
+
# Alias for callers that want to make the exact-length contract explicit
|
|
179
|
+
# at the call site.
|
|
180
|
+
alias read_exact read
|
|
181
|
+
|
|
174
182
|
def write(bytes)
|
|
175
183
|
@sock.write(bytes)
|
|
176
184
|
rescue SystemCallError, *TIMEOUT_ERRORS, *SSL_ERRORS, IOError => e
|
|
@@ -193,8 +201,8 @@ module Dalli
|
|
|
193
201
|
|
|
194
202
|
# Non-blocking read. Here to support the operation
|
|
195
203
|
# of the get_multi operation
|
|
196
|
-
def
|
|
197
|
-
@sock.read_available
|
|
204
|
+
def read_available(...)
|
|
205
|
+
@sock.read_available(...)
|
|
198
206
|
end
|
|
199
207
|
|
|
200
208
|
def max_allowed_failures
|
|
@@ -252,7 +260,7 @@ module Dalli
|
|
|
252
260
|
end
|
|
253
261
|
|
|
254
262
|
def fork_detected?
|
|
255
|
-
@pid && @pid !=
|
|
263
|
+
@pid && @pid != Process.pid
|
|
256
264
|
end
|
|
257
265
|
|
|
258
266
|
def log_down_detected
|
|
@@ -273,6 +281,22 @@ module Dalli
|
|
|
273
281
|
time = Time.now - @down_at
|
|
274
282
|
Dalli.logger.warn { format('%<name>s is back (downtime was %<time>.3f seconds)', name: name, time: time) }
|
|
275
283
|
end
|
|
284
|
+
|
|
285
|
+
private
|
|
286
|
+
|
|
287
|
+
# Reads exactly `count` bytes. IO#read(count) on a blocking socket blocks
|
|
288
|
+
# until it has `count` bytes, accumulating across TCP chunks internally,
|
|
289
|
+
# and only hands back a shorter (or nil) buffer when the stream hits EOF.
|
|
290
|
+
# So a short read means the peer closed mid-response: raise EOFError and
|
|
291
|
+
# let read's existing `rescue EOFError` route it through
|
|
292
|
+
# error_on_request!, which closes the dirty socket for a retry on a fresh
|
|
293
|
+
# connection (and preserves the $ERROR_INFO context down! relies on).
|
|
294
|
+
def read_bytes(count)
|
|
295
|
+
buffer = @sock.read(count)
|
|
296
|
+
return buffer if buffer && buffer.bytesize == count
|
|
297
|
+
|
|
298
|
+
raise EOFError, "EOF reading #{count} bytes; received #{buffer ? buffer.bytesize : 0}"
|
|
299
|
+
end
|
|
276
300
|
end
|
|
277
301
|
end
|
|
278
302
|
end
|
|
@@ -9,19 +9,18 @@ module Dalli
|
|
|
9
9
|
# allowed.
|
|
10
10
|
# memcached supports the use of base64 hashes for keys containing
|
|
11
11
|
# whitespace or non-ASCII characters, provided the 'b' flag is included in the request.
|
|
12
|
-
|
|
13
|
-
|
|
12
|
+
module KeyRegularizer
|
|
13
|
+
module_function
|
|
14
14
|
|
|
15
|
-
def
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
strict_base64_encoded = [key].pack('m0')
|
|
19
|
-
[strict_base64_encoded, true]
|
|
15
|
+
def required?(key)
|
|
16
|
+
!key.ascii_only? || /\s/.match?(key)
|
|
20
17
|
end
|
|
21
18
|
|
|
22
|
-
def
|
|
23
|
-
|
|
19
|
+
def encode(key)
|
|
20
|
+
[key].pack('m0')
|
|
21
|
+
end
|
|
24
22
|
|
|
23
|
+
def decode(encoded_key)
|
|
25
24
|
strict_base64_decoded = encoded_key.unpack1('m0')
|
|
26
25
|
strict_base64_decoded.force_encoding(Encoding::UTF_8)
|
|
27
26
|
end
|
data/lib/dalli/protocol/meta.rb
CHANGED
|
@@ -24,34 +24,29 @@ module Dalli
|
|
|
24
24
|
|
|
25
25
|
# Retrieval Commands
|
|
26
26
|
def get(key, options = nil)
|
|
27
|
-
encoded_key, base64 = KeyRegularizer.encode(key)
|
|
28
27
|
# Skip bitflags in raw mode - saves 2 bytes per request and skips parsing
|
|
29
28
|
skip_flags = raw_mode? || (options && options[:raw])
|
|
30
|
-
req = RequestFormatter.meta_get(key:
|
|
29
|
+
req = RequestFormatter.meta_get(key: key, skip_flags: skip_flags)
|
|
31
30
|
flushed_write(req)
|
|
32
31
|
response_processor.meta_get_with_value(cache_nils: cache_nils?(options))
|
|
33
32
|
end
|
|
34
33
|
|
|
35
34
|
def quiet_get_request(key)
|
|
36
|
-
encoded_key, base64 = KeyRegularizer.encode(key)
|
|
37
35
|
# Skip bitflags in raw mode - saves 2 bytes per request and skips parsing
|
|
38
|
-
RequestFormatter.meta_get(key:
|
|
39
|
-
skip_flags: raw_mode?)
|
|
36
|
+
RequestFormatter.meta_get(key: key, return_cas: true, quiet: true, skip_flags: raw_mode?)
|
|
40
37
|
end
|
|
41
38
|
|
|
42
39
|
def gat(key, ttl, options = nil)
|
|
43
40
|
ttl = TtlSanitizer.sanitize(ttl)
|
|
44
|
-
encoded_key, base64 = KeyRegularizer.encode(key)
|
|
45
41
|
skip_flags = raw_mode? || (options && options[:raw])
|
|
46
|
-
req = RequestFormatter.meta_get(key:
|
|
42
|
+
req = RequestFormatter.meta_get(key: key, ttl: ttl, skip_flags: skip_flags)
|
|
47
43
|
flushed_write(req)
|
|
48
44
|
response_processor.meta_get_with_value(cache_nils: cache_nils?(options))
|
|
49
45
|
end
|
|
50
46
|
|
|
51
47
|
def touch(key, ttl)
|
|
52
48
|
ttl = TtlSanitizer.sanitize(ttl)
|
|
53
|
-
|
|
54
|
-
req = RequestFormatter.meta_get(key: encoded_key, ttl: ttl, value: false, base64: base64)
|
|
49
|
+
req = RequestFormatter.meta_get(key: key, ttl: ttl, value: false)
|
|
55
50
|
flushed_write(req)
|
|
56
51
|
response_processor.meta_get_without_value
|
|
57
52
|
end
|
|
@@ -59,8 +54,7 @@ module Dalli
|
|
|
59
54
|
# TODO: This is confusing, as there's a cas command in memcached
|
|
60
55
|
# and this isn't it. Maybe rename? Maybe eliminate?
|
|
61
56
|
def cas(key)
|
|
62
|
-
|
|
63
|
-
req = RequestFormatter.meta_get(key: encoded_key, value: true, return_cas: true, base64: base64)
|
|
57
|
+
req = RequestFormatter.meta_get(key: key, value: true, return_cas: true)
|
|
64
58
|
flushed_write(req)
|
|
65
59
|
response_processor.meta_get_with_value_and_cas
|
|
66
60
|
end
|
|
@@ -90,9 +84,8 @@ module Dalli
|
|
|
90
84
|
# - :hit_before - true/false if previously accessed (only if return_hit_status: true)
|
|
91
85
|
# - :last_access - seconds since last access (only if return_last_access: true)
|
|
92
86
|
def meta_get(key, options = {})
|
|
93
|
-
encoded_key, base64 = KeyRegularizer.encode(key)
|
|
94
87
|
req = RequestFormatter.meta_get(
|
|
95
|
-
key:
|
|
88
|
+
key: key, value: true, return_cas: true,
|
|
96
89
|
vivify_ttl: options[:vivify_ttl], recache_ttl: options[:recache_ttl],
|
|
97
90
|
return_hit_status: options[:return_hit_status],
|
|
98
91
|
return_last_access: options[:return_last_access], skip_lru_bump: options[:skip_lru_bump]
|
|
@@ -112,8 +105,7 @@ module Dalli
|
|
|
112
105
|
# @param cas [Integer] optional CAS value for compare-and-swap
|
|
113
106
|
# @return [Boolean] true if successful
|
|
114
107
|
def delete_stale(key, cas = nil)
|
|
115
|
-
|
|
116
|
-
req = RequestFormatter.meta_delete(key: encoded_key, cas: cas, base64: base64, stale: true)
|
|
108
|
+
req = RequestFormatter.meta_delete(key: key, cas: cas, stale: true)
|
|
117
109
|
flushed_write(req)
|
|
118
110
|
response_processor.meta_delete
|
|
119
111
|
end
|
|
@@ -140,14 +132,12 @@ module Dalli
|
|
|
140
132
|
response_processor.meta_set_with_cas unless quiet?
|
|
141
133
|
end
|
|
142
134
|
|
|
143
|
-
# rubocop:disable Metrics/ParameterLists
|
|
144
135
|
def write_storage_req(mode, key, raw_value, ttl = nil, cas = nil, options = {}, quiet: quiet?)
|
|
145
136
|
(value, bitflags) = @value_marshaller.store(key, raw_value, options)
|
|
146
137
|
ttl = TtlSanitizer.sanitize(ttl) if ttl
|
|
147
|
-
|
|
148
|
-
req = RequestFormatter.meta_set(key: encoded_key, value: value,
|
|
138
|
+
req = RequestFormatter.meta_set(key: key, value: value,
|
|
149
139
|
bitflags: bitflags, cas: cas,
|
|
150
|
-
ttl: ttl, mode: mode, quiet: quiet
|
|
140
|
+
ttl: ttl, mode: mode, quiet: quiet)
|
|
151
141
|
write("#{req}#{value}#{TERMINATOR}")
|
|
152
142
|
@connection_manager.flush unless quiet
|
|
153
143
|
end
|
|
@@ -166,8 +156,7 @@ module Dalli
|
|
|
166
156
|
# rubocop:disable Metrics/ParameterLists
|
|
167
157
|
def write_append_prepend_req(mode, key, value, ttl = nil, cas = nil, _options = {})
|
|
168
158
|
ttl = TtlSanitizer.sanitize(ttl) if ttl
|
|
169
|
-
|
|
170
|
-
req = RequestFormatter.meta_set(key: encoded_key, value: value, base64: base64,
|
|
159
|
+
req = RequestFormatter.meta_set(key: key, value: value,
|
|
171
160
|
cas: cas, ttl: ttl, mode: mode, quiet: quiet?)
|
|
172
161
|
write("#{req}#{value}#{TERMINATOR}")
|
|
173
162
|
@connection_manager.flush unless quiet?
|
|
@@ -176,9 +165,7 @@ module Dalli
|
|
|
176
165
|
|
|
177
166
|
# Delete Commands
|
|
178
167
|
def delete(key, cas)
|
|
179
|
-
|
|
180
|
-
req = RequestFormatter.meta_delete(key: encoded_key, cas: cas,
|
|
181
|
-
base64: base64, quiet: quiet?)
|
|
168
|
+
req = RequestFormatter.meta_delete(key: key, cas: cas, quiet: quiet?)
|
|
182
169
|
write(req)
|
|
183
170
|
@connection_manager.flush unless quiet?
|
|
184
171
|
response_processor.meta_delete unless quiet?
|
|
@@ -187,11 +174,15 @@ module Dalli
|
|
|
187
174
|
# Pipelined delete - writes a quiet delete request without reading response.
|
|
188
175
|
# Used by PipelinedDeleter for bulk operations.
|
|
189
176
|
def pipelined_delete(key)
|
|
190
|
-
|
|
191
|
-
req = RequestFormatter.meta_delete(key: encoded_key, base64: base64, quiet: true)
|
|
177
|
+
req = RequestFormatter.meta_delete(key: key, quiet: true)
|
|
192
178
|
write(req)
|
|
193
179
|
end
|
|
194
180
|
|
|
181
|
+
def finish_pipelined_delete(sent)
|
|
182
|
+
write_noop
|
|
183
|
+
sent - response_processor.pipelined_delete_non_deletions
|
|
184
|
+
end
|
|
185
|
+
|
|
195
186
|
# Arithmetic Commands
|
|
196
187
|
def decr(key, count, ttl, initial)
|
|
197
188
|
decr_incr false, key, count, ttl, initial
|
|
@@ -203,9 +194,8 @@ module Dalli
|
|
|
203
194
|
|
|
204
195
|
def decr_incr(incr, key, delta, ttl, initial)
|
|
205
196
|
ttl = initial ? TtlSanitizer.sanitize(ttl) : nil # Only set a TTL if we want to set a value on miss
|
|
206
|
-
|
|
207
|
-
|
|
208
|
-
quiet: quiet?, base64: base64))
|
|
197
|
+
write(RequestFormatter.meta_arithmetic(key: key, delta: delta, initial: initial, incr: incr, ttl: ttl,
|
|
198
|
+
quiet: quiet?))
|
|
209
199
|
@connection_manager.flush unless quiet?
|
|
210
200
|
response_processor.decr_incr unless quiet?
|
|
211
201
|
end
|
|
@@ -248,23 +238,9 @@ module Dalli
|
|
|
248
238
|
# machinery (IO.select, response buffering, server grouping).
|
|
249
239
|
def read_multi_req(keys)
|
|
250
240
|
is_raw = raw_mode?
|
|
251
|
-
|
|
252
|
-
# In raw mode: "mg <key> v k q s\r\n" (no f flag, key at index 2)
|
|
253
|
-
# Normal mode: "mg <key> v f k q s\r\n" (key at index 3)
|
|
254
|
-
post_get = is_raw ? " v k q s\r\n" : " v f k q s\r\n"
|
|
255
|
-
buffer = ''.b
|
|
256
|
-
keys.each do |key|
|
|
257
|
-
encoded_key, base64 = KeyRegularizer.encode(key)
|
|
258
|
-
if base64
|
|
259
|
-
buffer << 'mg ' << encoded_key << ' b' << post_get
|
|
260
|
-
else
|
|
261
|
-
buffer << 'mg ' << encoded_key << post_get
|
|
262
|
-
end
|
|
263
|
-
end
|
|
264
|
-
buffer << 'mn' << TERMINATOR
|
|
241
|
+
buffer = RequestFormatter.multi_meta_get(keys, skip_flags: is_raw)
|
|
265
242
|
flushed_write(buffer)
|
|
266
243
|
buffer.clear
|
|
267
|
-
|
|
268
244
|
read_multi_get_responses(is_raw)
|
|
269
245
|
end
|
|
270
246
|
|
|
@@ -287,52 +263,33 @@ module Dalli
|
|
|
287
263
|
raw_key = tokens[key_index]
|
|
288
264
|
return unless raw_key
|
|
289
265
|
|
|
290
|
-
key =
|
|
266
|
+
key = raw_key[1..]
|
|
267
|
+
key = KeyRegularizer.decode(key) if tokens.include?('b')
|
|
291
268
|
bitflags = is_raw ? 0 : response_processor.bitflags_from_tokens(tokens)
|
|
292
269
|
[key, @value_marshaller.retrieve(value, bitflags)]
|
|
293
270
|
end
|
|
294
271
|
|
|
295
|
-
# rubocop:disable Metrics/AbcSize
|
|
296
|
-
|
|
297
272
|
# Single-server fast path for set_multi. Inlines request formatting to
|
|
298
273
|
# minimize per-key overhead. Avoids PipelinedSetter server grouping.
|
|
299
274
|
def write_multi_req(pairs, ttl, req_options)
|
|
300
275
|
ttl = TtlSanitizer.sanitize(ttl) if ttl
|
|
301
|
-
|
|
302
|
-
|
|
303
|
-
(value, bitflags) = @value_marshaller.store(key, raw_value, req_options)
|
|
304
|
-
encoded_key, base64 = KeyRegularizer.encode(key)
|
|
305
|
-
# Inline format: "ms <key> <size> c [b] F<flags> T<ttl> MS q\r\n"
|
|
306
|
-
buffer << "ms #{encoded_key} #{value.bytesize} c"
|
|
307
|
-
buffer << ' b' if base64
|
|
308
|
-
buffer << " F#{bitflags}" if bitflags
|
|
309
|
-
buffer << " T#{ttl}" if ttl
|
|
310
|
-
buffer << ' MS q' << TERMINATOR << value << TERMINATOR
|
|
276
|
+
entries = pairs.map do |key, raw_value|
|
|
277
|
+
[key, @value_marshaller.store(key, raw_value, req_options)]
|
|
311
278
|
end
|
|
312
|
-
|
|
279
|
+
|
|
280
|
+
buffer = RequestFormatter.multi_meta_set(entries, ttl: ttl)
|
|
313
281
|
flushed_write(buffer)
|
|
314
282
|
buffer.clear
|
|
315
283
|
response_processor.consume_all_responses_until_mn
|
|
316
284
|
end
|
|
317
|
-
# rubocop:enable Metrics/AbcSize
|
|
318
285
|
|
|
319
286
|
# Single-server fast path for delete_multi. Writes all quiet delete requests
|
|
320
287
|
# terminated by a noop, then consumes all responses.
|
|
321
288
|
def delete_multi_req(keys)
|
|
322
|
-
buffer =
|
|
323
|
-
keys.each do |key|
|
|
324
|
-
encoded_key, base64 = KeyRegularizer.encode(key)
|
|
325
|
-
# Inline format: "md <key> [b] q\r\n"
|
|
326
|
-
if base64
|
|
327
|
-
buffer << 'md ' << encoded_key << ' b q' << TERMINATOR
|
|
328
|
-
else
|
|
329
|
-
buffer << 'md ' << encoded_key << ' q' << TERMINATOR
|
|
330
|
-
end
|
|
331
|
-
end
|
|
332
|
-
buffer << RequestFormatter.meta_noop
|
|
289
|
+
buffer = RequestFormatter.multi_meta_delete(keys)
|
|
333
290
|
flushed_write(buffer)
|
|
334
291
|
buffer.clear
|
|
335
|
-
response_processor.
|
|
292
|
+
keys.size - response_processor.pipelined_delete_non_deletions
|
|
336
293
|
end
|
|
337
294
|
|
|
338
295
|
require_relative 'key_regularizer'
|
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
# frozen_string_literal:
|
|
1
|
+
# frozen_string_literal: true
|
|
2
2
|
|
|
3
3
|
module Dalli
|
|
4
4
|
module Protocol
|
|
@@ -7,14 +7,13 @@ module Dalli
|
|
|
7
7
|
# Class that encapsulates logic for formatting meta protocol requests
|
|
8
8
|
# to memcached.
|
|
9
9
|
##
|
|
10
|
-
|
|
10
|
+
module RequestFormatter
|
|
11
|
+
extend self
|
|
12
|
+
|
|
11
13
|
# Since these are string construction methods, we're going to disable these
|
|
12
14
|
# Rubocop directives. We really can't make this construction much simpler,
|
|
13
15
|
# and introducing an intermediate object seems like overkill.
|
|
14
16
|
#
|
|
15
|
-
# rubocop:disable Metrics/CyclomaticComplexity
|
|
16
|
-
# rubocop:disable Metrics/ParameterLists
|
|
17
|
-
# rubocop:disable Metrics/PerceivedComplexity
|
|
18
17
|
#
|
|
19
18
|
# Meta get flags:
|
|
20
19
|
#
|
|
@@ -35,16 +34,15 @@ module Dalli
|
|
|
35
34
|
# - Z: Client lost the recache race (another client is already regenerating)
|
|
36
35
|
# - h0/h1: Hit status (0 = first access, 1 = previously accessed)
|
|
37
36
|
# - l<N>: Seconds since last access
|
|
38
|
-
def
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
cmd = "mg #{key}"
|
|
37
|
+
def meta_get(key:, value: true, return_cas: false, ttl: nil, quiet: false,
|
|
38
|
+
vivify_ttl: nil, recache_ttl: nil,
|
|
39
|
+
return_hit_status: false, return_last_access: false, skip_lru_bump: false,
|
|
40
|
+
skip_flags: false)
|
|
41
|
+
cmd = "mg #{encoded_key(key)}"
|
|
43
42
|
# In raw mode (skip_flags: true), we don't request bitflags since they're not used.
|
|
44
43
|
# This saves 2 bytes per request and skips parsing on response.
|
|
45
44
|
cmd << (skip_flags ? ' v' : ' v f') if value
|
|
46
45
|
cmd << ' c' if return_cas
|
|
47
|
-
cmd << ' b' if base64
|
|
48
46
|
cmd << " T#{ttl}" if ttl
|
|
49
47
|
cmd << ' k q s' if quiet # Return the key in the response if quiet
|
|
50
48
|
cmd << " N#{vivify_ttl}" if vivify_ttl # Thundering herd: vivify on miss
|
|
@@ -55,9 +53,24 @@ module Dalli
|
|
|
55
53
|
cmd << TERMINATOR
|
|
56
54
|
end
|
|
57
55
|
|
|
58
|
-
def
|
|
56
|
+
def multi_meta_get(keys, skip_flags: false)
|
|
57
|
+
# In raw mode: "mg <key> v k q s\r\n" (no f flag, key at index 2)
|
|
58
|
+
# Normal mode: "mg <key> v f k q s\r\n" (key at index 3)
|
|
59
|
+
post_get = skip_flags ? " v k q s\r\n" : " v f k q s\r\n"
|
|
60
|
+
buffer = ''.b
|
|
61
|
+
keys.each do |key|
|
|
62
|
+
buffer << 'mg ' << encoded_key(key) << post_get
|
|
63
|
+
end
|
|
64
|
+
buffer << 'mn' << TERMINATOR
|
|
65
|
+
end
|
|
66
|
+
|
|
67
|
+
def meta_set(key:, value:, bitflags: nil, cas: nil, ttl: nil, mode: :set, quiet: false)
|
|
68
|
+
base64 = KeyRegularizer.required?(key)
|
|
69
|
+
key = KeyRegularizer.encode(key) if base64
|
|
59
70
|
cmd = "ms #{key} #{value.bytesize}"
|
|
60
|
-
|
|
71
|
+
# Skip the cas-return flag in quiet mode: the response is suppressed,
|
|
72
|
+
# so requesting it only adds bytes to the request.
|
|
73
|
+
cmd << ' c' if !quiet && !%i[append prepend].include?(mode)
|
|
61
74
|
cmd << ' b' if base64
|
|
62
75
|
cmd << " F#{bitflags}" if bitflags
|
|
63
76
|
cmd << cas_string(cas)
|
|
@@ -67,12 +80,29 @@ module Dalli
|
|
|
67
80
|
cmd << TERMINATOR
|
|
68
81
|
end
|
|
69
82
|
|
|
83
|
+
def multi_meta_set(entries, ttl: nil)
|
|
84
|
+
buffer = ''.b
|
|
85
|
+
entries.each do |key, pair|
|
|
86
|
+
value, bitflags = pair
|
|
87
|
+
|
|
88
|
+
base64 = KeyRegularizer.required?(key)
|
|
89
|
+
key = KeyRegularizer.encode(key) if base64
|
|
90
|
+
|
|
91
|
+
# Inline format: "ms <key> <size> c [b] F<flags> T<ttl> MS q\r\n"
|
|
92
|
+
buffer << "ms #{key} #{value.bytesize} c"
|
|
93
|
+
buffer << ' b' if base64
|
|
94
|
+
buffer << " F#{bitflags}" if bitflags
|
|
95
|
+
buffer << " T#{ttl}" if ttl
|
|
96
|
+
buffer << ' MS q' << TERMINATOR << value << TERMINATOR
|
|
97
|
+
end
|
|
98
|
+
buffer << META_NOOP
|
|
99
|
+
end
|
|
100
|
+
|
|
70
101
|
# Thundering herd protection flag:
|
|
71
102
|
# - stale (I flag): Instead of deleting the item, mark it as stale. Other clients
|
|
72
103
|
# using N/R flags will see the X flag and know the item is being regenerated.
|
|
73
|
-
def
|
|
74
|
-
cmd = "md #{key}"
|
|
75
|
-
cmd << ' b' if base64
|
|
104
|
+
def meta_delete(key:, cas: nil, ttl: nil, quiet: false, stale: false)
|
|
105
|
+
cmd = "md #{encoded_key(key)}"
|
|
76
106
|
cmd << cas_string(cas)
|
|
77
107
|
cmd << " T#{ttl}" if ttl
|
|
78
108
|
cmd << ' I' if stale # Mark stale instead of deleting
|
|
@@ -80,9 +110,16 @@ module Dalli
|
|
|
80
110
|
cmd << TERMINATOR
|
|
81
111
|
end
|
|
82
112
|
|
|
83
|
-
def
|
|
84
|
-
|
|
85
|
-
|
|
113
|
+
def multi_meta_delete(keys)
|
|
114
|
+
buffer = ''.b
|
|
115
|
+
keys.each do |key|
|
|
116
|
+
buffer << 'md ' << encoded_key(key) << ' q' << TERMINATOR
|
|
117
|
+
end
|
|
118
|
+
buffer << META_NOOP
|
|
119
|
+
end
|
|
120
|
+
|
|
121
|
+
def meta_arithmetic(key:, delta:, initial:, incr: true, cas: nil, ttl: nil, quiet: false)
|
|
122
|
+
cmd = "ma #{encoded_key(key)} v"
|
|
86
123
|
cmd << " D#{delta}" if delta
|
|
87
124
|
cmd << " J#{initial}" if initial
|
|
88
125
|
# Always set a TTL if an initial value is specified
|
|
@@ -97,15 +134,15 @@ module Dalli
|
|
|
97
134
|
# rubocop:enable Metrics/PerceivedComplexity
|
|
98
135
|
|
|
99
136
|
META_NOOP = "mn#{TERMINATOR}".freeze
|
|
100
|
-
def
|
|
137
|
+
def meta_noop
|
|
101
138
|
META_NOOP
|
|
102
139
|
end
|
|
103
140
|
|
|
104
|
-
def
|
|
141
|
+
def version
|
|
105
142
|
"version#{TERMINATOR}"
|
|
106
143
|
end
|
|
107
144
|
|
|
108
|
-
def
|
|
145
|
+
def flush(delay: nil, quiet: false)
|
|
109
146
|
cmd = +'flush_all'
|
|
110
147
|
cmd << " #{parse_to_64_bit_int(delay, 0)}" if delay
|
|
111
148
|
cmd << ' noreply' if quiet
|
|
@@ -114,7 +151,7 @@ module Dalli
|
|
|
114
151
|
|
|
115
152
|
ALLOWED_STATS_ARGS = [nil, '', 'items', 'slabs', 'settings', 'reset'].freeze
|
|
116
153
|
|
|
117
|
-
def
|
|
154
|
+
def stats(arg = nil)
|
|
118
155
|
raise ArgumentError, "Invalid stats argument: #{arg.inspect}" unless ALLOWED_STATS_ARGS.include?(arg)
|
|
119
156
|
|
|
120
157
|
cmd = +'stats'
|
|
@@ -122,7 +159,17 @@ module Dalli
|
|
|
122
159
|
cmd << TERMINATOR
|
|
123
160
|
end
|
|
124
161
|
|
|
125
|
-
def
|
|
162
|
+
def encoded_key(key)
|
|
163
|
+
if KeyRegularizer.required?(key)
|
|
164
|
+
KeyRegularizer.encode(key) << ' b'
|
|
165
|
+
else
|
|
166
|
+
key
|
|
167
|
+
end
|
|
168
|
+
end
|
|
169
|
+
|
|
170
|
+
private
|
|
171
|
+
|
|
172
|
+
def mode_to_token(mode)
|
|
126
173
|
case mode
|
|
127
174
|
when :add
|
|
128
175
|
'E'
|
|
@@ -137,12 +184,12 @@ module Dalli
|
|
|
137
184
|
end
|
|
138
185
|
end
|
|
139
186
|
|
|
140
|
-
def
|
|
187
|
+
def cas_string(cas)
|
|
141
188
|
cas = parse_to_64_bit_int(cas, nil)
|
|
142
189
|
cas.nil? || cas.zero? ? '' : " C#{cas}"
|
|
143
190
|
end
|
|
144
191
|
|
|
145
|
-
def
|
|
192
|
+
def parse_to_64_bit_int(val, default)
|
|
146
193
|
val.nil? ? nil : Integer(val)
|
|
147
194
|
rescue ArgumentError
|
|
148
195
|
# Sanitize to default if it isn't parsable as an integer
|
|
@@ -5,16 +5,16 @@ require 'timeout'
|
|
|
5
5
|
|
|
6
6
|
module Dalli
|
|
7
7
|
module Protocol
|
|
8
|
+
# Compact the buffer when the consumed portion exceeds this
|
|
9
|
+
# threshold and represents more than half the buffer
|
|
10
|
+
COMPACT_THRESHOLD = 4096
|
|
11
|
+
|
|
8
12
|
##
|
|
9
13
|
# Manages the buffer for responses from memcached.
|
|
10
14
|
# Uses an offset-based approach to avoid string allocations
|
|
11
15
|
# when advancing through parsed responses.
|
|
12
16
|
##
|
|
13
17
|
class ResponseBuffer
|
|
14
|
-
# Compact the buffer when the consumed portion exceeds this
|
|
15
|
-
# threshold and represents more than half the buffer
|
|
16
|
-
COMPACT_THRESHOLD = 4096
|
|
17
|
-
|
|
18
18
|
def initialize(io_source, response_processor)
|
|
19
19
|
@io_source = io_source
|
|
20
20
|
@response_processor = response_processor
|
|
@@ -23,22 +23,32 @@ module Dalli
|
|
|
23
23
|
end
|
|
24
24
|
|
|
25
25
|
def read
|
|
26
|
-
@buffer
|
|
26
|
+
remaining_bytes = @buffer.bytesize - @offset
|
|
27
|
+
if remaining_bytes.zero?
|
|
28
|
+
@offset = 0
|
|
29
|
+
@buffer = @io_source.read_available(@buffer)
|
|
30
|
+
else
|
|
31
|
+
if @offset > COMPACT_THRESHOLD && @offset > (@buffer.bytesize / 2)
|
|
32
|
+
@buffer.bytesplice(0, @offset, '')
|
|
33
|
+
@offset = 0
|
|
34
|
+
end
|
|
35
|
+
@buffer << @io_source.read_available
|
|
36
|
+
end
|
|
27
37
|
end
|
|
28
38
|
|
|
29
39
|
# Attempts to process a single response from the buffer,
|
|
30
40
|
# advancing the offset past the consumed bytes.
|
|
31
41
|
def process_single_getk_response
|
|
32
|
-
|
|
33
|
-
@offset +=
|
|
34
|
-
|
|
35
|
-
[status, cas, key, value]
|
|
42
|
+
response = @response_processor.getk_response_from_buffer(@buffer, @offset)
|
|
43
|
+
@offset += response.pop
|
|
44
|
+
response
|
|
36
45
|
end
|
|
37
46
|
|
|
38
47
|
# Resets the internal buffer to an empty state,
|
|
39
48
|
# so that we're ready to read pipelined responses
|
|
40
49
|
def reset
|
|
41
|
-
@buffer
|
|
50
|
+
@buffer&.clear
|
|
51
|
+
@buffer ||= ''.b
|
|
42
52
|
@offset = 0
|
|
43
53
|
end
|
|
44
54
|
|
|
@@ -54,6 +64,7 @@ module Dalli
|
|
|
54
64
|
|
|
55
65
|
# Clear the internal response buffer
|
|
56
66
|
def clear
|
|
67
|
+
@buffer&.clear
|
|
57
68
|
@buffer = nil
|
|
58
69
|
@offset = 0
|
|
59
70
|
end
|
|
@@ -61,18 +72,6 @@ module Dalli
|
|
|
61
72
|
def in_progress?
|
|
62
73
|
!@buffer.nil?
|
|
63
74
|
end
|
|
64
|
-
|
|
65
|
-
private
|
|
66
|
-
|
|
67
|
-
# Only compact when we've consumed a significant portion of the buffer.
|
|
68
|
-
# This avoids per-response string allocation while preventing unbounded
|
|
69
|
-
# memory growth for large pipelines.
|
|
70
|
-
def compact_if_needed
|
|
71
|
-
return unless @offset > COMPACT_THRESHOLD && @offset > @buffer.bytesize / 2
|
|
72
|
-
|
|
73
|
-
@buffer = @buffer.byteslice(@offset..)
|
|
74
|
-
@offset = 0
|
|
75
|
-
end
|
|
76
75
|
end
|
|
77
76
|
end
|
|
78
77
|
end
|
|
@@ -157,9 +157,26 @@ module Dalli
|
|
|
157
157
|
true
|
|
158
158
|
end
|
|
159
159
|
|
|
160
|
+
# Consumes the responses to a batch of quiet (pipelined) delete
|
|
161
|
+
# requests, which are terminated by a noop (MN). In quiet mode
|
|
162
|
+
# memcached suppresses the success response for each deleted key, so
|
|
163
|
+
# every line received before the terminator corresponds to a key that
|
|
164
|
+
# was NOT deleted -- a miss (NF) or an error. Returns that count so
|
|
165
|
+
# callers can derive the number of successful deletes as
|
|
166
|
+
# (keys_sent - non_deletions).
|
|
167
|
+
def pipelined_delete_non_deletions
|
|
168
|
+
non_deletions = 0
|
|
169
|
+
tokens = next_line_to_tokens
|
|
170
|
+
until tokens.first == MN
|
|
171
|
+
non_deletions += 1
|
|
172
|
+
tokens = next_line_to_tokens
|
|
173
|
+
end
|
|
174
|
+
non_deletions
|
|
175
|
+
end
|
|
176
|
+
|
|
160
177
|
def full_response_from_buffer(tokens, body, resp_size)
|
|
161
178
|
value = @value_marshaller.retrieve(body, bitflags_from_tokens(tokens))
|
|
162
|
-
[
|
|
179
|
+
[tokens.first == VA, cas_from_tokens(tokens), key_from_tokens(tokens), value, resp_size]
|
|
163
180
|
end
|
|
164
181
|
|
|
165
182
|
##
|
|
@@ -174,24 +191,26 @@ module Dalli
|
|
|
174
191
|
##
|
|
175
192
|
def getk_response_from_buffer(buf, offset = 0)
|
|
176
193
|
# Find the header terminator starting from offset
|
|
177
|
-
term_idx = buf.
|
|
178
|
-
return [0
|
|
194
|
+
term_idx = buf.byteindex(TERMINATOR, offset)
|
|
195
|
+
return [0] unless term_idx
|
|
179
196
|
|
|
180
197
|
header = buf.byteslice(offset, term_idx - offset)
|
|
181
198
|
tokens = header.split
|
|
182
199
|
header_len = header.bytesize + TERMINATOR.length
|
|
200
|
+
|
|
201
|
+
# The body len is removed from the tokens array
|
|
183
202
|
body_len = body_len_from_tokens(tokens)
|
|
184
203
|
|
|
185
204
|
# We have a complete response that has no body.
|
|
186
205
|
# This is either the response to the terminating
|
|
187
206
|
# noop or, if the status is not MN, an intermediate
|
|
188
207
|
# error response that needs to be discarded.
|
|
189
|
-
return [
|
|
208
|
+
return [true, header_len] if body_len.zero?
|
|
190
209
|
|
|
191
210
|
resp_size = header_len + body_len + TERMINATOR.length
|
|
192
211
|
# The header is in the buffer, but the body is not. As we don't have
|
|
193
212
|
# a complete response, don't advance the buffer
|
|
194
|
-
return [0
|
|
213
|
+
return [0] unless buf.bytesize >= offset + resp_size
|
|
195
214
|
|
|
196
215
|
# The full response is in our buffer, so parse it and return
|
|
197
216
|
# the values
|
|
@@ -210,17 +229,20 @@ module Dalli
|
|
|
210
229
|
end
|
|
211
230
|
|
|
212
231
|
def bitflags_from_tokens(tokens)
|
|
213
|
-
value_from_tokens(tokens, 'f')
|
|
232
|
+
value_from_tokens(tokens, 'f').to_i
|
|
214
233
|
end
|
|
215
234
|
|
|
216
235
|
def cas_from_tokens(tokens)
|
|
217
|
-
value_from_tokens(tokens, 'c')
|
|
236
|
+
value_from_tokens(tokens, 'c').to_i
|
|
218
237
|
end
|
|
219
238
|
|
|
220
239
|
def key_from_tokens(tokens)
|
|
221
240
|
encoded_key = value_from_tokens(tokens, 'k')
|
|
222
|
-
|
|
223
|
-
|
|
241
|
+
if tokens.delete('b')
|
|
242
|
+
KeyRegularizer.decode(encoded_key)
|
|
243
|
+
else
|
|
244
|
+
encoded_key
|
|
245
|
+
end
|
|
224
246
|
end
|
|
225
247
|
|
|
226
248
|
# Returns true if item was previously hit, false if first access, nil if not requested
|
|
@@ -235,18 +257,22 @@ module Dalli
|
|
|
235
257
|
# Returns seconds since last access, or nil if not requested
|
|
236
258
|
# The l flag returns l<seconds>
|
|
237
259
|
def last_access_from_tokens(tokens)
|
|
238
|
-
value_from_tokens(tokens, 'l')
|
|
260
|
+
value_from_tokens(tokens, 'l').to_i
|
|
239
261
|
end
|
|
240
262
|
|
|
241
263
|
def body_len_from_tokens(tokens)
|
|
242
|
-
value_from_tokens(tokens, 's')
|
|
264
|
+
value_from_tokens(tokens, 's').to_i
|
|
243
265
|
end
|
|
244
266
|
|
|
245
267
|
def value_from_tokens(tokens, flag)
|
|
246
|
-
|
|
247
|
-
|
|
248
|
-
|
|
249
|
-
|
|
268
|
+
# NB: as an optimization, we're mutating the matching token in place
|
|
269
|
+
# so there is a baked assumption that we're only accessing each token at most once.
|
|
270
|
+
index = tokens.find_index { |t| t.start_with?(flag) }
|
|
271
|
+
if index
|
|
272
|
+
tokens.delete_at(index).delete_prefix!(flag)
|
|
273
|
+
else
|
|
274
|
+
0
|
|
275
|
+
end
|
|
250
276
|
end
|
|
251
277
|
|
|
252
278
|
def read_line
|
data/lib/dalli/socket.rb
CHANGED
|
@@ -12,48 +12,63 @@ module Dalli
|
|
|
12
12
|
# Common methods for all socket implementations.
|
|
13
13
|
##
|
|
14
14
|
module InstanceMethods
|
|
15
|
-
def
|
|
16
|
-
|
|
17
|
-
|
|
18
|
-
|
|
19
|
-
|
|
20
|
-
|
|
15
|
+
def read_available(reusable_buffer = nil)
|
|
16
|
+
if reusable_buffer
|
|
17
|
+
value = read_nonblock(8196, reusable_buffer, exception: false)
|
|
18
|
+
case value
|
|
19
|
+
when :wait_writable, :wait_readable
|
|
20
|
+
return reusable_buffer.clear
|
|
21
|
+
when nil
|
|
22
|
+
raise Errno::ECONNRESET, "Connection reset: #{logged_options.inspect}"
|
|
23
|
+
end
|
|
24
|
+
else
|
|
25
|
+
value = ''.b
|
|
21
26
|
end
|
|
22
|
-
value
|
|
23
|
-
end
|
|
24
27
|
|
|
25
|
-
|
|
26
|
-
value = +''
|
|
28
|
+
buffer = ''.b
|
|
27
29
|
loop do
|
|
28
|
-
result = read_nonblock(8196, exception: false)
|
|
29
|
-
|
|
30
|
-
|
|
31
|
-
|
|
32
|
-
|
|
30
|
+
result = read_nonblock(8196, buffer, exception: false)
|
|
31
|
+
case result
|
|
32
|
+
when :wait_writable, :wait_readable
|
|
33
|
+
buffer.clear
|
|
34
|
+
return value
|
|
35
|
+
when nil
|
|
36
|
+
raise Errno::ECONNRESET, "Connection reset: #{logged_options.inspect}"
|
|
37
|
+
else
|
|
38
|
+
value << result
|
|
39
|
+
end
|
|
33
40
|
end
|
|
34
|
-
value
|
|
35
|
-
end
|
|
36
|
-
|
|
37
|
-
WAIT_RCS = %i[wait_writable wait_readable].freeze
|
|
38
|
-
|
|
39
|
-
def append_to_buffer?(result)
|
|
40
|
-
raise Timeout::Error, "IO timeout: #{logged_options.inspect}" if nonblock_timed_out?(result)
|
|
41
|
-
raise Errno::ECONNRESET, "Connection reset: #{logged_options.inspect}" unless result
|
|
42
|
-
|
|
43
|
-
!WAIT_RCS.include?(result)
|
|
44
|
-
end
|
|
45
|
-
|
|
46
|
-
def nonblock_timed_out?(result)
|
|
47
|
-
return true if result == :wait_readable && !wait_readable(options[:socket_timeout])
|
|
48
|
-
|
|
49
|
-
# TODO: Do we actually need this? Looks to be only used in read_nonblock
|
|
50
|
-
result == :wait_writable && !wait_writable(options[:socket_timeout])
|
|
51
41
|
end
|
|
52
42
|
|
|
53
43
|
FILTERED_OUT_OPTIONS = %i[username password].freeze
|
|
54
44
|
def logged_options
|
|
55
45
|
options.except(*FILTERED_OUT_OPTIONS)
|
|
56
46
|
end
|
|
47
|
+
|
|
48
|
+
# JRuby doesn't support IO#timeout=, so use custom readfull implementation
|
|
49
|
+
# CRuby 3.3+ has IO#timeout= which makes IO#read work with timeouts
|
|
50
|
+
if RUBY_ENGINE == 'jruby'
|
|
51
|
+
def readfull(count)
|
|
52
|
+
value = String.new(capacity: count + 1)
|
|
53
|
+
|
|
54
|
+
until value.bytesize == count
|
|
55
|
+
result = read_nonblock(count - value.bytesize, exception: false)
|
|
56
|
+
case result
|
|
57
|
+
when :wait_readable
|
|
58
|
+
wait_readable(options[:socket_timeout]) or raise Timeout::Error, "IO timeout: #{logged_options.inspect}"
|
|
59
|
+
when :wait_writable
|
|
60
|
+
wait_writable(options[:socket_timeout]) or raise Timeout::Error, "IO timeout: #{logged_options.inspect}"
|
|
61
|
+
when nil
|
|
62
|
+
raise Errno::ECONNRESET, "Connection reset: #{logged_options.inspect}"
|
|
63
|
+
else
|
|
64
|
+
value << result
|
|
65
|
+
end
|
|
66
|
+
end
|
|
67
|
+
|
|
68
|
+
value
|
|
69
|
+
end
|
|
70
|
+
# rubocop:enable Metrics/AbcSize
|
|
71
|
+
end
|
|
57
72
|
end
|
|
58
73
|
|
|
59
74
|
##
|
data/lib/dalli/version.rb
CHANGED
metadata
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: dalli
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 5.0.
|
|
4
|
+
version: 5.0.6
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Peter M. Goldstein
|
|
@@ -44,7 +44,6 @@ files:
|
|
|
44
44
|
- lib/dalli/instrumentation.rb
|
|
45
45
|
- lib/dalli/key_manager.rb
|
|
46
46
|
- lib/dalli/options.rb
|
|
47
|
-
- lib/dalli/pid_cache.rb
|
|
48
47
|
- lib/dalli/pipelined_deleter.rb
|
|
49
48
|
- lib/dalli/pipelined_getter.rb
|
|
50
49
|
- lib/dalli/pipelined_setter.rb
|
|
@@ -88,7 +87,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
|
88
87
|
- !ruby/object:Gem::Version
|
|
89
88
|
version: '0'
|
|
90
89
|
requirements: []
|
|
91
|
-
rubygems_version: 4.0.
|
|
90
|
+
rubygems_version: 4.0.16
|
|
92
91
|
specification_version: 4
|
|
93
92
|
summary: High performance memcached client for Ruby
|
|
94
93
|
test_files: []
|
data/lib/dalli/pid_cache.rb
DELETED
|
@@ -1,40 +0,0 @@
|
|
|
1
|
-
# frozen_string_literal: true
|
|
2
|
-
|
|
3
|
-
module Dalli
|
|
4
|
-
##
|
|
5
|
-
# Dalli::PIDCache is a wrapper class for PID checking to avoid system calls when checking the PID.
|
|
6
|
-
##
|
|
7
|
-
module PIDCache
|
|
8
|
-
if !Process.respond_to?(:fork) # JRuby or TruffleRuby
|
|
9
|
-
@pid = Process.pid
|
|
10
|
-
singleton_class.attr_reader(:pid)
|
|
11
|
-
elsif Process.respond_to?(:_fork) # Ruby 3.1+
|
|
12
|
-
class << self
|
|
13
|
-
attr_reader :pid
|
|
14
|
-
|
|
15
|
-
def update!
|
|
16
|
-
@pid = Process.pid # rubocop:disable ThreadSafety/ClassInstanceVariable
|
|
17
|
-
end
|
|
18
|
-
end
|
|
19
|
-
update!
|
|
20
|
-
|
|
21
|
-
##
|
|
22
|
-
# Dalli::PIDCache::CoreExt hooks into Process to be able to reset the PID cache after fork
|
|
23
|
-
##
|
|
24
|
-
module CoreExt
|
|
25
|
-
def _fork
|
|
26
|
-
child_pid = super
|
|
27
|
-
PIDCache.update! if child_pid.zero?
|
|
28
|
-
child_pid
|
|
29
|
-
end
|
|
30
|
-
end
|
|
31
|
-
Process.singleton_class.prepend(CoreExt)
|
|
32
|
-
else # Ruby 3.0 or older
|
|
33
|
-
class << self
|
|
34
|
-
def pid
|
|
35
|
-
Process.pid
|
|
36
|
-
end
|
|
37
|
-
end
|
|
38
|
-
end
|
|
39
|
-
end
|
|
40
|
-
end
|