tuber 0.7.0 → 0.8.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +10 -0
- data/README.md +26 -1
- data/lib/tuber/configuration.rb +28 -2
- data/lib/tuber/connection.rb +105 -21
- data/lib/tuber/version.rb +1 -1
- data/test/connection_test.rb +214 -1
- metadata +1 -1
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: a1d30e48b1dd0a0e43c0d5b028605ba85481353fcfcd0c8b4429cee30f55ae5a
|
|
4
|
+
data.tar.gz: 21e6f5e9e165c513b1b55d482b828d2f0d1e03f545667466f4a153fc131972f6
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 4160816477011c902e40ef12b601ffda5d155612e4b1a98c3cb36bbab335b1f9b28e7cfefb8d7b8b74278c54c54eeace1d7f75fe1e5a0f3bc700af83646251eb
|
|
7
|
+
data.tar.gz: 5b299b53bc6cab47d96d2afc4dd532517000025d35659480977b871c912e013a4f464e8c718efda9b2f50b9e51435a9381b98f180f729e2f4433052271927352
|
data/CHANGELOG.md
CHANGED
|
@@ -4,6 +4,16 @@ The tuber gem is a fork of [beaneater](https://github.com/beanstalkd/beaneater),
|
|
|
4
4
|
beaneater 1.1.4. Entries from 1.1.4 down are beaneater's history, kept for reference.
|
|
5
5
|
(0.0.1 was an empty placeholder published to claim the gem name.)
|
|
6
6
|
|
|
7
|
+
## 0.8.0 (November 22 2026)
|
|
8
|
+
|
|
9
|
+
Surviving a server whose host vanishes. A consumer blocked in `reserve` used to
|
|
10
|
+
wait forever on a connection whose server had gone without closing it, as after
|
|
11
|
+
a host reboot. It now notices in about a minute, reconnects and carries on.
|
|
12
|
+
|
|
13
|
+
* Turn on TCP keepalive by default, so a consumer blocked in `reserve` notices a server whose host vanished without closing the connection (a reboot that takes the network down first, a power cut, a partition) in about 60 seconds, instead of waiting forever on a half-open socket. Tune it with `keepalive_idle` (30), `keepalive_interval` (10) and `keepalive_count` (3), or turn it off with `keepalive = false`. (#2)
|
|
14
|
+
* Heal the connection after `Errno::ETIMEDOUT` (a failed keepalive) and `IO::TimeoutError` (a read or write timeout), as for a dropped socket: an idempotent command such as `reserve` reconnects and is re-sent, and the caller never sees the error. A timeout is re-sent at most once. A second one on a fresh socket means the server is up but not answering, and is raised.
|
|
15
|
+
* Fix `read_timeout` and `write_timeout`, which did nothing on Ruby 3.0 and newer: Ruby's sockets are non-blocking from 3.0 on, and ignore `SO_RCVTIMEO`/`SO_SNDTIMEO`. They now use `IO#timeout` on Ruby 3.2 and newer. Older Rubies keep the `setsockopt` fallback. Behaviour change: with `read_timeout` set, a reply that takes longer now raises `IO::TimeoutError` after one re-send. `reserve-with-timeout` and `reserve_batch` with a timeout get their own timeout added to `read_timeout`, and a plain `reserve` has no read deadline.
|
|
16
|
+
|
|
7
17
|
## 0.7.0 (October 11 2026)
|
|
8
18
|
|
|
9
19
|
Recurring schedules. A Tuber with the scheduler runs an app's recurring jobs
|
data/README.md
CHANGED
|
@@ -72,6 +72,10 @@ Tuber.configure do |config|
|
|
|
72
72
|
# config.resolv_timeout = nil
|
|
73
73
|
# config.read_timeout = nil
|
|
74
74
|
# config.write_timeout = nil
|
|
75
|
+
# config.keepalive = true
|
|
76
|
+
# config.keepalive_idle = 30
|
|
77
|
+
# config.keepalive_interval = 10
|
|
78
|
+
# config.keepalive_count = 3
|
|
75
79
|
# config.connect_retries = 0
|
|
76
80
|
# config.connect_retry_interval = 1
|
|
77
81
|
end
|
|
@@ -79,7 +83,11 @@ end
|
|
|
79
83
|
|
|
80
84
|
The above options are all defaults, so only include a configuration block if you need to make changes.
|
|
81
85
|
|
|
82
|
-
`connect_timeout` and `resolv_timeout` are passed through to `TCPSocket.new` on Ruby 3.0 and newer (ignored on Ruby < 3.0 for compatibility).
|
|
86
|
+
`connect_timeout` and `resolv_timeout` are passed through to `TCPSocket.new` on Ruby 3.0 and newer (ignored on Ruby < 3.0 for compatibility).
|
|
87
|
+
|
|
88
|
+
`read_timeout` is how long to wait for a reply before raising `IO::TimeoutError`, *beyond* any wait the command asks for: `reserve-with-timeout 30` and `reserve_batch(10, 30)` get their 30 seconds added on, and a plain `reserve` has no client deadline at all. `write_timeout` is how long a write may block on a server that has stopped reading. Both need Ruby 3.2 or newer (`IO#timeout`). On older Rubies they fall back to `SO_RCVTIMEO`/`SO_SNDTIMEO`, which Ruby's non-blocking sockets ignore from 3.0 on.
|
|
89
|
+
|
|
90
|
+
`keepalive` turns on TCP keepalive (see "Surviving a server restart" below). The `keepalive_*` timings are in seconds, and each is set only where the platform has the option.
|
|
83
91
|
|
|
84
92
|
### Connection
|
|
85
93
|
|
|
@@ -115,6 +123,16 @@ end
|
|
|
115
123
|
down) and reconnects on a client you already hold. It defaults to `0` — a
|
|
116
124
|
single attempt — which is the historical behaviour.
|
|
117
125
|
|
|
126
|
+
A consumer blocked in `reserve` is waiting on the server, so it only notices
|
|
127
|
+
the socket has died if the server's host says so. A clean restart closes the
|
|
128
|
+
connection, but when the host vanishes without a word — a reboot that takes the
|
|
129
|
+
network down before the server, a power cut, a partition — the old connection
|
|
130
|
+
is left half-open. TCP keepalive (on by default) probes an idle connection after
|
|
131
|
+
`keepalive_idle` seconds, every `keepalive_interval` seconds, and gives up after
|
|
132
|
+
`keepalive_count` unanswered probes. The blocked read then fails (about 60
|
|
133
|
+
seconds with the defaults) and the connection heals as above: the `reserve` is
|
|
134
|
+
re-sent on a fresh socket, and the caller never sees the error.
|
|
135
|
+
|
|
118
136
|
When a command does surface `Tuber::NotConnected`, heal the client you have
|
|
119
137
|
rather than building a new one:
|
|
120
138
|
|
|
@@ -485,6 +503,13 @@ Tuber.configure do |config|
|
|
|
485
503
|
# Connect attempts; see "Surviving a server restart" above.
|
|
486
504
|
config.connect_retries = 0
|
|
487
505
|
config.connect_retry_interval = 1
|
|
506
|
+
|
|
507
|
+
# TCP keepalive: notice a server whose host vanished in about
|
|
508
|
+
# idle + interval * count seconds. See "Surviving a server restart".
|
|
509
|
+
config.keepalive = true
|
|
510
|
+
config.keepalive_idle = 30
|
|
511
|
+
config.keepalive_interval = 10
|
|
512
|
+
config.keepalive_count = 3
|
|
488
513
|
end
|
|
489
514
|
```
|
|
490
515
|
|
data/lib/tuber/configuration.rb
CHANGED
|
@@ -12,8 +12,30 @@ class Tuber
|
|
|
12
12
|
alias_method :beanstalkd_url=, :tuber_url=
|
|
13
13
|
attr_accessor :connect_timeout # TCP connect timeout in seconds
|
|
14
14
|
attr_accessor :resolv_timeout # DNS resolve timeout in seconds
|
|
15
|
-
|
|
16
|
-
|
|
15
|
+
|
|
16
|
+
# Seconds to wait for a reply before raising IO::TimeoutError, beyond any
|
|
17
|
+
# wait the command itself asks for: reserve-with-timeout and reserve-batch
|
|
18
|
+
# get their own timeout added on, and a plain reserve has no deadline at
|
|
19
|
+
# all. Needs Ruby >= 3.2 (IO#timeout); older Rubies fall back to
|
|
20
|
+
# SO_RCVTIMEO, which Ruby's non-blocking sockets ignore from 3.0 on.
|
|
21
|
+
attr_accessor :read_timeout
|
|
22
|
+
|
|
23
|
+
# Seconds a write may block (the server not reading) before raising
|
|
24
|
+
# IO::TimeoutError. Same Ruby caveat as read_timeout.
|
|
25
|
+
attr_accessor :write_timeout
|
|
26
|
+
|
|
27
|
+
# TCP keepalive, on by default. A blocked read waits on the server, so it
|
|
28
|
+
# never learns the server is gone if the server's host vanished without
|
|
29
|
+
# closing the connection: a reboot that takes the network down first, a
|
|
30
|
+
# power cut, a partition. Keepalive probes an idle connection after
|
|
31
|
+
# keepalive_idle seconds, every keepalive_interval seconds, and gives up
|
|
32
|
+
# after keepalive_count unanswered probes, so the read fails (and the
|
|
33
|
+
# connection heals) after about idle + interval * count seconds: 60 with
|
|
34
|
+
# the defaults.
|
|
35
|
+
attr_accessor :keepalive
|
|
36
|
+
attr_accessor :keepalive_idle # idle seconds before the first probe
|
|
37
|
+
attr_accessor :keepalive_interval # seconds between probes
|
|
38
|
+
attr_accessor :keepalive_count # unanswered probes before giving up
|
|
17
39
|
|
|
18
40
|
# How hard a client tries to connect — "how long should this ride out a
|
|
19
41
|
# server restart", in one knob.
|
|
@@ -41,6 +63,10 @@ class Tuber
|
|
|
41
63
|
@resolv_timeout = nil
|
|
42
64
|
@read_timeout = nil
|
|
43
65
|
@write_timeout = nil
|
|
66
|
+
@keepalive = true
|
|
67
|
+
@keepalive_idle = 30
|
|
68
|
+
@keepalive_interval = 10
|
|
69
|
+
@keepalive_count = 3
|
|
44
70
|
@connect_retries = 0
|
|
45
71
|
# Matches Tuber::Connection::DEFAULT_RETRY_INTERVAL, which is not loaded yet.
|
|
46
72
|
@connect_retry_interval = 1
|
data/lib/tuber/connection.rb
CHANGED
|
@@ -78,6 +78,21 @@ class Tuber
|
|
|
78
78
|
(defined?(IO::TimeoutError) ? [IO::TimeoutError] : [])
|
|
79
79
|
).freeze
|
|
80
80
|
|
|
81
|
+
# What a blocked read or write raises once the server is judged gone
|
|
82
|
+
# without having said so: Errno::ETIMEDOUT when keepalive probes go
|
|
83
|
+
# unanswered, IO::TimeoutError (Ruby >= 3.2) when read_timeout or
|
|
84
|
+
# write_timeout runs out. The socket is useless afterwards either way: a
|
|
85
|
+
# reply still in flight could arrive later and be read as the answer to
|
|
86
|
+
# the next command.
|
|
87
|
+
TIMEOUT_ERRORS = (
|
|
88
|
+
[Errno::ETIMEDOUT] + (defined?(IO::TimeoutError) ? [IO::TimeoutError] : [])
|
|
89
|
+
).freeze
|
|
90
|
+
|
|
91
|
+
# Whether this Ruby has IO#timeout (3.2 and newer), the only read and
|
|
92
|
+
# write timeout its non-blocking sockets honour. SO_RCVTIMEO and
|
|
93
|
+
# SO_SNDTIMEO are kept for older Rubies but have no effect from 3.0 on.
|
|
94
|
+
IO_TIMEOUTS = IO.method_defined?(:timeout=)
|
|
95
|
+
|
|
81
96
|
# Initializes new connection.
|
|
82
97
|
#
|
|
83
98
|
# @param [String] address beanstalkd instance address.
|
|
@@ -127,7 +142,7 @@ class Tuber
|
|
|
127
142
|
_raise_not_connected! unless connection
|
|
128
143
|
|
|
129
144
|
command = command.dup.force_encoding('ASCII-8BIT') if command.respond_to?(:force_encoding)
|
|
130
|
-
|
|
145
|
+
_write(command.to_s + "\r\n", read_timeout: _read_timeout_for(command))
|
|
131
146
|
res = connection.readline
|
|
132
147
|
parse_response(command, res)
|
|
133
148
|
end
|
|
@@ -152,7 +167,7 @@ class Tuber
|
|
|
152
167
|
_raise_not_connected! unless connection
|
|
153
168
|
|
|
154
169
|
cmd = timeout ? "reserve-batch #{count} #{timeout}" : "reserve-batch #{count}"
|
|
155
|
-
|
|
170
|
+
_write(cmd + "\r\n", read_timeout: _read_timeout_for(cmd))
|
|
156
171
|
|
|
157
172
|
header = connection.readline.chomp
|
|
158
173
|
status, actual_count_str = header.split(/\s/, 2)
|
|
@@ -188,7 +203,7 @@ class Tuber
|
|
|
188
203
|
_raise_not_connected! unless connection
|
|
189
204
|
|
|
190
205
|
cmd = "delete-batch #{ids.join(' ')}"
|
|
191
|
-
|
|
206
|
+
_write(cmd + "\r\n")
|
|
192
207
|
|
|
193
208
|
res = connection.readline.chomp
|
|
194
209
|
status, deleted, not_found = res.split(/\s/)
|
|
@@ -214,7 +229,7 @@ class Tuber
|
|
|
214
229
|
@mutex.synchronize do
|
|
215
230
|
_raise_not_connected! unless connection
|
|
216
231
|
|
|
217
|
-
|
|
232
|
+
_write(command + "\r\n")
|
|
218
233
|
status, bytes = connection.readline.chomp.split(/\s/, 2)
|
|
219
234
|
raise UnexpectedResponse.from_status(status, command) unless status == 'OK'
|
|
220
235
|
|
|
@@ -387,8 +402,11 @@ class Tuber
|
|
|
387
402
|
end
|
|
388
403
|
|
|
389
404
|
begin
|
|
390
|
-
socket
|
|
391
|
-
|
|
405
|
+
_enable_keepalive(socket) if config.keepalive
|
|
406
|
+
unless IO_TIMEOUTS # IO#timeout is set per exchange instead, in #_write
|
|
407
|
+
socket.setsockopt(Socket::SOL_SOCKET, Socket::SO_RCVTIMEO, _timeval_for(config.read_timeout)) if config.read_timeout
|
|
408
|
+
socket.setsockopt(Socket::SOL_SOCKET, Socket::SO_SNDTIMEO, _timeval_for(config.write_timeout)) if config.write_timeout
|
|
409
|
+
end
|
|
392
410
|
@connection = socket
|
|
393
411
|
rescue
|
|
394
412
|
socket.close rescue nil
|
|
@@ -475,7 +493,7 @@ class Tuber
|
|
|
475
493
|
def _round_trip(command, body = nil)
|
|
476
494
|
frame = "#{command}\r\n".b
|
|
477
495
|
frame << body.b << "\r\n" if body
|
|
478
|
-
|
|
496
|
+
_write(frame)
|
|
479
497
|
connection.readline.chomp
|
|
480
498
|
end
|
|
481
499
|
|
|
@@ -489,22 +507,88 @@ class Tuber
|
|
|
489
507
|
# @param [Integer] tries The maximum number of tries in draining mode
|
|
490
508
|
# @return [Object] Result of the block passed
|
|
491
509
|
#
|
|
510
|
+
# A timeout (see TIMEOUT_ERRORS) is retransmitted at most once. One fresh
|
|
511
|
+
# socket heals a connection whose server vanished; a second timeout on it
|
|
512
|
+
# means the server is up but not answering, and the caller hears so
|
|
513
|
+
# rather than waiting on a loop that never ends.
|
|
514
|
+
#
|
|
492
515
|
def _with_retry(retry_interval: DEFAULT_RETRY_INTERVAL, init: true, tries: MAX_RETRIES, retransmit: true, &block)
|
|
493
|
-
|
|
494
|
-
|
|
495
|
-
|
|
496
|
-
|
|
497
|
-
|
|
498
|
-
|
|
499
|
-
|
|
500
|
-
|
|
501
|
-
|
|
502
|
-
|
|
503
|
-
|
|
504
|
-
|
|
516
|
+
timed_out = false
|
|
517
|
+
begin
|
|
518
|
+
yield
|
|
519
|
+
rescue EOFError, Errno::ECONNRESET, Errno::EPIPE,
|
|
520
|
+
Errno::ECONNREFUSED, *TIMEOUT_ERRORS => ex
|
|
521
|
+
_reconnect(tries: _connect_tries, retry_interval: retry_interval)
|
|
522
|
+
_initialize_tubes if init
|
|
523
|
+
raise ex unless retransmit
|
|
524
|
+
if TIMEOUT_ERRORS.any? { |klass| ex.is_a?(klass) }
|
|
525
|
+
raise ex if timed_out
|
|
526
|
+
timed_out = true
|
|
527
|
+
end
|
|
528
|
+
retry
|
|
529
|
+
rescue Tuber::DrainingError
|
|
530
|
+
tries -= 1
|
|
531
|
+
if tries.zero?
|
|
532
|
+
close
|
|
533
|
+
raise
|
|
534
|
+
end
|
|
535
|
+
sleep(retry_interval)
|
|
536
|
+
retry
|
|
537
|
+
end
|
|
538
|
+
end
|
|
539
|
+
|
|
540
|
+
# Writes a command, and any body framed with it, then sets how long the
|
|
541
|
+
# reads of its reply may block. Callers hold the mutex.
|
|
542
|
+
#
|
|
543
|
+
# On Ruby >= 3.2 both limits are IO#timeout, swapped between the write and
|
|
544
|
+
# the reads; older Rubies get the SO_*TIMEO set at connect instead.
|
|
545
|
+
#
|
|
546
|
+
# @param [String] data The bytes to send
|
|
547
|
+
# @param [Numeric, nil] read_timeout Seconds each read of the reply may
|
|
548
|
+
# block, nil for no limit (see #_read_timeout_for)
|
|
549
|
+
def _write(data, read_timeout: config.read_timeout)
|
|
550
|
+
connection.timeout = config.write_timeout if IO_TIMEOUTS
|
|
551
|
+
connection.write(data)
|
|
552
|
+
connection.timeout = read_timeout if IO_TIMEOUTS
|
|
553
|
+
end
|
|
554
|
+
|
|
555
|
+
# How long the reply to +command+ may take: read_timeout, plus however
|
|
556
|
+
# long the command asks the server to hold it before answering. A plain
|
|
557
|
+
# reserve waits as long as it takes by design, so it has no deadline;
|
|
558
|
+
# keepalive still notices a server that has vanished under it.
|
|
559
|
+
#
|
|
560
|
+
# @param [String] command The command line sent
|
|
561
|
+
# @return [Numeric, nil] Seconds, or nil for no limit
|
|
562
|
+
def _read_timeout_for(command)
|
|
563
|
+
limit = config.read_timeout
|
|
564
|
+
return limit unless limit
|
|
565
|
+
|
|
566
|
+
words = command.to_s[/\A[^\r\n]*/].split
|
|
567
|
+
case words[0]
|
|
568
|
+
when 'reserve' then nil
|
|
569
|
+
when 'reserve-with-timeout' then limit + words[1].to_i
|
|
570
|
+
when 'reserve-batch' then limit + words[2].to_i # count, then timeout
|
|
571
|
+
else limit
|
|
572
|
+
end
|
|
573
|
+
end
|
|
574
|
+
|
|
575
|
+
# Turns on TCP keepalive with the configured probe timings, so a read
|
|
576
|
+
# blocked on a server that vanished without closing the connection fails
|
|
577
|
+
# with Errno::ETIMEDOUT (or ECONNRESET, if the host is back and refuses
|
|
578
|
+
# the probe) instead of waiting forever. Each timing is set only where the
|
|
579
|
+
# platform has the option; macOS calls the idle time TCP_KEEPALIVE.
|
|
580
|
+
#
|
|
581
|
+
# @param [TCPSocket] socket A freshly connected socket
|
|
582
|
+
def _enable_keepalive(socket)
|
|
583
|
+
socket.setsockopt(Socket::SOL_SOCKET, Socket::SO_KEEPALIVE, true)
|
|
584
|
+
{
|
|
585
|
+
(Socket.const_defined?(:TCP_KEEPIDLE) ? :TCP_KEEPIDLE : :TCP_KEEPALIVE) => config.keepalive_idle,
|
|
586
|
+
:TCP_KEEPINTVL => config.keepalive_interval,
|
|
587
|
+
:TCP_KEEPCNT => config.keepalive_count,
|
|
588
|
+
}.each do |name, value|
|
|
589
|
+
next unless value && Socket.const_defined?(name)
|
|
590
|
+
socket.setsockopt(Socket::IPPROTO_TCP, Socket.const_get(name), value.ceil)
|
|
505
591
|
end
|
|
506
|
-
sleep(retry_interval)
|
|
507
|
-
retry
|
|
508
592
|
end
|
|
509
593
|
|
|
510
594
|
# Connect attempts to allow when healing a connection that was already
|
data/lib/tuber/version.rb
CHANGED
data/test/connection_test.rb
CHANGED
|
@@ -99,7 +99,11 @@ describe Tuber::Connection do
|
|
|
99
99
|
assert_same real_socket, bc.connection
|
|
100
100
|
end
|
|
101
101
|
|
|
102
|
-
it "should set SO_RCVTIMEO and SO_SNDTIMEO when read/write timeouts are configured" do
|
|
102
|
+
it "should set SO_RCVTIMEO and SO_SNDTIMEO when read/write timeouts are configured on Ruby < 3.2" do
|
|
103
|
+
# Ruby's sockets are non-blocking from 3.0 on and ignore these; from 3.2
|
|
104
|
+
# IO#timeout does the job (see 'for read and write timeouts').
|
|
105
|
+
skip("IO#timeout replaces SO_RCVTIMEO/SO_SNDTIMEO on Ruby >= 3.2") if Tuber::Connection::IO_TIMEOUTS
|
|
106
|
+
|
|
103
107
|
Tuber.configure do |config|
|
|
104
108
|
config.read_timeout = 3
|
|
105
109
|
config.write_timeout = 5
|
|
@@ -114,6 +118,215 @@ describe Tuber::Connection do
|
|
|
114
118
|
|
|
115
119
|
end # timeout configuration
|
|
116
120
|
|
|
121
|
+
# Settings changed in these blocks are put back by swapping in a copy of the
|
|
122
|
+
# configuration and restoring the original afterwards.
|
|
123
|
+
def swap_in_config_copy!
|
|
124
|
+
@saved_config = Tuber.configuration
|
|
125
|
+
Tuber.instance_variable_set(:@_configuration, @saved_config.dup)
|
|
126
|
+
end
|
|
127
|
+
|
|
128
|
+
def restore_config!
|
|
129
|
+
Tuber.instance_variable_set(:@_configuration, @saved_config) if @saved_config
|
|
130
|
+
end
|
|
131
|
+
|
|
132
|
+
# A server that accepts connections and then never reads or answers: the
|
|
133
|
+
# client side of a peer that has vanished without a FIN, as far as any one
|
|
134
|
+
# read or write can tell. Returns its address.
|
|
135
|
+
def start_silent_server
|
|
136
|
+
@silent_server = TCPServer.new('127.0.0.1', 0)
|
|
137
|
+
@silent_peers = []
|
|
138
|
+
@silent_thread = Thread.new do
|
|
139
|
+
loop { @silent_peers << @silent_server.accept }
|
|
140
|
+
rescue IOError
|
|
141
|
+
# closed by stop_silent_server
|
|
142
|
+
end
|
|
143
|
+
"127.0.0.1:#{@silent_server.addr[1]}"
|
|
144
|
+
end
|
|
145
|
+
|
|
146
|
+
def stop_silent_server
|
|
147
|
+
return unless @silent_server
|
|
148
|
+
@silent_server.close rescue nil
|
|
149
|
+
@silent_thread.join(1)
|
|
150
|
+
@silent_peers.each { |s| s.close rescue nil }
|
|
151
|
+
end
|
|
152
|
+
|
|
153
|
+
# The longest a test lets a call block before calling it a hang. Without the
|
|
154
|
+
# guard a read the client never gives up on would stall the whole suite.
|
|
155
|
+
def hang_guard
|
|
156
|
+
10
|
|
157
|
+
end
|
|
158
|
+
|
|
159
|
+
def int_sockopt(socket, level, name)
|
|
160
|
+
socket.getsockopt(level, name).int
|
|
161
|
+
end
|
|
162
|
+
|
|
163
|
+
describe 'for TCP keepalive' do
|
|
164
|
+
before { swap_in_config_copy! }
|
|
165
|
+
|
|
166
|
+
after do
|
|
167
|
+
@bc.close if @bc
|
|
168
|
+
restore_config!
|
|
169
|
+
end
|
|
170
|
+
|
|
171
|
+
it "enables keepalive by default, so a vanished server is noticed in about a minute" do
|
|
172
|
+
@bc = Tuber::Connection.new(tuber_address)
|
|
173
|
+
sock = @bc.connection
|
|
174
|
+
|
|
175
|
+
assert sock.getsockopt(Socket::SOL_SOCKET, Socket::SO_KEEPALIVE).bool,
|
|
176
|
+
"expected SO_KEEPALIVE on a fresh connection"
|
|
177
|
+
if Socket.const_defined?(:TCP_KEEPIDLE)
|
|
178
|
+
assert_equal 30, int_sockopt(sock, Socket::IPPROTO_TCP, Socket::TCP_KEEPIDLE)
|
|
179
|
+
end
|
|
180
|
+
if Socket.const_defined?(:TCP_KEEPINTVL)
|
|
181
|
+
assert_equal 10, int_sockopt(sock, Socket::IPPROTO_TCP, Socket::TCP_KEEPINTVL)
|
|
182
|
+
end
|
|
183
|
+
if Socket.const_defined?(:TCP_KEEPCNT)
|
|
184
|
+
assert_equal 3, int_sockopt(sock, Socket::IPPROTO_TCP, Socket::TCP_KEEPCNT)
|
|
185
|
+
end
|
|
186
|
+
end
|
|
187
|
+
|
|
188
|
+
it "uses the configured probe timings" do
|
|
189
|
+
skip "no TCP_KEEPIDLE on this platform" unless Socket.const_defined?(:TCP_KEEPIDLE)
|
|
190
|
+
Tuber.configure do |config|
|
|
191
|
+
config.keepalive_idle = 7
|
|
192
|
+
config.keepalive_interval = 2
|
|
193
|
+
config.keepalive_count = 5
|
|
194
|
+
end
|
|
195
|
+
|
|
196
|
+
@bc = Tuber::Connection.new(tuber_address)
|
|
197
|
+
sock = @bc.connection
|
|
198
|
+
|
|
199
|
+
assert_equal 7, int_sockopt(sock, Socket::IPPROTO_TCP, Socket::TCP_KEEPIDLE)
|
|
200
|
+
assert_equal 2, int_sockopt(sock, Socket::IPPROTO_TCP, Socket::TCP_KEEPINTVL)
|
|
201
|
+
assert_equal 5, int_sockopt(sock, Socket::IPPROTO_TCP, Socket::TCP_KEEPCNT)
|
|
202
|
+
end
|
|
203
|
+
|
|
204
|
+
it "keeps keepalive on the socket a reconnect builds" do
|
|
205
|
+
@bc = Tuber::Connection.new(tuber_address)
|
|
206
|
+
@bc.reconnect!
|
|
207
|
+
|
|
208
|
+
assert @bc.connection.getsockopt(Socket::SOL_SOCKET, Socket::SO_KEEPALIVE).bool
|
|
209
|
+
end
|
|
210
|
+
|
|
211
|
+
it "leaves keepalive off when it is turned off" do
|
|
212
|
+
Tuber.configure { |config| config.keepalive = false }
|
|
213
|
+
|
|
214
|
+
@bc = Tuber::Connection.new(tuber_address)
|
|
215
|
+
|
|
216
|
+
refute @bc.connection.getsockopt(Socket::SOL_SOCKET, Socket::SO_KEEPALIVE).bool
|
|
217
|
+
end
|
|
218
|
+
end # TCP keepalive
|
|
219
|
+
|
|
220
|
+
describe 'for a dead peer noticed mid-command' do
|
|
221
|
+
# What a blocked read raises once the kernel gives up on the peer:
|
|
222
|
+
# Errno::ETIMEDOUT from failed keepalive probes, IO::TimeoutError from a
|
|
223
|
+
# read_timeout. Either way the socket is dead and must be replaced.
|
|
224
|
+
timeout_errors = [Errno::ETIMEDOUT]
|
|
225
|
+
timeout_errors << IO::TimeoutError if defined?(IO::TimeoutError)
|
|
226
|
+
|
|
227
|
+
before { @bc = Tuber::Connection.new(tuber_address) }
|
|
228
|
+
after { @bc.close rescue nil }
|
|
229
|
+
|
|
230
|
+
timeout_errors.each do |error_class|
|
|
231
|
+
it "reconnects and retransmits an idempotent command after #{error_class}" do
|
|
232
|
+
old_socket = @bc.connection
|
|
233
|
+
TCPSocket.any_instance.expects(:readline).times(2).
|
|
234
|
+
raises(error_class.new).then.
|
|
235
|
+
returns("USING foo\n")
|
|
236
|
+
|
|
237
|
+
res = @bc.transmit "use foo"
|
|
238
|
+
|
|
239
|
+
assert_equal 'USING', res[:status]
|
|
240
|
+
refute_same old_socket, @bc.connection, "expected a fresh socket after #{error_class}"
|
|
241
|
+
end
|
|
242
|
+
|
|
243
|
+
it "heals the connection but re-raises #{error_class} for a non-idempotent command" do
|
|
244
|
+
old_socket = @bc.connection
|
|
245
|
+
TCPSocket.any_instance.expects(:readline).times(1).raises(error_class.new)
|
|
246
|
+
|
|
247
|
+
assert_raises(error_class) { @bc.transmit "delete 56" }
|
|
248
|
+
refute_same old_socket, @bc.connection, "expected a fresh socket after #{error_class}"
|
|
249
|
+
end
|
|
250
|
+
end
|
|
251
|
+
end # dead peer
|
|
252
|
+
|
|
253
|
+
describe 'for read and write timeouts' do
|
|
254
|
+
before do
|
|
255
|
+
skip "IO#timeout= (and with it working read/write timeouts) needs Ruby >= 3.2" unless IO.method_defined?(:timeout=)
|
|
256
|
+
swap_in_config_copy!
|
|
257
|
+
end
|
|
258
|
+
|
|
259
|
+
after do
|
|
260
|
+
@bc&.close
|
|
261
|
+
stop_silent_server
|
|
262
|
+
restore_config!
|
|
263
|
+
end
|
|
264
|
+
|
|
265
|
+
it "read_timeout raises against a server that never answers" do
|
|
266
|
+
address = start_silent_server
|
|
267
|
+
Tuber.configure { |config| config.read_timeout = 0.3 }
|
|
268
|
+
@bc = Tuber::Connection.new(address)
|
|
269
|
+
|
|
270
|
+
started = Time.now
|
|
271
|
+
Timeout.timeout(hang_guard) do
|
|
272
|
+
assert_raises(IO::TimeoutError) { @bc.transmit 'stats' }
|
|
273
|
+
end
|
|
274
|
+
# One retransmit on a fresh connection, then give up: about two reads' worth.
|
|
275
|
+
assert_operator Time.now - started, :<, 3
|
|
276
|
+
end
|
|
277
|
+
|
|
278
|
+
it "write_timeout raises when the server stops reading" do
|
|
279
|
+
address = start_silent_server
|
|
280
|
+
Tuber.configure { |config| config.write_timeout = 0.3 }
|
|
281
|
+
@bc = Tuber::Connection.new(address)
|
|
282
|
+
body = 'x' * (64 * 1024 * 1024) # far past any socket buffer
|
|
283
|
+
|
|
284
|
+
Timeout.timeout(hang_guard) do
|
|
285
|
+
assert_raises(IO::TimeoutError) { @bc.transmit "put 0 0 100 #{body.bytesize}\r\n#{body}" }
|
|
286
|
+
end
|
|
287
|
+
end
|
|
288
|
+
|
|
289
|
+
it "lets reserve-with-timeout wait out its own timeout past read_timeout" do
|
|
290
|
+
Tuber.configure { |config| config.read_timeout = 0.3 }
|
|
291
|
+
client = Tuber.new(tuber_address)
|
|
292
|
+
@bc = client.connection
|
|
293
|
+
client.tubes.watch!("rt_empty_#{Process.pid}")
|
|
294
|
+
|
|
295
|
+
Timeout.timeout(hang_guard) do
|
|
296
|
+
assert_raises(Tuber::TimedOutError) { client.tubes.reserve(1) }
|
|
297
|
+
end
|
|
298
|
+
end
|
|
299
|
+
|
|
300
|
+
it "lets a plain reserve block past read_timeout until a job arrives" do
|
|
301
|
+
Tuber.configure { |config| config.read_timeout = 0.3 }
|
|
302
|
+
tube = "rt_plain_#{Process.pid}"
|
|
303
|
+
client = Tuber.new(tuber_address)
|
|
304
|
+
@bc = client.connection
|
|
305
|
+
client.tubes.watch!(tube)
|
|
306
|
+
producer = Tuber.new(tuber_address)
|
|
307
|
+
Thread.new { sleep 1; producer.tubes[tube].put("late") }
|
|
308
|
+
|
|
309
|
+
job = Timeout.timeout(hang_guard) { client.tubes.reserve }
|
|
310
|
+
|
|
311
|
+
assert_equal "late", job.body
|
|
312
|
+
job.delete
|
|
313
|
+
ensure
|
|
314
|
+
producer.close if producer
|
|
315
|
+
end
|
|
316
|
+
|
|
317
|
+
it "lets reserve_batch long-poll past read_timeout" do
|
|
318
|
+
require_tuber!
|
|
319
|
+
Tuber.configure { |config| config.read_timeout = 0.3 }
|
|
320
|
+
client = Tuber.new(tuber_address)
|
|
321
|
+
@bc = client.connection
|
|
322
|
+
client.tubes.watch!("rt_batch_#{Process.pid}")
|
|
323
|
+
|
|
324
|
+
jobs = Timeout.timeout(hang_guard) { client.tubes.reserve_batch(5, 1) }
|
|
325
|
+
|
|
326
|
+
assert_equal [], jobs
|
|
327
|
+
end
|
|
328
|
+
end # read and write timeouts
|
|
329
|
+
|
|
117
330
|
describe 'for #transmit' do
|
|
118
331
|
before do
|
|
119
332
|
@host = 'localhost'
|