tuber 0.6.0 → 0.8.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +25 -0
- data/README.md +120 -1
- data/lib/tuber/configuration.rb +28 -2
- data/lib/tuber/connection.rb +189 -19
- data/lib/tuber/errors.rb +57 -3
- data/lib/tuber/schedules.rb +288 -0
- data/lib/tuber/tube/record.rb +21 -10
- data/lib/tuber/version.rb +1 -1
- data/lib/tuber.rb +10 -1
- data/test/connection_test.rb +234 -1
- data/test/errors_test.rb +7 -0
- data/test/schedules_test.rb +382 -0
- data/test/test_helper.rb +18 -0
- metadata +4 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: a1d30e48b1dd0a0e43c0d5b028605ba85481353fcfcd0c8b4429cee30f55ae5a
|
|
4
|
+
data.tar.gz: 21e6f5e9e165c513b1b55d482b828d2f0d1e03f545667466f4a153fc131972f6
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 4160816477011c902e40ef12b601ffda5d155612e4b1a98c3cb36bbab335b1f9b28e7cfefb8d7b8b74278c54c54eeace1d7f75fe1e5a0f3bc700af83646251eb
|
|
7
|
+
data.tar.gz: 5b299b53bc6cab47d96d2afc4dd532517000025d35659480977b871c912e013a4f464e8c718efda9b2f50b9e51435a9381b98f180f729e2f4433052271927352
|
data/CHANGELOG.md
CHANGED
|
@@ -4,6 +4,31 @@ The tuber gem is a fork of [beaneater](https://github.com/beanstalkd/beaneater),
|
|
|
4
4
|
beaneater 1.1.4. Entries from 1.1.4 down are beaneater's history, kept for reference.
|
|
5
5
|
(0.0.1 was an empty placeholder published to claim the gem name.)
|
|
6
6
|
|
|
7
|
+
## 0.8.0 (November 22 2026)
|
|
8
|
+
|
|
9
|
+
Surviving a server whose host vanishes. A consumer blocked in `reserve` used to
|
|
10
|
+
wait forever on a connection whose server had gone without closing it, as after
|
|
11
|
+
a host reboot. It now notices in about a minute, reconnects and carries on.
|
|
12
|
+
|
|
13
|
+
* Turn on TCP keepalive by default, so a consumer blocked in `reserve` notices a server whose host vanished without closing the connection (a reboot that takes the network down first, a power cut, a partition) in about 60 seconds, instead of waiting forever on a half-open socket. Tune it with `keepalive_idle` (30), `keepalive_interval` (10) and `keepalive_count` (3), or turn it off with `keepalive = false`. (#2)
|
|
14
|
+
* Heal the connection after `Errno::ETIMEDOUT` (a failed keepalive) and `IO::TimeoutError` (a read or write timeout), as for a dropped socket: an idempotent command such as `reserve` reconnects and is re-sent, and the caller never sees the error. A timeout is re-sent at most once. A second one on a fresh socket means the server is up but not answering, and is raised.
|
|
15
|
+
* Fix `read_timeout` and `write_timeout`, which did nothing on Ruby 3.0 and newer: Ruby's sockets are non-blocking from 3.0 on, and ignore `SO_RCVTIMEO`/`SO_SNDTIMEO`. They now use `IO#timeout` on Ruby 3.2 and newer. Older Rubies keep the `setsockopt` fallback. Behaviour change: with `read_timeout` set, a reply that takes longer now raises `IO::TimeoutError` after one re-send. `reserve-with-timeout` and `reserve_batch` with a timeout get their own timeout added to `read_timeout`, and a plain `reserve` has no read deadline.
|
|
16
|
+
|
|
17
|
+
## 0.7.0 (October 11 2026)
|
|
18
|
+
|
|
19
|
+
Recurring schedules. A Tuber with the scheduler runs an app's recurring jobs
|
|
20
|
+
itself, so the app can drop its scheduler process. This release adds the client
|
|
21
|
+
side. Nothing changes for code that doesn't call it.
|
|
22
|
+
|
|
23
|
+
* Add `Tuber#schedules`: `sync(owner) { |s| s.add ... }` replaces an owner's whole set of entries in one step, and `list` and `stats` show what the server holds. `s.add` takes `cron:` (with `zone:`), `every:` (with `offset:`) or a raw `spec:`, the put options with their usual defaults and value shapes, and `catchup:`. See "Recurring Schedules" in the README.
|
|
24
|
+
* `sync` refuses an empty set unless given `allow_empty: true`. An empty sync removes every entry the owner has, and an app that loaded an empty config by mistake shouldn't wipe its schedule.
|
|
25
|
+
* `sync` runs on a short-lived connection of its own, closed afterwards. A worker blocked in `reserve` on the client's connection can't hold it up, and an error partway can't leave that connection mid-sync. If the socket drops, the whole sync is re-run from the start, never resumed partway.
|
|
26
|
+
* Add `Tuber::ScheduleSyncError` for a sync tuber refuses. It names the owner, the refused entry (`#entry`) and tuber's reply (`#status`). The gem aborts the sync before raising, so the previous schedule keeps running. `#applied?` is true for the one exception: a commit that took effect but couldn't be written to disk.
|
|
27
|
+
* `schedules.stats` returns a `StatStruct`, like tube and job stats: `stats.next_fire` or `stats["next-fire"]`. Times come back as `Time` in the entry's own zone, counters as Integers, and empty fields as nil.
|
|
28
|
+
* Add `Connection#transmit_raw`, which returns an `OK <bytes>` body unparsed. `transmit`'s YAML handling rewrites `:`, `-` and `~` inside values, and schedule stats are full of them.
|
|
29
|
+
* Fix a put refused with `OUT_OF_STORAGE` coming back as a result hash, as though it had succeeded. Tuber sends that reply once a persistent server's `--max-storage-bytes` budget is spent. It now raises the new `Tuber::OutOfStorageError`, as `OUT_OF_MEMORY` raises `OutOfMemoryError`.
|
|
30
|
+
* `Tuber::UnexpectedResponse.new` takes an optional message as a third argument.
|
|
31
|
+
|
|
7
32
|
## 0.6.0 (September 16 2026)
|
|
8
33
|
|
|
9
34
|
Reconnection release. The gem retried reconnects but not connects, so a
|
data/README.md
CHANGED
|
@@ -72,6 +72,10 @@ Tuber.configure do |config|
|
|
|
72
72
|
# config.resolv_timeout = nil
|
|
73
73
|
# config.read_timeout = nil
|
|
74
74
|
# config.write_timeout = nil
|
|
75
|
+
# config.keepalive = true
|
|
76
|
+
# config.keepalive_idle = 30
|
|
77
|
+
# config.keepalive_interval = 10
|
|
78
|
+
# config.keepalive_count = 3
|
|
75
79
|
# config.connect_retries = 0
|
|
76
80
|
# config.connect_retry_interval = 1
|
|
77
81
|
end
|
|
@@ -79,7 +83,11 @@ end
|
|
|
79
83
|
|
|
80
84
|
The above options are all defaults, so only include a configuration block if you need to make changes.
|
|
81
85
|
|
|
82
|
-
`connect_timeout` and `resolv_timeout` are passed through to `TCPSocket.new` on Ruby 3.0 and newer (ignored on Ruby < 3.0 for compatibility).
|
|
86
|
+
`connect_timeout` and `resolv_timeout` are passed through to `TCPSocket.new` on Ruby 3.0 and newer (ignored on Ruby < 3.0 for compatibility).
|
|
87
|
+
|
|
88
|
+
`read_timeout` is how long to wait for a reply before raising `IO::TimeoutError`, *beyond* any wait the command asks for: `reserve-with-timeout 30` and `reserve_batch(10, 30)` get their 30 seconds added on, and a plain `reserve` has no client deadline at all. `write_timeout` is how long a write may block on a server that has stopped reading. Both need Ruby 3.2 or newer (`IO#timeout`). On older Rubies they fall back to `SO_RCVTIMEO`/`SO_SNDTIMEO`, which Ruby's non-blocking sockets ignore from 3.0 on.
|
|
89
|
+
|
|
90
|
+
`keepalive` turns on TCP keepalive (see "Surviving a server restart" below). The `keepalive_*` timings are in seconds, and each is set only where the platform has the option.
|
|
83
91
|
|
|
84
92
|
### Connection
|
|
85
93
|
|
|
@@ -115,6 +123,16 @@ end
|
|
|
115
123
|
down) and reconnects on a client you already hold. It defaults to `0` — a
|
|
116
124
|
single attempt — which is the historical behaviour.
|
|
117
125
|
|
|
126
|
+
A consumer blocked in `reserve` is waiting on the server, so it only notices
|
|
127
|
+
the socket has died if the server's host says so. A clean restart closes the
|
|
128
|
+
connection, but when the host vanishes without a word — a reboot that takes the
|
|
129
|
+
network down before the server, a power cut, a partition — the old connection
|
|
130
|
+
is left half-open. TCP keepalive (on by default) probes an idle connection after
|
|
131
|
+
`keepalive_idle` seconds, every `keepalive_interval` seconds, and gives up after
|
|
132
|
+
`keepalive_count` unanswered probes. The blocked read then fails (about 60
|
|
133
|
+
seconds with the defaults) and the connection heals as above: the `reserve` is
|
|
134
|
+
re-sent on a fresh socket, and the caller never sees the error.
|
|
135
|
+
|
|
118
136
|
When a command does surface `Tuber::NotConnected`, heal the client you have
|
|
119
137
|
rather than building a new one:
|
|
120
138
|
|
|
@@ -371,6 +389,98 @@ lower than expected means jobs hit their TTR and went back to the queue while th
|
|
|
371
389
|
worker was busy — otherwise invisible, since nothing notifies a worker that it
|
|
372
390
|
lost a job.
|
|
373
391
|
|
|
392
|
+
### Recurring Schedules (Tuber only)
|
|
393
|
+
|
|
394
|
+
Tuber can run an app's recurring jobs itself, so the app needs no scheduler
|
|
395
|
+
process (whenever, rufus-scheduler, sidekiq-cron, a `bin/scheduler` container).
|
|
396
|
+
The app sends its complete schedule under a name of its choosing, the *owner*,
|
|
397
|
+
and tuber puts each entry's body into its tube at the times the entry names. A
|
|
398
|
+
fire is an ordinary put: workers reserve, delete and bury it like any other job.
|
|
399
|
+
|
|
400
|
+
```ruby
|
|
401
|
+
tuber = Tuber.new
|
|
402
|
+
|
|
403
|
+
result = tuber.schedules.sync("splat") do |s|
|
|
404
|
+
s.add "monitor_evaluation", tube: "splat.checkins", every: 60,
|
|
405
|
+
body: { class: "Monitors::EvaluateJob", args: [] }.to_json,
|
|
406
|
+
idp: "monitor_evaluation", pri: 1024
|
|
407
|
+
s.add "histogram_rollup", tube: "splat.maintenance", cron: "5 * * * *",
|
|
408
|
+
body: { class: "Analytics::HistogramRollupJob", args: [] }.to_json,
|
|
409
|
+
idp: "histogram_rollup", con: "maintenance", pri: 1024
|
|
410
|
+
s.add "data_retention", tube: "splat.maintenance",
|
|
411
|
+
cron: "0 2 * * *", zone: "Australia/Melbourne",
|
|
412
|
+
body: { class: "Maintenance::RetentionJob", args: [] }.to_json,
|
|
413
|
+
idp: "data_retention", con: "maintenance", pri: 1024
|
|
414
|
+
end
|
|
415
|
+
result.to_h # => {added: 3, changed: 0, removed: 0, unchanged: 0}
|
|
416
|
+
```
|
|
417
|
+
|
|
418
|
+
- **Send the complete set every time.** An entry you leave out is removed. An
|
|
419
|
+
edited entry keeps its name and is updated in place. An identical sync changes
|
|
420
|
+
nothing, not even when entries next fire, so every replica may run it.
|
|
421
|
+
- **An empty set is refused** with `ArgumentError`, since it would remove every
|
|
422
|
+
entry the owner has. Pass `allow_empty: true` when that is what you mean.
|
|
423
|
+
- **Sync when a release starts, not on reconnect.** Use a release step (a Kamal
|
|
424
|
+
`pre-deploy` hook, a Heroku `release` phase) or sync once at process boot.
|
|
425
|
+
Tuber keeps schedules across its own restarts (with `-b`), and during a
|
|
426
|
+
rolling deploy an old process re-syncing after a reconnect would put back
|
|
427
|
+
entries the new release removed. The gem never syncs on its own.
|
|
428
|
+
- A sync uses a short-lived connection of its own, so a worker blocked in
|
|
429
|
+
`reserve` on the client's connection doesn't hold it up.
|
|
430
|
+
|
|
431
|
+
`s.add` takes the entry's name and:
|
|
432
|
+
|
|
433
|
+
| Option | |
|
|
434
|
+
| --- | --- |
|
|
435
|
+
| `tube:`, `body:` | Required. The body goes through `job_serializer`, as for `put` |
|
|
436
|
+
| `cron:` | 5-field cron or a macro (`"@daily"`); `zone:` adds an IANA zone (default UTC) |
|
|
437
|
+
| `every:` | Seconds, or `"10m"`, `"1h"`, `"1d"`, aligned to the Unix epoch; `offset:` shifts it |
|
|
438
|
+
| `spec:` | The spec as tuber reads it, instead of `cron:` or `every:` |
|
|
439
|
+
| `pri:`, `delay:`, `ttr:` | As for `put`, with the same defaults |
|
|
440
|
+
| `idp:`, `con:`, `grp:`, `aft:` | As for `put`: `idp: "key"` or `["key", ttl]`, `con: "key"` or `["key", limit]` |
|
|
441
|
+
| `catchup:` | For ticks missed while tuber was down: `:once` (default) fires once for the gap, an Integer fires it only if at most that many seconds late, `:never` skips it |
|
|
442
|
+
|
|
443
|
+
A `con:` value is a key, not a number: every job with `con: "maintenance"`
|
|
444
|
+
shares one slot, so entries that share it run one after another.
|
|
445
|
+
|
|
446
|
+
Anything tuber would refuse that the gem can see — a bad name, two specs, a body
|
|
447
|
+
that isn't a String — raises `ArgumentError` before anything is sent. When tuber
|
|
448
|
+
refuses the sync, it raises `Tuber::ScheduleSyncError`, and nothing changed: the
|
|
449
|
+
previous schedule keeps running. `#entry` names the refused entry (nil when tuber
|
|
450
|
+
refused the sync as a whole) and `#status` gives tuber's reply, such as
|
|
451
|
+
`BAD_SCHEDULE` for a zone typo like `Australia/Melborne`. The one exception is
|
|
452
|
+
`#applied?`: the commit took effect but tuber couldn't write it to disk
|
|
453
|
+
(`INTERNAL_ERROR`). It runs until tuber restarts, so sync again once the disk is
|
|
454
|
+
fixed.
|
|
455
|
+
|
|
456
|
+
```ruby
|
|
457
|
+
begin
|
|
458
|
+
tuber.schedules.sync("splat") { |s| ... }
|
|
459
|
+
rescue Tuber::ScheduleSyncError => e
|
|
460
|
+
Rails.logger.error(e.message) # names the entry and the reason
|
|
461
|
+
end
|
|
462
|
+
```
|
|
463
|
+
|
|
464
|
+
To see what tuber holds:
|
|
465
|
+
|
|
466
|
+
```ruby
|
|
467
|
+
tuber.schedules.list # => [["splat", "data_retention"], ...] across owners
|
|
468
|
+
tuber.schedules.list("splat")
|
|
469
|
+
stats = tuber.schedules.stats("splat", "data_retention") # nil if there's no such entry
|
|
470
|
+
stats.next_fire # => 2026-10-12 02:00:00 +1100 (next_fire_2, next_fire_3 too)
|
|
471
|
+
stats.last_result # => "INSERTED 4711"
|
|
472
|
+
stats.buried_streak # => 0
|
|
473
|
+
```
|
|
474
|
+
|
|
475
|
+
**`idp:` and buried jobs.** With `idp:`, a fire is skipped while the previous
|
|
476
|
+
fire's job is still in the tube, which is what you want when a run outlasts its
|
|
477
|
+
interval. But a *buried* job keeps its key too, so every later fire dedups
|
|
478
|
+
against it and queues nothing until it is kicked or deleted. `buried_streak`
|
|
479
|
+
counts those fires: alert when it is above 0.
|
|
480
|
+
|
|
481
|
+
Tuber's [scheduling guide](https://github.com/tuberq/tuber/blob/main/docs/scheduling.md)
|
|
482
|
+
covers spec syntax, daylight saving, catch-up and monitoring.
|
|
483
|
+
|
|
374
484
|
### Stats
|
|
375
485
|
|
|
376
486
|
```ruby
|
|
@@ -393,6 +503,13 @@ Tuber.configure do |config|
|
|
|
393
503
|
# Connect attempts; see "Surviving a server restart" above.
|
|
394
504
|
config.connect_retries = 0
|
|
395
505
|
config.connect_retry_interval = 1
|
|
506
|
+
|
|
507
|
+
# TCP keepalive: notice a server whose host vanished in about
|
|
508
|
+
# idle + interval * count seconds. See "Surviving a server restart".
|
|
509
|
+
config.keepalive = true
|
|
510
|
+
config.keepalive_idle = 30
|
|
511
|
+
config.keepalive_interval = 10
|
|
512
|
+
config.keepalive_count = 3
|
|
396
513
|
end
|
|
397
514
|
```
|
|
398
515
|
|
|
@@ -413,6 +530,8 @@ end
|
|
|
413
530
|
| Tuber::NotFoundError | Job or tube not found. |
|
|
414
531
|
| Tuber::TimedOutError | Reserve timed out. |
|
|
415
532
|
| Tuber::JobNotReserved | Action requires a reserved job. |
|
|
533
|
+
| Tuber::OutOfStorageError | A persistent Tuber's disk budget (`--max-storage-bytes`) is spent. |
|
|
534
|
+
| Tuber::ScheduleSyncError | Tuber refused a schedule sync; see Recurring Schedules. |
|
|
416
535
|
|
|
417
536
|
See the [Tuber protocol](https://github.com/tuberq/tuber/blob/main/docs/protocol.md) (a superset of the [beanstalk protocol](https://github.com/beanstalkd/beanstalkd/blob/master/doc/protocol.txt)) for additional error types.
|
|
418
537
|
|
data/lib/tuber/configuration.rb
CHANGED
|
@@ -12,8 +12,30 @@ class Tuber
|
|
|
12
12
|
alias_method :beanstalkd_url=, :tuber_url=
|
|
13
13
|
attr_accessor :connect_timeout # TCP connect timeout in seconds
|
|
14
14
|
attr_accessor :resolv_timeout # DNS resolve timeout in seconds
|
|
15
|
-
|
|
16
|
-
|
|
15
|
+
|
|
16
|
+
# Seconds to wait for a reply before raising IO::TimeoutError, beyond any
|
|
17
|
+
# wait the command itself asks for: reserve-with-timeout and reserve-batch
|
|
18
|
+
# get their own timeout added on, and a plain reserve has no deadline at
|
|
19
|
+
# all. Needs Ruby >= 3.2 (IO#timeout); older Rubies fall back to
|
|
20
|
+
# SO_RCVTIMEO, which Ruby's non-blocking sockets ignore from 3.0 on.
|
|
21
|
+
attr_accessor :read_timeout
|
|
22
|
+
|
|
23
|
+
# Seconds a write may block (the server not reading) before raising
|
|
24
|
+
# IO::TimeoutError. Same Ruby caveat as read_timeout.
|
|
25
|
+
attr_accessor :write_timeout
|
|
26
|
+
|
|
27
|
+
# TCP keepalive, on by default. A blocked read waits on the server, so it
|
|
28
|
+
# never learns the server is gone if the server's host vanished without
|
|
29
|
+
# closing the connection: a reboot that takes the network down first, a
|
|
30
|
+
# power cut, a partition. Keepalive probes an idle connection after
|
|
31
|
+
# keepalive_idle seconds, every keepalive_interval seconds, and gives up
|
|
32
|
+
# after keepalive_count unanswered probes, so the read fails (and the
|
|
33
|
+
# connection heals) after about idle + interval * count seconds: 60 with
|
|
34
|
+
# the defaults.
|
|
35
|
+
attr_accessor :keepalive
|
|
36
|
+
attr_accessor :keepalive_idle # idle seconds before the first probe
|
|
37
|
+
attr_accessor :keepalive_interval # seconds between probes
|
|
38
|
+
attr_accessor :keepalive_count # unanswered probes before giving up
|
|
17
39
|
|
|
18
40
|
# How hard a client tries to connect — "how long should this ride out a
|
|
19
41
|
# server restart", in one knob.
|
|
@@ -41,6 +63,10 @@ class Tuber
|
|
|
41
63
|
@resolv_timeout = nil
|
|
42
64
|
@read_timeout = nil
|
|
43
65
|
@write_timeout = nil
|
|
66
|
+
@keepalive = true
|
|
67
|
+
@keepalive_idle = 30
|
|
68
|
+
@keepalive_interval = 10
|
|
69
|
+
@keepalive_count = 3
|
|
44
70
|
@connect_retries = 0
|
|
45
71
|
# Matches Tuber::Connection::DEFAULT_RETRY_INTERVAL, which is not loaded yet.
|
|
46
72
|
@connect_retry_interval = 1
|
data/lib/tuber/connection.rb
CHANGED
|
@@ -78,6 +78,21 @@ class Tuber
|
|
|
78
78
|
(defined?(IO::TimeoutError) ? [IO::TimeoutError] : [])
|
|
79
79
|
).freeze
|
|
80
80
|
|
|
81
|
+
# What a blocked read or write raises once the server is judged gone
|
|
82
|
+
# without having said so: Errno::ETIMEDOUT when keepalive probes go
|
|
83
|
+
# unanswered, IO::TimeoutError (Ruby >= 3.2) when read_timeout or
|
|
84
|
+
# write_timeout runs out. The socket is useless afterwards either way: a
|
|
85
|
+
# reply still in flight could arrive later and be read as the answer to
|
|
86
|
+
# the next command.
|
|
87
|
+
TIMEOUT_ERRORS = (
|
|
88
|
+
[Errno::ETIMEDOUT] + (defined?(IO::TimeoutError) ? [IO::TimeoutError] : [])
|
|
89
|
+
).freeze
|
|
90
|
+
|
|
91
|
+
# Whether this Ruby has IO#timeout (3.2 and newer), the only read and
|
|
92
|
+
# write timeout its non-blocking sockets honour. SO_RCVTIMEO and
|
|
93
|
+
# SO_SNDTIMEO are kept for older Rubies but have no effect from 3.0 on.
|
|
94
|
+
IO_TIMEOUTS = IO.method_defined?(:timeout=)
|
|
95
|
+
|
|
81
96
|
# Initializes new connection.
|
|
82
97
|
#
|
|
83
98
|
# @param [String] address beanstalkd instance address.
|
|
@@ -127,7 +142,7 @@ class Tuber
|
|
|
127
142
|
_raise_not_connected! unless connection
|
|
128
143
|
|
|
129
144
|
command = command.dup.force_encoding('ASCII-8BIT') if command.respond_to?(:force_encoding)
|
|
130
|
-
|
|
145
|
+
_write(command.to_s + "\r\n", read_timeout: _read_timeout_for(command))
|
|
131
146
|
res = connection.readline
|
|
132
147
|
parse_response(command, res)
|
|
133
148
|
end
|
|
@@ -152,7 +167,7 @@ class Tuber
|
|
|
152
167
|
_raise_not_connected! unless connection
|
|
153
168
|
|
|
154
169
|
cmd = timeout ? "reserve-batch #{count} #{timeout}" : "reserve-batch #{count}"
|
|
155
|
-
|
|
170
|
+
_write(cmd + "\r\n", read_timeout: _read_timeout_for(cmd))
|
|
156
171
|
|
|
157
172
|
header = connection.readline.chomp
|
|
158
173
|
status, actual_count_str = header.split(/\s/, 2)
|
|
@@ -188,7 +203,7 @@ class Tuber
|
|
|
188
203
|
_raise_not_connected! unless connection
|
|
189
204
|
|
|
190
205
|
cmd = "delete-batch #{ids.join(' ')}"
|
|
191
|
-
|
|
206
|
+
_write(cmd + "\r\n")
|
|
192
207
|
|
|
193
208
|
res = connection.readline.chomp
|
|
194
209
|
status, deleted, not_found = res.split(/\s/)
|
|
@@ -200,6 +215,81 @@ class Tuber
|
|
|
200
215
|
end
|
|
201
216
|
end
|
|
202
217
|
|
|
218
|
+
# Sends a command answered with `OK <bytes>` and returns the body
|
|
219
|
+
# unparsed. For replies whose values #parse_response would mangle:
|
|
220
|
+
# its Psych workaround rewrites `:`, `-` and `~` inside every value, and
|
|
221
|
+
# stats-schedule's times, specs and tags are full of them.
|
|
222
|
+
#
|
|
223
|
+
# @param [String] command A command whose success reply is `OK <bytes>`
|
|
224
|
+
# @return [String] The reply body (the YAML document), without its CRLF
|
|
225
|
+
# @raise [Tuber::UnexpectedResponse] Any other reply, e.g. NotFoundError
|
|
226
|
+
#
|
|
227
|
+
def transmit_raw(command)
|
|
228
|
+
_with_retry do
|
|
229
|
+
@mutex.synchronize do
|
|
230
|
+
_raise_not_connected! unless connection
|
|
231
|
+
|
|
232
|
+
_write(command + "\r\n")
|
|
233
|
+
status, bytes = connection.readline.chomp.split(/\s/, 2)
|
|
234
|
+
raise UnexpectedResponse.from_status(status, command) unless status == 'OK'
|
|
235
|
+
|
|
236
|
+
body = connection.read(bytes.to_i)
|
|
237
|
+
crlf = connection.read(2)
|
|
238
|
+
raise ExpectedCrlfError.new('EXPECTED_CRLF', command) unless crlf == "\r\n"
|
|
239
|
+
body.force_encoding(Encoding::UTF_8)
|
|
240
|
+
end
|
|
241
|
+
end
|
|
242
|
+
end
|
|
243
|
+
|
|
244
|
+
# Replaces +owner+'s recurring schedule with +entries+ in one exchange:
|
|
245
|
+
# schedule-begin, a schedule-put per entry, schedule-commit. The mutex is
|
|
246
|
+
# held throughout, so no other command interleaves with it on the socket.
|
|
247
|
+
#
|
|
248
|
+
# A dropped socket makes the server discard what was staged, so the whole
|
|
249
|
+
# exchange is re-run on a fresh connection — never resumed partway. That
|
|
250
|
+
# is safe because a sync is idempotent; if the drop came after the commit
|
|
251
|
+
# landed, the re-run reports every entry as unchanged.
|
|
252
|
+
#
|
|
253
|
+
# Run it on a connection of its own, and close that afterwards, as
|
|
254
|
+
# Tuber::Schedules#sync does: any other error partway can leave the server
|
|
255
|
+
# mid-sync on this socket with a reply unread.
|
|
256
|
+
#
|
|
257
|
+
# @param [String] owner The owner whose entries are replaced
|
|
258
|
+
# @param [Array<Tuber::Schedules::Entry>] entries The complete new set
|
|
259
|
+
# @return [Tuber::Schedules::SyncResult]
|
|
260
|
+
# @raise [Tuber::ScheduleSyncError] The server refused the sync
|
|
261
|
+
# @api private Use Tuber::Schedules#sync, which validates the entries
|
|
262
|
+
#
|
|
263
|
+
def schedule_sync(owner, entries)
|
|
264
|
+
_with_retry do
|
|
265
|
+
@mutex.synchronize do
|
|
266
|
+
_raise_not_connected! unless connection
|
|
267
|
+
|
|
268
|
+
cmd = "schedule-begin #{owner}"
|
|
269
|
+
reply = _round_trip(cmd)
|
|
270
|
+
raise ScheduleSyncError.new(reply[/\A\S*/], cmd, owner: owner) unless reply == "SYNCING #{owner}"
|
|
271
|
+
|
|
272
|
+
entries.each do |entry|
|
|
273
|
+
cmd = "schedule-put #{entry.line}"
|
|
274
|
+
reply = _round_trip(cmd, entry.body)
|
|
275
|
+
next if reply == 'STAGED'
|
|
276
|
+
|
|
277
|
+
# The refusal has already doomed the sync; the commit would only
|
|
278
|
+
# say ABORTED. Abort now so the error names the entry, and so the
|
|
279
|
+
# connection leaves sync mode before the caller sees it.
|
|
280
|
+
_round_trip('schedule-abort')
|
|
281
|
+
raise ScheduleSyncError.new(reply, cmd, owner: owner, entry: entry.name)
|
|
282
|
+
end
|
|
283
|
+
|
|
284
|
+
status, *counts = _round_trip('schedule-commit').split(' ')
|
|
285
|
+
unless status == 'SYNCED' && counts.size == 4
|
|
286
|
+
raise ScheduleSyncError.new(status.to_s, 'schedule-commit', owner: owner)
|
|
287
|
+
end
|
|
288
|
+
Schedules::SyncResult.new(*counts.map(&:to_i))
|
|
289
|
+
end
|
|
290
|
+
end
|
|
291
|
+
end
|
|
292
|
+
|
|
203
293
|
# Close connection with beanstalkd server.
|
|
204
294
|
#
|
|
205
295
|
# @example
|
|
@@ -312,8 +402,11 @@ class Tuber
|
|
|
312
402
|
end
|
|
313
403
|
|
|
314
404
|
begin
|
|
315
|
-
socket
|
|
316
|
-
|
|
405
|
+
_enable_keepalive(socket) if config.keepalive
|
|
406
|
+
unless IO_TIMEOUTS # IO#timeout is set per exchange instead, in #_write
|
|
407
|
+
socket.setsockopt(Socket::SOL_SOCKET, Socket::SO_RCVTIMEO, _timeval_for(config.read_timeout)) if config.read_timeout
|
|
408
|
+
socket.setsockopt(Socket::SOL_SOCKET, Socket::SO_SNDTIMEO, _timeval_for(config.write_timeout)) if config.write_timeout
|
|
409
|
+
end
|
|
317
410
|
@connection = socket
|
|
318
411
|
rescue
|
|
319
412
|
socket.close rescue nil
|
|
@@ -393,6 +486,17 @@ class Tuber
|
|
|
393
486
|
transmit("reserve-mode #{@reserve_mode}", init: false) if @reserve_mode && @reserve_mode.to_s != 'fifo'
|
|
394
487
|
end
|
|
395
488
|
|
|
489
|
+
# Writes one command, and a body framed as for put when given, and reads
|
|
490
|
+
# the one-line reply. Callers hold the mutex.
|
|
491
|
+
#
|
|
492
|
+
# @return [String] The reply line without its CRLF
|
|
493
|
+
def _round_trip(command, body = nil)
|
|
494
|
+
frame = "#{command}\r\n".b
|
|
495
|
+
frame << body.b << "\r\n" if body
|
|
496
|
+
_write(frame)
|
|
497
|
+
connection.readline.chomp
|
|
498
|
+
end
|
|
499
|
+
|
|
396
500
|
# Wrapper method for capturing certain failures and retry the payload block
|
|
397
501
|
#
|
|
398
502
|
# @param [Proc] block The command to execute.
|
|
@@ -403,22 +507,88 @@ class Tuber
|
|
|
403
507
|
# @param [Integer] tries The maximum number of tries in draining mode
|
|
404
508
|
# @return [Object] Result of the block passed
|
|
405
509
|
#
|
|
510
|
+
# A timeout (see TIMEOUT_ERRORS) is retransmitted at most once. One fresh
|
|
511
|
+
# socket heals a connection whose server vanished; a second timeout on it
|
|
512
|
+
# means the server is up but not answering, and the caller hears so
|
|
513
|
+
# rather than waiting on a loop that never ends.
|
|
514
|
+
#
|
|
406
515
|
def _with_retry(retry_interval: DEFAULT_RETRY_INTERVAL, init: true, tries: MAX_RETRIES, retransmit: true, &block)
|
|
407
|
-
|
|
408
|
-
|
|
409
|
-
|
|
410
|
-
|
|
411
|
-
|
|
412
|
-
|
|
413
|
-
|
|
414
|
-
|
|
415
|
-
|
|
416
|
-
|
|
417
|
-
|
|
418
|
-
|
|
516
|
+
timed_out = false
|
|
517
|
+
begin
|
|
518
|
+
yield
|
|
519
|
+
rescue EOFError, Errno::ECONNRESET, Errno::EPIPE,
|
|
520
|
+
Errno::ECONNREFUSED, *TIMEOUT_ERRORS => ex
|
|
521
|
+
_reconnect(tries: _connect_tries, retry_interval: retry_interval)
|
|
522
|
+
_initialize_tubes if init
|
|
523
|
+
raise ex unless retransmit
|
|
524
|
+
if TIMEOUT_ERRORS.any? { |klass| ex.is_a?(klass) }
|
|
525
|
+
raise ex if timed_out
|
|
526
|
+
timed_out = true
|
|
527
|
+
end
|
|
528
|
+
retry
|
|
529
|
+
rescue Tuber::DrainingError
|
|
530
|
+
tries -= 1
|
|
531
|
+
if tries.zero?
|
|
532
|
+
close
|
|
533
|
+
raise
|
|
534
|
+
end
|
|
535
|
+
sleep(retry_interval)
|
|
536
|
+
retry
|
|
537
|
+
end
|
|
538
|
+
end
|
|
539
|
+
|
|
540
|
+
# Writes a command, and any body framed with it, then sets how long the
|
|
541
|
+
# reads of its reply may block. Callers hold the mutex.
|
|
542
|
+
#
|
|
543
|
+
# On Ruby >= 3.2 both limits are IO#timeout, swapped between the write and
|
|
544
|
+
# the reads; older Rubies get the SO_*TIMEO set at connect instead.
|
|
545
|
+
#
|
|
546
|
+
# @param [String] data The bytes to send
|
|
547
|
+
# @param [Numeric, nil] read_timeout Seconds each read of the reply may
|
|
548
|
+
# block, nil for no limit (see #_read_timeout_for)
|
|
549
|
+
def _write(data, read_timeout: config.read_timeout)
|
|
550
|
+
connection.timeout = config.write_timeout if IO_TIMEOUTS
|
|
551
|
+
connection.write(data)
|
|
552
|
+
connection.timeout = read_timeout if IO_TIMEOUTS
|
|
553
|
+
end
|
|
554
|
+
|
|
555
|
+
# How long the reply to +command+ may take: read_timeout, plus however
|
|
556
|
+
# long the command asks the server to hold it before answering. A plain
|
|
557
|
+
# reserve waits as long as it takes by design, so it has no deadline;
|
|
558
|
+
# keepalive still notices a server that has vanished under it.
|
|
559
|
+
#
|
|
560
|
+
# @param [String] command The command line sent
|
|
561
|
+
# @return [Numeric, nil] Seconds, or nil for no limit
|
|
562
|
+
def _read_timeout_for(command)
|
|
563
|
+
limit = config.read_timeout
|
|
564
|
+
return limit unless limit
|
|
565
|
+
|
|
566
|
+
words = command.to_s[/\A[^\r\n]*/].split
|
|
567
|
+
case words[0]
|
|
568
|
+
when 'reserve' then nil
|
|
569
|
+
when 'reserve-with-timeout' then limit + words[1].to_i
|
|
570
|
+
when 'reserve-batch' then limit + words[2].to_i # count, then timeout
|
|
571
|
+
else limit
|
|
572
|
+
end
|
|
573
|
+
end
|
|
574
|
+
|
|
575
|
+
# Turns on TCP keepalive with the configured probe timings, so a read
|
|
576
|
+
# blocked on a server that vanished without closing the connection fails
|
|
577
|
+
# with Errno::ETIMEDOUT (or ECONNRESET, if the host is back and refuses
|
|
578
|
+
# the probe) instead of waiting forever. Each timing is set only where the
|
|
579
|
+
# platform has the option; macOS calls the idle time TCP_KEEPALIVE.
|
|
580
|
+
#
|
|
581
|
+
# @param [TCPSocket] socket A freshly connected socket
|
|
582
|
+
def _enable_keepalive(socket)
|
|
583
|
+
socket.setsockopt(Socket::SOL_SOCKET, Socket::SO_KEEPALIVE, true)
|
|
584
|
+
{
|
|
585
|
+
(Socket.const_defined?(:TCP_KEEPIDLE) ? :TCP_KEEPIDLE : :TCP_KEEPALIVE) => config.keepalive_idle,
|
|
586
|
+
:TCP_KEEPINTVL => config.keepalive_interval,
|
|
587
|
+
:TCP_KEEPCNT => config.keepalive_count,
|
|
588
|
+
}.each do |name, value|
|
|
589
|
+
next unless value && Socket.const_defined?(name)
|
|
590
|
+
socket.setsockopt(Socket::IPPROTO_TCP, Socket.const_get(name), value.ceil)
|
|
419
591
|
end
|
|
420
|
-
sleep(retry_interval)
|
|
421
|
-
retry
|
|
422
592
|
end
|
|
423
593
|
|
|
424
594
|
# Connect attempts to allow when healing a connection that was already
|
data/lib/tuber/errors.rb
CHANGED
|
@@ -11,7 +11,7 @@ class Tuber
|
|
|
11
11
|
# Abstract class for errors that occur when a command does not complete successfully.
|
|
12
12
|
class UnexpectedResponse < RuntimeError
|
|
13
13
|
# Set of status states that are considered errors
|
|
14
|
-
ERROR_STATES = %w(OUT_OF_MEMORY INTERNAL_ERROR
|
|
14
|
+
ERROR_STATES = %w(OUT_OF_MEMORY OUT_OF_STORAGE INTERNAL_ERROR
|
|
15
15
|
BAD_FORMAT UNKNOWN_COMMAND JOB_TOO_BIG DRAINING
|
|
16
16
|
TIMED_OUT DEADLINE_SOON NOT_FOUND NOT_IGNORED EXPECTED_CRLF)
|
|
17
17
|
|
|
@@ -27,13 +27,14 @@ class Tuber
|
|
|
27
27
|
#
|
|
28
28
|
# @param [Tuber::UnexpectedResponse] status Unexpected response object
|
|
29
29
|
# @param [String] cmd Beanstalkd request command
|
|
30
|
+
# @param [String, nil] message Overrides the default "Response failed with: <status>"
|
|
30
31
|
#
|
|
31
32
|
# @example
|
|
32
33
|
# Tuber::UnexpectedResponse.new(NotFoundError, 'bury 123')
|
|
33
34
|
#
|
|
34
|
-
def initialize(status, cmd)
|
|
35
|
+
def initialize(status, cmd, message = nil)
|
|
35
36
|
@status, @cmd = status, cmd
|
|
36
|
-
super("Response failed with: #{status}")
|
|
37
|
+
super(message || "Response failed with: #{status}")
|
|
37
38
|
end
|
|
38
39
|
|
|
39
40
|
# Translate beanstalkd error status to ruby Exeception
|
|
@@ -55,6 +56,8 @@ class Tuber
|
|
|
55
56
|
|
|
56
57
|
# Raises when the beanstalkd instance runs out of memory
|
|
57
58
|
class OutOfMemoryError < UnexpectedResponse; end
|
|
59
|
+
# Raises when a persistent Tuber server's disk budget (--max-storage-bytes) is spent
|
|
60
|
+
class OutOfStorageError < UnexpectedResponse; end
|
|
58
61
|
# Raises when the beanstalkd instance is draining and new jobs cannot be inserted
|
|
59
62
|
class DrainingError < UnexpectedResponse; end
|
|
60
63
|
# Raises when the job or tube cannot be found
|
|
@@ -75,4 +78,55 @@ class Tuber
|
|
|
75
78
|
class TimedOutError < UnexpectedResponse; end
|
|
76
79
|
# Raises when a tube could not be ignored because it is the last watched tube.
|
|
77
80
|
class NotIgnoredError < UnexpectedResponse; end
|
|
81
|
+
|
|
82
|
+
# Raises when a Tuber server refuses a schedule sync (see Tuber::Schedules#sync).
|
|
83
|
+
#
|
|
84
|
+
# Unless #applied? is true, nothing changed: the owner's previous schedule
|
|
85
|
+
# is still the one in effect.
|
|
86
|
+
class ScheduleSyncError < UnexpectedResponse
|
|
87
|
+
# @!attribute owner
|
|
88
|
+
# @return [String] the owner whose sync was refused
|
|
89
|
+
# @!attribute entry
|
|
90
|
+
# @return [String, nil] the entry tuber refused, nil when it refused the
|
|
91
|
+
# sync as a whole (on begin or commit)
|
|
92
|
+
attr_reader :owner, :entry
|
|
93
|
+
|
|
94
|
+
# @param [String] status The server's reply, e.g. "BAD_SCHEDULE"
|
|
95
|
+
# @param [String] cmd The command it refused
|
|
96
|
+
# @param [String] owner The owner being synced
|
|
97
|
+
# @param [String, nil] entry The refused entry's name
|
|
98
|
+
def initialize(status, cmd, owner:, entry: nil)
|
|
99
|
+
@owner, @entry = owner, entry
|
|
100
|
+
super(status, cmd, describe(status, cmd, owner, entry))
|
|
101
|
+
end
|
|
102
|
+
|
|
103
|
+
# True when the new set is live despite the error: the commit took effect
|
|
104
|
+
# in memory but tuber couldn't write it to disk (INTERNAL_ERROR). A tuber
|
|
105
|
+
# restart before the next successful sync brings back the previous set.
|
|
106
|
+
#
|
|
107
|
+
# @return [Boolean]
|
|
108
|
+
def applied?
|
|
109
|
+
cmd == 'schedule-commit' && status == 'INTERNAL_ERROR'
|
|
110
|
+
end
|
|
111
|
+
|
|
112
|
+
private
|
|
113
|
+
|
|
114
|
+
def describe(status, cmd, owner, entry)
|
|
115
|
+
if entry
|
|
116
|
+
"tuber refused schedule #{entry.inspect} (#{status}): #{cmd}. " \
|
|
117
|
+
"Nothing changed: #{owner.inspect}'s previous schedule is still in effect."
|
|
118
|
+
elsif cmd == 'schedule-commit' && status == 'INTERNAL_ERROR'
|
|
119
|
+
"#{owner.inspect}'s new schedule is live, but tuber couldn't write it to disk " \
|
|
120
|
+
"(INTERNAL_ERROR). A tuber restart before the next successful sync brings back the previous one."
|
|
121
|
+
elsif cmd == 'schedule-commit'
|
|
122
|
+
"tuber refused #{owner.inspect}'s schedule on commit (#{status}). " \
|
|
123
|
+
"Nothing changed: the previous schedule is still in effect."
|
|
124
|
+
elsif status == 'UNKNOWN_COMMAND'
|
|
125
|
+
"this server has no recurring schedules (UNKNOWN_COMMAND to #{cmd.inspect}): " \
|
|
126
|
+
"it is beanstalkd, or a tuber older than the scheduler."
|
|
127
|
+
else
|
|
128
|
+
"tuber refused to begin a schedule sync for #{owner.inspect} (#{status})."
|
|
129
|
+
end
|
|
130
|
+
end
|
|
131
|
+
end
|
|
78
132
|
end
|