robin 0.0.4 → 0.1.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/CHANGELOG.md +372 -0
- data/COPYING +674 -0
- data/LICENSE +165 -0
- data/README.md +1490 -0
- data/assets/favicon.svg +31 -0
- data/assets/robin-logo.svg +37 -0
- data/assets/robin-mono.svg +12 -0
- data/assets/robin-wordmark.svg +4 -0
- data/assets/robin.svg +34 -0
- data/exe/robin +5 -0
- data/ext/robin_resp/extconf.rb +29 -0
- data/ext/robin_resp/robin_resp.c +645 -0
- data/lib/active_job/queue_adapters/robin_adapter.rb +3 -0
- data/lib/generators/robin/install/install_generator.rb +39 -0
- data/lib/generators/robin/install/templates/initializer.rb.tt +31 -0
- data/lib/generators/robin/install/templates/robin.yml.tt +25 -0
- data/lib/generators/robin/job/job_generator.rb +78 -0
- data/lib/generators/robin/job/templates/job.rb.tt +13 -0
- data/lib/generators/robin/job/templates/job_spec.rb.tt +21 -0
- data/lib/generators/robin/job/templates/job_test.rb.tt +21 -0
- data/lib/puma/plugin/robin.rb +131 -0
- data/lib/robin/active_job.rb +144 -0
- data/lib/robin/api.rb +777 -0
- data/lib/robin/batch.rb +700 -0
- data/lib/robin/cancellation.rb +62 -0
- data/lib/robin/cli.rb +325 -0
- data/lib/robin/client.rb +355 -21
- data/lib/robin/config.rb +454 -4
- data/lib/robin/cron.rb +325 -0
- data/lib/robin/debounce.rb +108 -0
- data/lib/robin/deploy.rb +65 -0
- data/lib/robin/embedded.rb +53 -0
- data/lib/robin/encryption.rb +147 -0
- data/lib/robin/events.rb +60 -0
- data/lib/robin/expiry.rb +41 -0
- data/lib/robin/fetcher.rb +441 -0
- data/lib/robin/gvl.rb +62 -0
- data/lib/robin/health.rb +76 -0
- data/lib/robin/iterable_job.rb +294 -0
- data/lib/robin/job.rb +258 -0
- data/lib/robin/job_class.rb +157 -0
- data/lib/robin/job_logger.rb +121 -0
- data/lib/robin/launcher.rb +658 -0
- data/lib/robin/leader.rb +86 -0
- data/lib/robin/limiter.rb +639 -0
- data/lib/robin/manager.rb +125 -0
- data/lib/robin/metrics.rb +303 -0
- data/lib/robin/middleware.rb +165 -0
- data/lib/robin/opentelemetry.rb +88 -0
- data/lib/robin/periodic.rb +327 -0
- data/lib/robin/processor.rb +317 -0
- data/lib/robin/profiler.rb +140 -0
- data/lib/robin/progress.rb +46 -0
- data/lib/robin/rails/app_loader.rb +23 -0
- data/lib/robin/rails/current_attributes.rb +88 -0
- data/lib/robin/rails/error_reporter.rb +31 -0
- data/lib/robin/rails/notifications.rb +22 -0
- data/lib/robin/rails/railtie.rb +87 -0
- data/lib/robin/rails/transactional_push.rb +49 -0
- data/lib/robin/rails.rb +23 -0
- data/lib/robin/recovery.rb +135 -0
- data/lib/robin/redis/connection.rb +639 -0
- data/lib/robin/redis/pool.rb +190 -0
- data/lib/robin/redis/resp.rb +77 -0
- data/lib/robin/reliable_push.rb +153 -0
- data/lib/robin/retrier.rb +284 -0
- data/lib/robin/scheduler.rb +102 -0
- data/lib/robin/scripts.rb +331 -0
- data/lib/robin/statsd.rb +135 -0
- data/lib/robin/swarm.rb +319 -0
- data/lib/robin/systemd.rb +47 -0
- data/lib/robin/testing.rb +158 -0
- data/lib/robin/top.rb +178 -0
- data/lib/robin/unique.rb +198 -0
- data/lib/robin/version.rb +5 -0
- data/lib/robin/warmup.rb +94 -0
- data/lib/robin/web/api.rb +580 -0
- data/lib/robin/web/batch_status.rb +44 -0
- data/lib/robin/web/dist/assets/ar-y7v9b75f.js +1 -0
- data/lib/robin/web/dist/assets/ar-y7v9b75f.js.gz +0 -0
- data/lib/robin/web/dist/assets/bricolage-0qmzcpnj.woff2 +0 -0
- data/lib/robin/web/dist/assets/cs-df88mq5e.js +1 -0
- data/lib/robin/web/dist/assets/cs-df88mq5e.js.gz +0 -0
- data/lib/robin/web/dist/assets/da-kemms74g.js +1 -0
- data/lib/robin/web/dist/assets/da-kemms74g.js.gz +0 -0
- data/lib/robin/web/dist/assets/de-z8kkqg65.js +1 -0
- data/lib/robin/web/dist/assets/de-z8kkqg65.js.gz +0 -0
- data/lib/robin/web/dist/assets/es-9ahrrn1s.js +1 -0
- data/lib/robin/web/dist/assets/es-9ahrrn1s.js.gz +0 -0
- data/lib/robin/web/dist/assets/fa-7r0x28nc.js +1 -0
- data/lib/robin/web/dist/assets/fa-7r0x28nc.js.gz +0 -0
- data/lib/robin/web/dist/assets/favicon-1ebfcc0182.svg +31 -0
- data/lib/robin/web/dist/assets/favicon-1ebfcc0182.svg.gz +0 -0
- data/lib/robin/web/dist/assets/fr-r7mqf3d9.js +1 -0
- data/lib/robin/web/dist/assets/fr-r7mqf3d9.js.gz +0 -0
- data/lib/robin/web/dist/assets/he-7t46b6rt.js +1 -0
- data/lib/robin/web/dist/assets/he-7t46b6rt.js.gz +0 -0
- data/lib/robin/web/dist/assets/hi-zj434tx4.js +1 -0
- data/lib/robin/web/dist/assets/hi-zj434tx4.js.gz +0 -0
- data/lib/robin/web/dist/assets/it-pqc2m7gm.js +1 -0
- data/lib/robin/web/dist/assets/it-pqc2m7gm.js.gz +0 -0
- data/lib/robin/web/dist/assets/ja-k9rgm1pj.js +1 -0
- data/lib/robin/web/dist/assets/ja-k9rgm1pj.js.gz +0 -0
- data/lib/robin/web/dist/assets/ko-bxersv91.js +1 -0
- data/lib/robin/web/dist/assets/ko-bxersv91.js.gz +0 -0
- data/lib/robin/web/dist/assets/main-anw3f6ba.css +1 -0
- data/lib/robin/web/dist/assets/main-anw3f6ba.css.gz +0 -0
- data/lib/robin/web/dist/assets/main-pdtvjejb.js +7 -0
- data/lib/robin/web/dist/assets/main-pdtvjejb.js.gz +0 -0
- data/lib/robin/web/dist/assets/main-zz8zzhwn.js +2 -0
- data/lib/robin/web/dist/assets/main-zz8zzhwn.js.gz +0 -0
- data/lib/robin/web/dist/assets/nb-fjnc5064.js +1 -0
- data/lib/robin/web/dist/assets/nb-fjnc5064.js.gz +0 -0
- data/lib/robin/web/dist/assets/nl-9p8b95b6.js +1 -0
- data/lib/robin/web/dist/assets/nl-9p8b95b6.js.gz +0 -0
- data/lib/robin/web/dist/assets/pl-8xsrdn8n.js +1 -0
- data/lib/robin/web/dist/assets/pl-8xsrdn8n.js.gz +0 -0
- data/lib/robin/web/dist/assets/pt-BR-650j7cvg.js +1 -0
- data/lib/robin/web/dist/assets/pt-BR-650j7cvg.js.gz +0 -0
- data/lib/robin/web/dist/assets/robin-ae4a4aedd9.svg +34 -0
- data/lib/robin/web/dist/assets/robin-ae4a4aedd9.svg.gz +0 -0
- data/lib/robin/web/dist/assets/robin-wordmark-5dde5d8e4b.svg +4 -0
- data/lib/robin/web/dist/assets/robin-wordmark-5dde5d8e4b.svg.gz +0 -0
- data/lib/robin/web/dist/assets/ru-a9ftyw3d.js +1 -0
- data/lib/robin/web/dist/assets/ru-a9ftyw3d.js.gz +0 -0
- data/lib/robin/web/dist/assets/sv-j9gqamr1.js +1 -0
- data/lib/robin/web/dist/assets/sv-j9gqamr1.js.gz +0 -0
- data/lib/robin/web/dist/assets/tr-y9z0q51g.js +1 -0
- data/lib/robin/web/dist/assets/tr-y9z0q51g.js.gz +0 -0
- data/lib/robin/web/dist/assets/uk-2dr9ew3p.js +1 -0
- data/lib/robin/web/dist/assets/uk-2dr9ew3p.js.gz +0 -0
- data/lib/robin/web/dist/assets/ur-h25nzq1s.js +1 -0
- data/lib/robin/web/dist/assets/ur-h25nzq1s.js.gz +0 -0
- data/lib/robin/web/dist/assets/vi-9y2ff1bd.js +1 -0
- data/lib/robin/web/dist/assets/vi-9y2ff1bd.js.gz +0 -0
- data/lib/robin/web/dist/assets/zh-CN-tbj9mf98.js +1 -0
- data/lib/robin/web/dist/assets/zh-CN-tbj9mf98.js.gz +0 -0
- data/lib/robin/web/dist/assets/zh-TW-59q8j9vw.js +1 -0
- data/lib/robin/web/dist/assets/zh-TW-59q8j9vw.js.gz +0 -0
- data/lib/robin/web/dist/index.html +19 -0
- data/lib/robin/web/extension.rb +193 -0
- data/lib/robin/web/permissions.rb +40 -0
- data/lib/robin/web/prometheus.rb +79 -0
- data/lib/robin/web/router.rb +150 -0
- data/lib/robin/web/static.rb +110 -0
- data/lib/robin/web.rb +196 -0
- data/lib/robin.rb +258 -8
- data/sig/manifest.yaml +9 -0
- data/sig/robin/api.rbs +442 -0
- data/sig/robin/batch.rbs +292 -0
- data/sig/robin/cancellation.rbs +25 -0
- data/sig/robin/cli.rbs +102 -0
- data/sig/robin/client.rbs +118 -0
- data/sig/robin/config.rbs +341 -0
- data/sig/robin/cron.rbs +149 -0
- data/sig/robin/debounce.rbs +36 -0
- data/sig/robin/deploy.rbs +24 -0
- data/sig/robin/embedded.rbs +25 -0
- data/sig/robin/encryption.rbs +49 -0
- data/sig/robin/events.rbs +10 -0
- data/sig/robin/expiry.rbs +17 -0
- data/sig/robin/fetcher.rbs +134 -0
- data/sig/robin/gvl.rbs +39 -0
- data/sig/robin/health.rbs +30 -0
- data/sig/robin/iterable_job.rbs +99 -0
- data/sig/robin/job.rbs +173 -0
- data/sig/robin/job_class.rbs +55 -0
- data/sig/robin/job_logger.rbs +56 -0
- data/sig/robin/launcher.rbs +191 -0
- data/sig/robin/leader.rbs +43 -0
- data/sig/robin/limiter.rbs +317 -0
- data/sig/robin/manager.rbs +54 -0
- data/sig/robin/metrics.rbs +122 -0
- data/sig/robin/middleware.rbs +106 -0
- data/sig/robin/opentelemetry.rbs +51 -0
- data/sig/robin/periodic.rbs +81 -0
- data/sig/robin/processor.rbs +107 -0
- data/sig/robin/profiler.rbs +60 -0
- data/sig/robin/progress.rbs +22 -0
- data/sig/robin/rails/app_loader.rbs +10 -0
- data/sig/robin/rails/current_attributes.rbs +44 -0
- data/sig/robin/rails/error_reporter.rbs +21 -0
- data/sig/robin/rails/notifications.rbs +9 -0
- data/sig/robin/rails/transactional_push.rbs +16 -0
- data/sig/robin/rails.rbs +8 -0
- data/sig/robin/recovery.rbs +29 -0
- data/sig/robin/redis/connection.rbs +224 -0
- data/sig/robin/redis/pool.rbs +89 -0
- data/sig/robin/redis/resp.rbs +29 -0
- data/sig/robin/reliable_push.rbs +77 -0
- data/sig/robin/retrier.rbs +72 -0
- data/sig/robin/scheduler.rbs +40 -0
- data/sig/robin/scripts.rbs +36 -0
- data/sig/robin/statsd.rbs +55 -0
- data/sig/robin/swarm.rbs +133 -0
- data/sig/robin/systemd.rbs +13 -0
- data/sig/robin/testing.rbs +80 -0
- data/sig/robin/top.rbs +79 -0
- data/sig/robin/unique.rbs +72 -0
- data/sig/robin/version.rbs +3 -0
- data/sig/robin/warmup.rbs +44 -0
- data/sig/robin/web/api.rbs +196 -0
- data/sig/robin/web/batch_status.rbs +25 -0
- data/sig/robin/web/extension.rbs +106 -0
- data/sig/robin/web/permissions.rbs +14 -0
- data/sig/robin/web/prometheus.rbs +33 -0
- data/sig/robin/web/router.rbs +53 -0
- data/sig/robin/web/static.rbs +42 -0
- data/sig/robin/web.rbs +140 -0
- data/sig/robin.rbs +116 -0
- data/sig/robin_resp.rbs +55 -0
- metadata +274 -44
- data/bin/robin +0 -5
- data/lib/robin/actions/authenticate.rb +0 -38
- data/lib/robin/actions/empty_tweet.rb +0 -7
- data/lib/robin/actions/followers.rb +0 -22
- data/lib/robin/actions/help.rb +0 -16
- data/lib/robin/actions/home_timeline.rb +0 -9
- data/lib/robin/actions/new_tweet.rb +0 -11
- data/lib/robin/actions/unknown_flag.rb +0 -11
- data/lib/robin/actions/user_timeline.rb +0 -19
- data/lib/robin/args_parser.rb +0 -70
- data/lib/robin/pin_authenticator.rb +0 -31
- data/lib/robin/user.rb +0 -36
checksums.yaml
ADDED
|
@@ -0,0 +1,7 @@
|
|
|
1
|
+
---
|
|
2
|
+
SHA256:
|
|
3
|
+
metadata.gz: 6bdccd19278640dbbdf172fd44f951335f227216873dabc210ec08e9d2d06260
|
|
4
|
+
data.tar.gz: d9920135f7c3b36b661fb60bdb323b0ca384088e6af442ed426c784bdf45feca
|
|
5
|
+
SHA512:
|
|
6
|
+
metadata.gz: f75f4120b7d0201ca5ba9f341311f551e1a835fd907d848ca6f9b2bfbb4557e6273617aab4bbe23a22bf2339091fc022a652ada1a1b976c82c08fc0d73d37184
|
|
7
|
+
data.tar.gz: 17554eef056b3125455aad16edc92d8e9ff1305b91916bcde1431722ec2bd42d722b83ce26fd973d4016341d63d8bc53c3564f1a328862042281a7412bcf8da8
|
data/CHANGELOG.md
ADDED
|
@@ -0,0 +1,372 @@
|
|
|
1
|
+
# Changelog
|
|
2
|
+
|
|
3
|
+
All notable changes to this project will be documented in this file.
|
|
4
|
+
|
|
5
|
+
The format is based on [Keep a Changelog](https://keepachangelog.com/en/2.0.0/),
|
|
6
|
+
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
|
7
|
+
|
|
8
|
+
## [Unreleased]
|
|
9
|
+
|
|
10
|
+
## [0.1.0] - 2026-10-24
|
|
11
|
+
|
|
12
|
+
The first release. Everything is open source under the LGPL-3.0.
|
|
13
|
+
Robin needs Ruby 3.4 or later (or JRuby 10).
|
|
14
|
+
|
|
15
|
+
### Added
|
|
16
|
+
|
|
17
|
+
- Core: multithreaded processing, capsules, weighted/strict queue priority,
|
|
18
|
+
scheduled jobs, exponential-backoff retries, dead set, middleware,
|
|
19
|
+
strict JSON arguments, bulk push, embedded mode, deferrable push.
|
|
20
|
+
- Reliability: reliable fetch with per-process in-flight jobs, crash
|
|
21
|
+
recovery, poison-pill detection, reliable push buffering.
|
|
22
|
+
- Batches with nested workflows and success/complete/death callbacks.
|
|
23
|
+
- Rate limiters: concurrent, bucket, window, leaky and points.
|
|
24
|
+
- Periodic (cron) jobs with time zones, unique jobs, debounced jobs,
|
|
25
|
+
expiring jobs, encrypted arguments with key rotation, iterable jobs.
|
|
26
|
+
- Leader election, swarm mode with rolling restarts and memory limits.
|
|
27
|
+
- Metrics: per-class timings and histograms, historical rollups, StatsD
|
|
28
|
+
and Prometheus output.
|
|
29
|
+
- Job progress reporting and cooperative cancellation.
|
|
30
|
+
- Web dashboard (TypeScript) with live updates, multi-cluster support
|
|
31
|
+
and authorization hooks.
|
|
32
|
+
- Optional C extension that encodes commands, parses Redis replies, and
|
|
33
|
+
sends commands and reads their replies in one release of the GVL,
|
|
34
|
+
checking for dropped connections without a read.
|
|
35
|
+
- RBS signatures for Robin's API (`sig/`), shipped in the gem, and
|
|
36
|
+
checked against the code with Steep in strict mode (`rake steep`).
|
|
37
|
+
- `config.fetch_timeout`: how long an idle worker waits for a job (2s).
|
|
38
|
+
- `Job#capsule` and `Robin::Capsule.current`: the capsule whose worker is
|
|
39
|
+
running the code. A worker thread emits a `worker.started` event as it
|
|
40
|
+
starts. After a resize, the capsule's `concurrency` is the new count.
|
|
41
|
+
- `Robin.cancel!` takes several jids, cancelling them in one round trip;
|
|
42
|
+
cancelling a batch with children takes one round trip, not one each.
|
|
43
|
+
- A server logs the Redis it uses at startup (its version, address and
|
|
44
|
+
key prefixes; never a password), so connecting to the wrong one shows.
|
|
45
|
+
- Iterable jobs whose enumerator has no size (the Active Record and CSV
|
|
46
|
+
enumerators) report how many items they've done, shown on the Busy page
|
|
47
|
+
under a moving bar; `progress!(nil, message)` reports progress without
|
|
48
|
+
a percentage.
|
|
49
|
+
- `config.internal_key_prefix` puts Robin's own keys under a prefix of
|
|
50
|
+
their own (the job keys keep `key_prefix`).
|
|
51
|
+
- `config.retry_backoff`, `config.recovery_handlers`,
|
|
52
|
+
`config.leader_candidate`, the `:fork` lifecycle event (in swarm
|
|
53
|
+
children), `batch.jobs(atomic: true)`, `around_iteration` for iterable
|
|
54
|
+
jobs, `Robin::Client.new(chain:)`, and CLI plugin hooks for config file
|
|
55
|
+
names and settings (`Robin::CLI.config_paths`, `config_file_hooks`).
|
|
56
|
+
ActiveJob jobs' retry hooks come from the class they wrap.
|
|
57
|
+
- Batched fetch: one round trip fetches a job for every worker of a
|
|
58
|
+
capsule asking at the same time (up to `config.fetch_batch`, 10), and
|
|
59
|
+
finished jobs are acknowledged with the next round trip. Extra jobs wait
|
|
60
|
+
in flight; ones no worker takes within a second, or from a queue that's
|
|
61
|
+
been paused, go back to their queue, as do those of a stopping process.
|
|
62
|
+
- Middleware can define `self.applies?(*args)` to be skipped (and never
|
|
63
|
+
built) for jobs it doesn't concern; the built-in server middleware do.
|
|
64
|
+
- A java-platform gem for JRuby, with a Java extension in place of the C
|
|
65
|
+
one: the same reply parser and socket reader (each socket waits on a
|
|
66
|
+
selector of its own), a command encoder, and lock-free bookkeeping for
|
|
67
|
+
the connection pool. Built with `ROBIN_PLATFORM=java gem build` after
|
|
68
|
+
`rake java:compile`; held to 100% JaCoCo coverage and 100% PIT mutation
|
|
69
|
+
coverage.
|
|
70
|
+
- `retry_for:` keeps retrying a job until a set time after its first
|
|
71
|
+
failure; `retry_queue:` runs a job's retries on another queue.
|
|
72
|
+
- Deploy markers (`Robin::Deploy.mark!`, `robin deploy LABEL`), drawn as
|
|
73
|
+
lines on the dashboard's metrics charts.
|
|
74
|
+
- Events: `config.event_handlers` are called with what Robin otherwise
|
|
75
|
+
only logs (expired, skipped, discarded and rate-limited jobs, recovered
|
|
76
|
+
and orphaned jobs, dead workers and swarm children, slow Redis, a lapsed
|
|
77
|
+
heartbeat, memory limits, slow iterations, reliable push buffering).
|
|
78
|
+
See `Robin::Events`.
|
|
79
|
+
- Reliable push reports jobs it drops from a full buffer, and jobs still
|
|
80
|
+
buffered when the process exits (after one last try to push them), to
|
|
81
|
+
the error handlers, as `Robin::ReliablePush::Dropped` and `Lost`.
|
|
82
|
+
- GVL contention monitoring (C extension): each heartbeat
|
|
83
|
+
records how much of the time the process's threads could wait for the
|
|
84
|
+
GVL they did, shown on the dashboard and sent to StatsD as
|
|
85
|
+
`gvl.contention`; contention kept high logs a warning and emits a
|
|
86
|
+
`gvl.contended` event. `config.gvl_monitoring = false` turns it off.
|
|
87
|
+
- The dashboard in 25 languages, right-to-left ones included, chosen from
|
|
88
|
+
the browser's languages or under Settings; each loads on demand.
|
|
89
|
+
- Dashboard extensions can bring their own JavaScript module, stylesheet
|
|
90
|
+
and other assets, JSON data endpoints, and translations (YAML files
|
|
91
|
+
used by `request.t`, the script's `t` and the navigation).
|
|
92
|
+
- StatsD counters for batches (created, complete, success, death,
|
|
93
|
+
cancelled, plus a duration timing) and for recovered, orphaned and
|
|
94
|
+
poison-pill jobs.
|
|
95
|
+
- Rails: `config.robin.current_attributes = ["Current"]` carries
|
|
96
|
+
`ActiveSupport::CurrentAttributes` from where a job is pushed into it.
|
|
97
|
+
- Rate limiters take `reschedule:` and `backoff:`, `config.rate_limit_backoff`
|
|
98
|
+
sets a backoff for all of them, and `Robin::Limiter.unlimited` is a
|
|
99
|
+
limiter that never limits.
|
|
100
|
+
- `perform_bulk(..., spread_interval: 3600)` (and `"spread_interval"` in
|
|
101
|
+
`Client#push_bulk`) spreads the jobs evenly over that many seconds.
|
|
102
|
+
- Iterable jobs: an `on_cancel` hook, a `current_object` reader, and
|
|
103
|
+
`max_iteration_runtime` (config or job option), after which a run
|
|
104
|
+
checkpoints and requeues itself.
|
|
105
|
+
- `Robin::Web.job_info(label) { |job| ... }` adds rows (text or links) to
|
|
106
|
+
job pages; `Robin::Web.assets_path` serves the dashboard's assets from a
|
|
107
|
+
CDN.
|
|
108
|
+
- `robin top`: a live terminal view of totals, queues, processes and
|
|
109
|
+
running jobs.
|
|
110
|
+
- The Busy page can show only processes or only running jobs
|
|
111
|
+
(`?only=processes`, `?only=jobs`); INFO (Ctrl-T on macOS and BSD) dumps
|
|
112
|
+
threads like TTIN; a suppressed duplicate unique job's log line names
|
|
113
|
+
the jid holding the lock.
|
|
114
|
+
- Batch tags (`batch.tags = [...]`), with `Robin::Batch.list(tag:)` and a
|
|
115
|
+
tag filter on the dashboard's Batches page.
|
|
116
|
+
- `Robin::Statsd.new(distributions: true)` sends Datadog distributions
|
|
117
|
+
(`jobs.perform_dist`, `batch.duration_dist`); under DogStatsD, batch
|
|
118
|
+
metrics carry the batch's tags.
|
|
119
|
+
- `Robin::Client.rewriters`: callables that may rewrite a job hash before
|
|
120
|
+
it's pushed. The Rails integration uses one so periodic jobs (and
|
|
121
|
+
`Client.push`) can name ActiveJob classes.
|
|
122
|
+
- `Robin::Periodic.verify!` checks a periodic schedule in an app's test
|
|
123
|
+
suite.
|
|
124
|
+
- Bucket and window limiters take `within_limit(used: n)`;
|
|
125
|
+
`Robin::Limiter.via(pool)` keeps limiters in another Redis.
|
|
126
|
+
- Limiters take `policy: :skip`: when over the limit, `within_limit`
|
|
127
|
+
returns nil without running the block instead of raising.
|
|
128
|
+
- `Robin::Queue#delete_by_class` deletes a class's jobs inside Redis; the
|
|
129
|
+
dashboard offers it when the selected jobs share a class.
|
|
130
|
+
- `Robin::Web::BatchStatus`: a Rack app your own pages can poll for a
|
|
131
|
+
batch's progress (counts and states only).
|
|
132
|
+
- `robin swarm -n 0.5` (or `ROBIN_SWARM_COUNT`): a decimal means that many
|
|
133
|
+
processes per CPU core.
|
|
134
|
+
- Benchmarks: `rake bench` (parser and throughput) and `rake bench:compare`
|
|
135
|
+
(Robin vs Sidekiq on the same workload, with optional added latency).
|
|
136
|
+
- Redis Sentinel (`sentinels:` with the master named in the URL), with a
|
|
137
|
+
role check on connect; sharding with `MyJob.set(pool:)` and
|
|
138
|
+
`Robin::Client.via(pool) { ... }`.
|
|
139
|
+
- Health checks: an HTTP endpoint per server (`config.health_port`),
|
|
140
|
+
`robin health` for exec probes, and `/health` on the dashboard.
|
|
141
|
+
- OpenTelemetry tracing (`require "robin/opentelemetry"`) with producer
|
|
142
|
+
and consumer spans and trace context carried in the job.
|
|
143
|
+
- Profiling single job runs with Vernier (`MyJob.set(profile: true)`),
|
|
144
|
+
stored in Redis for the Firefox Profiler.
|
|
145
|
+
- Operations: process titles, a warning when Redis round trips are slow,
|
|
146
|
+
systemd notify and watchdog support, `REDIS_PROVIDER`, connections
|
|
147
|
+
named `robin`, passwords never shown by `inspect`, Capistrano- and
|
|
148
|
+
Heroku-aware tags and hostnames, previous signal traps still run,
|
|
149
|
+
capsules frozen after boot, optional idle connection reaping
|
|
150
|
+
(`idle_timeout`).
|
|
151
|
+
- Jobs: per-job `log_level` and `logging: false`.
|
|
152
|
+
- Dashboard extensions (`Robin::Web::Extension`, `Robin::Web.register`):
|
|
153
|
+
pages of your own, built from blocks, with actions.
|
|
154
|
+
- A Profiles page in the dashboard, to download job profiles for the
|
|
155
|
+
Firefox Profiler.
|
|
156
|
+
- Works with any Ruby or Rack app. Extension points: `config.push_deferrer`
|
|
157
|
+
(defer delivery, e.g. until a transaction commits), `Robin::Job.inclusion_checks`,
|
|
158
|
+
and `robin` CLI plugins (`robin/plugins/*.rb` in any gem) that add app
|
|
159
|
+
loaders and environment variables.
|
|
160
|
+
- Resizing a running process: a capsule's worker count changes without a
|
|
161
|
+
restart, from the dashboard's Busy page, `POST
|
|
162
|
+
api/processes/:identity/resize`, or `Robin::ProcessRecord#resize!`.
|
|
163
|
+
New workers start at once, extra ones finish their job first, and a
|
|
164
|
+
Redis pool Robin sized grows to fit (a `workers.resized` event).
|
|
165
|
+
- A Puma plugin (`plugin :robin`, optionally `robin_config { |config| ... }`)
|
|
166
|
+
runs an embedded server in every worker once the app has loaded, and
|
|
167
|
+
stops it gracefully on shutdown and on hot, phased and fork-worker
|
|
168
|
+
restarts. Robin is never loaded in Puma's master, so phased restarts
|
|
169
|
+
with `prune_bundler` pick up changed gem versions. `Embedded#stop`
|
|
170
|
+
waits for a stop already in progress (including one asked for from the
|
|
171
|
+
dashboard) instead of stopping twice.
|
|
172
|
+
- The Rails integration, part of the robin gem. It turns itself on in a
|
|
173
|
+
Rails app (Bundler loads robin after Rails); elsewhere it loads
|
|
174
|
+
nothing of Rails. `require "robin/rails"` turns it on where robin was
|
|
175
|
+
required before Rails:
|
|
176
|
+
- Railtie: jobs run in the Rails executor and reloader, `Rails.logger`
|
|
177
|
+
outside the server, `Robin.config.environment` from `Rails.env`.
|
|
178
|
+
- ActiveJob adapter (`:robin`) with `robin_options`, pipelined
|
|
179
|
+
`perform_all_later` and continuation support.
|
|
180
|
+
- `bundle exec robin` boots the Rails app and reads `RAILS_ENV`.
|
|
181
|
+
- Errors reported to `Rails.error` (`config.robin.report_errors`).
|
|
182
|
+
- Transaction-aware pushes (`config.robin.transactional_push`).
|
|
183
|
+
- Generators: `robin:install` and `robin:job`.
|
|
184
|
+
- Failed jobs' backtraces go through `Rails.backtrace_cleaner`.
|
|
185
|
+
- What jobs log through `Rails.logger` is tagged with their class and jid.
|
|
186
|
+
- Mail deliveries show as the mail they send (`UserMailer#welcome`).
|
|
187
|
+
- Robin's events are published through `ActiveSupport::Notifications` as
|
|
188
|
+
`"<event>.robin"` (`config.robin.notifications`).
|
|
189
|
+
|
|
190
|
+
### Changed
|
|
191
|
+
|
|
192
|
+
- Performance: at most one Redis round trip per job (a finished job is
|
|
193
|
+
acknowledged by the round trip that fetches the next), replies awaited
|
|
194
|
+
rather than polled for, middleware chains that build no closures per
|
|
195
|
+
job, bulk pushes normalized once per batch, and job ids drawn from
|
|
196
|
+
batched random bytes.
|
|
197
|
+
- Fewer Redis round trips outside the job path:
|
|
198
|
+
- A heartbeat is one round trip (its metrics, leader lease and in-flight
|
|
199
|
+
sweep go in its transaction; it reports the previous heartbeat's round
|
|
200
|
+
trip time), and buffered jobs of every capsule go back in one more.
|
|
201
|
+
- Recovering dead processes takes three round trips however many there
|
|
202
|
+
are, not one per process and job; pruning processes and sampling queue
|
|
203
|
+
sizes take one each (two), and periodic jobs take three per tick
|
|
204
|
+
however many are due.
|
|
205
|
+
- A debounced push, a batch callback, a deploy mark, and a limiter that
|
|
206
|
+
gives up (it counts itself over its limit) take one round trip fewer;
|
|
207
|
+
so does loading a Lua script Redis lost. A debounced job's claim is
|
|
208
|
+
made with its push, so a push a later middleware stops no longer
|
|
209
|
+
unschedules the job pushed before it.
|
|
210
|
+
- `retry_all` and `kill_all`, and the dashboard's actions on selected or
|
|
211
|
+
all jobs, take a few round trips per page of 100, not two per job;
|
|
212
|
+
`Batch#cancel!` takes two whatever the depth of children.
|
|
213
|
+
- The dashboard's stats, processes, busy jobs and limiters are read in
|
|
214
|
+
one round trip each, and every page in a fixed number.
|
|
215
|
+
- `Robin::Client.together { ... }` sends the jobs its block pushes in one
|
|
216
|
+
round trip. The Rails integration uses it for `perform_all_later` (one round trip
|
|
217
|
+
for all its classes and queues) and for transactional pushes (the jobs
|
|
218
|
+
of one transaction go together when it commits).
|
|
219
|
+
- Tests pin the round trips of each of these with `assert_roundtrips`.
|
|
220
|
+
- A finished job is acknowledged with the next round trip, or within 2
|
|
221
|
+
milliseconds by a thread of its own if no round trip comes (a worker
|
|
222
|
+
that takes a job fetched for it makes none, and long jobs can keep
|
|
223
|
+
every worker busy), so a crash re-runs only jobs that finished just
|
|
224
|
+
before it.
|
|
225
|
+
- An idle worker of a capsule with one queue blocks on it again without
|
|
226
|
+
a non-blocking fetch first, halving the commands idle processes send.
|
|
227
|
+
- JRuby: hash arguments with keys that aren't symbols take braces, as
|
|
228
|
+
JRuby 10.1 can't compile a method with a call passing one without
|
|
229
|
+
(enforced by the `Robin/BracedHashArgument` RuboCop cop), and
|
|
230
|
+
`push_bulk` does its per-job work in methods of their own, compiled
|
|
231
|
+
after 50 jobs. Warm pushes are about 12% faster.
|
|
232
|
+
- `robin stats`, `robin top` and `robin deploy` boot the app as a client,
|
|
233
|
+
so its `configure_client` blocks (Redis settings, say) apply to them.
|
|
234
|
+
- Redis formats shared with other Ruby job tools: schedule, retry and
|
|
235
|
+
dead set scores (and a job's "at") in epoch seconds; scheduled and
|
|
236
|
+
failed jobs stored without "at" (enqueued_at is set when they reach
|
|
237
|
+
their queue); a 0-based "retry_count" in place of "failures"
|
|
238
|
+
(Job#retry_count is now Job#failures); compressed backtraces as plain
|
|
239
|
+
base64; process records at the process's identity, with signals named
|
|
240
|
+
TSTP, TERM and TTIN; daily totals kept five years. Older float-second
|
|
241
|
+
timestamps are still read.
|
|
242
|
+
- In-flight jobs are acknowledged by key, not by payload: each fetched job
|
|
243
|
+
goes into a key of its own (`inflight:<identity>:<n>`, registered in the
|
|
244
|
+
process's `inflight:<identity>` set and reused once acknowledged), so a
|
|
245
|
+
job's payload is never sent back to Redis, and batches of large jobs
|
|
246
|
+
are taken with plain commands instead of copying them through Lua. With
|
|
247
|
+
100 KB arguments a process runs about 4x as many jobs per second, with a
|
|
248
|
+
fifth of the Redis CPU per job; with 10 KB, about twice as many. No-op
|
|
249
|
+
jobs cost a few percent more (a key per job is created and deleted).
|
|
250
|
+
Recovery still reads the single in-flight list of earlier development
|
|
251
|
+
versions.
|
|
252
|
+
- Job log lines the logger would drop cost nothing: no timing is
|
|
253
|
+
formatted and the logger isn't called.
|
|
254
|
+
- The C extension encodes commands and, in one release of the GVL,
|
|
255
|
+
checks the connection, writes the commands and reads the start of the
|
|
256
|
+
replies (`NativeIO.roundtrip`).
|
|
257
|
+
- No shared locks per job: workers keep their running job to themselves,
|
|
258
|
+
metrics are recorded per thread and merged on flush, so threads that run
|
|
259
|
+
in parallel (JRuby) never wait on each other.
|
|
260
|
+
- Capsules with a single queue fetch with pipelined plain commands instead
|
|
261
|
+
of a Lua script, which costs Redis about a third less per job.
|
|
262
|
+
- Pipelines run Lua scripts by their SHA, as single calls do: a
|
|
263
|
+
connection remembers which scripts its server has, and loads one it
|
|
264
|
+
hasn't seen in the same round trip, so a script's source is sent once
|
|
265
|
+
per connection, not with every pipeline. (Transactions still send the
|
|
266
|
+
source, so a server that lost its scripts can't apply half of one.)
|
|
267
|
+
- Error handlers run in the failed job's logging context, and the
|
|
268
|
+
default handler logs an error, its context and backtrace as one entry.
|
|
269
|
+
- `push_bulk` inside a batch's `jobs` block adds its jobs to the batch in
|
|
270
|
+
one round trip, not one per job; if it raises, none are added.
|
|
271
|
+
- Cancelling a batch (`Robin::Batch#cancel!`, or from the dashboard)
|
|
272
|
+
cancels its child batches and their jobs too; the batch completes
|
|
273
|
+
without succeeding.
|
|
274
|
+
- `Client.timestamp_ms(nil)` now raises instead of meaning "now".
|
|
275
|
+
- `logger` is a declared dependency (a bundled gem since Ruby 3.5).
|
|
276
|
+
- Tested against Redis 7.4, 8.2 and 8.10, Valkey 9.1 and Dragonfly 2.0.
|
|
277
|
+
- `rake test` runs test files in parallel processes (`rake test:serial` in
|
|
278
|
+
one).
|
|
279
|
+
- Robin may never get slower, after [WebKit's performance
|
|
280
|
+
policy](https://webkit.org/performance/) ([CONTRIBUTING.md](CONTRIBUTING.md)):
|
|
281
|
+
`rake perf` runs Robin's benchmarks (`duel/`) with
|
|
282
|
+
[Duel](https://github.com/sferik/duel-ruby), on every pull request and
|
|
283
|
+
push in CI (CRuby with YJIT and without, and JRuby), and fails if a
|
|
284
|
+
change is slower than its baseline at any of them, and each night if
|
|
285
|
+
`main` is slower than the last release. CPU time, instructions and
|
|
286
|
+
Redis CPU are judged by a 99% confidence interval for paired ratios,
|
|
287
|
+
over three processes of each version and confirmed in fresh ones;
|
|
288
|
+
allocations, round trips and Redis commands are counted, and peak
|
|
289
|
+
memory measured; a regression comes with a profile of each version.
|
|
290
|
+
The benchmarks cover pushing, running jobs (one worker's loop and a
|
|
291
|
+
whole server), scheduling (and the leader's wake-up), heartbeats,
|
|
292
|
+
crash recovery, periodic jobs, rate limiters, the dashboard's
|
|
293
|
+
requests, the protocol and the JIT warm-up.
|
|
294
|
+
- Performance: idle workers block 2s per fetch, and check whether their
|
|
295
|
+
queue was paused in the same round trip, as of the moment a job
|
|
296
|
+
arrives (Redis runs the check once the blocking fetch returns); built-in client
|
|
297
|
+
middleware (unique, debounce, batch, encryption) is skipped for jobs
|
|
298
|
+
that don't use it (bulk pushes about 10% faster, twice as fast before
|
|
299
|
+
JRuby has warmed up); `robin swarm` loads all of Robin (`Robin.eager_load!`) and
|
|
300
|
+
runs `Process.warmup` before forking, so children share it
|
|
301
|
+
copy-on-write (about 1 MB less private memory each); built-in
|
|
302
|
+
middleware is built once instead of per job (`Middleware::Stateless`);
|
|
303
|
+
stored backtraces are compressed, with `config.backtrace_cleaner`;
|
|
304
|
+
`retry_all`, `kill_all` and dashboard bulk actions page through sets;
|
|
305
|
+
timings and histograms leave out failed runs.
|
|
306
|
+
- Idle time is spent for busy time: the cluster's leader wakes its
|
|
307
|
+
scheduler when the next scheduled job or retry is due, rather than at
|
|
308
|
+
its next poll (jobs scheduled ahead start within milliseconds, not
|
|
309
|
+
seconds, late); a standalone server runs `Process.warmup` before it
|
|
310
|
+
starts, and, if asked, a full GC once it has gone idle
|
|
311
|
+
(`config.idle_gc`: off by default; `true`, or the fewest jobs since the
|
|
312
|
+
last collection, turns it on); and off CRuby a server warms up the JIT before its workers start
|
|
313
|
+
(`config.jit_warmup`; on JRuby a new server's first pushes are about
|
|
314
|
+
3x as fast, and it reaches full speed sooner).
|
|
315
|
+
- Jobs: `Robin::Job::Timeout` is no longer a `StandardError`.
|
|
316
|
+
- Test modes set with a block are thread-local.
|
|
317
|
+
- Finding jobs by jid searches queues inside Redis; `Robin::JobLocator`
|
|
318
|
+
and an optional jid index (`config.jid_index`) point it at the right
|
|
319
|
+
queue.
|
|
320
|
+
- Iterable jobs make one Redis round trip every five seconds: the
|
|
321
|
+
checkpoint, progress and cancellation check share a transaction, so a
|
|
322
|
+
cancelled job stops at its next checkpoint (within about five seconds,
|
|
323
|
+
was one). An interruption saves the checkpoint and requeues the job in
|
|
324
|
+
one transaction, and finishing deletes the state and reports progress
|
|
325
|
+
in one pipeline. A limiter's definition is written once per process and
|
|
326
|
+
Redis (and again when it changes) rather than once per limiter object,
|
|
327
|
+
so limiters built inside `perform` cost no extra round trip.
|
|
328
|
+
- Fewer Redis round trips around jobs: a blocking fetch checks for a
|
|
329
|
+
pause in the same round trip; `unique_until: :start` jobs release
|
|
330
|
+
their lock once; expired jobs outside a batch and skipped cancelled
|
|
331
|
+
jobs cost none. A failing batch job's failure goes in its retry's
|
|
332
|
+
transaction, and a job that dies or is discarded is moved, acknowledged,
|
|
333
|
+
unlocked and recorded with its batch in one transaction (was up to
|
|
334
|
+
five round trips). A push inside a batch's `jobs` block registers and
|
|
335
|
+
sends the job in one transaction, a new batch (child or not) is
|
|
336
|
+
created in one, and `push_bulk` takes its jobs' unique locks and makes
|
|
337
|
+
their debounce claims in one round trip each instead of one per job.
|
|
338
|
+
`unique_for` of zero or less takes no lock.
|
|
339
|
+
|
|
340
|
+
### Fixed
|
|
341
|
+
|
|
342
|
+
- A forked child (a Puma worker, a swarm child) lets go of the TLS
|
|
343
|
+
connections it inherited without a TLS goodbye, which made Redis hang
|
|
344
|
+
up on the parent's connection; its parent's next command failed.
|
|
345
|
+
- An iterable job with nothing to iterate finishes at 100% instead of
|
|
346
|
+
failing on its progress.
|
|
347
|
+
- Expiring jobs count `expires_in` from when the job is due, not when it
|
|
348
|
+
was pushed, so a scheduled job no longer expires before it runs.
|
|
349
|
+
- A job that dies or is discarded now gives up its unique lock in the
|
|
350
|
+
retrier, whatever ended it.
|
|
351
|
+
- Hardening: graceful-shutdown requeues don't count toward poison-pill
|
|
352
|
+
detection; failure recording never strands a job (unreadable or binary
|
|
353
|
+
error messages, bad `robin_retry_in` values); fetch errors back off
|
|
354
|
+
instead of respawning threads; connections drop and reconnect after
|
|
355
|
+
READONLY/LOADING/failover replies and never resend a partially written
|
|
356
|
+
pipeline; the heartbeat never writes decrypted arguments; retries and
|
|
357
|
+
dead jobs are stored as pushed (not as the job mutated them); iterating
|
|
358
|
+
while deleting visits every job; batch jobs are never deferred; one
|
|
359
|
+
shared grace period for interrupted jobs, which are only interrupted
|
|
360
|
+
inside `perform`; errors caused by the shutdown don't count as failures;
|
|
361
|
+
iterable jobs checkpoint when an iteration raises; jitter on custom retry
|
|
362
|
+
delays; death handlers run for discarded jobs (`discarded_at`); failing
|
|
363
|
+
`:startup` hooks abort the boot; boot checks for Redis 6.2+, eviction
|
|
364
|
+
policy, pool size, concurrency, timeout and duplicate queues; orphaned
|
|
365
|
+
in-flight jobs on live processes are swept back to their queues; the
|
|
366
|
+
heartbeat and leader duties run on their own threads and Redis
|
|
367
|
+
connections, and a lapsed heartbeat is reported.
|
|
368
|
+
|
|
369
|
+
### Security
|
|
370
|
+
|
|
371
|
+
- Invalid Redis URLs no longer show their password in the error.
|
|
372
|
+
- The dashboard rejects cross-site writes using `Sec-Fetch-Site` too.
|