robin 0.0.4 → 0.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (225) hide show
  1. checksums.yaml +7 -0
  2. data/CHANGELOG.md +372 -0
  3. data/COPYING +674 -0
  4. data/LICENSE +165 -0
  5. data/README.md +1490 -0
  6. data/assets/favicon.svg +31 -0
  7. data/assets/robin-logo.svg +37 -0
  8. data/assets/robin-mono.svg +12 -0
  9. data/assets/robin-wordmark.svg +4 -0
  10. data/assets/robin.svg +34 -0
  11. data/exe/robin +5 -0
  12. data/ext/robin_resp/extconf.rb +29 -0
  13. data/ext/robin_resp/robin_resp.c +645 -0
  14. data/lib/active_job/queue_adapters/robin_adapter.rb +3 -0
  15. data/lib/generators/robin/install/install_generator.rb +39 -0
  16. data/lib/generators/robin/install/templates/initializer.rb.tt +31 -0
  17. data/lib/generators/robin/install/templates/robin.yml.tt +25 -0
  18. data/lib/generators/robin/job/job_generator.rb +78 -0
  19. data/lib/generators/robin/job/templates/job.rb.tt +13 -0
  20. data/lib/generators/robin/job/templates/job_spec.rb.tt +21 -0
  21. data/lib/generators/robin/job/templates/job_test.rb.tt +21 -0
  22. data/lib/puma/plugin/robin.rb +131 -0
  23. data/lib/robin/active_job.rb +144 -0
  24. data/lib/robin/api.rb +777 -0
  25. data/lib/robin/batch.rb +700 -0
  26. data/lib/robin/cancellation.rb +62 -0
  27. data/lib/robin/cli.rb +325 -0
  28. data/lib/robin/client.rb +355 -21
  29. data/lib/robin/config.rb +454 -4
  30. data/lib/robin/cron.rb +325 -0
  31. data/lib/robin/debounce.rb +108 -0
  32. data/lib/robin/deploy.rb +65 -0
  33. data/lib/robin/embedded.rb +53 -0
  34. data/lib/robin/encryption.rb +147 -0
  35. data/lib/robin/events.rb +60 -0
  36. data/lib/robin/expiry.rb +41 -0
  37. data/lib/robin/fetcher.rb +441 -0
  38. data/lib/robin/gvl.rb +62 -0
  39. data/lib/robin/health.rb +76 -0
  40. data/lib/robin/iterable_job.rb +294 -0
  41. data/lib/robin/job.rb +258 -0
  42. data/lib/robin/job_class.rb +157 -0
  43. data/lib/robin/job_logger.rb +121 -0
  44. data/lib/robin/launcher.rb +658 -0
  45. data/lib/robin/leader.rb +86 -0
  46. data/lib/robin/limiter.rb +639 -0
  47. data/lib/robin/manager.rb +125 -0
  48. data/lib/robin/metrics.rb +303 -0
  49. data/lib/robin/middleware.rb +165 -0
  50. data/lib/robin/opentelemetry.rb +88 -0
  51. data/lib/robin/periodic.rb +327 -0
  52. data/lib/robin/processor.rb +317 -0
  53. data/lib/robin/profiler.rb +140 -0
  54. data/lib/robin/progress.rb +46 -0
  55. data/lib/robin/rails/app_loader.rb +23 -0
  56. data/lib/robin/rails/current_attributes.rb +88 -0
  57. data/lib/robin/rails/error_reporter.rb +31 -0
  58. data/lib/robin/rails/notifications.rb +22 -0
  59. data/lib/robin/rails/railtie.rb +87 -0
  60. data/lib/robin/rails/transactional_push.rb +49 -0
  61. data/lib/robin/rails.rb +23 -0
  62. data/lib/robin/recovery.rb +135 -0
  63. data/lib/robin/redis/connection.rb +639 -0
  64. data/lib/robin/redis/pool.rb +190 -0
  65. data/lib/robin/redis/resp.rb +77 -0
  66. data/lib/robin/reliable_push.rb +153 -0
  67. data/lib/robin/retrier.rb +284 -0
  68. data/lib/robin/scheduler.rb +102 -0
  69. data/lib/robin/scripts.rb +331 -0
  70. data/lib/robin/statsd.rb +135 -0
  71. data/lib/robin/swarm.rb +319 -0
  72. data/lib/robin/systemd.rb +47 -0
  73. data/lib/robin/testing.rb +158 -0
  74. data/lib/robin/top.rb +178 -0
  75. data/lib/robin/unique.rb +198 -0
  76. data/lib/robin/version.rb +5 -0
  77. data/lib/robin/warmup.rb +94 -0
  78. data/lib/robin/web/api.rb +580 -0
  79. data/lib/robin/web/batch_status.rb +44 -0
  80. data/lib/robin/web/dist/assets/ar-y7v9b75f.js +1 -0
  81. data/lib/robin/web/dist/assets/ar-y7v9b75f.js.gz +0 -0
  82. data/lib/robin/web/dist/assets/bricolage-0qmzcpnj.woff2 +0 -0
  83. data/lib/robin/web/dist/assets/cs-df88mq5e.js +1 -0
  84. data/lib/robin/web/dist/assets/cs-df88mq5e.js.gz +0 -0
  85. data/lib/robin/web/dist/assets/da-kemms74g.js +1 -0
  86. data/lib/robin/web/dist/assets/da-kemms74g.js.gz +0 -0
  87. data/lib/robin/web/dist/assets/de-z8kkqg65.js +1 -0
  88. data/lib/robin/web/dist/assets/de-z8kkqg65.js.gz +0 -0
  89. data/lib/robin/web/dist/assets/es-9ahrrn1s.js +1 -0
  90. data/lib/robin/web/dist/assets/es-9ahrrn1s.js.gz +0 -0
  91. data/lib/robin/web/dist/assets/fa-7r0x28nc.js +1 -0
  92. data/lib/robin/web/dist/assets/fa-7r0x28nc.js.gz +0 -0
  93. data/lib/robin/web/dist/assets/favicon-1ebfcc0182.svg +31 -0
  94. data/lib/robin/web/dist/assets/favicon-1ebfcc0182.svg.gz +0 -0
  95. data/lib/robin/web/dist/assets/fr-r7mqf3d9.js +1 -0
  96. data/lib/robin/web/dist/assets/fr-r7mqf3d9.js.gz +0 -0
  97. data/lib/robin/web/dist/assets/he-7t46b6rt.js +1 -0
  98. data/lib/robin/web/dist/assets/he-7t46b6rt.js.gz +0 -0
  99. data/lib/robin/web/dist/assets/hi-zj434tx4.js +1 -0
  100. data/lib/robin/web/dist/assets/hi-zj434tx4.js.gz +0 -0
  101. data/lib/robin/web/dist/assets/it-pqc2m7gm.js +1 -0
  102. data/lib/robin/web/dist/assets/it-pqc2m7gm.js.gz +0 -0
  103. data/lib/robin/web/dist/assets/ja-k9rgm1pj.js +1 -0
  104. data/lib/robin/web/dist/assets/ja-k9rgm1pj.js.gz +0 -0
  105. data/lib/robin/web/dist/assets/ko-bxersv91.js +1 -0
  106. data/lib/robin/web/dist/assets/ko-bxersv91.js.gz +0 -0
  107. data/lib/robin/web/dist/assets/main-anw3f6ba.css +1 -0
  108. data/lib/robin/web/dist/assets/main-anw3f6ba.css.gz +0 -0
  109. data/lib/robin/web/dist/assets/main-pdtvjejb.js +7 -0
  110. data/lib/robin/web/dist/assets/main-pdtvjejb.js.gz +0 -0
  111. data/lib/robin/web/dist/assets/main-zz8zzhwn.js +2 -0
  112. data/lib/robin/web/dist/assets/main-zz8zzhwn.js.gz +0 -0
  113. data/lib/robin/web/dist/assets/nb-fjnc5064.js +1 -0
  114. data/lib/robin/web/dist/assets/nb-fjnc5064.js.gz +0 -0
  115. data/lib/robin/web/dist/assets/nl-9p8b95b6.js +1 -0
  116. data/lib/robin/web/dist/assets/nl-9p8b95b6.js.gz +0 -0
  117. data/lib/robin/web/dist/assets/pl-8xsrdn8n.js +1 -0
  118. data/lib/robin/web/dist/assets/pl-8xsrdn8n.js.gz +0 -0
  119. data/lib/robin/web/dist/assets/pt-BR-650j7cvg.js +1 -0
  120. data/lib/robin/web/dist/assets/pt-BR-650j7cvg.js.gz +0 -0
  121. data/lib/robin/web/dist/assets/robin-ae4a4aedd9.svg +34 -0
  122. data/lib/robin/web/dist/assets/robin-ae4a4aedd9.svg.gz +0 -0
  123. data/lib/robin/web/dist/assets/robin-wordmark-5dde5d8e4b.svg +4 -0
  124. data/lib/robin/web/dist/assets/robin-wordmark-5dde5d8e4b.svg.gz +0 -0
  125. data/lib/robin/web/dist/assets/ru-a9ftyw3d.js +1 -0
  126. data/lib/robin/web/dist/assets/ru-a9ftyw3d.js.gz +0 -0
  127. data/lib/robin/web/dist/assets/sv-j9gqamr1.js +1 -0
  128. data/lib/robin/web/dist/assets/sv-j9gqamr1.js.gz +0 -0
  129. data/lib/robin/web/dist/assets/tr-y9z0q51g.js +1 -0
  130. data/lib/robin/web/dist/assets/tr-y9z0q51g.js.gz +0 -0
  131. data/lib/robin/web/dist/assets/uk-2dr9ew3p.js +1 -0
  132. data/lib/robin/web/dist/assets/uk-2dr9ew3p.js.gz +0 -0
  133. data/lib/robin/web/dist/assets/ur-h25nzq1s.js +1 -0
  134. data/lib/robin/web/dist/assets/ur-h25nzq1s.js.gz +0 -0
  135. data/lib/robin/web/dist/assets/vi-9y2ff1bd.js +1 -0
  136. data/lib/robin/web/dist/assets/vi-9y2ff1bd.js.gz +0 -0
  137. data/lib/robin/web/dist/assets/zh-CN-tbj9mf98.js +1 -0
  138. data/lib/robin/web/dist/assets/zh-CN-tbj9mf98.js.gz +0 -0
  139. data/lib/robin/web/dist/assets/zh-TW-59q8j9vw.js +1 -0
  140. data/lib/robin/web/dist/assets/zh-TW-59q8j9vw.js.gz +0 -0
  141. data/lib/robin/web/dist/index.html +19 -0
  142. data/lib/robin/web/extension.rb +193 -0
  143. data/lib/robin/web/permissions.rb +40 -0
  144. data/lib/robin/web/prometheus.rb +79 -0
  145. data/lib/robin/web/router.rb +150 -0
  146. data/lib/robin/web/static.rb +110 -0
  147. data/lib/robin/web.rb +196 -0
  148. data/lib/robin.rb +258 -8
  149. data/sig/manifest.yaml +9 -0
  150. data/sig/robin/api.rbs +442 -0
  151. data/sig/robin/batch.rbs +292 -0
  152. data/sig/robin/cancellation.rbs +25 -0
  153. data/sig/robin/cli.rbs +102 -0
  154. data/sig/robin/client.rbs +118 -0
  155. data/sig/robin/config.rbs +341 -0
  156. data/sig/robin/cron.rbs +149 -0
  157. data/sig/robin/debounce.rbs +36 -0
  158. data/sig/robin/deploy.rbs +24 -0
  159. data/sig/robin/embedded.rbs +25 -0
  160. data/sig/robin/encryption.rbs +49 -0
  161. data/sig/robin/events.rbs +10 -0
  162. data/sig/robin/expiry.rbs +17 -0
  163. data/sig/robin/fetcher.rbs +134 -0
  164. data/sig/robin/gvl.rbs +39 -0
  165. data/sig/robin/health.rbs +30 -0
  166. data/sig/robin/iterable_job.rbs +99 -0
  167. data/sig/robin/job.rbs +173 -0
  168. data/sig/robin/job_class.rbs +55 -0
  169. data/sig/robin/job_logger.rbs +56 -0
  170. data/sig/robin/launcher.rbs +191 -0
  171. data/sig/robin/leader.rbs +43 -0
  172. data/sig/robin/limiter.rbs +317 -0
  173. data/sig/robin/manager.rbs +54 -0
  174. data/sig/robin/metrics.rbs +122 -0
  175. data/sig/robin/middleware.rbs +106 -0
  176. data/sig/robin/opentelemetry.rbs +51 -0
  177. data/sig/robin/periodic.rbs +81 -0
  178. data/sig/robin/processor.rbs +107 -0
  179. data/sig/robin/profiler.rbs +60 -0
  180. data/sig/robin/progress.rbs +22 -0
  181. data/sig/robin/rails/app_loader.rbs +10 -0
  182. data/sig/robin/rails/current_attributes.rbs +44 -0
  183. data/sig/robin/rails/error_reporter.rbs +21 -0
  184. data/sig/robin/rails/notifications.rbs +9 -0
  185. data/sig/robin/rails/transactional_push.rbs +16 -0
  186. data/sig/robin/rails.rbs +8 -0
  187. data/sig/robin/recovery.rbs +29 -0
  188. data/sig/robin/redis/connection.rbs +224 -0
  189. data/sig/robin/redis/pool.rbs +89 -0
  190. data/sig/robin/redis/resp.rbs +29 -0
  191. data/sig/robin/reliable_push.rbs +77 -0
  192. data/sig/robin/retrier.rbs +72 -0
  193. data/sig/robin/scheduler.rbs +40 -0
  194. data/sig/robin/scripts.rbs +36 -0
  195. data/sig/robin/statsd.rbs +55 -0
  196. data/sig/robin/swarm.rbs +133 -0
  197. data/sig/robin/systemd.rbs +13 -0
  198. data/sig/robin/testing.rbs +80 -0
  199. data/sig/robin/top.rbs +79 -0
  200. data/sig/robin/unique.rbs +72 -0
  201. data/sig/robin/version.rbs +3 -0
  202. data/sig/robin/warmup.rbs +44 -0
  203. data/sig/robin/web/api.rbs +196 -0
  204. data/sig/robin/web/batch_status.rbs +25 -0
  205. data/sig/robin/web/extension.rbs +106 -0
  206. data/sig/robin/web/permissions.rbs +14 -0
  207. data/sig/robin/web/prometheus.rbs +33 -0
  208. data/sig/robin/web/router.rbs +53 -0
  209. data/sig/robin/web/static.rbs +42 -0
  210. data/sig/robin/web.rbs +140 -0
  211. data/sig/robin.rbs +116 -0
  212. data/sig/robin_resp.rbs +55 -0
  213. metadata +274 -44
  214. data/bin/robin +0 -5
  215. data/lib/robin/actions/authenticate.rb +0 -38
  216. data/lib/robin/actions/empty_tweet.rb +0 -7
  217. data/lib/robin/actions/followers.rb +0 -22
  218. data/lib/robin/actions/help.rb +0 -16
  219. data/lib/robin/actions/home_timeline.rb +0 -9
  220. data/lib/robin/actions/new_tweet.rb +0 -11
  221. data/lib/robin/actions/unknown_flag.rb +0 -11
  222. data/lib/robin/actions/user_timeline.rb +0 -19
  223. data/lib/robin/args_parser.rb +0 -70
  224. data/lib/robin/pin_authenticator.rb +0 -31
  225. data/lib/robin/user.rb +0 -36
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA256:
3
+ metadata.gz: 6bdccd19278640dbbdf172fd44f951335f227216873dabc210ec08e9d2d06260
4
+ data.tar.gz: d9920135f7c3b36b661fb60bdb323b0ca384088e6af442ed426c784bdf45feca
5
+ SHA512:
6
+ metadata.gz: f75f4120b7d0201ca5ba9f341311f551e1a835fd907d848ca6f9b2bfbb4557e6273617aab4bbe23a22bf2339091fc022a652ada1a1b976c82c08fc0d73d37184
7
+ data.tar.gz: 17554eef056b3125455aad16edc92d8e9ff1305b91916bcde1431722ec2bd42d722b83ce26fd973d4016341d63d8bc53c3564f1a328862042281a7412bcf8da8
data/CHANGELOG.md ADDED
@@ -0,0 +1,372 @@
1
+ # Changelog
2
+
3
+ All notable changes to this project will be documented in this file.
4
+
5
+ The format is based on [Keep a Changelog](https://keepachangelog.com/en/2.0.0/),
6
+ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
7
+
8
+ ## [Unreleased]
9
+
10
+ ## [0.1.0] - 2026-10-24
11
+
12
+ The first release. Everything is open source under the LGPL-3.0.
13
+ Robin needs Ruby 3.4 or later (or JRuby 10).
14
+
15
+ ### Added
16
+
17
+ - Core: multithreaded processing, capsules, weighted/strict queue priority,
18
+ scheduled jobs, exponential-backoff retries, dead set, middleware,
19
+ strict JSON arguments, bulk push, embedded mode, deferrable push.
20
+ - Reliability: reliable fetch with per-process in-flight jobs, crash
21
+ recovery, poison-pill detection, reliable push buffering.
22
+ - Batches with nested workflows and success/complete/death callbacks.
23
+ - Rate limiters: concurrent, bucket, window, leaky and points.
24
+ - Periodic (cron) jobs with time zones, unique jobs, debounced jobs,
25
+ expiring jobs, encrypted arguments with key rotation, iterable jobs.
26
+ - Leader election, swarm mode with rolling restarts and memory limits.
27
+ - Metrics: per-class timings and histograms, historical rollups, StatsD
28
+ and Prometheus output.
29
+ - Job progress reporting and cooperative cancellation.
30
+ - Web dashboard (TypeScript) with live updates, multi-cluster support
31
+ and authorization hooks.
32
+ - Optional C extension that encodes commands, parses Redis replies, and
33
+ sends commands and reads their replies in one release of the GVL,
34
+ checking for dropped connections without a read.
35
+ - RBS signatures for Robin's API (`sig/`), shipped in the gem, and
36
+ checked against the code with Steep in strict mode (`rake steep`).
37
+ - `config.fetch_timeout`: how long an idle worker waits for a job (2s).
38
+ - `Job#capsule` and `Robin::Capsule.current`: the capsule whose worker is
39
+ running the code. A worker thread emits a `worker.started` event as it
40
+ starts. After a resize, the capsule's `concurrency` is the new count.
41
+ - `Robin.cancel!` takes several jids, cancelling them in one round trip;
42
+ cancelling a batch with children takes one round trip, not one each.
43
+ - A server logs the Redis it uses at startup (its version, address and
44
+ key prefixes; never a password), so connecting to the wrong one shows.
45
+ - Iterable jobs whose enumerator has no size (the Active Record and CSV
46
+ enumerators) report how many items they've done, shown on the Busy page
47
+ under a moving bar; `progress!(nil, message)` reports progress without
48
+ a percentage.
49
+ - `config.internal_key_prefix` puts Robin's own keys under a prefix of
50
+ their own (the job keys keep `key_prefix`).
51
+ - `config.retry_backoff`, `config.recovery_handlers`,
52
+ `config.leader_candidate`, the `:fork` lifecycle event (in swarm
53
+ children), `batch.jobs(atomic: true)`, `around_iteration` for iterable
54
+ jobs, `Robin::Client.new(chain:)`, and CLI plugin hooks for config file
55
+ names and settings (`Robin::CLI.config_paths`, `config_file_hooks`).
56
+ ActiveJob jobs' retry hooks come from the class they wrap.
57
+ - Batched fetch: one round trip fetches a job for every worker of a
58
+ capsule asking at the same time (up to `config.fetch_batch`, 10), and
59
+ finished jobs are acknowledged with the next round trip. Extra jobs wait
60
+ in flight; ones no worker takes within a second, or from a queue that's
61
+ been paused, go back to their queue, as do those of a stopping process.
62
+ - Middleware can define `self.applies?(*args)` to be skipped (and never
63
+ built) for jobs it doesn't concern; the built-in server middleware do.
64
+ - A java-platform gem for JRuby, with a Java extension in place of the C
65
+ one: the same reply parser and socket reader (each socket waits on a
66
+ selector of its own), a command encoder, and lock-free bookkeeping for
67
+ the connection pool. Built with `ROBIN_PLATFORM=java gem build` after
68
+ `rake java:compile`; held to 100% JaCoCo coverage and 100% PIT mutation
69
+ coverage.
70
+ - `retry_for:` keeps retrying a job until a set time after its first
71
+ failure; `retry_queue:` runs a job's retries on another queue.
72
+ - Deploy markers (`Robin::Deploy.mark!`, `robin deploy LABEL`), drawn as
73
+ lines on the dashboard's metrics charts.
74
+ - Events: `config.event_handlers` are called with what Robin otherwise
75
+ only logs (expired, skipped, discarded and rate-limited jobs, recovered
76
+ and orphaned jobs, dead workers and swarm children, slow Redis, a lapsed
77
+ heartbeat, memory limits, slow iterations, reliable push buffering).
78
+ See `Robin::Events`.
79
+ - Reliable push reports jobs it drops from a full buffer, and jobs still
80
+ buffered when the process exits (after one last try to push them), to
81
+ the error handlers, as `Robin::ReliablePush::Dropped` and `Lost`.
82
+ - GVL contention monitoring (C extension): each heartbeat
83
+ records how much of the time the process's threads could wait for the
84
+ GVL they did, shown on the dashboard and sent to StatsD as
85
+ `gvl.contention`; contention kept high logs a warning and emits a
86
+ `gvl.contended` event. `config.gvl_monitoring = false` turns it off.
87
+ - The dashboard in 25 languages, right-to-left ones included, chosen from
88
+ the browser's languages or under Settings; each loads on demand.
89
+ - Dashboard extensions can bring their own JavaScript module, stylesheet
90
+ and other assets, JSON data endpoints, and translations (YAML files
91
+ used by `request.t`, the script's `t` and the navigation).
92
+ - StatsD counters for batches (created, complete, success, death,
93
+ cancelled, plus a duration timing) and for recovered, orphaned and
94
+ poison-pill jobs.
95
+ - Rails: `config.robin.current_attributes = ["Current"]` carries
96
+ `ActiveSupport::CurrentAttributes` from where a job is pushed into it.
97
+ - Rate limiters take `reschedule:` and `backoff:`, `config.rate_limit_backoff`
98
+ sets a backoff for all of them, and `Robin::Limiter.unlimited` is a
99
+ limiter that never limits.
100
+ - `perform_bulk(..., spread_interval: 3600)` (and `"spread_interval"` in
101
+ `Client#push_bulk`) spreads the jobs evenly over that many seconds.
102
+ - Iterable jobs: an `on_cancel` hook, a `current_object` reader, and
103
+ `max_iteration_runtime` (config or job option), after which a run
104
+ checkpoints and requeues itself.
105
+ - `Robin::Web.job_info(label) { |job| ... }` adds rows (text or links) to
106
+ job pages; `Robin::Web.assets_path` serves the dashboard's assets from a
107
+ CDN.
108
+ - `robin top`: a live terminal view of totals, queues, processes and
109
+ running jobs.
110
+ - The Busy page can show only processes or only running jobs
111
+ (`?only=processes`, `?only=jobs`); INFO (Ctrl-T on macOS and BSD) dumps
112
+ threads like TTIN; a suppressed duplicate unique job's log line names
113
+ the jid holding the lock.
114
+ - Batch tags (`batch.tags = [...]`), with `Robin::Batch.list(tag:)` and a
115
+ tag filter on the dashboard's Batches page.
116
+ - `Robin::Statsd.new(distributions: true)` sends Datadog distributions
117
+ (`jobs.perform_dist`, `batch.duration_dist`); under DogStatsD, batch
118
+ metrics carry the batch's tags.
119
+ - `Robin::Client.rewriters`: callables that may rewrite a job hash before
120
+ it's pushed. The Rails integration uses one so periodic jobs (and
121
+ `Client.push`) can name ActiveJob classes.
122
+ - `Robin::Periodic.verify!` checks a periodic schedule in an app's test
123
+ suite.
124
+ - Bucket and window limiters take `within_limit(used: n)`;
125
+ `Robin::Limiter.via(pool)` keeps limiters in another Redis.
126
+ - Limiters take `policy: :skip`: when over the limit, `within_limit`
127
+ returns nil without running the block instead of raising.
128
+ - `Robin::Queue#delete_by_class` deletes a class's jobs inside Redis; the
129
+ dashboard offers it when the selected jobs share a class.
130
+ - `Robin::Web::BatchStatus`: a Rack app your own pages can poll for a
131
+ batch's progress (counts and states only).
132
+ - `robin swarm -n 0.5` (or `ROBIN_SWARM_COUNT`): a decimal means that many
133
+ processes per CPU core.
134
+ - Benchmarks: `rake bench` (parser and throughput) and `rake bench:compare`
135
+ (Robin vs Sidekiq on the same workload, with optional added latency).
136
+ - Redis Sentinel (`sentinels:` with the master named in the URL), with a
137
+ role check on connect; sharding with `MyJob.set(pool:)` and
138
+ `Robin::Client.via(pool) { ... }`.
139
+ - Health checks: an HTTP endpoint per server (`config.health_port`),
140
+ `robin health` for exec probes, and `/health` on the dashboard.
141
+ - OpenTelemetry tracing (`require "robin/opentelemetry"`) with producer
142
+ and consumer spans and trace context carried in the job.
143
+ - Profiling single job runs with Vernier (`MyJob.set(profile: true)`),
144
+ stored in Redis for the Firefox Profiler.
145
+ - Operations: process titles, a warning when Redis round trips are slow,
146
+ systemd notify and watchdog support, `REDIS_PROVIDER`, connections
147
+ named `robin`, passwords never shown by `inspect`, Capistrano- and
148
+ Heroku-aware tags and hostnames, previous signal traps still run,
149
+ capsules frozen after boot, optional idle connection reaping
150
+ (`idle_timeout`).
151
+ - Jobs: per-job `log_level` and `logging: false`.
152
+ - Dashboard extensions (`Robin::Web::Extension`, `Robin::Web.register`):
153
+ pages of your own, built from blocks, with actions.
154
+ - A Profiles page in the dashboard, to download job profiles for the
155
+ Firefox Profiler.
156
+ - Works with any Ruby or Rack app. Extension points: `config.push_deferrer`
157
+ (defer delivery, e.g. until a transaction commits), `Robin::Job.inclusion_checks`,
158
+ and `robin` CLI plugins (`robin/plugins/*.rb` in any gem) that add app
159
+ loaders and environment variables.
160
+ - Resizing a running process: a capsule's worker count changes without a
161
+ restart, from the dashboard's Busy page, `POST
162
+ api/processes/:identity/resize`, or `Robin::ProcessRecord#resize!`.
163
+ New workers start at once, extra ones finish their job first, and a
164
+ Redis pool Robin sized grows to fit (a `workers.resized` event).
165
+ - A Puma plugin (`plugin :robin`, optionally `robin_config { |config| ... }`)
166
+ runs an embedded server in every worker once the app has loaded, and
167
+ stops it gracefully on shutdown and on hot, phased and fork-worker
168
+ restarts. Robin is never loaded in Puma's master, so phased restarts
169
+ with `prune_bundler` pick up changed gem versions. `Embedded#stop`
170
+ waits for a stop already in progress (including one asked for from the
171
+ dashboard) instead of stopping twice.
172
+ - The Rails integration, part of the robin gem. It turns itself on in a
173
+ Rails app (Bundler loads robin after Rails); elsewhere it loads
174
+ nothing of Rails. `require "robin/rails"` turns it on where robin was
175
+ required before Rails:
176
+ - Railtie: jobs run in the Rails executor and reloader, `Rails.logger`
177
+ outside the server, `Robin.config.environment` from `Rails.env`.
178
+ - ActiveJob adapter (`:robin`) with `robin_options`, pipelined
179
+ `perform_all_later` and continuation support.
180
+ - `bundle exec robin` boots the Rails app and reads `RAILS_ENV`.
181
+ - Errors reported to `Rails.error` (`config.robin.report_errors`).
182
+ - Transaction-aware pushes (`config.robin.transactional_push`).
183
+ - Generators: `robin:install` and `robin:job`.
184
+ - Failed jobs' backtraces go through `Rails.backtrace_cleaner`.
185
+ - What jobs log through `Rails.logger` is tagged with their class and jid.
186
+ - Mail deliveries show as the mail they send (`UserMailer#welcome`).
187
+ - Robin's events are published through `ActiveSupport::Notifications` as
188
+ `"<event>.robin"` (`config.robin.notifications`).
189
+
190
+ ### Changed
191
+
192
+ - Performance: at most one Redis round trip per job (a finished job is
193
+ acknowledged by the round trip that fetches the next), replies awaited
194
+ rather than polled for, middleware chains that build no closures per
195
+ job, bulk pushes normalized once per batch, and job ids drawn from
196
+ batched random bytes.
197
+ - Fewer Redis round trips outside the job path:
198
+ - A heartbeat is one round trip (its metrics, leader lease and in-flight
199
+ sweep go in its transaction; it reports the previous heartbeat's round
200
+ trip time), and buffered jobs of every capsule go back in one more.
201
+ - Recovering dead processes takes three round trips however many there
202
+ are, not one per process and job; pruning processes and sampling queue
203
+ sizes take one each (two), and periodic jobs take three per tick
204
+ however many are due.
205
+ - A debounced push, a batch callback, a deploy mark, and a limiter that
206
+ gives up (it counts itself over its limit) take one round trip fewer;
207
+ so does loading a Lua script Redis lost. A debounced job's claim is
208
+ made with its push, so a push a later middleware stops no longer
209
+ unschedules the job pushed before it.
210
+ - `retry_all` and `kill_all`, and the dashboard's actions on selected or
211
+ all jobs, take a few round trips per page of 100, not two per job;
212
+ `Batch#cancel!` takes two whatever the depth of children.
213
+ - The dashboard's stats, processes, busy jobs and limiters are read in
214
+ one round trip each, and every page in a fixed number.
215
+ - `Robin::Client.together { ... }` sends the jobs its block pushes in one
216
+ round trip. The Rails integration uses it for `perform_all_later` (one round trip
217
+ for all its classes and queues) and for transactional pushes (the jobs
218
+ of one transaction go together when it commits).
219
+ - Tests pin the round trips of each of these with `assert_roundtrips`.
220
+ - A finished job is acknowledged with the next round trip, or within 2
221
+ milliseconds by a thread of its own if no round trip comes (a worker
222
+ that takes a job fetched for it makes none, and long jobs can keep
223
+ every worker busy), so a crash re-runs only jobs that finished just
224
+ before it.
225
+ - An idle worker of a capsule with one queue blocks on it again without
226
+ a non-blocking fetch first, halving the commands idle processes send.
227
+ - JRuby: hash arguments with keys that aren't symbols take braces, as
228
+ JRuby 10.1 can't compile a method with a call passing one without
229
+ (enforced by the `Robin/BracedHashArgument` RuboCop cop), and
230
+ `push_bulk` does its per-job work in methods of their own, compiled
231
+ after 50 jobs. Warm pushes are about 12% faster.
232
+ - `robin stats`, `robin top` and `robin deploy` boot the app as a client,
233
+ so its `configure_client` blocks (Redis settings, say) apply to them.
234
+ - Redis formats shared with other Ruby job tools: schedule, retry and
235
+ dead set scores (and a job's "at") in epoch seconds; scheduled and
236
+ failed jobs stored without "at" (enqueued_at is set when they reach
237
+ their queue); a 0-based "retry_count" in place of "failures"
238
+ (Job#retry_count is now Job#failures); compressed backtraces as plain
239
+ base64; process records at the process's identity, with signals named
240
+ TSTP, TERM and TTIN; daily totals kept five years. Older float-second
241
+ timestamps are still read.
242
+ - In-flight jobs are acknowledged by key, not by payload: each fetched job
243
+ goes into a key of its own (`inflight:<identity>:<n>`, registered in the
244
+ process's `inflight:<identity>` set and reused once acknowledged), so a
245
+ job's payload is never sent back to Redis, and batches of large jobs
246
+ are taken with plain commands instead of copying them through Lua. With
247
+ 100 KB arguments a process runs about 4x as many jobs per second, with a
248
+ fifth of the Redis CPU per job; with 10 KB, about twice as many. No-op
249
+ jobs cost a few percent more (a key per job is created and deleted).
250
+ Recovery still reads the single in-flight list of earlier development
251
+ versions.
252
+ - Job log lines the logger would drop cost nothing: no timing is
253
+ formatted and the logger isn't called.
254
+ - The C extension encodes commands and, in one release of the GVL,
255
+ checks the connection, writes the commands and reads the start of the
256
+ replies (`NativeIO.roundtrip`).
257
+ - No shared locks per job: workers keep their running job to themselves,
258
+ metrics are recorded per thread and merged on flush, so threads that run
259
+ in parallel (JRuby) never wait on each other.
260
+ - Capsules with a single queue fetch with pipelined plain commands instead
261
+ of a Lua script, which costs Redis about a third less per job.
262
+ - Pipelines run Lua scripts by their SHA, as single calls do: a
263
+ connection remembers which scripts its server has, and loads one it
264
+ hasn't seen in the same round trip, so a script's source is sent once
265
+ per connection, not with every pipeline. (Transactions still send the
266
+ source, so a server that lost its scripts can't apply half of one.)
267
+ - Error handlers run in the failed job's logging context, and the
268
+ default handler logs an error, its context and backtrace as one entry.
269
+ - `push_bulk` inside a batch's `jobs` block adds its jobs to the batch in
270
+ one round trip, not one per job; if it raises, none are added.
271
+ - Cancelling a batch (`Robin::Batch#cancel!`, or from the dashboard)
272
+ cancels its child batches and their jobs too; the batch completes
273
+ without succeeding.
274
+ - `Client.timestamp_ms(nil)` now raises instead of meaning "now".
275
+ - `logger` is a declared dependency (a bundled gem since Ruby 3.5).
276
+ - Tested against Redis 7.4, 8.2 and 8.10, Valkey 9.1 and Dragonfly 2.0.
277
+ - `rake test` runs test files in parallel processes (`rake test:serial` in
278
+ one).
279
+ - Robin may never get slower, after [WebKit's performance
280
+ policy](https://webkit.org/performance/) ([CONTRIBUTING.md](CONTRIBUTING.md)):
281
+ `rake perf` runs Robin's benchmarks (`duel/`) with
282
+ [Duel](https://github.com/sferik/duel-ruby), on every pull request and
283
+ push in CI (CRuby with YJIT and without, and JRuby), and fails if a
284
+ change is slower than its baseline at any of them, and each night if
285
+ `main` is slower than the last release. CPU time, instructions and
286
+ Redis CPU are judged by a 99% confidence interval for paired ratios,
287
+ over three processes of each version and confirmed in fresh ones;
288
+ allocations, round trips and Redis commands are counted, and peak
289
+ memory measured; a regression comes with a profile of each version.
290
+ The benchmarks cover pushing, running jobs (one worker's loop and a
291
+ whole server), scheduling (and the leader's wake-up), heartbeats,
292
+ crash recovery, periodic jobs, rate limiters, the dashboard's
293
+ requests, the protocol and the JIT warm-up.
294
+ - Performance: idle workers block 2s per fetch, and check whether their
295
+ queue was paused in the same round trip, as of the moment a job
296
+ arrives (Redis runs the check once the blocking fetch returns); built-in client
297
+ middleware (unique, debounce, batch, encryption) is skipped for jobs
298
+ that don't use it (bulk pushes about 10% faster, twice as fast before
299
+ JRuby has warmed up); `robin swarm` loads all of Robin (`Robin.eager_load!`) and
300
+ runs `Process.warmup` before forking, so children share it
301
+ copy-on-write (about 1 MB less private memory each); built-in
302
+ middleware is built once instead of per job (`Middleware::Stateless`);
303
+ stored backtraces are compressed, with `config.backtrace_cleaner`;
304
+ `retry_all`, `kill_all` and dashboard bulk actions page through sets;
305
+ timings and histograms leave out failed runs.
306
+ - Idle time is spent for busy time: the cluster's leader wakes its
307
+ scheduler when the next scheduled job or retry is due, rather than at
308
+ its next poll (jobs scheduled ahead start within milliseconds, not
309
+ seconds, late); a standalone server runs `Process.warmup` before it
310
+ starts, and, if asked, a full GC once it has gone idle
311
+ (`config.idle_gc`: off by default; `true`, or the fewest jobs since the
312
+ last collection, turns it on); and off CRuby a server warms up the JIT before its workers start
313
+ (`config.jit_warmup`; on JRuby a new server's first pushes are about
314
+ 3x as fast, and it reaches full speed sooner).
315
+ - Jobs: `Robin::Job::Timeout` is no longer a `StandardError`.
316
+ - Test modes set with a block are thread-local.
317
+ - Finding jobs by jid searches queues inside Redis; `Robin::JobLocator`
318
+ and an optional jid index (`config.jid_index`) point it at the right
319
+ queue.
320
+ - Iterable jobs make one Redis round trip every five seconds: the
321
+ checkpoint, progress and cancellation check share a transaction, so a
322
+ cancelled job stops at its next checkpoint (within about five seconds,
323
+ was one). An interruption saves the checkpoint and requeues the job in
324
+ one transaction, and finishing deletes the state and reports progress
325
+ in one pipeline. A limiter's definition is written once per process and
326
+ Redis (and again when it changes) rather than once per limiter object,
327
+ so limiters built inside `perform` cost no extra round trip.
328
+ - Fewer Redis round trips around jobs: a blocking fetch checks for a
329
+ pause in the same round trip; `unique_until: :start` jobs release
330
+ their lock once; expired jobs outside a batch and skipped cancelled
331
+ jobs cost none. A failing batch job's failure goes in its retry's
332
+ transaction, and a job that dies or is discarded is moved, acknowledged,
333
+ unlocked and recorded with its batch in one transaction (was up to
334
+ five round trips). A push inside a batch's `jobs` block registers and
335
+ sends the job in one transaction, a new batch (child or not) is
336
+ created in one, and `push_bulk` takes its jobs' unique locks and makes
337
+ their debounce claims in one round trip each instead of one per job.
338
+ `unique_for` of zero or less takes no lock.
339
+
340
+ ### Fixed
341
+
342
+ - A forked child (a Puma worker, a swarm child) lets go of the TLS
343
+ connections it inherited without a TLS goodbye, which made Redis hang
344
+ up on the parent's connection; its parent's next command failed.
345
+ - An iterable job with nothing to iterate finishes at 100% instead of
346
+ failing on its progress.
347
+ - Expiring jobs count `expires_in` from when the job is due, not when it
348
+ was pushed, so a scheduled job no longer expires before it runs.
349
+ - A job that dies or is discarded now gives up its unique lock in the
350
+ retrier, whatever ended it.
351
+ - Hardening: graceful-shutdown requeues don't count toward poison-pill
352
+ detection; failure recording never strands a job (unreadable or binary
353
+ error messages, bad `robin_retry_in` values); fetch errors back off
354
+ instead of respawning threads; connections drop and reconnect after
355
+ READONLY/LOADING/failover replies and never resend a partially written
356
+ pipeline; the heartbeat never writes decrypted arguments; retries and
357
+ dead jobs are stored as pushed (not as the job mutated them); iterating
358
+ while deleting visits every job; batch jobs are never deferred; one
359
+ shared grace period for interrupted jobs, which are only interrupted
360
+ inside `perform`; errors caused by the shutdown don't count as failures;
361
+ iterable jobs checkpoint when an iteration raises; jitter on custom retry
362
+ delays; death handlers run for discarded jobs (`discarded_at`); failing
363
+ `:startup` hooks abort the boot; boot checks for Redis 6.2+, eviction
364
+ policy, pool size, concurrency, timeout and duplicate queues; orphaned
365
+ in-flight jobs on live processes are swept back to their queues; the
366
+ heartbeat and leader duties run on their own threads and Redis
367
+ connections, and a lapsed heartbeat is reported.
368
+
369
+ ### Security
370
+
371
+ - Invalid Redis URLs no longer show their password in the error.
372
+ - The dashboard rejects cross-site writes using `Sec-Fetch-Site` too.