sidekiq 6.1.1 → 6.5.10

Sign up to get free protection for your applications and to get access to all the features.
Files changed (118) hide show
  1. checksums.yaml +4 -4
  2. data/Changes.md +250 -3
  3. data/LICENSE +3 -3
  4. data/README.md +10 -6
  5. data/bin/sidekiq +3 -3
  6. data/bin/sidekiqload +70 -66
  7. data/bin/sidekiqmon +1 -1
  8. data/lib/generators/sidekiq/job_generator.rb +57 -0
  9. data/lib/generators/sidekiq/templates/{worker.rb.erb → job.rb.erb} +2 -2
  10. data/lib/generators/sidekiq/templates/{worker_spec.rb.erb → job_spec.rb.erb} +1 -1
  11. data/lib/generators/sidekiq/templates/{worker_test.rb.erb → job_test.rb.erb} +1 -1
  12. data/lib/sidekiq/api.rb +352 -156
  13. data/lib/sidekiq/cli.rb +86 -41
  14. data/lib/sidekiq/client.rb +49 -73
  15. data/lib/sidekiq/{util.rb → component.rb} +12 -14
  16. data/lib/sidekiq/delay.rb +3 -1
  17. data/lib/sidekiq/extensions/action_mailer.rb +3 -2
  18. data/lib/sidekiq/extensions/active_record.rb +1 -1
  19. data/lib/sidekiq/extensions/generic_proxy.rb +4 -2
  20. data/lib/sidekiq/fetch.rb +31 -20
  21. data/lib/sidekiq/job.rb +13 -0
  22. data/lib/sidekiq/job_logger.rb +16 -28
  23. data/lib/sidekiq/job_retry.rb +79 -59
  24. data/lib/sidekiq/job_util.rb +71 -0
  25. data/lib/sidekiq/launcher.rb +126 -65
  26. data/lib/sidekiq/logger.rb +11 -20
  27. data/lib/sidekiq/manager.rb +35 -34
  28. data/lib/sidekiq/metrics/deploy.rb +47 -0
  29. data/lib/sidekiq/metrics/query.rb +153 -0
  30. data/lib/sidekiq/metrics/shared.rb +94 -0
  31. data/lib/sidekiq/metrics/tracking.rb +134 -0
  32. data/lib/sidekiq/middleware/chain.rb +88 -42
  33. data/lib/sidekiq/middleware/current_attributes.rb +63 -0
  34. data/lib/sidekiq/middleware/i18n.rb +6 -4
  35. data/lib/sidekiq/middleware/modules.rb +21 -0
  36. data/lib/sidekiq/monitor.rb +2 -2
  37. data/lib/sidekiq/paginator.rb +17 -9
  38. data/lib/sidekiq/processor.rb +47 -41
  39. data/lib/sidekiq/rails.rb +32 -4
  40. data/lib/sidekiq/redis_client_adapter.rb +154 -0
  41. data/lib/sidekiq/redis_connection.rb +84 -55
  42. data/lib/sidekiq/ring_buffer.rb +29 -0
  43. data/lib/sidekiq/scheduled.rb +96 -32
  44. data/lib/sidekiq/testing/inline.rb +4 -4
  45. data/lib/sidekiq/testing.rb +38 -39
  46. data/lib/sidekiq/transaction_aware_client.rb +45 -0
  47. data/lib/sidekiq/version.rb +1 -1
  48. data/lib/sidekiq/web/action.rb +3 -3
  49. data/lib/sidekiq/web/application.rb +41 -16
  50. data/lib/sidekiq/web/csrf_protection.rb +32 -5
  51. data/lib/sidekiq/web/helpers.rb +52 -30
  52. data/lib/sidekiq/web/router.rb +4 -1
  53. data/lib/sidekiq/web.rb +38 -78
  54. data/lib/sidekiq/worker.rb +142 -16
  55. data/lib/sidekiq.rb +114 -31
  56. data/sidekiq.gemspec +12 -4
  57. data/web/assets/images/apple-touch-icon.png +0 -0
  58. data/web/assets/javascripts/application.js +114 -60
  59. data/web/assets/javascripts/chart.min.js +13 -0
  60. data/web/assets/javascripts/chartjs-plugin-annotation.min.js +7 -0
  61. data/web/assets/javascripts/dashboard.js +50 -67
  62. data/web/assets/javascripts/graph.js +16 -0
  63. data/web/assets/javascripts/metrics.js +262 -0
  64. data/web/assets/stylesheets/application-dark.css +61 -51
  65. data/web/assets/stylesheets/application-rtl.css +0 -4
  66. data/web/assets/stylesheets/application.css +84 -243
  67. data/web/locales/ar.yml +8 -2
  68. data/web/locales/el.yml +43 -19
  69. data/web/locales/en.yml +11 -1
  70. data/web/locales/es.yml +18 -2
  71. data/web/locales/fr.yml +8 -1
  72. data/web/locales/ja.yml +10 -0
  73. data/web/locales/lt.yml +1 -1
  74. data/web/locales/pt-br.yml +27 -9
  75. data/web/locales/ru.yml +4 -0
  76. data/web/locales/zh-cn.yml +36 -11
  77. data/web/locales/zh-tw.yml +32 -7
  78. data/web/views/_footer.erb +1 -1
  79. data/web/views/_job_info.erb +1 -1
  80. data/web/views/_nav.erb +1 -1
  81. data/web/views/_poll_link.erb +2 -5
  82. data/web/views/_summary.erb +7 -7
  83. data/web/views/busy.erb +57 -21
  84. data/web/views/dashboard.erb +23 -14
  85. data/web/views/dead.erb +1 -1
  86. data/web/views/layout.erb +2 -1
  87. data/web/views/metrics.erb +69 -0
  88. data/web/views/metrics_for_job.erb +87 -0
  89. data/web/views/morgue.erb +6 -6
  90. data/web/views/queue.erb +15 -11
  91. data/web/views/queues.erb +4 -4
  92. data/web/views/retries.erb +7 -7
  93. data/web/views/retry.erb +1 -1
  94. data/web/views/scheduled.erb +1 -1
  95. metadata +52 -39
  96. data/.circleci/config.yml +0 -71
  97. data/.github/contributing.md +0 -32
  98. data/.github/issue_template.md +0 -11
  99. data/.gitignore +0 -13
  100. data/.standard.yml +0 -20
  101. data/3.0-Upgrade.md +0 -70
  102. data/4.0-Upgrade.md +0 -53
  103. data/5.0-Upgrade.md +0 -56
  104. data/6.0-Upgrade.md +0 -72
  105. data/COMM-LICENSE +0 -97
  106. data/Ent-2.0-Upgrade.md +0 -37
  107. data/Ent-Changes.md +0 -275
  108. data/Gemfile +0 -24
  109. data/Gemfile.lock +0 -208
  110. data/Pro-2.0-Upgrade.md +0 -138
  111. data/Pro-3.0-Upgrade.md +0 -44
  112. data/Pro-4.0-Upgrade.md +0 -35
  113. data/Pro-5.0-Upgrade.md +0 -25
  114. data/Pro-Changes.md +0 -795
  115. data/Rakefile +0 -10
  116. data/code_of_conduct.md +0 -50
  117. data/lib/generators/sidekiq/worker_generator.rb +0 -57
  118. data/lib/sidekiq/exception_handler.rb +0 -27
data/Pro-2.0-Upgrade.md DELETED
@@ -1,138 +0,0 @@
1
- # Upgrading to Sidekiq Pro 2.0
2
-
3
- Sidekiq Pro 2.0 allows nested batches for more complex job workflows
4
- and provides a new reliable scheduler which uses Lua to guarantee
5
- atomicity and much higher performance.
6
-
7
- It also removes deprecated APIs, changes the batch data format and
8
- how features are activated. Read carefully to ensure your upgrade goes
9
- smoothly.
10
-
11
- Sidekiq Pro 2.0 requires Sidekiq 3.3.2 or greater. Redis 2.8 is
12
- recommended; Redis 2.4 or 2.6 will work but some functionality will not be
13
- available.
14
-
15
- **Note that you CANNOT go back to Pro 1.x once you've created batches
16
- with 2.x. The new batches will not process correctly with 1.x.**
17
-
18
- **If you are on a version of Sidekiq Pro <1.5, you should upgrade to the
19
- latest 1.x version and run it for a week before upgrading to 2.0.**
20
-
21
- ## Nested Batches
22
-
23
- Batches can now be nested within the `jobs` method.
24
- This feature enables Sidekiq Pro to handle workflow processing of any size
25
- and complexity!
26
-
27
- ```ruby
28
- a = Sidekiq::Batch.new
29
- a.on(:success, SomeCallback)
30
- a.jobs do
31
- SomeWork.perform_async
32
-
33
- b = Sidekiq::Batch.new
34
- b.on(:success, MyCallback)
35
- b.jobs do
36
- OtherWork.perform_async
37
- end
38
- end
39
- ```
40
-
41
- Parent batch callbacks are not processed until all child batch callbacks have
42
- run successfully. In the example above, `MyCallback` will always fire
43
- before `SomeCallback` because `b` is considered a child of `a`.
44
-
45
- Of course you can dynamically add child batches while a batch job is executing.
46
-
47
- ```ruby
48
- def perform(*args)
49
- do_something(args)
50
-
51
- if more_work?
52
- # Sidekiq::Worker#batch returns the Batch this job is part of.
53
- batch.jobs do
54
- b = Sidekiq::Batch.new
55
- b.on(:success, MyCallback)
56
- b.jobs do
57
- OtherWork.perform_async
58
- end
59
- end
60
- end
61
- end
62
- ```
63
-
64
- More context: [#1485]
65
-
66
- ## Batch Data
67
-
68
- The batch data model was overhauled. Batch data should take
69
- significantly less space in Redis now. A simple benchmark shows 25%
70
- savings but real world savings should be even greater.
71
-
72
- * Batch 2.x BIDs are 14 character URL-safe Base64-encoded strings, e.g.
73
- "vTF1-9QvLPnREQ". Batch 1.x BIDs were 16 character hex-encoded
74
- strings, e.g. "4a3fc67d30370edf".
75
- * In 1.x, batch data was not removed until it naturally expired in Redis.
76
- In 2.x, all data for a batch is removed from Redis once the batch has
77
- run any success callbacks.
78
- * Because of the former point, batch expiry is no longer a concern.
79
- Batch expiry is hardcoded to 30 days and is no longer user-tunable.
80
- * Failed batch jobs no longer automatically store any associated
81
- backtrace in Redis.
82
-
83
- **There's no data migration required. Sidekiq Pro 2.0 transparently handles
84
- both old and new format.**
85
-
86
- More context: [#2130]
87
-
88
- ## Reliability
89
-
90
- 2.0 brings a new reliable scheduler which uses Lua inside Redis so enqueuing
91
- scheduled jobs is atomic. Benchmarks show it 50x faster when enqueuing
92
- lots of jobs.
93
-
94
- **Two caveats**:
95
- - Client-side middleware is not executed
96
- for each job when enqueued with the reliable scheduler. No Sidekiq or
97
- Sidekiq Pro functionality is affected by this change but some 3rd party
98
- plugins might be.
99
- - The Lua script used inside the reliable scheduler is not safe for use
100
- with Redis Cluster or other multi-master Redis solutions.
101
- It is safe to use with Redis Sentinel or a typical master/slave replication setup.
102
-
103
- **You no longer require anything to use the Reliability features.**
104
-
105
- * Activate reliable fetch and/or the new reliable scheduler:
106
- ```ruby
107
- Sidekiq.configure_server do |config|
108
- config.reliable_fetch!
109
- config.reliable_scheduler!
110
- end
111
- ```
112
- * Activate reliable push:
113
- ```ruby
114
- Sidekiq::Client.reliable_push!
115
- ```
116
-
117
- More context: [#2130]
118
-
119
- ## Other Changes
120
-
121
- * You must require `sidekiq/pro/notifications` if you want to use the
122
- existing notification schemes. I don't recommend using them as the
123
- newer-style `Sidekiq::Batch#on` method is simpler and more flexible.
124
- * Several classes have been renamed. Generally these classes are ones
125
- you should not need to require/use in your own code, e.g. the Batch
126
- middleware.
127
- * You can add `attr_accessor :jid` to a Batch callback class and Sidekiq
128
- Pro will set it to the jid of the callback job. [#2178]
129
- * There's now an official API to iterate all known Batches [#2191]
130
- ```ruby
131
- Sidekiq::BatchSet.new.each {|status| p status.bid }
132
- ```
133
- * The Web UI now shows the Sidekiq Pro version in the footer. [#1991]
134
-
135
- ## Thanks
136
-
137
- Adam Prescott, Luke van der Hoeven and Jon Hyman all provided valuable
138
- feedback during the release process. Thank you guys!
data/Pro-3.0-Upgrade.md DELETED
@@ -1,44 +0,0 @@
1
- # Welcome to Sidekiq Pro 3.0!
2
-
3
- Sidekiq Pro 3.0 is designed to work with Sidekiq 4.0.
4
-
5
- ## What's New
6
-
7
- * **Redis 2.8.0 or greater is required.** Redis 2.8 was released two years
8
- ago and contains **many** useful features which Sidekiq couldn't
9
- leverage until now. **Redis 3.0.3 or greater is recommended** for large
10
- scale use.
11
-
12
- * Sidekiq Pro no longer uses Celluloid. If your application code uses Celluloid,
13
- you will need to pull it in yourself.
14
-
15
- * Pausing and unpausing queues is now instantaneous, no more polling!
16
-
17
- * Reliable fetch has been re-implemented due to the fetch changes in
18
- Sidekiq 4.0.
19
-
20
- * Support for platforms without persistent hostnames. Since the reliable\_fetch
21
- algorithm requires a persistent hostname, an alternative reliability
22
- algorithm is now available for platforms like Heroku and Docker:
23
- ```ruby
24
- Sidekiq.configure_server do |config|
25
- config.timed_fetch!
26
- end
27
- ```
28
- The wiki contains [much more detail about each reliability option](https://github.com/mperham/sidekiq/wiki/Pro-Reliability-Server).
29
-
30
- * The old 'sidekiq/notifications' features have been removed.
31
-
32
- ## Upgrade
33
-
34
- First, make sure you are using Redis 2.8 or greater. Next:
35
-
36
- * Upgrade to the latest Sidekiq Pro 2.x.
37
- ```ruby
38
- gem 'sidekiq-pro', '< 3'
39
- ```
40
- * Fix any deprecation warnings you see.
41
- * Upgrade to 3.x.
42
- ```ruby
43
- gem 'sidekiq-pro', '< 4'
44
- ```
data/Pro-4.0-Upgrade.md DELETED
@@ -1,35 +0,0 @@
1
- # Welcome to Sidekiq Pro 4.0!
2
-
3
- Sidekiq Pro 4.0 is designed to work with Sidekiq 5.0.
4
-
5
- ## What's New
6
-
7
- * Batches now "die" if any of their jobs die. You can enumerate the set
8
- of dead batches and their associated dead jobs. The success callback
9
- for a dead batch will never fire unless these jobs are fixed.
10
- ```ruby
11
- Sidekiq::Batch::DeadSet.new.each do |status|
12
- status.dead? # => true
13
- status.dead_jobs # => [...]
14
- end
15
- ```
16
- This API allows you to enumerate the batches which need help.
17
- If you fix the issue and the dead jobs succeed, the batch will succeed.
18
- * The older `reliable_fetch` and `timed_fetch` algorithms have been
19
- removed. Only super\_fetch is available in 4.0.
20
- * The statsd middleware has been tweaked to remove support for legacy,
21
- pre-3.6.0 configuration and add relevant tags.
22
- * Requires Sidekiq 5.0.5+.
23
-
24
- ## Upgrade
25
-
26
- * Upgrade to the latest Sidekiq Pro 3.x.
27
- ```ruby
28
- gem 'sidekiq-pro', '< 4'
29
- ```
30
- * Fix any deprecation warnings you see.
31
- * Upgrade to 4.x.
32
- ```ruby
33
- gem 'sidekiq-pro', '< 5'
34
- ```
35
-
data/Pro-5.0-Upgrade.md DELETED
@@ -1,25 +0,0 @@
1
- # Welcome to Sidekiq Pro 5.0!
2
-
3
- Sidekiq Pro 5.0 is mainly a cleanup release for Sidekiq 6.0. The
4
- migration should be as close to trivial as a major version bump can be.
5
- Note that Sidekiq 6.0 does have major breaking changes.
6
-
7
- ## What's New
8
-
9
- * New localizations for the Sidekiq Pro Web UI: ES, ZH, PT, JA, RU
10
- * Removed deprecated APIs and warnings.
11
- * Various changes for Sidekiq 6.0
12
- * Requires Ruby 2.5+ and Redis 4.0+
13
- * Requires Sidekiq 6.0+.
14
-
15
- ## Upgrade
16
-
17
- * Upgrade to the latest Sidekiq Pro 4.x.
18
- ```ruby
19
- gem 'sidekiq-pro', '< 5'
20
- ```
21
- * Fix any deprecation warnings you see.
22
- * Upgrade to 5.x.
23
- ```ruby
24
- gem 'sidekiq-pro', '< 6'
25
- ```