sidekiq 6.4.1 → 7.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 99c9e264c092b88ea726be158fafe5bbab91f82f4b5864dee406280622e98e4b
-  data.tar.gz: acd72bd99929d7c9d129cb9662276cc5adb7214de07cd4fc8accf6b9d521994a
+  metadata.gz: 9c6762c97172b3f8e4b7cc1fd940756ee8796faf70ece8d5e9ede9e2dc7797fe
+  data.tar.gz: 905e0e1f381e1c40832ab2137589161fcccfb63c847ad36c9cca15765d952659
 SHA512:
-  metadata.gz: 622c25276c017302c1a9d144e9366043ba359b2c3b0c57d4e7baad8f9de2e9c9969a86c91acdbefcf736af92e297c4e1fbe2008aa41e0c1accadda77dd0724f5
-  data.tar.gz: 7e64012a5368cb0158ecaa50cdea6447709a64dd3a2816b36a31e7f17d70fffff81bd8d317c0cc1f9a6317adcffad9c200c48f9ca4bf208afba819ff7a07738e
+  metadata.gz: acfc9bb74585cc65ad1af27e263668ec154ac791baa99de2a1542cfe462a7b13544a758a7bd574d623088c9e683f705437cc4e4495f05e57770037d030214a44
+  data.tar.gz: 31320e6cd2fab0d253f3a32616f51360c4ad6a9ef30acef5f88a6ccc6eaf75e04b76ea12dd053dd29442d96fa49234f65e14801e7600cf0f300c023d75aecf43

data/Changes.md

@@ -2,7 +2,102 @@
 
 [Sidekiq Changes](https://github.com/mperham/sidekiq/blob/main/Changes.md) | [Sidekiq Pro Changes](https://github.com/mperham/sidekiq/blob/main/Pro-Changes.md) | [Sidekiq Enterprise Changes](https://github.com/mperham/sidekiq/blob/main/Ent-Changes.md)
 
-HEAD
+main
+----------
+
+- Embedded mode!
+- Capsules!!
+- Job Execution metrics!!!
+- See `docs/7.0-Upgrade.md` for release notes
+
+6-x
+----------
+
+- Add pagination to Busy page [#5556]
+- Speed up WorkSet#each [#5559]
+- Adjust CurrentAttributes to work with the String class name so we aren't referencing
+the Class within a Rails initializer [#5536]
+
+6.5.7
+----------
+
+- Updates for JA and ZH locales
+- Further optimizations for scheduled polling [#5513]
+
+6.5.6
+----------
+
+- Fix deprecation warnings with redis-rb 4.8.0 [#5484]
+- Lock redis-rb to < 5.0 as we are moving to redis-client in Sidekiq 7.0
+
+6.5.5
+----------
+
+- Fix require issue with job_retry.rb [#5462]
+- Improve Sidekiq::Web compatibility with Rack 3.x
+
+6.5.4
+----------
+
+- Fix invalid code on Ruby 2.5 [#5460]
+- Fix further metrics dependency issues [#5457]
+
+6.5.3
+----------
+
+- Don't require metrics code without explicit opt-in [#5456]
+
+6.5.2
+----------
+
+- [Job Metrics are under active development, help wanted!](https://github.com/mperham/sidekiq/wiki/Metrics#contributing) **BETA**
+- Add `Context` column on queue page which shows any CurrentAttributes [#5450]
+- `sidekiq_retry_in` may now return `:discard` or `:kill` to dynamically stop job retries [#5406]
+- Smarter sorting of processes in /busy Web UI [#5398]
+- Fix broken hamburger menu in mobile UI [#5428]
+- Require redis-rb 4.5.0. Note that Sidekiq will break if you use the
+  [`Redis.exists_returns_integer = false`](https://github.com/redis/redis-rb/blob/master/CHANGELOG.md#450) flag. [#5394]
+
+6.5.1
+----------
+
+- Fix `push_bulk` breakage [#5387]
+
+6.5.0
+---------
+
+- Substantial refactoring of Sidekiq server internals, part of a larger effort
+  to reduce Sidekiq's internal usage of global methods and data, see [docs/global_to_local.md](docs/global_to_local.md) and [docs/middleware.md](docs/middleware.md).
+- **Add beta support for the `redis-client` gem**. This will become the default Redis driver in Sidekiq 7.0. [#5298]
+  Read more: https://github.com/mperham/sidekiq/wiki/Using-redis-client
+- **Add beta support for DB transaction-aware client** [#5291]
+  Add this line to your initializer and any jobs created during a transaction
+  will only be pushed to Redis **after the transaction commits**. You will need to add the
+  `after_commit_everywhere` gem to your Gemfile.
+```ruby
+Sidekiq.transactional_push!
+```
+  This feature does not have a lot of production usage yet; please try it out and let us
+  know if you have any issues. It will be fully supported in Sidekiq 7.0 or removed if it
+  proves problematic.
+- Fix regression with middleware arguments [#5312]
+
+6.4.2
+---------
+
+- Strict argument checking now runs after client-side middleware [#5246]
+- Fix page events with live polling [#5184]
+- Many under-the-hood changes to remove all usage of the term "worker"
+  from the Sidekiq codebase and APIs. This mostly involved RDoc and local
+  variable names but a few constants and public APIs were changed. The old
+  APIs will be removed in Sidekiq 7.0.
+```
+Sidekiq::DEFAULT_WORKER_OPTIONS -> Sidekiq.default_job_options
+Sidekiq.default_worker_options -> Sidekiq.default_job_options
+Sidekiq::Queues["default"].jobs_by_worker(HardJob) -> Sidekiq::Queues["default"].jobs_by_class(HardJob)
+```
+
+6.4.1
 ---------
 
 - Fix pipeline/multi deprecations in redis-rb 4.6
@@ -319,6 +414,13 @@ See the [Logging wiki page](https://github.com/mperham/sidekiq/wiki/Logging) for
 - Integrate the StandardRB code formatter to ensure consistent code
   styling. [#4114, gearnode]
 
+5.2.10
+---------
+
+- Backport fix for CVE-2022-23837.
+- Migrate to `exists?` for redis-rb.
+- Lock redis-rb to <4.6 to avoid deprecations.
+
 5.2.9
 ---------
 
@@ -484,7 +586,7 @@ Sidekiq::Middleware::Server::Logging -> Sidekiq::JobLogger
 - The `SomeWorker.set(options)` API was re-written to avoid thread-local state. [#2152]
 - Sidekiq Enterprise's encrypted jobs now display "[encrypted data]" in the Web UI instead
   of random hex bytes.
-- Please see the [5.0 Upgrade notes](5.0-Upgrade.md) for more detail.
+- Please see the [5.0 Upgrade notes](docs/5.0-Upgrade.md) for more detail.
 
 4.2.10
 -----------
@@ -702,7 +804,7 @@ Sidekiq::Queues.clear_all
 - Sidekiq's internals have been completely overhauled for performance
   and to remove dependencies.  This has resulted in major speedups, as
   [detailed on my blog](http://www.mikeperham.com/2015/10/14/optimizing-sidekiq/).
-- See the [4.0 upgrade notes](4.0-Upgrade.md) for more detail.
+- See the [4.0 upgrade notes](docs/4.0-Upgrade.md) for more detail.
 
 3.5.4
 -----------
@@ -969,7 +1071,7 @@ sidekiq_options :dead => false, :retry => 5
 3.0.0
 -----------
 
-Please see [3.0-Upgrade.md](3.0-Upgrade.md) for more comprehensive upgrade notes.
+Please see [3.0-Upgrade.md](docs/3.0-Upgrade.md) for more comprehensive upgrade notes.
 
 - **Dead Job Queue** - jobs which run out of retries are now moved to a dead
   job queue.  These jobs must be retried manually or they will expire
@@ -1013,7 +1115,7 @@ Sidekiq::Client.via(ConnectionPool.new { Redis.new }) do
 end
 ```
   **Sharding support does require a breaking change to client-side
-middleware, see 3.0-Upgrade.md.**
+middleware, see docs/3.0-Upgrade.md.**
 - New Chinese, Greek, Swedish and Czech translations for the Web UI.
 - Updated most languages translations for the new UI features.
 - **Remove official Capistrano integration** - this integration has been

data/README.md

@@ -27,16 +27,16 @@ This benchmark can be found in `bin/sidekiqload` and assumes a Redis network lat
 Requirements
 -----------------
 
-- Redis: 4.0+
-- Ruby: MRI 2.5+ or JRuby 9.2+.
+- Redis: 6.2+
+- Ruby: MRI 2.7+ or JRuby 9.3+.
 
-Sidekiq 6.0 supports Rails 5.0+ but does not require it.
+Sidekiq 7.0 supports Rails 6.0+ but does not require it.
 
 
 Installation
 -----------------
 
-    gem install sidekiq
+    bundle add sidekiq
 
 
 Getting Started
@@ -52,24 +52,26 @@ Sidekiq and see its features in action.  Here's the Web UI:
 Want to Upgrade?
 -------------------
 
+Use `bundle up sidekiq` to upgrade Sidekiq and all its dependencies.
+Upgrade notes between each major version can be found in the `docs/` directory.
+
 I also sell Sidekiq Pro and Sidekiq Enterprise, extensions to Sidekiq which provide more
 features, a commercial-friendly license and allow you to support high
 quality open source development all at the same time.  Please see the
 [Sidekiq](https://sidekiq.org/) homepage for more detail.
 
-Subscribe to the **[quarterly newsletter](https://tinyletter.com/sidekiq)** to stay informed about the latest
-features and changes to Sidekiq and its bigger siblings.
-
 
 Problems?
 -----------------
 
-**Please do not directly email any Sidekiq committers with questions or problems.**  A community is best served when discussions are held in public.
+**Please do not directly email any Sidekiq committers with questions or problems.**
+A community is best served when discussions are held in public.
 
 If you have a problem, please review the [FAQ](https://github.com/mperham/sidekiq/wiki/FAQ) and [Troubleshooting](https://github.com/mperham/sidekiq/wiki/Problems-and-Troubleshooting) wiki pages.
 Searching the [issues](https://github.com/mperham/sidekiq/issues) for your problem is also a good idea.
 
-Sidekiq Pro and Sidekiq Enterprise customers get private email support.  You can purchase at https://sidekiq.org; email support@contribsys.com for help.
+Sidekiq Pro and Sidekiq Enterprise customers get private email support.
+You can purchase at https://sidekiq.org; email support@contribsys.com for help.
 
 Useful resources:
 
@@ -77,7 +79,7 @@ Useful resources:
 * Occasional announcements are made to the [@sidekiq](https://twitter.com/sidekiq) Twitter account.
 * The [Sidekiq tag](https://stackoverflow.com/questions/tagged/sidekiq) on Stack Overflow has lots of useful Q & A.
 
-Every Friday morning is Sidekiq happy hour: I video chat and answer questions.
+Every Friday morning is Sidekiq office hour: I video chat and answer questions.
 See the [Sidekiq support page](https://sidekiq.org/support.html) for details.
 
 Contributing
@@ -85,12 +87,11 @@ Contributing
 
 Please see [the contributing guidelines](https://github.com/mperham/sidekiq/blob/main/.github/contributing.md).
 
-
 License
 -----------------
 
-Please see [LICENSE](https://github.com/mperham/sidekiq/blob/main/LICENSE) for licensing details.
-
+Please see [LICENSE.txt](https://github.com/mperham/sidekiq/blob/main/LICENSE.txt) for licensing details.
+The license for Sidekiq Pro and Sidekiq Enterprise can be found in [COMM-LICENSE.txt](https://github.com/mperham/sidekiq/blob/main/COMM-LICENSE.txt).
 
 Author
 -----------------

data/bin/sidekiq

@@ -10,7 +10,7 @@ def integrate_with_systemd
   return unless ENV["NOTIFY_SOCKET"]
 
   Sidekiq.configure_server do |config|
-    Sidekiq.logger.info "Enabling systemd notification integration"
+    config.logger.info "Enabling systemd notification integration"
     require "sidekiq/sd_notify"
     config.on(:startup) do
       Sidekiq::SdNotify.ready
@@ -31,12 +31,7 @@ begin
   cli.run
 rescue => e
   raise e if $DEBUG
-  if Sidekiq.error_handlers.length == 0
-    warn e.message
-    warn e.backtrace.join("\n")
-  else
-    cli.handle_exception e
-  end
-
+  warn e.message
+  warn e.backtrace.join("\n")
   exit 1
 end

data/bin/sidekiqload

@@ -8,21 +8,18 @@ $TESTING = false
 require "bundler/setup"
 Bundler.require(:default, :load_test)
 
-require_relative "../lib/sidekiq/cli"
-require_relative "../lib/sidekiq/launcher"
-
-Sidekiq.configure_server do |config|
-  config.options[:concurrency] = 10
+x = Sidekiq.configure_embed do |config|
   config.redis = {db: 13, port: 6380}
+  config.concurrency = 10
   # config.redis = { db: 13, port: 6380, driver: :hiredis}
-  config.options[:queues] << "default"
+  config.queues = %w[default]
   config.logger.level = Logger::ERROR
   config.average_scheduled_poll_interval = 2
   config.reliable! if defined?(Sidekiq::Pro)
 end
 
 class LoadWorker
-  include Sidekiq::Worker
+  include Sidekiq::Job
   sidekiq_options retry: 1
   sidekiq_retry_in do |x|
     1
@@ -36,7 +33,6 @@ end
 
 # brew tap shopify/shopify
 # brew install toxiproxy
-# gem install toxiproxy
 # run `toxiproxy-server` in a separate terminal window.
 require "toxiproxy"
 # simulate a non-localhost network for realer-world conditions.
@@ -86,16 +82,11 @@ def Process.rss
   `ps -o rss= -p #{Process.pid}`.chomp.to_i
 end
 
-iter = 50
+iter = 10
 count = 10_000
 
 iter.times do
-  arr = Array.new(count) do
-    []
-  end
-  count.times do |idx|
-    arr[idx][0] = idx
-  end
+  arr = Array.new(count) { |idx| [idx] }
   Sidekiq::Client.push_bulk("class" => LoadWorker, "args" => arr)
 end
 Sidekiq.logger.error "Created #{count * iter} jobs"
@@ -103,8 +94,8 @@ Sidekiq.logger.error "Created #{count * iter} jobs"
 start = Time.now
 
 Monitoring = Thread.new do
-  while true
-    sleep 0.2
+  loop do
+    sleep 1.0
     qsize = Sidekiq.redis do |conn|
       conn.llen "queue:default"
     end
@@ -124,32 +115,38 @@ Monitoring = Thread.new do
   end
 end
 
+def with_latency(latency, &block)
+  Sidekiq.logger.error "Simulating #{latency}ms of latency between Sidekiq and redis"
+  if latency > 0
+    Toxiproxy[:redis].downstream(:latency, latency: latency).apply(&block)
+  else
+    yield
+  end
+end
+
 begin
-  # RubyProf::exclude_threads = [ Monitoring ]
+  # RubyProf.exclude_threads = [Monitoring]
   # RubyProf.start
-  events = Sidekiq.options[:lifecycle_events][:startup]
-  events.each(&:call)
-  events.clear
 
-  Sidekiq.logger.error "Simulating 1ms of latency between Sidekiq and redis"
-  Toxiproxy[:redis].downstream(:latency, latency: 1).apply do
-    launcher = Sidekiq::Launcher.new(Sidekiq.options)
-    launcher.run
+  with_latency(Integer(ENV.fetch("LATENCY", "1"))) do
+    x.run
 
-    while readable_io = IO.select([self_read])
+    while (readable_io = IO.select([self_read]))
       signal = readable_io.first[0].gets.strip
-      handle_signal(launcher, signal)
+      handle_signal(x, signal)
     end
   end
-rescue SystemExit => e
+rescue SystemExit
   # Sidekiq.logger.error("Profiling...")
   # result = RubyProf.stop
   # printer = RubyProf::GraphHtmlPrinter.new(result)
-  # printer.print(File.new("output.html", "w"), :min_percent => 1)
+  # printer.print(File.new("output.html", "w"), min_percent: 1)
   # normal
 rescue => e
   raise e if $DEBUG
   warn e.message
   warn e.backtrace.join("\n")
   exit 1
+ensure
+  x.stop
 end