sidekiq 5.0.0 → 6.0.0

data/Ent-Changes.md

@@ -4,6 +4,82 @@
 
 Please see [http://sidekiq.org/](http://sidekiq.org/) for more details and how to buy.
 
+2.0.0
+-------------
+
+- Except for the [newly required credentials](/mperham/sidekiq/issue/4232), Sidekiq Enterprise 2.0 does
+  not have any significant migration steps.
+- Sidekiq Enterprise must now be started with valid license credentials. [#4232]
+- Call `GC.compact` if possible in sidekiqswarm before forking [#4181]
+- Changes for forward-compatibility with Sidekiq 6.0.
+- Add death handler to remove any lingering unique locks [#4162]
+- Backoff can now be customized per rate limiter [#4219]
+- Code formatting changes for StandardRB
+
+1.8.1
+-------------
+
+- Fix excessive lock reclaims with concurrent limiter [#4105]
+- Add ES translations, see issues [#3949](https://github.com/mperham/sidekiq/issues/3949) and [#3951](https://github.com/mperham/sidekiq/issues/3951) to add your own language.
+
+1.8.0
+-------------
+
+- Require Sidekiq Pro 4.0 and Sidekiq 5.2.
+- Refactor historical metrics API to use revamped Statsd support in Sidekiq Pro
+- Add a gauge to historical metrics for `default` queue latency [#4079]
+
+1.7.2
+-------------
+
+- Add PT and JA translations
+- Fix elapsed time calculations to use monotonic clock [#4000, sj26]
+- Fix edge case where flapping leadership would cause old periodic
+  jobs to be fired once [#3974]
+- Add support for sidekiqswarm memory monitoring on FreeBSD [#3884]
+
+1.7.1
+-------------
+
+- Fix Lua error in concurrent rate limiter under heavy contention
+- Remove superfluous `freeze` calls on Strings [#3759]
+
+1.7.0
+-------------
+
+- **NEW FEATURE** [Rolling restarts](https://github.com/mperham/sidekiq/wiki/Ent-Rolling-Restarts) - great for long running jobs!
+- Adjust middleware so unique jobs that don't push aren't registered in a Batch [#3662]
+- Add new unlimited rate limiter, useful for testing [#3743]
+```ruby
+limiter = Sidekiq::Limiter.unlimited(...any args...)
+```
+
+1.6.1
+-------------
+
+- Fix crash in rate limiter middleware when used with custom exceptions [#3604]
+
+1.6.0
+-------------
+
+- Show process "leader" tag on Busy page, requires Sidekiq 5.0.2 [#2867]
+- Capture custom metrics with the `save_history` API. [#2815]
+- Implement new `unique_until: 'start'` policy option. [#3471]
+
+1.5.4
+-------------
+
+- Fix broken Cron page in Web UI [#3458]
+
+1.5.3
+-------------
+
+- Remove dependency on the algorithms gem [#3446]
+- Allow user to specify max memory in megabytes with SIDEKIQ\_MAXMEM\_MB [#3451]
+- Implement logic to detect app startup failure, sidekiqswarm will exit
+  rather than try to restart the app forever [#3450]
+- Another fix for doubly-encrypted arguments [#3368]
+
 1.5.2
 -------------
 

data/Gemfile

@@ -1,29 +1,24 @@
-source 'https://rubygems.org'
+source "https://rubygems.org"
+
 gemspec
 
-gem 'rails', '>= 5.0.1'
-gem "hiredis"
-gem 'simplecov'
-gem 'minitest'
-#gem 'minitest-utils'
-gem 'toxiproxy'
+gem "rake"
+gem "redis-namespace"
+gem "rails"
+gem "sqlite3", platforms: :ruby
+gem "activerecord-jdbcsqlite3-adapter", platforms: :jruby
 
-platforms :rbx do
-  gem 'rubysl', '~> 2.0'         # if using anything in the ruby standard library
-  gem 'psych'                    # if using yaml
-  gem 'rubinius-developer_tools' # if using any of coverage, debugger, profiler
+group :test do
+  gem "minitest"
+  gem "simplecov"
 end
 
-platforms :ruby do
-  gem 'sqlite3'
+group :development, :test do
+  gem "pry-byebug", platforms: :mri
+  gem "standard"
 end
 
-platforms :mri do
-  gem 'pry-byebug'
-  gem 'ruby-prof'
+group :load_test do
+  gem "hiredis"
+  gem "toxiproxy"
 end
-
-#platforms :jruby do
-  #gem 'jruby-openssl'
-  #gem 'activerecord-jdbcsqlite3-adapter'
-#end

data/Gemfile.lock

@@ -0,0 +1,196 @@
+PATH
+  remote: .
+  specs:
+    sidekiq (6.0.0)
+      connection_pool (>= 2.2.2)
+      rack (>= 2.0.0)
+      rack-protection (>= 2.0.0)
+      redis (>= 4.1.0)
+
+GEM
+  remote: https://rubygems.org/
+  specs:
+    actioncable (6.0.0)
+      actionpack (= 6.0.0)
+      nio4r (~> 2.0)
+      websocket-driver (>= 0.6.1)
+    actionmailbox (6.0.0)
+      actionpack (= 6.0.0)
+      activejob (= 6.0.0)
+      activerecord (= 6.0.0)
+      activestorage (= 6.0.0)
+      activesupport (= 6.0.0)
+      mail (>= 2.7.1)
+    actionmailer (6.0.0)
+      actionpack (= 6.0.0)
+      actionview (= 6.0.0)
+      activejob (= 6.0.0)
+      mail (~> 2.5, >= 2.5.4)
+      rails-dom-testing (~> 2.0)
+    actionpack (6.0.0)
+      actionview (= 6.0.0)
+      activesupport (= 6.0.0)
+      rack (~> 2.0)
+      rack-test (>= 0.6.3)
+      rails-dom-testing (~> 2.0)
+      rails-html-sanitizer (~> 1.0, >= 1.2.0)
+    actiontext (6.0.0)
+      actionpack (= 6.0.0)
+      activerecord (= 6.0.0)
+      activestorage (= 6.0.0)
+      activesupport (= 6.0.0)
+      nokogiri (>= 1.8.5)
+    actionview (6.0.0)
+      activesupport (= 6.0.0)
+      builder (~> 3.1)
+      erubi (~> 1.4)
+      rails-dom-testing (~> 2.0)
+      rails-html-sanitizer (~> 1.1, >= 1.2.0)
+    activejob (6.0.0)
+      activesupport (= 6.0.0)
+      globalid (>= 0.3.6)
+    activemodel (6.0.0)
+      activesupport (= 6.0.0)
+    activerecord (6.0.0)
+      activemodel (= 6.0.0)
+      activesupport (= 6.0.0)
+    activestorage (6.0.0)
+      actionpack (= 6.0.0)
+      activejob (= 6.0.0)
+      activerecord (= 6.0.0)
+      marcel (~> 0.3.1)
+    activesupport (6.0.0)
+      concurrent-ruby (~> 1.0, >= 1.0.2)
+      i18n (>= 0.7, < 2)
+      minitest (~> 5.1)
+      tzinfo (~> 1.1)
+      zeitwerk (~> 2.1, >= 2.1.8)
+    ast (2.4.0)
+    builder (3.2.3)
+    byebug (11.0.1)
+    coderay (1.1.2)
+    concurrent-ruby (1.1.5)
+    connection_pool (2.2.2)
+    crass (1.0.4)
+    docile (1.3.2)
+    erubi (1.8.0)
+    globalid (0.4.2)
+      activesupport (>= 4.2.0)
+    hiredis (0.6.3)
+    i18n (1.6.0)
+      concurrent-ruby (~> 1.0)
+    jaro_winkler (1.5.3)
+    json (2.2.0)
+    loofah (2.2.3)
+      crass (~> 1.0.2)
+      nokogiri (>= 1.5.9)
+    mail (2.7.1)
+      mini_mime (>= 0.1.1)
+    marcel (0.3.3)
+      mimemagic (~> 0.3.2)
+    method_source (0.9.2)
+    mimemagic (0.3.3)
+    mini_mime (1.0.2)
+    mini_portile2 (2.4.0)
+    minitest (5.11.3)
+    nio4r (2.5.1)
+    nokogiri (1.10.4)
+      mini_portile2 (~> 2.4.0)
+    parallel (1.17.0)
+    parser (2.6.3.0)
+      ast (~> 2.4.0)
+    pry (0.12.2)
+      coderay (~> 1.1.0)
+      method_source (~> 0.9.0)
+    pry-byebug (3.7.0)
+      byebug (~> 11.0)
+      pry (~> 0.10)
+    rack (2.0.7)
+    rack-protection (2.0.7)
+      rack
+    rack-test (1.1.0)
+      rack (>= 1.0, < 3)
+    rails (6.0.0)
+      actioncable (= 6.0.0)
+      actionmailbox (= 6.0.0)
+      actionmailer (= 6.0.0)
+      actionpack (= 6.0.0)
+      actiontext (= 6.0.0)
+      actionview (= 6.0.0)
+      activejob (= 6.0.0)
+      activemodel (= 6.0.0)
+      activerecord (= 6.0.0)
+      activestorage (= 6.0.0)
+      activesupport (= 6.0.0)
+      bundler (>= 1.3.0)
+      railties (= 6.0.0)
+      sprockets-rails (>= 2.0.0)
+    rails-dom-testing (2.0.3)
+      activesupport (>= 4.2.0)
+      nokogiri (>= 1.6)
+    rails-html-sanitizer (1.2.0)
+      loofah (~> 2.2, >= 2.2.2)
+    railties (6.0.0)
+      actionpack (= 6.0.0)
+      activesupport (= 6.0.0)
+      method_source
+      rake (>= 0.8.7)
+      thor (>= 0.20.3, < 2.0)
+    rainbow (3.0.0)
+    rake (12.3.3)
+    redis (4.1.2)
+    redis-namespace (1.6.0)
+      redis (>= 3.0.4)
+    rubocop (0.72.0)
+      jaro_winkler (~> 1.5.1)
+      parallel (~> 1.10)
+      parser (>= 2.6)
+      rainbow (>= 2.2.2, < 4.0)
+      ruby-progressbar (~> 1.7)
+      unicode-display_width (>= 1.4.0, < 1.7)
+    rubocop-performance (1.4.1)
+      rubocop (>= 0.71.0)
+    ruby-progressbar (1.10.1)
+    simplecov (0.17.0)
+      docile (~> 1.1)
+      json (>= 1.8, < 3)
+      simplecov-html (~> 0.10.0)
+    simplecov-html (0.10.2)
+    sprockets (3.7.2)
+      concurrent-ruby (~> 1.0)
+      rack (> 1, < 3)
+    sprockets-rails (3.2.1)
+      actionpack (>= 4.0)
+      activesupport (>= 4.0)
+      sprockets (>= 3.0.0)
+    sqlite3 (1.4.1)
+    standard (0.1.2)
+      rubocop (~> 0.72.0)
+      rubocop-performance (~> 1.4.0)
+    thor (0.20.3)
+    thread_safe (0.3.6)
+    toxiproxy (1.0.3)
+    tzinfo (1.2.5)
+      thread_safe (~> 0.1)
+    unicode-display_width (1.6.0)
+    websocket-driver (0.7.1)
+      websocket-extensions (>= 0.1.0)
+    websocket-extensions (0.1.4)
+    zeitwerk (2.1.9)
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  activerecord-jdbcsqlite3-adapter
+  hiredis
+  minitest
+  pry-byebug
+  rails
+  rake
+  redis-namespace
+  sidekiq!
+  simplecov
+  sqlite3
+  standard
+  toxiproxy

data/LICENSE

@@ -5,5 +5,5 @@ the LGPLv3 license.  Please see
 for license text.
 
 Sidekiq Pro has a commercial-friendly license allowing private forks
-and modifications of Sidekiq.  Please see http://sidekiq.org/pro/ for
+and modifications of Sidekiq.  Please see https://sidekiq.org/products/pro.html for
 more detail.  You can find the commercial license terms in COMM-LICENSE.

data/Pro-4.0-Upgrade.md

@@ -0,0 +1,35 @@
+# Welcome to Sidekiq Pro 4.0!
+
+Sidekiq Pro 4.0 is designed to work with Sidekiq 5.0.
+
+## What's New
+
+* Batches now "die" if any of their jobs die.  You can enumerate the set
+  of dead batches and their associated dead jobs.  The success callback
+  for a dead batch will never fire unless these jobs are fixed.
+```ruby
+Sidekiq::Batch::DeadSet.new.each do |status|
+  status.dead? # => true
+  status.dead_jobs # => [...]
+end
+```
+This API allows you to enumerate the batches which need help.
+If you fix the issue and the dead jobs succeed, the batch will succeed.
+* The older `reliable_fetch` and `timed_fetch` algorithms have been
+  removed.  Only super\_fetch is available in 4.0.
+* The statsd middleware has been tweaked to remove support for legacy,
+  pre-3.6.0 configuration and add relevant tags.
+* Requires Sidekiq 5.0.5+.
+
+## Upgrade
+
+* Upgrade to the latest Sidekiq Pro 3.x.
+```ruby
+gem 'sidekiq-pro', '< 4'
+```
+* Fix any deprecation warnings you see.
+* Upgrade to 4.x.
+```ruby
+gem 'sidekiq-pro', '< 5'
+```
+

data/Pro-5.0-Upgrade.md

@@ -0,0 +1,25 @@
+# Welcome to Sidekiq Pro 5.0!
+
+Sidekiq Pro 5.0 is mainly a cleanup release for Sidekiq 6.0. The
+migration should be as close to trivial as a major version bump can be.
+Note that Sidekiq 6.0 does have major breaking changes.
+
+## What's New
+
+* New localizations for the Sidekiq Pro Web UI: ES, ZH, PT, JA, RU
+* Removed deprecated APIs and warnings.
+* Various changes for Sidekiq 6.0
+* Requires Ruby 2.5+ and Redis 4.0+
+* Requires Sidekiq 6.0+.
+
+## Upgrade
+
+* Upgrade to the latest Sidekiq Pro 4.x.
+```ruby
+gem 'sidekiq-pro', '< 5'
+```
+* Fix any deprecation warnings you see.
+* Upgrade to 5.x.
+```ruby
+gem 'sidekiq-pro', '< 6'
+```

data/Pro-Changes.md

@@ -4,9 +4,145 @@
 
 Please see [http://sidekiq.org/](http://sidekiq.org/) for more details and how to buy.
 
-HEAD
+5.0.0
 ---------
 
+- There is no significant migration from Sidekiq Pro 4.0 to 5.0
+  but make sure you read the [update notes for Sidekiq
+6.0](/mperham/sidekiq/blob/master/6.0-Upgrade.md).
+- Removed various deprecated APIs and associated warnings.
+- **BREAKING CHANGE** Remove the `Sidekiq::Batch::Status#dead_jobs` API in favor of
+  `Sidekiq::Batch::Status#dead_jids`. [#4217]
+- Update Sidekiq Pro codebase to use StandardRB formatting
+- Fix lingering "b-XXX-died" elements in Redis which could cause
+  excessive memory usage. [#4217]
+- Add ES translations, see issues [#3949](https://github.com/mperham/sidekiq/issues/3949) and [#3951](https://github.com/mperham/sidekiq/issues/3951) to add your own language.
+
+4.0.5
+---------
+
+- Increase super\_fetch retriever thread count from 1 to 2 to make it
+  less sensitive to Redis latency.
+- Better handling of invalid job JSON by reliable scheduler [#4053]
+- Added ZH, PT, JA and RU translations.
+
+4.0.4
+---------
+
+- Update Sidekiq::Client patches to work with new Module#prepend
+  mechanism in Sidekiq 5.2.0. [#3930]
+
+4.0.3
+---------
+
+- Add at\_exit handler to push any saved jobs in `reliable_push` when exiting. [#3823]
+- Implement batch death callback.  This is fired the first time a job within a batch dies. [#3841]
+```ruby
+batch = Sidekiq::Batch.new
+batch.on(:death, ...)
+```
+
+4.0.2
+---------
+
+- Remove super\_fetch edge case leading to an unnecessary `sleep(1)`
+  call and resulting latency [#3790]
+- Fix possible bad statsd metric call on super\_fetch startup
+- Remove superfluous `freeze` calls on Strings [#3759]
+
+4.0.1
+---------
+
+- Fix incompatibility with the statsd-ruby gem [#3740]
+- Add tags to Statsd metrics when using Datadog [#3744]
+
+4.0.0
+---------
+
+- See the [Sidekiq Pro 4.0](Pro-4.0-Upgrade.md) release notes.
+
+
+3.7.1
+---------
+
+- Deprecate timed\_fetch.  Switch to super\_fetch:
+```ruby
+config.super_fetch!
+```
+
+
+3.7.0
+---------
+
+- Refactor batch job success/failure to gracefully handle several edge
+  cases with regard to Sidekiq::Shutdown.  This should greatly reduce
+  the chances of seeing the long-standing "negative pending count" problem. [#3710]
+
+
+3.6.1
+---------
+
+- Add support for Datadog::Statsd, it is the recommended Statsd client.  [#3699]
+```ruby
+Sidekiq::Pro.dogstatsd = ->{ Datadog::Statsd.new("metrics.example.com", 8125) }
+```
+- Size the statsd connection pool based on Sidekiq's concurrency [#3700]
+
+
+3.6.0
+---------
+
+This release overhauls the Statsd metrics support and adds more
+metrics for tracking Pro feature usage.  In your initializer:
+```ruby
+Sidekiq::Pro.statsd = ->{ ::Statsd.new("127.0.0.1", 8125) }
+```
+Sidekiq Pro will emit more metrics to Statsd:
+```
+jobs.expired - when a job is expired
+jobs.recovered.push - when a job is recovered by reliable_push after network outage
+jobs.recovered.fetch - when a job is recovered by super_fetch after process crash
+batch.created - when a batch is created
+batch.complete - when a batch is completed
+batch.success - when a batch is successful
+```
+Sidekiq Pro's existing Statsd middleware has been rewritten to leverage the new API.
+Everything should be backwards compatible with one deprecation notice.
+
+
+3.5.4
+---------
+
+- Fix case in SuperFetch where Redis downtime can lead to processor thread death [#3684]
+- Fix case where TimedFetch might not recover some pending jobs
+- Fix edge case in Batch::Status#poll leading to premature completion [#3640]
+- Adjust scan API to check 100 elements at a time, to minimize network round trips
+  when scanning large sets.
+
+3.5.3
+---------
+
+- Restore error check for super\_fetch's job ack [#3601]
+- Trim error messages saved in Batch's failure hash, preventing huge
+  messages from bloating Redis. [#3570]
+
+3.5.2
+---------
+
+- Fix `Status#completed?` when run against a Batch that had succeeded
+  and was deleted. [#3519]
+
+3.5.1
+---------
+
+- Work with Sidekiq 5.0.2+
+- Improve performance of super\_fetch with weighted queues [#3489]
+
+3.5.0
+---------
+
+- Add queue pause/unpause endpoints for scripting via curl [#3445]
+- Change how super\_fetch names private queues to avoid hostname/queue clashes. [#3443]
 - Re-implement `Sidekiq::Queue#delete_job` to avoid O(n) runtime [#3408]
 - Batch page displays Pending JIDs if less than 10 [#3130]
 - Batch page has a Search button to find associated Retries [#3130]