sidekiq-ultimate 0.0.1.alpha

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA256:
3
+ metadata.gz: 21eb153f8f9af214ece524d28299bb23e2a1918d90a8b99e41a63031eeb4037b
4
+ data.tar.gz: 9b930c25ddedb5177da9ee3bad6756c37f156355e9c639c00ecf8f5803016aeb
5
+ SHA512:
6
+ metadata.gz: b38096795e63949611c87f0c42f35314ba7c0ec64b459fb91e6bb794e65aef2994c807a7d06873bedf92e78a343fedfcb95ece065478c5c1e0abf6a0b9d614c3
7
+ data.tar.gz: f36223cef6b03277f7090c0e5b1d3b33aeb70b7157c31205adbfc24d7a5ce4199499bd49ca215acfeac008a21d86973f321736326651e117594d0cd0fa031b9e
data/.gitignore ADDED
@@ -0,0 +1,10 @@
1
+ /Gemfile.lock
2
+
3
+ /.autoenv.zsh
4
+ /.bundle
5
+ /.yardoc
6
+ /_yardoc
7
+ /coverage
8
+ /doc
9
+ /pkg
10
+ /tmp
data/.rspec ADDED
@@ -0,0 +1 @@
1
+ --require spec_helper
data/.rubocop.yml ADDED
@@ -0,0 +1,46 @@
1
+ AllCops:
2
+ TargetRubyVersion: 2.3
3
+ DisplayCopNames: true
4
+
5
+
6
+ ## Layout ######################################################################
7
+
8
+ Layout/DotPosition:
9
+ EnforcedStyle: trailing
10
+
11
+ Layout/IndentArray:
12
+ EnforcedStyle: consistent
13
+
14
+ Layout/IndentHash:
15
+ EnforcedStyle: consistent
16
+
17
+
18
+ ## Metrics #####################################################################
19
+
20
+ Metrics/BlockLength:
21
+ Exclude:
22
+ - "spec/**/*"
23
+
24
+
25
+ ## Style #######################################################################
26
+
27
+ Style/BracesAroundHashParameters:
28
+ Enabled: false
29
+
30
+ Style/HashSyntax:
31
+ EnforcedStyle: hash_rockets
32
+
33
+ Style/RegexpLiteral:
34
+ EnforcedStyle: percent_r
35
+
36
+ Style/RescueStandardError:
37
+ EnforcedStyle: implicit
38
+
39
+ Style/SafeNavigation:
40
+ Enabled: false
41
+
42
+ Style/StringLiterals:
43
+ EnforcedStyle: double_quotes
44
+
45
+ Style/YodaCondition:
46
+ Enabled: false
data/.travis.yml ADDED
@@ -0,0 +1,28 @@
1
+ language: ruby
2
+ sudo: false
3
+
4
+ services:
5
+ - redis-server
6
+
7
+ cache: bundler
8
+
9
+ rvm:
10
+ - 2.3
11
+ - 2.4
12
+ - 2.5
13
+
14
+ matrix:
15
+ fast_finish: true
16
+ include:
17
+ - rvm: 2.4
18
+ env: TEST_SUITE="rubocop"
19
+
20
+ before_install:
21
+ - gem update --system
22
+ - gem --version
23
+ - gem install bundler --no-rdoc --no-ri
24
+ - bundle --version
25
+
26
+ install: bundle install --without development doc
27
+
28
+ script: bundle exec rake $TEST_SUITE
data/ARCHITECTURE.md ADDED
@@ -0,0 +1,115 @@
1
+ # Decisions and Assumptions (AKA Architecture)
2
+
3
+ This is a very proof of concept version of the architecture for reliable fetch
4
+ strategy. IMplementation will be based on reliable queue pattern described in
5
+ redis documentation. In short fetch will look like this:
6
+
7
+ ``` ruby
8
+ COOLDOWN = 2
9
+ IDENTITY = Object.new.tap { |o| o.extend Sidekiq::Util }.identity
10
+
11
+ def retrieve
12
+ Sidekiq.redis do
13
+ queue_names.each do |name|
14
+ pending = "queue:#{name}"
15
+ inproc = "inproc:#{IDENTITY}:#{name}"
16
+ job = redis.rpoplpush(pending, inproc)
17
+ return job if job
18
+ end
19
+ end
20
+
21
+ sleep COOLDOWN
22
+ end
23
+ ```
24
+
25
+ The above means that we will have inproc queue per queue and sidekiq server
26
+ process. Naturally we will need a process that will monitor "orphan" queues.
27
+ We will run such on each Sidekiq server in a `Concurrent::TimerTask` thread.
28
+ We can easily check which sidekiq processes are currently alive with:
29
+
30
+ ``` ruby
31
+ redis.exists(process_identity)
32
+ ```
33
+
34
+ Sidekiq keeps it's own set of all known processes and clears it out upon first
35
+ web view, so we need our own way to track all ever-running sidekiq process
36
+ identities. So we can subscribe to `startup` event like so:
37
+
38
+ ``` ruby
39
+ Sidekiq.on :startup do
40
+ Sidekiq.redis do |redis|
41
+ redis.sadd("ultimate:identities", IDENTITY)
42
+ end
43
+ end
44
+ ```
45
+
46
+ So now our casualties monitor can get all known identities and check which of
47
+ them are still alive:
48
+
49
+ ``` ruby
50
+ casualties = []
51
+
52
+ Sidekiq.redis do |redis|
53
+ identities = redis.smembers("ultimate:identities")
54
+ heartbeats = redis.pipelined do
55
+ identities.each { |key| redis.exists(key) }
56
+ end
57
+
58
+ heartbeats.each_with_index do |exists, idx|
59
+ casualties << identities[idx] unless exists
60
+ end
61
+ end
62
+ ```
63
+
64
+ I want to put lost but found jobs back to the queue. But I want them to appear
65
+ at the head of the pending queue so that thwey will be retried. So, I want some
66
+ sort of LPOPRPUSH command, which does not exist, so we will use LUA script for
67
+ that to guarantee atomic execution:
68
+
69
+ ``` lua
70
+ local src = KEYS[1]
71
+ local dst = KEYS[2]
72
+ local val = redis.call("LPOP", src)
73
+
74
+ if val then
75
+ redis.call("RPUSH", dst, val)
76
+ end
77
+
78
+ return val
79
+ ```
80
+
81
+ So now our casualties monitor can start resurrecting them:
82
+
83
+ ``` ruby
84
+ def resurrect
85
+ Sidekiq.redis do |redis|
86
+ casualties.each do |identity|
87
+ queue_names.each do |name|
88
+ src = "inproc:#{identity}:#{name}"
89
+ dst = "queue:#{name}"
90
+ loop while redis.eval(LPOPRPUSH, :keys => [src, dst])
91
+ redis.del(src)
92
+ end
93
+ end
94
+ end
95
+ end
96
+ ```
97
+
98
+ All good, but here's the problem: dead process could have differnet set of
99
+ queues it was serving. So, instead of relying on `Sidekiq.options` we can
100
+ save queues in the hash in redis, so our startup event will look like this:
101
+
102
+ ``` ruby
103
+ Sidekiq.on :startup do
104
+ Sidekiq.redis do |redis|
105
+ queues = JSON.dump(Sidekiq.options[:queues].uniq)
106
+ redis.hmset("ultimate:identities", IDENTITY, queues)
107
+ end
108
+ end
109
+ ```
110
+
111
+ Now, to get casualties we will use `HKEYS` instead of `SMEMBERS`. But they have
112
+ same complexity.
113
+
114
+ In addition to the above we will be using redis-based locks to guarantee only
115
+ one sidekiq process is handling resurrection at a time.
data/Gemfile ADDED
@@ -0,0 +1,28 @@
1
+ # frozen_string_literal: true
2
+
3
+ source "https://rubygems.org"
4
+ ruby RUBY_VERSION
5
+
6
+ gem "rake"
7
+ gem "rspec"
8
+ gem "rubocop", "~> 0.52.0", :require => false
9
+
10
+ group :development do
11
+ gem "guard", :require => false
12
+ gem "guard-rspec", :require => false
13
+ gem "guard-rubocop", :require => false
14
+ gem "pry", :require => false
15
+ end
16
+
17
+ group :test do
18
+ gem "codecov", :require => false
19
+ gem "simplecov", :require => false
20
+ end
21
+
22
+ group :doc do
23
+ gem "redcarpet"
24
+ gem "yard"
25
+ end
26
+
27
+ # Specify your gem's dependencies in redis-prescription.gemspec
28
+ gemspec
data/Guardfile ADDED
@@ -0,0 +1,21 @@
1
+ # frozen_string_literal: true
2
+
3
+ guard :rspec, :cmd => "bundle exec rspec" do
4
+ require "guard/rspec/dsl"
5
+ dsl = Guard::RSpec::Dsl.new(self)
6
+
7
+ # RSpec files
8
+ rspec = dsl.rspec
9
+ watch(rspec.spec_helper) { rspec.spec_dir }
10
+ watch(rspec.spec_support) { rspec.spec_dir }
11
+ watch(rspec.spec_files)
12
+
13
+ # Ruby files
14
+ ruby = dsl.ruby
15
+ dsl.watch_spec_files_for(ruby.lib_files)
16
+ end
17
+
18
+ guard :rubocop do
19
+ watch(%r{.+\.rb$})
20
+ watch(%r{(?:.+/)?\.rubocop(?:_todo)?\.yml$}) { |m| File.dirname(m[0]) }
21
+ end
data/LICENSE.txt ADDED
@@ -0,0 +1,21 @@
1
+ The MIT License (MIT)
2
+
3
+ Copyright (c) 2018 SensorTower Inc.
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining a copy
6
+ of this software and associated documentation files (the "Software"), to deal
7
+ in the Software without restriction, including without limitation the rights
8
+ to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
9
+ copies of the Software, and to permit persons to whom the Software is
10
+ furnished to do so, subject to the following conditions:
11
+
12
+ The above copyright notice and this permission notice shall be included in
13
+ all copies or substantial portions of the Software.
14
+
15
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
16
+ IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
17
+ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
18
+ AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
19
+ LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
20
+ OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
21
+ THE SOFTWARE.
data/README.md ADDED
@@ -0,0 +1,116 @@
1
+ # Sidekiq::Ultimate
2
+
3
+ Sidekiq ultimate experience.
4
+
5
+ ---
6
+
7
+ **WARNING**
8
+
9
+ This ia an alpha/preview software. Lots of changes will be made and eventually
10
+ it will overtake [sidekiq-throttled][] and will become truly ultimate sidekiq
11
+ extension one will need. :D
12
+
13
+ ---
14
+
15
+
16
+ ## Installation
17
+
18
+ Add this line to your application's Gemfile:
19
+
20
+ ```ruby
21
+ gem "sidekiq-ultimate", ">= 0.0.1.alpha"
22
+ ```
23
+
24
+ And then execute:
25
+
26
+ $ bundle
27
+
28
+ Or install it yourself as:
29
+
30
+ $ gem install sidekiq-ultimate
31
+
32
+
33
+ ## Usage
34
+
35
+ Add somewhere in your app's bootstrap (e.g. `config/initializers/sidekiq.rb` if
36
+ you are using Rails):
37
+
38
+ ``` ruby
39
+ require "sidekiq/ultimate"
40
+ Sidekiq::Ultimate.setup!
41
+ ```
42
+
43
+ ---
44
+
45
+ **NOTICE**
46
+
47
+ Throttling is brought by [sidekiq-throttled][] and it's automatically set up
48
+ by the command above - don't run `Sidekiq::Throttled.setup!` yourself.
49
+
50
+ Thus look up it's README for throttling configuration details.
51
+
52
+ ---
53
+
54
+
55
+ ## Supported Ruby Versions
56
+
57
+ This library aims to support and is [tested against][travis-ci] the following
58
+ Ruby and Redis client versions:
59
+
60
+ * Ruby
61
+ * 2.3.x
62
+ * 2.4.x
63
+ * 2.5.x
64
+
65
+ * [redis-rb](https://github.com/redis/redis-rb)
66
+ * 4.x
67
+
68
+ * [redis-namespace](https://github.com/resque/redis-namespace)
69
+ * 1.6
70
+
71
+
72
+ If something doesn't work on one of these versions, it's a bug.
73
+
74
+ This library may inadvertently work (or seem to work) on other Ruby versions,
75
+ however support will only be provided for the versions listed above.
76
+
77
+ If you would like this library to support another Ruby version or
78
+ implementation, you may volunteer to be a maintainer. Being a maintainer
79
+ entails making sure all tests run and pass on that implementation. When
80
+ something breaks on your implementation, you will be responsible for providing
81
+ patches in a timely fashion. If critical issues for a particular implementation
82
+ exist at the time of a major release, support for that Ruby version may be
83
+ dropped.
84
+
85
+
86
+ ## Development
87
+
88
+ After checking out the repo, run `bundle install` to install dependencies.
89
+ Then, run `bundle exec rake spec` to run the tests with ruby-rb client.
90
+
91
+ To install this gem onto your local machine, run `bundle exec rake install`.
92
+ To release a new version, update the version number in `version.rb`, and then
93
+ run `bundle exec rake release`, which will create a git tag for the version,
94
+ push git commits and tags, and push the `.gem` file to [rubygems.org][].
95
+
96
+
97
+ ## Contributing
98
+
99
+ * Fork sidekiq-ultimate on GitHub
100
+ * Make your changes
101
+ * Ensure all tests pass (`bundle exec rake`)
102
+ * Send a pull request
103
+ * If we like them we'll merge them
104
+ * If we've accepted a patch, feel free to ask for commit access!
105
+
106
+
107
+ ## Copyright
108
+
109
+ Copyright (c) 2018 SensorTower Inc.<br>
110
+ See [LICENSE.md][] for further details.
111
+
112
+
113
+ [travis.ci]: http://travis-ci.org/sensortower/sidekiq-ultimate
114
+ [rubygems.org]: https://rubygems.org
115
+ [LICENSE.md]: https://github.com/sensortower/sidekiq-ultimate/blob/master/LICENSE.txt
116
+ [sidekiq-throttled]: http://travis-ci.org/sensortower/sidekiq-throttled
data/Rakefile ADDED
@@ -0,0 +1,15 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "bundler/gem_tasks"
4
+
5
+ require "rspec/core/rake_task"
6
+ RSpec::Core::RakeTask.new
7
+
8
+ require "rubocop/rake_task"
9
+ RuboCop::RakeTask.new
10
+
11
+ if ENV["CI"]
12
+ task :default => :spec
13
+ else
14
+ task :default => %i[rubocop spec]
15
+ end
@@ -0,0 +1,29 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "sidekiq/throttled"
4
+
5
+ require "sidekiq/ultimate/version"
6
+
7
+ module Sidekiq
8
+ # Sidekiq ultimate experience.
9
+ module Ultimate
10
+ class << self
11
+ # Sets up reliable throttled fetch and friends.
12
+ # @return [void]
13
+ def setup!
14
+ Sidekiq::Throttled::Communicator.instance.setup!
15
+ Sidekiq::Throttled::QueuesPauser.instance.setup!
16
+
17
+ Sidekiq.configure_server do |config|
18
+ require "sidekiq/ultimate/fetch"
19
+ Sidekiq::Ultimate::Fetch.setup!
20
+
21
+ require "sidekiq/throttled/middleware"
22
+ config.server_middleware do |chain|
23
+ chain.add Sidekiq::Throttled::Middleware
24
+ end
25
+ end
26
+ end
27
+ end
28
+ end
29
+ end
@@ -0,0 +1,74 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "monitor"
4
+
5
+ require "concurrent/utility/monotonic_time"
6
+
7
+ module Sidekiq
8
+ module Ultimate
9
+ # List that tracks when elements were added and enumerates over those not
10
+ # older than `ttl` seconds ago.
11
+ #
12
+ # ## Implementation
13
+ #
14
+ # Internally list holds an array of arrays. Thus ecah element is a tuple of
15
+ # monotonic timestamp (when element was added) and element itself:
16
+ #
17
+ # [
18
+ # [ 123456.7890, "default" ],
19
+ # [ 123456.7891, "urgent" ],
20
+ # [ 123457.9621, "urgent" ],
21
+ # ...
22
+ # ]
23
+ #
24
+ # It does not deduplicates elements. Eviction happens only upon elements
25
+ # retrieval (see {#each}).
26
+ #
27
+ # @see http://ruby-concurrency.github.io/concurrent-ruby/Concurrent.html#monotonic_time-class_method
28
+ # @see https://ruby-doc.org/core/Process.html#method-c-clock_gettime
29
+ # @see https://linux.die.net/man/3/clock_gettime
30
+ #
31
+ # @private
32
+ class ExpirableList
33
+ include Enumerable
34
+
35
+ # Create a new ExpirableList instance.
36
+ #
37
+ # @param ttl [Float] elements time-to-live in seconds
38
+ def initialize(ttl)
39
+ @ttl = ttl.to_f
40
+ @arr = []
41
+ @mon = Monitor.new
42
+ end
43
+
44
+ # Pushes given element into the list.
45
+ #
46
+ # @params element [Object]
47
+ # @return [ExpirableList] self
48
+ def <<(element)
49
+ @mon.synchronize { @arr << [Concurrent.monotonic_time, element] }
50
+ self
51
+ end
52
+
53
+ # Evicts expired elements and calls the given block once for each element
54
+ # left, passing that element as a parameter.
55
+ #
56
+ # @yield [element]
57
+ # @return [Enumerator] if no block given
58
+ # @return [ExpirableList] self if block given
59
+ def each
60
+ return to_enum __method__ unless block_given?
61
+
62
+ # Evict expired elements
63
+ @mon.synchronize do
64
+ horizon = Concurrent.monotonic_time - @ttl
65
+ @arr.shift while @arr[0] && @arr[0][0] < horizon
66
+ end
67
+
68
+ @arr.dup.each { |element| yield element[1] }
69
+
70
+ self
71
+ end
72
+ end
73
+ end
74
+ end
@@ -0,0 +1,77 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "sidekiq/throttled"
4
+
5
+ require "sidekiq/ultimate/expirable_list"
6
+ require "sidekiq/ultimate/queue_name"
7
+ require "sidekiq/ultimate/resurrector"
8
+ require "sidekiq/ultimate/unit_of_work"
9
+
10
+ module Sidekiq
11
+ module Ultimate
12
+ # Throttled reliable fetcher implementing reliable queue pattern.
13
+ class Fetch
14
+ # Timeout to sleep between fetch retries in case of no job received,
15
+ # as well as timeout to wait for redis to give us something to work.
16
+ TIMEOUT = 2
17
+
18
+ def initialize(options)
19
+ @exhausted = ExpirableList.new(10 * TIMEOUT)
20
+
21
+ @strict = options[:strict] ? true : false
22
+ @queues = options[:queues].map { |name| QueueName.new(name) }
23
+
24
+ @queues.uniq! if @strict
25
+ end
26
+
27
+ def retrieve_work
28
+ work = retrieve
29
+
30
+ return unless work
31
+ return work unless work.throttled?
32
+
33
+ work.requeue_throttled
34
+ @exhausted << work.queue
35
+ end
36
+
37
+ def self.bulk_requeue(*)
38
+ # do nothing
39
+ end
40
+
41
+ def self.setup!
42
+ Sidekiq.options[:fetch] = self
43
+ Resurrector.setup!
44
+ end
45
+
46
+ private
47
+
48
+ def retrieve
49
+ Sidekiq.redis do |redis|
50
+ queues.each do |queue|
51
+ job = redis.rpoplpush(queue.pending, queue.inproc)
52
+ return UnitOfWork.new(queue, job) if job
53
+
54
+ @exhausted << queue
55
+ end
56
+ end
57
+
58
+ sleep TIMEOUT
59
+ nil
60
+ end
61
+
62
+ def queues
63
+ (@strict ? @queues : @queues.shuffle.uniq) - exhausted - paused_queues
64
+ end
65
+
66
+ def exhausted
67
+ @exhausted.to_a
68
+ end
69
+
70
+ def paused_queues
71
+ Sidekiq::Throttled::QueuesPauser.instance.
72
+ instance_variable_get(:@paused_queues).
73
+ map { |q| QueueName[q] }
74
+ end
75
+ end
76
+ end
77
+ end
@@ -0,0 +1,9 @@
1
+ local src = KEYS[1]
2
+ local dst = KEYS[2]
3
+ local val = redis.call("LPOP", src)
4
+
5
+ if val then
6
+ redis.call("RPUSH", dst, val)
7
+ end
8
+
9
+ return val
@@ -0,0 +1,122 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "sidekiq/util"
4
+
5
+ module Sidekiq
6
+ module Ultimate
7
+ # Helper object that extend queue name string with redis keys.
8
+ #
9
+ # @private
10
+ class QueueName
11
+ # Regexp used to normalize (possibly) expanded queue name, e.g. the one
12
+ # that is returned upon redis BRPOP
13
+ QUEUE_PREFIX_RE = %r{.*queue:}
14
+ private_constant :QUEUE_PREFIX_RE
15
+
16
+ # Internal helper context.
17
+ Helper = Module.new { extend Sidekiq::Util }
18
+ private_constant :Helper
19
+
20
+ # Original stringified queue name.
21
+ #
22
+ # @example
23
+ #
24
+ # queue_name.normalized # => "foobar"
25
+ #
26
+ # @return [String]
27
+ attr_reader :normalized
28
+ alias to_s normalized
29
+
30
+ # Create a new QueueName instance.
31
+ #
32
+ # @param normalized [#to_s] Normalized (without any namespaces or `queue:`
33
+ # prefixes) queue name.
34
+ # @param identity [#to_s] Sidekiq process identity.
35
+ def initialize(normalized, identity: self.class.process_identity)
36
+ @normalized = -normalized.to_s
37
+ @identity = -identity.to_s
38
+ end
39
+
40
+ # @!attribute [r] hash
41
+ #
42
+ # A hash based on the normalized queue name.
43
+ #
44
+ # @see https://ruby-doc.org/core/Object.html#method-i-hash
45
+ # @return [Integer]
46
+ def hash
47
+ @hash ||= @normalized.hash
48
+ end
49
+
50
+ # @!attribute [r] pending
51
+ #
52
+ # Redis key of queue list.
53
+ #
54
+ # @example
55
+ #
56
+ # queue_name.pending # => "queue:foobar"
57
+ #
58
+ # @return [String]
59
+ def pending
60
+ @pending ||= -"queue:#{@normalized}"
61
+ end
62
+
63
+ # @!attribute [r] inproc
64
+ #
65
+ # Redis key of in-process jobs list.
66
+ #
67
+ # @example
68
+ #
69
+ # queue_name.inproc # => "inproc:argentum:12345:a9b8c7d6e5f4:foobar"
70
+ #
71
+ # @return [String]
72
+ def inproc
73
+ @inproc ||= -"inproc:#{@identity}:#{@normalized}"
74
+ end
75
+
76
+ # Check if `other` is the {QueueName} representing same queue.
77
+ #
78
+ # @example
79
+ #
80
+ # QueueName.new("abc").eql? QueueName.new("abc") # => true
81
+ # QueueName.new("abc").eql? QueueName.new("xyz") # => false
82
+ #
83
+ # @param other [Object]
84
+ # @return [Boolean]
85
+ def ==(other)
86
+ other.is_a?(self.class) && @normalized == other.normalized
87
+ end
88
+ alias eql? ==
89
+
90
+ # Returns human-friendly printable QueueName representation.
91
+ #
92
+ # @example
93
+ #
94
+ # QueueName.new("foobar").inspect # => QueueName["foobar"]
95
+ # QueueName["queue:foobar"].inspect # => QueueName["foobar"]
96
+ #
97
+ # @return [String]
98
+ def inspect
99
+ "#{self.class}[#{@normalized.inspect}]"
100
+ end
101
+
102
+ # Returns new QueueName instance with normalized queue name. Use this
103
+ # when you're not sure if queue name is normalized or not (e.g. with
104
+ # queue name received as part of BRPOP command).
105
+ #
106
+ # @example
107
+ #
108
+ # QueueName["ns:queue:foobar"].normalized # => "foobar"
109
+ #
110
+ # @param name [#to_s] Queue name
111
+ # @param kwargs (see #initialize for details on possible options)
112
+ # @return [QueueName]
113
+ def self.[](name, **kwargs)
114
+ new(name.to_s.sub(QUEUE_PREFIX_RE, ""), **kwargs)
115
+ end
116
+
117
+ def self.process_identity
118
+ Helper.identity
119
+ end
120
+ end
121
+ end
122
+ end
@@ -0,0 +1,99 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "redis/lockers"
4
+ require "redis/prescription"
5
+
6
+ module Sidekiq
7
+ module Ultimate
8
+ # Lost jobs resurrector.
9
+ module Resurrector
10
+ LPOPRPUSH = Redis::Prescription.read("#{__dir__}/lpoprpush.lua")
11
+ private_constant :LPOPRPUSH
12
+
13
+ MAIN_KEY = "ultimate:resurrector"
14
+ private_constant :MAIN_KEY
15
+
16
+ LOCK_KEY = "#{MAIN_KEY}:lock"
17
+ private_constant :LOCK_KEY
18
+
19
+ class << self
20
+ def setup!
21
+ ctulhu = Concurrent::TimerTask.new(:execution_interval => 5) do
22
+ resurrect!
23
+ end
24
+
25
+ Sidekiq.on(:startup) do
26
+ register_process!
27
+ ctulhu.execute
28
+ end
29
+
30
+ Sidekiq.on(:shutdown) { ctulhu.shutdown }
31
+ end
32
+
33
+ def resurrect!
34
+ lock do
35
+ casualties.each do |identity|
36
+ queues(identity).each { |queue| resurrect(queue) }
37
+ cleanup(identity)
38
+ end
39
+ end
40
+ end
41
+
42
+ private
43
+
44
+ def register_process!
45
+ Sidekiq.redis do |redis|
46
+ queues = JSON.dump(Sidekiq.options[:queues].uniq)
47
+ identity = Object.new.tap { |o| o.extend Sidekiq::Util }.identity
48
+
49
+ redis.hset(MAIN_KEY, identity, queues)
50
+ end
51
+ end
52
+
53
+ def lock(&block)
54
+ Sidekiq.redis do |redis|
55
+ Redis::Lockers.acquire(redis, LOCK_KEY, :ttl => 30_000, &block)
56
+ end
57
+ end
58
+
59
+ def casualties
60
+ Sidekiq.redis do |redis|
61
+ casualties = []
62
+ identities = redis.hkeys(MAIN_KEY)
63
+
64
+ redis.pipelined { identities.each { |k| redis.exists k } }.
65
+ each_with_index { |v, i| casualties << identities[i] unless v }
66
+
67
+ casualties
68
+ end
69
+ end
70
+
71
+ def queues(identity)
72
+ Sidekiq.redis do |redis|
73
+ queues = redis.hget(MAIN_KEY, identity)
74
+
75
+ return [] unless queues
76
+
77
+ JSON.parse(queues).map do |q|
78
+ QueueName.new(q, :identity => identity)
79
+ end
80
+ end
81
+ end
82
+
83
+ def resurrect(queue)
84
+ Sidekiq.redis do |redis|
85
+ kwargs = { :keys => [queue.inproc, queue.pending] }
86
+ count = 0
87
+
88
+ count += 1 while LPOPRPUSH.eval(redis, **kwargs)
89
+ redis.del(queue.inproc)
90
+ end
91
+ end
92
+
93
+ def cleanup(identity)
94
+ Sidekiq.redis { |redis| redis.hdel(MAIN_KEY, identity) }
95
+ end
96
+ end
97
+ end
98
+ end
99
+ end
@@ -0,0 +1,80 @@
1
+ # frozen_string_literal: true
2
+
3
+ require "sidekiq/throttled"
4
+
5
+ module Sidekiq
6
+ module Ultimate
7
+ # Job message envelope.
8
+ #
9
+ # @private
10
+ class UnitOfWork
11
+ # JSON payload
12
+ #
13
+ # @return [String]
14
+ attr_reader :job
15
+
16
+ # @param [QueueName] queue where job was pulled from
17
+ # @param [String] job JSON payload
18
+ def initialize(queue, job)
19
+ @queue = queue
20
+ @job = job
21
+ end
22
+
23
+ # Pending jobs queue key name.
24
+ #
25
+ # @return [String]
26
+ def queue
27
+ @queue.pending
28
+ end
29
+
30
+ # Normalized `queue` name.
31
+ #
32
+ # @see QueueName#normalized
33
+ # @return [String]
34
+ def queue_name
35
+ @queue.normalized
36
+ end
37
+
38
+ # Remove job from the inproc list.
39
+ #
40
+ # Sidekiq calls this when it thinks jobs was performed with no mistakes.
41
+ #
42
+ # @return [void]
43
+ def acknowledge
44
+ Sidekiq.redis { |redis| redis.lrem(@queue.inproc, -1, @job) }
45
+ end
46
+
47
+ # We gonna resurrect jobs that were inside inproc queue upon process
48
+ # start, so no point in doing anything here.
49
+ #
50
+ # @return [void]
51
+ def requeue
52
+ # do nothing
53
+ end
54
+
55
+ # Pushes job back to the head of the queue, so that job won't be tried
56
+ # immediately after it was requeued (in most cases).
57
+ #
58
+ # @note This is triggered when job is throttled. So it is same operation
59
+ # Sidekiq performs upon `Sidekiq::Worker.perform_async` call.
60
+ #
61
+ # @return [void]
62
+ def requeue_throttled
63
+ Sidekiq.redis do |redis|
64
+ redis.pipeline do
65
+ redis.lpush(@queue.pending, @job)
66
+ acknowledge
67
+ end
68
+ end
69
+ end
70
+
71
+ # Tells whenever job should be pushed back to queue (throttled) or not.
72
+ #
73
+ # @see Sidekiq::Throttled.throttled?
74
+ # @return [Boolean]
75
+ def throttled?
76
+ Sidekiq::Throttled.throttled?(@job)
77
+ end
78
+ end
79
+ end
80
+ end
@@ -0,0 +1,8 @@
1
+ # frozen_string_literal: true
2
+
3
+ module Sidekiq
4
+ module Ultimate
5
+ # Gem version.
6
+ VERSION = "0.0.1.alpha"
7
+ end
8
+ end
@@ -0,0 +1,39 @@
1
+ # frozen_string_literal: true
2
+
3
+ lib = File.expand_path("../lib", __FILE__)
4
+ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
5
+
6
+ require "sidekiq/ultimate/version"
7
+
8
+ Gem::Specification.new do |spec|
9
+ spec.name = "sidekiq-ultimate"
10
+ spec.version = Sidekiq::Ultimate::VERSION
11
+ spec.authors = ["Alexey Zapparov"]
12
+ spec.email = ["ixti@member.fsf.org"]
13
+
14
+ spec.summary = "Sidekiq ultimate experience."
15
+ spec.description = "Sidekiq ultimate experience."
16
+
17
+ spec.homepage = "https://github.com/sensortower/sidekiq-ultimate"
18
+ spec.license = "MIT"
19
+
20
+ spec.files = `git ls-files -z`.split("\x0").reject do |f|
21
+ f.match(%r{^(test|spec|features)/})
22
+ end
23
+ spec.bindir = "exe"
24
+ spec.executables = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
25
+ spec.require_paths = ["lib"]
26
+
27
+ spec.add_runtime_dependency "concurrent-ruby", "~> 1.0"
28
+ spec.add_runtime_dependency "redis-lockers", "~> 1.1"
29
+ spec.add_runtime_dependency "redis-prescription", "~> 1.0"
30
+ spec.add_runtime_dependency "sidekiq", "~> 5.0"
31
+
32
+ # temporary couple this with sidekiq-throttled until it will be merged into
33
+ # this gem instead.
34
+ spec.add_runtime_dependency "sidekiq-throttled", "~> 0.8.2"
35
+
36
+ spec.add_development_dependency "bundler", "~> 1.16"
37
+
38
+ spec.required_ruby_version = "~> 2.3"
39
+ end
metadata ADDED
@@ -0,0 +1,147 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: sidekiq-ultimate
3
+ version: !ruby/object:Gem::Version
4
+ version: 0.0.1.alpha
5
+ platform: ruby
6
+ authors:
7
+ - Alexey Zapparov
8
+ autorequire:
9
+ bindir: exe
10
+ cert_chain: []
11
+ date: 2018-02-22 00:00:00.000000000 Z
12
+ dependencies:
13
+ - !ruby/object:Gem::Dependency
14
+ name: concurrent-ruby
15
+ requirement: !ruby/object:Gem::Requirement
16
+ requirements:
17
+ - - "~>"
18
+ - !ruby/object:Gem::Version
19
+ version: '1.0'
20
+ type: :runtime
21
+ prerelease: false
22
+ version_requirements: !ruby/object:Gem::Requirement
23
+ requirements:
24
+ - - "~>"
25
+ - !ruby/object:Gem::Version
26
+ version: '1.0'
27
+ - !ruby/object:Gem::Dependency
28
+ name: redis-lockers
29
+ requirement: !ruby/object:Gem::Requirement
30
+ requirements:
31
+ - - "~>"
32
+ - !ruby/object:Gem::Version
33
+ version: '1.1'
34
+ type: :runtime
35
+ prerelease: false
36
+ version_requirements: !ruby/object:Gem::Requirement
37
+ requirements:
38
+ - - "~>"
39
+ - !ruby/object:Gem::Version
40
+ version: '1.1'
41
+ - !ruby/object:Gem::Dependency
42
+ name: redis-prescription
43
+ requirement: !ruby/object:Gem::Requirement
44
+ requirements:
45
+ - - "~>"
46
+ - !ruby/object:Gem::Version
47
+ version: '1.0'
48
+ type: :runtime
49
+ prerelease: false
50
+ version_requirements: !ruby/object:Gem::Requirement
51
+ requirements:
52
+ - - "~>"
53
+ - !ruby/object:Gem::Version
54
+ version: '1.0'
55
+ - !ruby/object:Gem::Dependency
56
+ name: sidekiq
57
+ requirement: !ruby/object:Gem::Requirement
58
+ requirements:
59
+ - - "~>"
60
+ - !ruby/object:Gem::Version
61
+ version: '5.0'
62
+ type: :runtime
63
+ prerelease: false
64
+ version_requirements: !ruby/object:Gem::Requirement
65
+ requirements:
66
+ - - "~>"
67
+ - !ruby/object:Gem::Version
68
+ version: '5.0'
69
+ - !ruby/object:Gem::Dependency
70
+ name: sidekiq-throttled
71
+ requirement: !ruby/object:Gem::Requirement
72
+ requirements:
73
+ - - "~>"
74
+ - !ruby/object:Gem::Version
75
+ version: 0.8.2
76
+ type: :runtime
77
+ prerelease: false
78
+ version_requirements: !ruby/object:Gem::Requirement
79
+ requirements:
80
+ - - "~>"
81
+ - !ruby/object:Gem::Version
82
+ version: 0.8.2
83
+ - !ruby/object:Gem::Dependency
84
+ name: bundler
85
+ requirement: !ruby/object:Gem::Requirement
86
+ requirements:
87
+ - - "~>"
88
+ - !ruby/object:Gem::Version
89
+ version: '1.16'
90
+ type: :development
91
+ prerelease: false
92
+ version_requirements: !ruby/object:Gem::Requirement
93
+ requirements:
94
+ - - "~>"
95
+ - !ruby/object:Gem::Version
96
+ version: '1.16'
97
+ description: Sidekiq ultimate experience.
98
+ email:
99
+ - ixti@member.fsf.org
100
+ executables: []
101
+ extensions: []
102
+ extra_rdoc_files: []
103
+ files:
104
+ - ".gitignore"
105
+ - ".rspec"
106
+ - ".rubocop.yml"
107
+ - ".travis.yml"
108
+ - ARCHITECTURE.md
109
+ - Gemfile
110
+ - Guardfile
111
+ - LICENSE.txt
112
+ - README.md
113
+ - Rakefile
114
+ - lib/sidekiq/ultimate.rb
115
+ - lib/sidekiq/ultimate/expirable_list.rb
116
+ - lib/sidekiq/ultimate/fetch.rb
117
+ - lib/sidekiq/ultimate/lpoprpush.lua
118
+ - lib/sidekiq/ultimate/queue_name.rb
119
+ - lib/sidekiq/ultimate/resurrector.rb
120
+ - lib/sidekiq/ultimate/unit_of_work.rb
121
+ - lib/sidekiq/ultimate/version.rb
122
+ - sidekiq-ultimate.gemspec
123
+ homepage: https://github.com/sensortower/sidekiq-ultimate
124
+ licenses:
125
+ - MIT
126
+ metadata: {}
127
+ post_install_message:
128
+ rdoc_options: []
129
+ require_paths:
130
+ - lib
131
+ required_ruby_version: !ruby/object:Gem::Requirement
132
+ requirements:
133
+ - - "~>"
134
+ - !ruby/object:Gem::Version
135
+ version: '2.3'
136
+ required_rubygems_version: !ruby/object:Gem::Requirement
137
+ requirements:
138
+ - - ">"
139
+ - !ruby/object:Gem::Version
140
+ version: 1.3.1
141
+ requirements: []
142
+ rubyforge_project:
143
+ rubygems_version: 2.7.3
144
+ signing_key:
145
+ specification_version: 4
146
+ summary: Sidekiq ultimate experience.
147
+ test_files: []