sidekiq-fast-enq 1.0.2 → 1.0.3
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +29 -0
- data/MIT_LICENSE.txt +1 -1
- data/README.md +44 -3
- data/VERSION +1 -1
- data/lib/sidekiq-fast-enq.rb +17 -16
- data/sidekiq_fast_enq.gemspec +27 -20
- metadata +15 -92
- data/.gitignore +0 -6
- data/.travis.yml +0 -28
- data/Appraisals +0 -9
- data/Gemfile +0 -2
- data/Gemfile.lock +0 -54
- data/HISTORY.md +0 -11
- data/Rakefile +0 -57
- data/gemfiles/.bundle/config +0 -2
- data/gemfiles/sidekiq_3.gemfile +0 -7
- data/gemfiles/sidekiq_3.gemfile.lock +0 -74
- data/gemfiles/sidekiq_4.gemfile +0 -7
- data/gemfiles/sidekiq_4.gemfile.lock +0 -56
- data/gemfiles/sidekiq_5.gemfile +0 -7
- data/gemfiles/sidekiq_5.gemfile.lock +0 -55
- data/gemfiles/sidekiq_6.gemfile +0 -7
- data/gemfiles/sidekiq_6.gemfile.lock +0 -55
- data/spec/sidekiq-fast-enq_spec.rb +0 -53
- data/spec/spec_helper.rb +0 -34
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 4c2ef4bb287f73b4dc43e32b604d16e98dc284822f399952e13cee7df1869836
|
4
|
+
data.tar.gz: b9c4efb591b505b94733ba6722b50c99ddd0a6151da9b54ab5d16534364cda15
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: d1ed8261dd525b1de1820f676b9c545a0cc6486cef1a3ed8e078ea9eec7ecf2e67f94aaba11d347b5d59f2a8d8956612d564560182f7d3652fe9ec00a167dcc7
|
7
|
+
data.tar.gz: dc9c18fc20a0a0ff75bacf4428c94f4eb5c8665824f1060755508a1273d05bb7889c73499cff4bd5a60c24b01360c5d5a7a52754a991a40996af39d9bfb29cd0
|
data/CHANGELOG.md
ADDED
@@ -0,0 +1,29 @@
|
|
1
|
+
# Changelog
|
2
|
+
All notable changes to this project will be documented in this file.
|
3
|
+
|
4
|
+
The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
|
5
|
+
and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
|
6
|
+
|
7
|
+
## 1.0.3
|
8
|
+
|
9
|
+
### Added
|
10
|
+
|
11
|
+
- Compatibility with Sidekiq 7
|
12
|
+
|
13
|
+
## 1.0.2
|
14
|
+
|
15
|
+
### Added
|
16
|
+
|
17
|
+
- Compatibility with Sidekiq 6
|
18
|
+
|
19
|
+
## 1.0.2
|
20
|
+
|
21
|
+
### Changed
|
22
|
+
|
23
|
+
- Update gemspec to specify compatibility with sidekiq 4
|
24
|
+
|
25
|
+
# 1.0.0
|
26
|
+
|
27
|
+
### Added
|
28
|
+
|
29
|
+
- Initial release
|
data/MIT_LICENSE.txt
CHANGED
data/README.md
CHANGED
@@ -1,5 +1,8 @@
|
|
1
|
-
|
2
|
-
|
1
|
+
Sidekiq Fast Enqueuing
|
2
|
+
|
3
|
+
[![Continuous Integration](https://github.com/bdurand/sidekiq-fast-enq/actions/workflows/continuous_integration.yml/badge.svg)](https://github.com/bdurand/sidekiq-fast-enq/actions/workflows/continuous_integration.yml)
|
4
|
+
[![Regression Test](https://github.com/bdurand/sidekiq-fast-enq/actions/workflows/regression_test.yml/badge.svg)](https://github.com/bdurand/sidekiq-fast-enq/actions/workflows/regression_test.yml)
|
5
|
+
[![Ruby Style Guide](https://img.shields.io/badge/code_style-standard-brightgreen.svg)](https://github.com/testdouble/standard)
|
3
6
|
|
4
7
|
This gem provides a much more efficien implementation for checking the Sidekiq scheduled and retry queues. This can provide a significant performance boost for large sidekiq implementations that utilize many processes. It can also reduce load on the redis server.
|
5
8
|
|
@@ -17,9 +20,47 @@ This plugin does not alter any sidekiq internal code or data structures.
|
|
17
20
|
|
18
21
|
In your sidekiq configuration you need to set the `:scheduled_enq` option to `SidekiqFastEnq` (only available in sidekiq 3.4.0 and later). You might also want to hard code a value for the `:poll_interval_average` option as well. If this option is not set the polling interval for checking the scheduled queues is based on the number of processes in an effort to reduce the effects of the race condition. It is not needed with this code and scheduled jobs will be enqueued closer to their scheduled time without it.
|
19
22
|
|
23
|
+
```ruby
|
24
|
+
Sidekiq.default_configuration[:scheduled_enq] = SidekiqFastEnq
|
25
|
+
Sidekiq.default_configuration[:poll_interval_average] = 30
|
26
|
+
```
|
27
|
+
|
28
|
+
For Sidekiq versions prior to version 7:
|
29
|
+
|
20
30
|
```ruby
|
21
31
|
Sidekiq.options[:scheduled_enq] = SidekiqFastEnq
|
22
32
|
Sidekiq.options[:poll_interval_average] = 30
|
23
33
|
```
|
24
34
|
|
25
|
-
|
35
|
+
|
36
|
+
### Redis requirement
|
37
|
+
|
38
|
+
Redis server 2.6 or greater is required for this code.
|
39
|
+
|
40
|
+
## Installation
|
41
|
+
|
42
|
+
Add this line to your application's Gemfile:
|
43
|
+
|
44
|
+
```ruby
|
45
|
+
gem 'sidekiq-fast-enq'
|
46
|
+
```
|
47
|
+
|
48
|
+
And then execute:
|
49
|
+
```bash
|
50
|
+
$ bundle
|
51
|
+
```
|
52
|
+
|
53
|
+
Or install it yourself as:
|
54
|
+
```bash
|
55
|
+
$ gem install sidekiq-fast-enq
|
56
|
+
```
|
57
|
+
|
58
|
+
## Contributing
|
59
|
+
|
60
|
+
Fork the repository and open a pull request on GitHub.
|
61
|
+
|
62
|
+
Please use the [standardrb](https://github.com/testdouble/standard) syntax and lint your code with `standardrb --fix` before submitting.
|
63
|
+
|
64
|
+
## License
|
65
|
+
|
66
|
+
The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
|
data/VERSION
CHANGED
@@ -1 +1 @@
|
|
1
|
-
1.0.
|
1
|
+
1.0.3
|
data/lib/sidekiq-fast-enq.rb
CHANGED
@@ -1,4 +1,6 @@
|
|
1
|
-
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
require "sidekiq"
|
2
4
|
|
3
5
|
# Implementation of the Sidekiq::Scheduled::Enq class that uses a server side Lua script
|
4
6
|
# to atomically get the next scheduled job to run and then pops it from the list. This
|
@@ -8,7 +10,8 @@ class SidekiqFastEnq
|
|
8
10
|
DEFAULT_BATCH_SIZE = 1000
|
9
11
|
|
10
12
|
def initialize(batch_size = nil)
|
11
|
-
|
13
|
+
options = (Sidekiq.respond_to?(:default_configuration) ? Sidekiq.default_configuration : Sidekiq.options)
|
14
|
+
batch_size ||= (options[:fast_enq_batch_size] || DEFAULT_BATCH_SIZE)
|
12
15
|
@script = lua_script(batch_size)
|
13
16
|
Sidekiq.redis do |conn|
|
14
17
|
@script_sha_1 = conn.script(:load, @script)
|
@@ -43,10 +46,10 @@ class SidekiqFastEnq
|
|
43
46
|
Sidekiq::Client.push(Sidekiq.load_json(job))
|
44
47
|
enqueue_time += (Time.now - t)
|
45
48
|
jobs_count += 1
|
46
|
-
logger.debug("enqueued #{sorted_set}: #{job}") if logger
|
49
|
+
logger.debug("enqueued #{sorted_set}: #{job}") if logger&.debug?
|
47
50
|
end
|
48
51
|
|
49
|
-
if jobs_count > 0 && logger
|
52
|
+
if jobs_count > 0 && logger&.info?
|
50
53
|
loop_time = Time.now - start_time
|
51
54
|
logger.info("SidekiqFastEnq enqueued #{jobs_count} from #{sorted_set} in #{loop_time.round(3)}s (pop: #{pop_time.round(3)}s; enqueue: #{enqueue_time.round(3)}s)")
|
52
55
|
end
|
@@ -63,18 +66,16 @@ class SidekiqFastEnq
|
|
63
66
|
end
|
64
67
|
|
65
68
|
# Evaluate and execute a Lua script on the redis server.
|
66
|
-
def eval_script(conn, script, sha1, argv=[])
|
67
|
-
|
68
|
-
|
69
|
-
|
70
|
-
|
71
|
-
|
72
|
-
|
73
|
-
|
74
|
-
|
75
|
-
|
76
|
-
raise e
|
77
|
-
end
|
69
|
+
def eval_script(conn, script, sha1, argv = [])
|
70
|
+
conn.evalsha(sha1, [], argv)
|
71
|
+
rescue Redis::CommandError => e
|
72
|
+
if e.message.include?("NOSCRIPT")
|
73
|
+
t = Time.now
|
74
|
+
sha1 = conn.script(:load, script)
|
75
|
+
Sidekiq::Logging.logger.info("loaded script #{sha1} in #{Time.now - t}s")
|
76
|
+
retry
|
77
|
+
else
|
78
|
+
raise e
|
78
79
|
end
|
79
80
|
end
|
80
81
|
|
data/sidekiq_fast_enq.gemspec
CHANGED
@@ -1,27 +1,34 @@
|
|
1
|
-
# coding: utf-8
|
2
|
-
lib = File.expand_path('../lib', __FILE__)
|
3
|
-
$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
|
4
|
-
|
5
1
|
Gem::Specification.new do |spec|
|
6
|
-
spec.name
|
7
|
-
spec.version
|
8
|
-
spec.authors
|
9
|
-
spec.email
|
10
|
-
|
11
|
-
spec.
|
12
|
-
spec.homepage
|
13
|
-
spec.license
|
2
|
+
spec.name = "sidekiq-fast-enq"
|
3
|
+
spec.version = File.read(File.expand_path("VERSION", __dir__)).strip
|
4
|
+
spec.authors = ["Brian Durand"]
|
5
|
+
spec.email = ["bbdurand@gmail.com"]
|
6
|
+
|
7
|
+
spec.summary = "More efficient scheduled job queue implementation for sidekiq to increase throughput in large installations."
|
8
|
+
spec.homepage = "https://github.com/bdurand/sidekiq-fast-enq"
|
9
|
+
spec.license = "MIT"
|
10
|
+
|
11
|
+
# Specify which files should be added to the gem when it is released.
|
12
|
+
# The `git ls-files -z` loads the files in the RubyGem that have been added into git.
|
13
|
+
ignore_files = %w[
|
14
|
+
.
|
15
|
+
Appraisals
|
16
|
+
Gemfile
|
17
|
+
Gemfile.lock
|
18
|
+
Rakefile
|
19
|
+
bin/
|
20
|
+
gemfiles/
|
21
|
+
spec/
|
22
|
+
]
|
23
|
+
spec.files = Dir.chdir(__dir__) do
|
24
|
+
`git ls-files -z`.split("\x0").reject { |f| ignore_files.any? { |path| f.start_with?(path) } }
|
25
|
+
end
|
14
26
|
|
15
|
-
spec.files = `git ls-files`.split($/)
|
16
|
-
spec.executables = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
|
17
|
-
spec.test_files = spec.files.grep(%r{^(test|spec|features)/})
|
18
27
|
spec.require_paths = ["lib"]
|
19
28
|
|
20
|
-
spec.add_dependency(
|
29
|
+
spec.add_dependency("sidekiq", ">=5.0")
|
21
30
|
|
22
31
|
spec.add_development_dependency "bundler"
|
23
|
-
|
24
|
-
spec.
|
25
|
-
spec.add_development_dependency "timecop"
|
26
|
-
spec.add_development_dependency "appraisal"
|
32
|
+
|
33
|
+
spec.required_ruby_version = ">= 2.5"
|
27
34
|
end
|
metadata
CHANGED
@@ -1,15 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: sidekiq-fast-enq
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 1.0.
|
4
|
+
version: 1.0.3
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
|
-
- We Heart It
|
8
7
|
- Brian Durand
|
9
|
-
autorequire:
|
8
|
+
autorequire:
|
10
9
|
bindir: bin
|
11
10
|
cert_chain: []
|
12
|
-
date:
|
11
|
+
date: 2023-11-12 00:00:00.000000000 Z
|
13
12
|
dependencies:
|
14
13
|
- !ruby/object:Gem::Dependency
|
15
14
|
name: sidekiq
|
@@ -17,14 +16,14 @@ dependencies:
|
|
17
16
|
requirements:
|
18
17
|
- - ">="
|
19
18
|
- !ruby/object:Gem::Version
|
20
|
-
version: '
|
19
|
+
version: '5.0'
|
21
20
|
type: :runtime
|
22
21
|
prerelease: false
|
23
22
|
version_requirements: !ruby/object:Gem::Requirement
|
24
23
|
requirements:
|
25
24
|
- - ">="
|
26
25
|
- !ruby/object:Gem::Version
|
27
|
-
version: '
|
26
|
+
version: '5.0'
|
28
27
|
- !ruby/object:Gem::Dependency
|
29
28
|
name: bundler
|
30
29
|
requirement: !ruby/object:Gem::Requirement
|
@@ -39,99 +38,24 @@ dependencies:
|
|
39
38
|
- - ">="
|
40
39
|
- !ruby/object:Gem::Version
|
41
40
|
version: '0'
|
42
|
-
|
43
|
-
name: rake
|
44
|
-
requirement: !ruby/object:Gem::Requirement
|
45
|
-
requirements:
|
46
|
-
- - ">="
|
47
|
-
- !ruby/object:Gem::Version
|
48
|
-
version: '0'
|
49
|
-
type: :development
|
50
|
-
prerelease: false
|
51
|
-
version_requirements: !ruby/object:Gem::Requirement
|
52
|
-
requirements:
|
53
|
-
- - ">="
|
54
|
-
- !ruby/object:Gem::Version
|
55
|
-
version: '0'
|
56
|
-
- !ruby/object:Gem::Dependency
|
57
|
-
name: rspec
|
58
|
-
requirement: !ruby/object:Gem::Requirement
|
59
|
-
requirements:
|
60
|
-
- - ">="
|
61
|
-
- !ruby/object:Gem::Version
|
62
|
-
version: '0'
|
63
|
-
type: :development
|
64
|
-
prerelease: false
|
65
|
-
version_requirements: !ruby/object:Gem::Requirement
|
66
|
-
requirements:
|
67
|
-
- - ">="
|
68
|
-
- !ruby/object:Gem::Version
|
69
|
-
version: '0'
|
70
|
-
- !ruby/object:Gem::Dependency
|
71
|
-
name: timecop
|
72
|
-
requirement: !ruby/object:Gem::Requirement
|
73
|
-
requirements:
|
74
|
-
- - ">="
|
75
|
-
- !ruby/object:Gem::Version
|
76
|
-
version: '0'
|
77
|
-
type: :development
|
78
|
-
prerelease: false
|
79
|
-
version_requirements: !ruby/object:Gem::Requirement
|
80
|
-
requirements:
|
81
|
-
- - ">="
|
82
|
-
- !ruby/object:Gem::Version
|
83
|
-
version: '0'
|
84
|
-
- !ruby/object:Gem::Dependency
|
85
|
-
name: appraisal
|
86
|
-
requirement: !ruby/object:Gem::Requirement
|
87
|
-
requirements:
|
88
|
-
- - ">="
|
89
|
-
- !ruby/object:Gem::Version
|
90
|
-
version: '0'
|
91
|
-
type: :development
|
92
|
-
prerelease: false
|
93
|
-
version_requirements: !ruby/object:Gem::Requirement
|
94
|
-
requirements:
|
95
|
-
- - ">="
|
96
|
-
- !ruby/object:Gem::Version
|
97
|
-
version: '0'
|
98
|
-
description: More efficient scheduled job queue implementation for sidekiq to increase
|
99
|
-
throughput in large installations.
|
41
|
+
description:
|
100
42
|
email:
|
101
|
-
- dev@weheartit.com
|
102
43
|
- bbdurand@gmail.com
|
103
44
|
executables: []
|
104
45
|
extensions: []
|
105
46
|
extra_rdoc_files: []
|
106
47
|
files:
|
107
|
-
-
|
108
|
-
- ".travis.yml"
|
109
|
-
- Appraisals
|
110
|
-
- Gemfile
|
111
|
-
- Gemfile.lock
|
112
|
-
- HISTORY.md
|
48
|
+
- CHANGELOG.md
|
113
49
|
- MIT_LICENSE.txt
|
114
50
|
- README.md
|
115
|
-
- Rakefile
|
116
51
|
- VERSION
|
117
|
-
- gemfiles/.bundle/config
|
118
|
-
- gemfiles/sidekiq_3.gemfile
|
119
|
-
- gemfiles/sidekiq_3.gemfile.lock
|
120
|
-
- gemfiles/sidekiq_4.gemfile
|
121
|
-
- gemfiles/sidekiq_4.gemfile.lock
|
122
|
-
- gemfiles/sidekiq_5.gemfile
|
123
|
-
- gemfiles/sidekiq_5.gemfile.lock
|
124
|
-
- gemfiles/sidekiq_6.gemfile
|
125
|
-
- gemfiles/sidekiq_6.gemfile.lock
|
126
52
|
- lib/sidekiq-fast-enq.rb
|
127
53
|
- sidekiq_fast_enq.gemspec
|
128
|
-
|
129
|
-
- spec/spec_helper.rb
|
130
|
-
homepage: https://github.com/weheartit/sidekiq_fast_enq
|
54
|
+
homepage: https://github.com/bdurand/sidekiq-fast-enq
|
131
55
|
licenses:
|
132
56
|
- MIT
|
133
57
|
metadata: {}
|
134
|
-
post_install_message:
|
58
|
+
post_install_message:
|
135
59
|
rdoc_options: []
|
136
60
|
require_paths:
|
137
61
|
- lib
|
@@ -139,17 +63,16 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
139
63
|
requirements:
|
140
64
|
- - ">="
|
141
65
|
- !ruby/object:Gem::Version
|
142
|
-
version: '
|
66
|
+
version: '2.5'
|
143
67
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
144
68
|
requirements:
|
145
69
|
- - ">="
|
146
70
|
- !ruby/object:Gem::Version
|
147
71
|
version: '0'
|
148
72
|
requirements: []
|
149
|
-
rubygems_version: 3.
|
150
|
-
signing_key:
|
73
|
+
rubygems_version: 3.4.12
|
74
|
+
signing_key:
|
151
75
|
specification_version: 4
|
152
|
-
summary: More efficient scheduled job queue implementation for sidekiq
|
153
|
-
|
154
|
-
|
155
|
-
- spec/spec_helper.rb
|
76
|
+
summary: More efficient scheduled job queue implementation for sidekiq to increase
|
77
|
+
throughput in large installations.
|
78
|
+
test_files: []
|
data/.gitignore
DELETED
data/.travis.yml
DELETED
@@ -1,28 +0,0 @@
|
|
1
|
-
language: ruby
|
2
|
-
|
3
|
-
rvm:
|
4
|
-
- "2.6"
|
5
|
-
|
6
|
-
services:
|
7
|
-
- redis-server
|
8
|
-
|
9
|
-
gemfile:
|
10
|
-
- Gemfile
|
11
|
-
|
12
|
-
matrix:
|
13
|
-
include:
|
14
|
-
- rvm: "2.6"
|
15
|
-
gemfile: gemfiles/sidekiq_6.gemfile
|
16
|
-
|
17
|
-
- rvm: "2.5"
|
18
|
-
gemfile: gemfiles/sidekiq_5.gemfile
|
19
|
-
|
20
|
-
- rvm: "2.5"
|
21
|
-
gemfile: gemfiles/sidekiq_4.gemfile
|
22
|
-
|
23
|
-
- rvm: "2.4"
|
24
|
-
gemfile: gemfiles/sidekiq_3.gemfile
|
25
|
-
|
26
|
-
script:
|
27
|
-
- gem list bundler
|
28
|
-
- bundle exec rake spec
|
data/Appraisals
DELETED
data/Gemfile
DELETED
data/Gemfile.lock
DELETED
@@ -1,54 +0,0 @@
|
|
1
|
-
PATH
|
2
|
-
remote: .
|
3
|
-
specs:
|
4
|
-
sidekiq-fast-enq (1.0.2)
|
5
|
-
sidekiq (>= 3.4)
|
6
|
-
|
7
|
-
GEM
|
8
|
-
remote: https://rubygems.org/
|
9
|
-
specs:
|
10
|
-
appraisal (2.2.0)
|
11
|
-
bundler
|
12
|
-
rake
|
13
|
-
thor (>= 0.14.0)
|
14
|
-
connection_pool (2.2.2)
|
15
|
-
diff-lcs (1.3)
|
16
|
-
rack (2.1.1)
|
17
|
-
rack-protection (2.0.8.1)
|
18
|
-
rack
|
19
|
-
rake (13.0.1)
|
20
|
-
redis (4.1.3)
|
21
|
-
rspec (3.9.0)
|
22
|
-
rspec-core (~> 3.9.0)
|
23
|
-
rspec-expectations (~> 3.9.0)
|
24
|
-
rspec-mocks (~> 3.9.0)
|
25
|
-
rspec-core (3.9.1)
|
26
|
-
rspec-support (~> 3.9.1)
|
27
|
-
rspec-expectations (3.9.0)
|
28
|
-
diff-lcs (>= 1.2.0, < 2.0)
|
29
|
-
rspec-support (~> 3.9.0)
|
30
|
-
rspec-mocks (3.9.1)
|
31
|
-
diff-lcs (>= 1.2.0, < 2.0)
|
32
|
-
rspec-support (~> 3.9.0)
|
33
|
-
rspec-support (3.9.2)
|
34
|
-
sidekiq (6.0.4)
|
35
|
-
connection_pool (>= 2.2.2)
|
36
|
-
rack (>= 2.0.0)
|
37
|
-
rack-protection (>= 2.0.0)
|
38
|
-
redis (>= 4.1.0)
|
39
|
-
thor (1.0.1)
|
40
|
-
timecop (0.9.1)
|
41
|
-
|
42
|
-
PLATFORMS
|
43
|
-
ruby
|
44
|
-
|
45
|
-
DEPENDENCIES
|
46
|
-
appraisal
|
47
|
-
bundler
|
48
|
-
rake
|
49
|
-
rspec
|
50
|
-
sidekiq-fast-enq!
|
51
|
-
timecop
|
52
|
-
|
53
|
-
BUNDLED WITH
|
54
|
-
2.1.4
|
data/HISTORY.md
DELETED
data/Rakefile
DELETED
@@ -1,57 +0,0 @@
|
|
1
|
-
require "bundler/gem_tasks"
|
2
|
-
require "rspec/core/rake_task"
|
3
|
-
|
4
|
-
RSpec::Core::RakeTask.new(:spec)
|
5
|
-
|
6
|
-
task :default => :appraisals
|
7
|
-
|
8
|
-
desc "run the specs using appraisal"
|
9
|
-
task :appraisals do
|
10
|
-
exec "bundle exec appraisal rake spec"
|
11
|
-
end
|
12
|
-
|
13
|
-
namespace :appraisals do
|
14
|
-
desc "install all the appraisal gemspecs"
|
15
|
-
task :install do
|
16
|
-
exec "bundle exec appraisal install"
|
17
|
-
end
|
18
|
-
end
|
19
|
-
|
20
|
-
task :load_test, [:jobs_size, :workers, :fast] do |t, args|
|
21
|
-
require 'celluloid'
|
22
|
-
require File.expand_path('../lib/sidekiq-fast-enq', __FILE__)
|
23
|
-
require 'sidekiq/scheduled'
|
24
|
-
require 'sidekiq/api'
|
25
|
-
|
26
|
-
class FastEnqLoadTestWorker
|
27
|
-
include Sidekiq::Worker
|
28
|
-
def perform()
|
29
|
-
end
|
30
|
-
end
|
31
|
-
|
32
|
-
jobs_size = args[:jobs_size].to_i
|
33
|
-
workers_size = args[:workers].to_i
|
34
|
-
klass = (args[:fast] == 'fast' ? SidekiqFastEnq : Sidekiq::Scheduled::Enq)
|
35
|
-
|
36
|
-
Sidekiq.configure_server do |config|
|
37
|
-
config.redis = {:namespace => "sidekiq_fast_enq_load_test"}
|
38
|
-
end
|
39
|
-
|
40
|
-
Sidekiq::ScheduledSet.new.clear
|
41
|
-
jobs_size.times do
|
42
|
-
FastEnqLoadTestWorker.perform_in(rand)
|
43
|
-
end
|
44
|
-
|
45
|
-
t = Time.now
|
46
|
-
workers_size.times do
|
47
|
-
fork do
|
48
|
-
klass.new.enqueue_jobs
|
49
|
-
end
|
50
|
-
end
|
51
|
-
|
52
|
-
workers_size.times do
|
53
|
-
Process.wait
|
54
|
-
end
|
55
|
-
|
56
|
-
puts "Enqueued #{jobs_size} jobs in #{Time.now - t} seconds"
|
57
|
-
end
|
data/gemfiles/.bundle/config
DELETED
data/gemfiles/sidekiq_3.gemfile
DELETED
@@ -1,74 +0,0 @@
|
|
1
|
-
PATH
|
2
|
-
remote: ..
|
3
|
-
specs:
|
4
|
-
sidekiq-fast-enq (1.0.2)
|
5
|
-
sidekiq (>= 3.4)
|
6
|
-
|
7
|
-
GEM
|
8
|
-
remote: https://rubygems.org/
|
9
|
-
specs:
|
10
|
-
appraisal (2.2.0)
|
11
|
-
bundler
|
12
|
-
rake
|
13
|
-
thor (>= 0.14.0)
|
14
|
-
celluloid (0.17.4)
|
15
|
-
celluloid-essentials
|
16
|
-
celluloid-extras
|
17
|
-
celluloid-fsm
|
18
|
-
celluloid-pool
|
19
|
-
celluloid-supervision
|
20
|
-
timers (>= 4.1.1)
|
21
|
-
celluloid-essentials (0.20.5)
|
22
|
-
timers (>= 4.1.1)
|
23
|
-
celluloid-extras (0.20.5)
|
24
|
-
timers (>= 4.1.1)
|
25
|
-
celluloid-fsm (0.20.5)
|
26
|
-
timers (>= 4.1.1)
|
27
|
-
celluloid-pool (0.20.5)
|
28
|
-
timers (>= 4.1.1)
|
29
|
-
celluloid-supervision (0.20.6)
|
30
|
-
timers (>= 4.1.1)
|
31
|
-
connection_pool (2.2.2)
|
32
|
-
diff-lcs (1.3)
|
33
|
-
json (1.8.6)
|
34
|
-
rake (13.0.1)
|
35
|
-
redis (3.3.5)
|
36
|
-
redis-namespace (1.7.0)
|
37
|
-
redis (>= 3.0.4)
|
38
|
-
rspec (3.9.0)
|
39
|
-
rspec-core (~> 3.9.0)
|
40
|
-
rspec-expectations (~> 3.9.0)
|
41
|
-
rspec-mocks (~> 3.9.0)
|
42
|
-
rspec-core (3.9.1)
|
43
|
-
rspec-support (~> 3.9.1)
|
44
|
-
rspec-expectations (3.9.0)
|
45
|
-
diff-lcs (>= 1.2.0, < 2.0)
|
46
|
-
rspec-support (~> 3.9.0)
|
47
|
-
rspec-mocks (3.9.1)
|
48
|
-
diff-lcs (>= 1.2.0, < 2.0)
|
49
|
-
rspec-support (~> 3.9.0)
|
50
|
-
rspec-support (3.9.2)
|
51
|
-
sidekiq (3.5.4)
|
52
|
-
celluloid (~> 0.17.2)
|
53
|
-
connection_pool (~> 2.2, >= 2.2.0)
|
54
|
-
json (~> 1.0)
|
55
|
-
redis (~> 3.2, >= 3.2.1)
|
56
|
-
redis-namespace (~> 1.5, >= 1.5.2)
|
57
|
-
thor (1.0.1)
|
58
|
-
timecop (0.9.1)
|
59
|
-
timers (4.3.0)
|
60
|
-
|
61
|
-
PLATFORMS
|
62
|
-
ruby
|
63
|
-
|
64
|
-
DEPENDENCIES
|
65
|
-
appraisal
|
66
|
-
bundler
|
67
|
-
rake
|
68
|
-
rspec
|
69
|
-
sidekiq (~> 3.0)
|
70
|
-
sidekiq-fast-enq!
|
71
|
-
timecop
|
72
|
-
|
73
|
-
BUNDLED WITH
|
74
|
-
2.1.4
|
data/gemfiles/sidekiq_4.gemfile
DELETED
@@ -1,56 +0,0 @@
|
|
1
|
-
PATH
|
2
|
-
remote: ..
|
3
|
-
specs:
|
4
|
-
sidekiq-fast-enq (1.0.2)
|
5
|
-
sidekiq (>= 3.4)
|
6
|
-
|
7
|
-
GEM
|
8
|
-
remote: https://rubygems.org/
|
9
|
-
specs:
|
10
|
-
appraisal (2.2.0)
|
11
|
-
bundler
|
12
|
-
rake
|
13
|
-
thor (>= 0.14.0)
|
14
|
-
concurrent-ruby (1.1.5)
|
15
|
-
connection_pool (2.2.2)
|
16
|
-
diff-lcs (1.3)
|
17
|
-
rack (2.1.1)
|
18
|
-
rack-protection (2.0.8.1)
|
19
|
-
rack
|
20
|
-
rake (13.0.1)
|
21
|
-
redis (3.3.5)
|
22
|
-
rspec (3.9.0)
|
23
|
-
rspec-core (~> 3.9.0)
|
24
|
-
rspec-expectations (~> 3.9.0)
|
25
|
-
rspec-mocks (~> 3.9.0)
|
26
|
-
rspec-core (3.9.1)
|
27
|
-
rspec-support (~> 3.9.1)
|
28
|
-
rspec-expectations (3.9.0)
|
29
|
-
diff-lcs (>= 1.2.0, < 2.0)
|
30
|
-
rspec-support (~> 3.9.0)
|
31
|
-
rspec-mocks (3.9.1)
|
32
|
-
diff-lcs (>= 1.2.0, < 2.0)
|
33
|
-
rspec-support (~> 3.9.0)
|
34
|
-
rspec-support (3.9.2)
|
35
|
-
sidekiq (4.2.10)
|
36
|
-
concurrent-ruby (~> 1.0)
|
37
|
-
connection_pool (~> 2.2, >= 2.2.0)
|
38
|
-
rack-protection (>= 1.5.0)
|
39
|
-
redis (~> 3.2, >= 3.2.1)
|
40
|
-
thor (1.0.1)
|
41
|
-
timecop (0.9.1)
|
42
|
-
|
43
|
-
PLATFORMS
|
44
|
-
ruby
|
45
|
-
|
46
|
-
DEPENDENCIES
|
47
|
-
appraisal
|
48
|
-
bundler
|
49
|
-
rake
|
50
|
-
rspec
|
51
|
-
sidekiq (~> 4.0)
|
52
|
-
sidekiq-fast-enq!
|
53
|
-
timecop
|
54
|
-
|
55
|
-
BUNDLED WITH
|
56
|
-
2.1.4
|
data/gemfiles/sidekiq_5.gemfile
DELETED
@@ -1,55 +0,0 @@
|
|
1
|
-
PATH
|
2
|
-
remote: ..
|
3
|
-
specs:
|
4
|
-
sidekiq-fast-enq (1.0.2)
|
5
|
-
sidekiq (>= 3.4)
|
6
|
-
|
7
|
-
GEM
|
8
|
-
remote: https://rubygems.org/
|
9
|
-
specs:
|
10
|
-
appraisal (2.2.0)
|
11
|
-
bundler
|
12
|
-
rake
|
13
|
-
thor (>= 0.14.0)
|
14
|
-
connection_pool (2.2.2)
|
15
|
-
diff-lcs (1.3)
|
16
|
-
rack (2.1.1)
|
17
|
-
rack-protection (2.0.8.1)
|
18
|
-
rack
|
19
|
-
rake (13.0.1)
|
20
|
-
redis (4.1.3)
|
21
|
-
rspec (3.9.0)
|
22
|
-
rspec-core (~> 3.9.0)
|
23
|
-
rspec-expectations (~> 3.9.0)
|
24
|
-
rspec-mocks (~> 3.9.0)
|
25
|
-
rspec-core (3.9.1)
|
26
|
-
rspec-support (~> 3.9.1)
|
27
|
-
rspec-expectations (3.9.0)
|
28
|
-
diff-lcs (>= 1.2.0, < 2.0)
|
29
|
-
rspec-support (~> 3.9.0)
|
30
|
-
rspec-mocks (3.9.1)
|
31
|
-
diff-lcs (>= 1.2.0, < 2.0)
|
32
|
-
rspec-support (~> 3.9.0)
|
33
|
-
rspec-support (3.9.2)
|
34
|
-
sidekiq (5.2.7)
|
35
|
-
connection_pool (~> 2.2, >= 2.2.2)
|
36
|
-
rack (>= 1.5.0)
|
37
|
-
rack-protection (>= 1.5.0)
|
38
|
-
redis (>= 3.3.5, < 5)
|
39
|
-
thor (1.0.1)
|
40
|
-
timecop (0.9.1)
|
41
|
-
|
42
|
-
PLATFORMS
|
43
|
-
ruby
|
44
|
-
|
45
|
-
DEPENDENCIES
|
46
|
-
appraisal
|
47
|
-
bundler
|
48
|
-
rake
|
49
|
-
rspec
|
50
|
-
sidekiq (~> 5.0)
|
51
|
-
sidekiq-fast-enq!
|
52
|
-
timecop
|
53
|
-
|
54
|
-
BUNDLED WITH
|
55
|
-
2.1.4
|
data/gemfiles/sidekiq_6.gemfile
DELETED
@@ -1,55 +0,0 @@
|
|
1
|
-
PATH
|
2
|
-
remote: ..
|
3
|
-
specs:
|
4
|
-
sidekiq-fast-enq (1.0.2)
|
5
|
-
sidekiq (>= 3.4)
|
6
|
-
|
7
|
-
GEM
|
8
|
-
remote: https://rubygems.org/
|
9
|
-
specs:
|
10
|
-
appraisal (2.2.0)
|
11
|
-
bundler
|
12
|
-
rake
|
13
|
-
thor (>= 0.14.0)
|
14
|
-
connection_pool (2.2.2)
|
15
|
-
diff-lcs (1.3)
|
16
|
-
rack (2.1.1)
|
17
|
-
rack-protection (2.0.8.1)
|
18
|
-
rack
|
19
|
-
rake (13.0.1)
|
20
|
-
redis (4.1.3)
|
21
|
-
rspec (3.9.0)
|
22
|
-
rspec-core (~> 3.9.0)
|
23
|
-
rspec-expectations (~> 3.9.0)
|
24
|
-
rspec-mocks (~> 3.9.0)
|
25
|
-
rspec-core (3.9.1)
|
26
|
-
rspec-support (~> 3.9.1)
|
27
|
-
rspec-expectations (3.9.0)
|
28
|
-
diff-lcs (>= 1.2.0, < 2.0)
|
29
|
-
rspec-support (~> 3.9.0)
|
30
|
-
rspec-mocks (3.9.1)
|
31
|
-
diff-lcs (>= 1.2.0, < 2.0)
|
32
|
-
rspec-support (~> 3.9.0)
|
33
|
-
rspec-support (3.9.2)
|
34
|
-
sidekiq (6.0.4)
|
35
|
-
connection_pool (>= 2.2.2)
|
36
|
-
rack (>= 2.0.0)
|
37
|
-
rack-protection (>= 2.0.0)
|
38
|
-
redis (>= 4.1.0)
|
39
|
-
thor (1.0.1)
|
40
|
-
timecop (0.9.1)
|
41
|
-
|
42
|
-
PLATFORMS
|
43
|
-
ruby
|
44
|
-
|
45
|
-
DEPENDENCIES
|
46
|
-
appraisal
|
47
|
-
bundler
|
48
|
-
rake
|
49
|
-
rspec
|
50
|
-
sidekiq (~> 6.0)
|
51
|
-
sidekiq-fast-enq!
|
52
|
-
timecop
|
53
|
-
|
54
|
-
BUNDLED WITH
|
55
|
-
2.1.4
|
@@ -1,53 +0,0 @@
|
|
1
|
-
require 'spec_helper'
|
2
|
-
|
3
|
-
describe SidekiqFastEnq do
|
4
|
-
|
5
|
-
let(:scheduled_set){ Sidekiq::ScheduledSet.new }
|
6
|
-
let(:retry_set){ Sidekiq::RetrySet.new }
|
7
|
-
let(:default_queue){ Sidekiq::Queue.new }
|
8
|
-
|
9
|
-
before :each do
|
10
|
-
scheduled_set.clear
|
11
|
-
retry_set.clear
|
12
|
-
default_queue.clear
|
13
|
-
end
|
14
|
-
|
15
|
-
it "should return without doing anything if there are no scheduled jobs" do
|
16
|
-
SidekiqFastEnq.new.enqueue_jobs
|
17
|
-
expect(scheduled_set.size).to eq(0)
|
18
|
-
expect(retry_set.size).to eq(0)
|
19
|
-
expect(default_queue.size).to eq(0)
|
20
|
-
end
|
21
|
-
|
22
|
-
it "should enqueue a single elligible job from the scheduled jobs queue" do
|
23
|
-
Timecop.travel(Time.now - 3600){ FastEnqTestWorker.perform_in(60, 'one') }
|
24
|
-
SidekiqFastEnq.new.enqueue_jobs
|
25
|
-
expect(scheduled_set.size).to eq(0)
|
26
|
-
expect(retry_set.size).to eq(0)
|
27
|
-
expect(default_queue.size).to eq(1)
|
28
|
-
end
|
29
|
-
|
30
|
-
it "should enqueue all elligible jobs from the scheduled jobs queue" do
|
31
|
-
Timecop.travel(Time.now - 3600){ FastEnqTestWorker.perform_in(60, 'one') }
|
32
|
-
Timecop.travel(Time.now - 3600){ FastEnqTestWorker.perform_in(900, 'two') }
|
33
|
-
FastEnqTestWorker.perform_in(10, 'three')
|
34
|
-
SidekiqFastEnq.new.enqueue_jobs
|
35
|
-
expect(scheduled_set.size).to eq(1)
|
36
|
-
expect(retry_set.size).to eq(0)
|
37
|
-
expect(default_queue.size).to eq(2)
|
38
|
-
end
|
39
|
-
|
40
|
-
it "should enqueue all elligible jobs from the scheduled jobs queue when there are a lot of them" do
|
41
|
-
Timecop.travel(Time.now - 3600) do
|
42
|
-
200.times do
|
43
|
-
FastEnqTestWorker.perform_in(60, 'one')
|
44
|
-
end
|
45
|
-
end
|
46
|
-
FastEnqTestWorker.perform_in(10, 'three')
|
47
|
-
SidekiqFastEnq.new.enqueue_jobs
|
48
|
-
expect(scheduled_set.size).to eq(1)
|
49
|
-
expect(retry_set.size).to eq(0)
|
50
|
-
expect(default_queue.size).to eq(200)
|
51
|
-
end
|
52
|
-
|
53
|
-
end
|
data/spec/spec_helper.rb
DELETED
@@ -1,34 +0,0 @@
|
|
1
|
-
# Breaks if not required. Sidekiq doesn't directly require in
|
2
|
-
# the load process.
|
3
|
-
|
4
|
-
require_relative '../lib/sidekiq-fast-enq'
|
5
|
-
|
6
|
-
require 'timecop'
|
7
|
-
require 'sidekiq/version'
|
8
|
-
require 'celluloid' if Sidekiq::VERSION.to_i < 4
|
9
|
-
require 'sidekiq/scheduled'
|
10
|
-
require 'sidekiq/api'
|
11
|
-
|
12
|
-
RSpec.configure do |config|
|
13
|
-
config.run_all_when_everything_filtered = true
|
14
|
-
config.filter_run :focus
|
15
|
-
|
16
|
-
# Run specs in random order to surface order dependencies. If you find an
|
17
|
-
# order dependency and want to debug it, you can fix the order by providing
|
18
|
-
# the seed, which is printed after each run.
|
19
|
-
# --seed 1234
|
20
|
-
config.order = 'random'
|
21
|
-
|
22
|
-
Sidekiq.configure_server do |config|
|
23
|
-
config.redis = {:namespace => "sidekiq_fast_enq_test"}
|
24
|
-
end
|
25
|
-
Sidekiq.options[:scheduled_enq] = SidekiqFastEnq
|
26
|
-
Sidekiq.logger.level = Logger::FATAL
|
27
|
-
end
|
28
|
-
|
29
|
-
class FastEnqTestWorker
|
30
|
-
include Sidekiq::Worker
|
31
|
-
|
32
|
-
def perform(arg)
|
33
|
-
end
|
34
|
-
end
|