sidekiq-fast-enq 1.0.1 → 1.0.3

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
- SHA1:
3
- metadata.gz: 42316bf316539012e9cd71544606ad581c47917d
4
- data.tar.gz: 9817a10809a564cec6ab8e87612a009c2a7ed8f7
2
+ SHA256:
3
+ metadata.gz: 4c2ef4bb287f73b4dc43e32b604d16e98dc284822f399952e13cee7df1869836
4
+ data.tar.gz: b9c4efb591b505b94733ba6722b50c99ddd0a6151da9b54ab5d16534364cda15
5
5
  SHA512:
6
- metadata.gz: 3eb9ce115d05af9f31166b3752bbc3bd5fd8260dcfce625328bcb5be6306261f766a1f5f773559a192c267af9069d9dde2f9f0ec6943bbd16cc3821a93777d40
7
- data.tar.gz: ec68435cbeb752ead5db389d518dba994af9b6f3f757c4693b620e2f668008deee51e1a5e88ca57a584312f5c7f2ada729f4b90705b0c7e8cd133b2dddb26286
6
+ metadata.gz: d1ed8261dd525b1de1820f676b9c545a0cc6486cef1a3ed8e078ea9eec7ecf2e67f94aaba11d347b5d59f2a8d8956612d564560182f7d3652fe9ec00a167dcc7
7
+ data.tar.gz: dc9c18fc20a0a0ff75bacf4428c94f4eb5c8665824f1060755508a1273d05bb7889c73499cff4bd5a60c24b01360c5d5a7a52754a991a40996af39d9bfb29cd0
data/CHANGELOG.md ADDED
@@ -0,0 +1,29 @@
1
+ # Changelog
2
+ All notable changes to this project will be documented in this file.
3
+
4
+ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.0.0/),
5
+ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0.html).
6
+
7
+ ## 1.0.3
8
+
9
+ ### Added
10
+
11
+ - Compatibility with Sidekiq 7
12
+
13
+ ## 1.0.2
14
+
15
+ ### Added
16
+
17
+ - Compatibility with Sidekiq 6
18
+
19
+ ## 1.0.2
20
+
21
+ ### Changed
22
+
23
+ - Update gemspec to specify compatibility with sidekiq 4
24
+
25
+ # 1.0.0
26
+
27
+ ### Added
28
+
29
+ - Initial release
data/MIT_LICENSE.txt CHANGED
@@ -1,4 +1,4 @@
1
- Copyright (c) 2015 WHI, Inc.
1
+ Copyright (c) 2015 Brian Durand
2
2
 
3
3
  MIT License
4
4
 
data/README.md CHANGED
@@ -1,3 +1,9 @@
1
+ Sidekiq Fast Enqueuing
2
+
3
+ [![Continuous Integration](https://github.com/bdurand/sidekiq-fast-enq/actions/workflows/continuous_integration.yml/badge.svg)](https://github.com/bdurand/sidekiq-fast-enq/actions/workflows/continuous_integration.yml)
4
+ [![Regression Test](https://github.com/bdurand/sidekiq-fast-enq/actions/workflows/regression_test.yml/badge.svg)](https://github.com/bdurand/sidekiq-fast-enq/actions/workflows/regression_test.yml)
5
+ [![Ruby Style Guide](https://img.shields.io/badge/code_style-standard-brightgreen.svg)](https://github.com/testdouble/standard)
6
+
1
7
  This gem provides a much more efficien implementation for checking the Sidekiq scheduled and retry queues. This can provide a significant performance boost for large sidekiq implementations that utilize many processes. It can also reduce load on the redis server.
2
8
 
3
9
  ### TL;DR
@@ -14,10 +20,47 @@ This plugin does not alter any sidekiq internal code or data structures.
14
20
 
15
21
  In your sidekiq configuration you need to set the `:scheduled_enq` option to `SidekiqFastEnq` (only available in sidekiq 3.4.0 and later). You might also want to hard code a value for the `:poll_interval_average` option as well. If this option is not set the polling interval for checking the scheduled queues is based on the number of processes in an effort to reduce the effects of the race condition. It is not needed with this code and scheduled jobs will be enqueued closer to their scheduled time without it.
16
22
 
23
+ ```ruby
24
+ Sidekiq.default_configuration[:scheduled_enq] = SidekiqFastEnq
25
+ Sidekiq.default_configuration[:poll_interval_average] = 30
26
+ ```
27
+
28
+ For Sidekiq versions prior to version 7:
29
+
17
30
  ```ruby
18
31
  Sidekiq.options[:scheduled_enq] = SidekiqFastEnq
19
32
  Sidekiq.options[:poll_interval_average] = 30
20
33
  ```
21
34
 
22
- Since this gem eliminates the race condition of having too many processes processing th
23
- Note: this gem utilizes server side Lua scripting so you must be using Redis Server 2.6.0 or later.
35
+
36
+ ### Redis requirement
37
+
38
+ Redis server 2.6 or greater is required for this code.
39
+
40
+ ## Installation
41
+
42
+ Add this line to your application's Gemfile:
43
+
44
+ ```ruby
45
+ gem 'sidekiq-fast-enq'
46
+ ```
47
+
48
+ And then execute:
49
+ ```bash
50
+ $ bundle
51
+ ```
52
+
53
+ Or install it yourself as:
54
+ ```bash
55
+ $ gem install sidekiq-fast-enq
56
+ ```
57
+
58
+ ## Contributing
59
+
60
+ Fork the repository and open a pull request on GitHub.
61
+
62
+ Please use the [standardrb](https://github.com/testdouble/standard) syntax and lint your code with `standardrb --fix` before submitting.
63
+
64
+ ## License
65
+
66
+ The gem is available as open source under the terms of the [MIT License](https://opensource.org/licenses/MIT).
data/VERSION CHANGED
@@ -1 +1 @@
1
- 1.0.1
1
+ 1.0.3
@@ -1,4 +1,6 @@
1
- require 'sidekiq'
1
+ # frozen_string_literal: true
2
+
3
+ require "sidekiq"
2
4
 
3
5
  # Implementation of the Sidekiq::Scheduled::Enq class that uses a server side Lua script
4
6
  # to atomically get the next scheduled job to run and then pops it from the list. This
@@ -6,19 +8,20 @@ require 'sidekiq'
6
8
  # race conditions checking the scheduled queues.
7
9
  class SidekiqFastEnq
8
10
  DEFAULT_BATCH_SIZE = 1000
9
-
11
+
10
12
  def initialize(batch_size = nil)
11
- batch_size ||= (Sidekiq.options[:fast_enq_batch_size] || DEFAULT_BATCH_SIZE)
13
+ options = (Sidekiq.respond_to?(:default_configuration) ? Sidekiq.default_configuration : Sidekiq.options)
14
+ batch_size ||= (options[:fast_enq_batch_size] || DEFAULT_BATCH_SIZE)
12
15
  @script = lua_script(batch_size)
13
16
  Sidekiq.redis do |conn|
14
17
  @script_sha_1 = conn.script(:load, @script)
15
18
  end
16
19
  end
17
-
20
+
18
21
  def enqueue_jobs(now = Time.now.to_f.to_s, sorted_sets = nil)
19
22
  sorted_sets ||= Sidekiq::Scheduled::SETS
20
- logger = Sidekiq::Logging.logger
21
-
23
+ logger = Sidekiq.logger
24
+
22
25
  # A job's "score" in Redis is the time at which it should be processed.
23
26
  # Just check Redis for the set of jobs with a timestamp before now.
24
27
  Sidekiq.redis do |conn|
@@ -29,7 +32,7 @@ class SidekiqFastEnq
29
32
  start_time = Time.now
30
33
  pop_time = 0.0
31
34
  enqueue_time = 0.0
32
-
35
+
33
36
  # Get the next item in the queue if it's score (time to execute) is <= now.
34
37
  # We need to go through the list one at a time to reduce the risk of something
35
38
  # going wrong between the time jobs are popped from the scheduled queue and when
@@ -43,10 +46,10 @@ class SidekiqFastEnq
43
46
  Sidekiq::Client.push(Sidekiq.load_json(job))
44
47
  enqueue_time += (Time.now - t)
45
48
  jobs_count += 1
46
- logger.debug("enqueued #{sorted_set}: #{job}") if logger && logger.debug?
49
+ logger.debug("enqueued #{sorted_set}: #{job}") if logger&.debug?
47
50
  end
48
-
49
- if jobs_count > 0 && logger && logger.info?
51
+
52
+ if jobs_count > 0 && logger&.info?
50
53
  loop_time = Time.now - start_time
51
54
  logger.info("SidekiqFastEnq enqueued #{jobs_count} from #{sorted_set} in #{loop_time.round(3)}s (pop: #{pop_time.round(3)}s; enqueue: #{enqueue_time.round(3)}s)")
52
55
  end
@@ -61,20 +64,18 @@ class SidekiqFastEnq
61
64
  def pop_job(conn, sorted_set, now)
62
65
  eval_script(conn, @script, @script_sha_1, [sorted_set, now])
63
66
  end
64
-
67
+
65
68
  # Evaluate and execute a Lua script on the redis server.
66
- def eval_script(conn, script, sha1, argv=[])
67
- begin
68
- conn.evalsha(sha1, [], argv)
69
- rescue Redis::CommandError => e
70
- if e.message.include?('NOSCRIPT'.freeze)
71
- t = Time.now
72
- sha1 = conn.script(:load, script)
73
- Sidekiq::Logging.logger.info("loaded script #{sha1} in #{Time.now - t}s")
74
- retry
75
- else
76
- raise e
77
- end
69
+ def eval_script(conn, script, sha1, argv = [])
70
+ conn.evalsha(sha1, [], argv)
71
+ rescue Redis::CommandError => e
72
+ if e.message.include?("NOSCRIPT")
73
+ t = Time.now
74
+ sha1 = conn.script(:load, script)
75
+ Sidekiq::Logging.logger.info("loaded script #{sha1} in #{Time.now - t}s")
76
+ retry
77
+ else
78
+ raise e
78
79
  end
79
80
  end
80
81
 
@@ -87,7 +88,7 @@ class SidekiqFastEnq
87
88
  local sorted_set = ARGV[1]
88
89
  local now = tonumber(ARGV[2])
89
90
  local ready_cache = sorted_set .. '.cache'
90
-
91
+
91
92
  while true do
92
93
  -- Check a cached list of jobs that are ready to execute
93
94
  local job = redis.call('lpop', ready_cache)
@@ -1,26 +1,34 @@
1
- # coding: utf-8
2
- lib = File.expand_path('../lib', __FILE__)
3
- $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
4
-
5
1
  Gem::Specification.new do |spec|
6
- spec.name = "sidekiq-fast-enq"
7
- spec.version = File.read(File.expand_path("../VERSION", __FILE__)).chomp
8
- spec.authors = ["We Heart It", "Brian Durand"]
9
- spec.email = ["dev@weheartit.com", "bbdurand@gmail.com"]
10
- spec.summary = "More efficient scheduled job queue implementation for sidekiq"
11
- spec.description = "More efficient scheduled job queue implementation for sidekiq to increase throughput in large installations."
12
- spec.homepage = "https://github.com/weheartit/sidekiq_fast_enq"
13
- spec.license = "MIT"
2
+ spec.name = "sidekiq-fast-enq"
3
+ spec.version = File.read(File.expand_path("VERSION", __dir__)).strip
4
+ spec.authors = ["Brian Durand"]
5
+ spec.email = ["bbdurand@gmail.com"]
6
+
7
+ spec.summary = "More efficient scheduled job queue implementation for sidekiq to increase throughput in large installations."
8
+ spec.homepage = "https://github.com/bdurand/sidekiq-fast-enq"
9
+ spec.license = "MIT"
10
+
11
+ # Specify which files should be added to the gem when it is released.
12
+ # The `git ls-files -z` loads the files in the RubyGem that have been added into git.
13
+ ignore_files = %w[
14
+ .
15
+ Appraisals
16
+ Gemfile
17
+ Gemfile.lock
18
+ Rakefile
19
+ bin/
20
+ gemfiles/
21
+ spec/
22
+ ]
23
+ spec.files = Dir.chdir(__dir__) do
24
+ `git ls-files -z`.split("\x0").reject { |f| ignore_files.any? { |path| f.start_with?(path) } }
25
+ end
14
26
 
15
- spec.files = `git ls-files`.split($/)
16
- spec.executables = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
17
- spec.test_files = spec.files.grep(%r{^(test|spec|features)/})
18
27
  spec.require_paths = ["lib"]
19
28
 
20
- spec.add_dependency('sidekiq', '>=3.4')
29
+ spec.add_dependency("sidekiq", ">=5.0")
30
+
31
+ spec.add_development_dependency "bundler"
21
32
 
22
- spec.add_development_dependency "bundler", "~> 1.3"
23
- spec.add_development_dependency "rake"
24
- spec.add_development_dependency "rspec"
25
- spec.add_development_dependency "timecop"
33
+ spec.required_ruby_version = ">= 2.5"
26
34
  end
metadata CHANGED
@@ -1,15 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: sidekiq-fast-enq
3
3
  version: !ruby/object:Gem::Version
4
- version: 1.0.1
4
+ version: 1.0.3
5
5
  platform: ruby
6
6
  authors:
7
- - We Heart It
8
7
  - Brian Durand
9
- autorequire:
8
+ autorequire:
10
9
  bindir: bin
11
10
  cert_chain: []
12
- date: 2015-11-19 00:00:00.000000000 Z
11
+ date: 2023-11-12 00:00:00.000000000 Z
13
12
  dependencies:
14
13
  - !ruby/object:Gem::Dependency
15
14
  name: sidekiq
@@ -17,44 +16,16 @@ dependencies:
17
16
  requirements:
18
17
  - - ">="
19
18
  - !ruby/object:Gem::Version
20
- version: '3.4'
19
+ version: '5.0'
21
20
  type: :runtime
22
21
  prerelease: false
23
22
  version_requirements: !ruby/object:Gem::Requirement
24
23
  requirements:
25
24
  - - ">="
26
25
  - !ruby/object:Gem::Version
27
- version: '3.4'
26
+ version: '5.0'
28
27
  - !ruby/object:Gem::Dependency
29
28
  name: bundler
30
- requirement: !ruby/object:Gem::Requirement
31
- requirements:
32
- - - "~>"
33
- - !ruby/object:Gem::Version
34
- version: '1.3'
35
- type: :development
36
- prerelease: false
37
- version_requirements: !ruby/object:Gem::Requirement
38
- requirements:
39
- - - "~>"
40
- - !ruby/object:Gem::Version
41
- version: '1.3'
42
- - !ruby/object:Gem::Dependency
43
- name: rake
44
- requirement: !ruby/object:Gem::Requirement
45
- requirements:
46
- - - ">="
47
- - !ruby/object:Gem::Version
48
- version: '0'
49
- type: :development
50
- prerelease: false
51
- version_requirements: !ruby/object:Gem::Requirement
52
- requirements:
53
- - - ">="
54
- - !ruby/object:Gem::Version
55
- version: '0'
56
- - !ruby/object:Gem::Dependency
57
- name: rspec
58
29
  requirement: !ruby/object:Gem::Requirement
59
30
  requirements:
60
31
  - - ">="
@@ -67,44 +38,24 @@ dependencies:
67
38
  - - ">="
68
39
  - !ruby/object:Gem::Version
69
40
  version: '0'
70
- - !ruby/object:Gem::Dependency
71
- name: timecop
72
- requirement: !ruby/object:Gem::Requirement
73
- requirements:
74
- - - ">="
75
- - !ruby/object:Gem::Version
76
- version: '0'
77
- type: :development
78
- prerelease: false
79
- version_requirements: !ruby/object:Gem::Requirement
80
- requirements:
81
- - - ">="
82
- - !ruby/object:Gem::Version
83
- version: '0'
84
- description: More efficient scheduled job queue implementation for sidekiq to increase
85
- throughput in large installations.
41
+ description:
86
42
  email:
87
- - dev@weheartit.com
88
43
  - bbdurand@gmail.com
89
44
  executables: []
90
45
  extensions: []
91
46
  extra_rdoc_files: []
92
47
  files:
93
- - ".gitignore"
94
- - HISTORY.txt
48
+ - CHANGELOG.md
95
49
  - MIT_LICENSE.txt
96
50
  - README.md
97
- - Rakefile
98
51
  - VERSION
99
52
  - lib/sidekiq-fast-enq.rb
100
53
  - sidekiq_fast_enq.gemspec
101
- - spec/sidekiq-fast-enq_spec.rb
102
- - spec/spec_helper.rb
103
- homepage: https://github.com/weheartit/sidekiq_fast_enq
54
+ homepage: https://github.com/bdurand/sidekiq-fast-enq
104
55
  licenses:
105
56
  - MIT
106
57
  metadata: {}
107
- post_install_message:
58
+ post_install_message:
108
59
  rdoc_options: []
109
60
  require_paths:
110
61
  - lib
@@ -112,18 +63,16 @@ required_ruby_version: !ruby/object:Gem::Requirement
112
63
  requirements:
113
64
  - - ">="
114
65
  - !ruby/object:Gem::Version
115
- version: '0'
66
+ version: '2.5'
116
67
  required_rubygems_version: !ruby/object:Gem::Requirement
117
68
  requirements:
118
69
  - - ">="
119
70
  - !ruby/object:Gem::Version
120
71
  version: '0'
121
72
  requirements: []
122
- rubyforge_project:
123
- rubygems_version: 2.4.5
124
- signing_key:
73
+ rubygems_version: 3.4.12
74
+ signing_key:
125
75
  specification_version: 4
126
- summary: More efficient scheduled job queue implementation for sidekiq
127
- test_files:
128
- - spec/sidekiq-fast-enq_spec.rb
129
- - spec/spec_helper.rb
76
+ summary: More efficient scheduled job queue implementation for sidekiq to increase
77
+ throughput in large installations.
78
+ test_files: []
data/.gitignore DELETED
@@ -1,17 +0,0 @@
1
- *.gem
2
- *.rbc
3
- .bundle
4
- .config
5
- .yardoc
6
- Gemfile.lock
7
- InstalledFiles
8
- _yardoc
9
- coverage
10
- doc/
11
- lib/bundler/man
12
- pkg
13
- rdoc
14
- spec/reports
15
- test/tmp
16
- test/version_tmp
17
- tmp
data/HISTORY.txt DELETED
@@ -1,7 +0,0 @@
1
- 1.0.1
2
-
3
- Update gemspec to specify compatibility with sidekiq 4.
4
-
5
- 1.0.0
6
-
7
- Initial release.
data/Rakefile DELETED
@@ -1,57 +0,0 @@
1
- require "bundler/gem_tasks"
2
-
3
- desc 'Default: run unit tests.'
4
- task :default => :test
5
-
6
- desc 'RVM likes to call it tests'
7
- task :tests => :test
8
-
9
- begin
10
- require 'rspec'
11
- require 'rspec/core/rake_task'
12
- desc 'Run the unit tests'
13
- RSpec::Core::RakeTask.new(:test)
14
- rescue LoadError
15
- task :test do
16
- STDERR.puts "You must have rspec 2.0 installed to run the tests"
17
- end
18
- end
19
-
20
- task :load_test, [:jobs_size, :workers, :fast] do |t, args|
21
- require 'celluloid'
22
- require File.expand_path('../lib/sidekiq-fast-enq', __FILE__)
23
- require 'sidekiq/scheduled'
24
- require 'sidekiq/api'
25
-
26
- class FastEnqLoadTestWorker
27
- include Sidekiq::Worker
28
- def perform()
29
- end
30
- end
31
-
32
- jobs_size = args[:jobs_size].to_i
33
- workers_size = args[:workers].to_i
34
- klass = (args[:fast] == 'fast' ? SidekiqFastEnq : Sidekiq::Scheduled::Enq)
35
-
36
- Sidekiq.configure_server do |config|
37
- config.redis = {:namespace => "sidekiq_fast_enq_load_test"}
38
- end
39
-
40
- Sidekiq::ScheduledSet.new.clear
41
- jobs_size.times do
42
- FastEnqLoadTestWorker.perform_in(rand)
43
- end
44
-
45
- t = Time.now
46
- workers_size.times do
47
- fork do
48
- klass.new.enqueue_jobs
49
- end
50
- end
51
-
52
- workers_size.times do
53
- Process.wait
54
- end
55
-
56
- puts "Enqueued #{jobs_size} jobs in #{Time.now - t} seconds"
57
- end
@@ -1,53 +0,0 @@
1
- require 'spec_helper'
2
-
3
- describe SidekiqFastEnq do
4
-
5
- let(:scheduled_set){ Sidekiq::ScheduledSet.new }
6
- let(:retry_set){ Sidekiq::RetrySet.new }
7
- let(:default_queue){ Sidekiq::Queue.new }
8
-
9
- before :each do
10
- scheduled_set.clear
11
- retry_set.clear
12
- default_queue.clear
13
- end
14
-
15
- it "should return without doing anything if there are no scheduled jobs" do
16
- SidekiqFastEnq.new.enqueue_jobs
17
- expect(scheduled_set.size).to eq(0)
18
- expect(retry_set.size).to eq(0)
19
- expect(default_queue.size).to eq(0)
20
- end
21
-
22
- it "should enqueue a single elligible job from the scheduled jobs queue" do
23
- Timecop.travel(Time.now - 3600){ FastEnqTestWorker.perform_in(60, 'one') }
24
- SidekiqFastEnq.new.enqueue_jobs
25
- expect(scheduled_set.size).to eq(0)
26
- expect(retry_set.size).to eq(0)
27
- expect(default_queue.size).to eq(1)
28
- end
29
-
30
- it "should enqueue all elligible jobs from the scheduled jobs queue" do
31
- Timecop.travel(Time.now - 3600){ FastEnqTestWorker.perform_in(60, 'one') }
32
- Timecop.travel(Time.now - 3600){ FastEnqTestWorker.perform_in(900, 'two') }
33
- FastEnqTestWorker.perform_in(10, 'three')
34
- SidekiqFastEnq.new.enqueue_jobs
35
- expect(scheduled_set.size).to eq(1)
36
- expect(retry_set.size).to eq(0)
37
- expect(default_queue.size).to eq(2)
38
- end
39
-
40
- it "should enqueue all elligible jobs from the scheduled jobs queue when there are a lot of them" do
41
- Timecop.travel(Time.now - 3600) do
42
- 200.times do
43
- FastEnqTestWorker.perform_in(60, 'one')
44
- end
45
- end
46
- FastEnqTestWorker.perform_in(10, 'three')
47
- SidekiqFastEnq.new.enqueue_jobs
48
- expect(scheduled_set.size).to eq(1)
49
- expect(retry_set.size).to eq(0)
50
- expect(default_queue.size).to eq(200)
51
- end
52
-
53
- end
data/spec/spec_helper.rb DELETED
@@ -1,36 +0,0 @@
1
- # Breaks if not required. Sidekiq doesn't directly require in
2
- # the load process.
3
-
4
- sidekiq_version = Array(ENV["SIDEKIQ_VERSION"] || ">=3.4")
5
- gem 'sidekiq', *sidekiq_version
6
-
7
- require File.expand_path('../../lib/sidekiq-fast-enq', __FILE__)
8
- require 'timecop'
9
- require 'sidekiq/version'
10
- require 'celluloid' if Sidekiq::VERSION.to_i < 4
11
- require 'sidekiq/scheduled'
12
- require 'sidekiq/api'
13
-
14
- RSpec.configure do |config|
15
- config.run_all_when_everything_filtered = true
16
- config.filter_run :focus
17
-
18
- # Run specs in random order to surface order dependencies. If you find an
19
- # order dependency and want to debug it, you can fix the order by providing
20
- # the seed, which is printed after each run.
21
- # --seed 1234
22
- config.order = 'random'
23
-
24
- Sidekiq.configure_server do |config|
25
- config.redis = {:namespace => "sidekiq_fast_enq_test"}
26
- end
27
- Sidekiq.options[:scheduled_enq] = SidekiqFastEnq
28
- Sidekiq::Logging.logger = Logger.new(StringIO.new)
29
- end
30
-
31
- class FastEnqTestWorker
32
- include Sidekiq::Worker
33
-
34
- def perform(arg)
35
- end
36
- end