resqutils 0.0.1 → 1.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: a63b6baf7b6534578500e949ccb71c6db6494794
4
- data.tar.gz: 1a59736bd38007a380c5935e089125cbec529840
3
+ metadata.gz: 63c56aed110c280ee955278e9d007f242482e15e
4
+ data.tar.gz: 07eac3dc61ff36df991829d3002d805da88bd3d3
5
5
  SHA512:
6
- metadata.gz: 16ca3aff7ae9da98e8a61295742e344fbbc76c139eeb309a13e0fd3a66c47644a7f6a4ec2396c242a10d460a13e6de91efbbe503f464ba82990174f1d8daa941
7
- data.tar.gz: 627e1e5b9a6f9e60854a41ee261270c625392dfc446b5dfe20ee68510b37a85df47deb1c82a15fb6ac57b2c6da8e6b269a088943e8acb82843e264eb304c07aa
6
+ metadata.gz: 91fc61241a2cd1c0b623f9473fab38ecfcc19f1d7b83c92a1c5c91f72e62c8378ce614950e530f7b535987dec8f3029f62cef0c1e573d139f6d3e8dc885481c2
7
+ data.tar.gz: 2f5f870492b27271bd29ebc34e4b9d6f24c0117dcf443a479175ca545a514cb881352168e12095a807ce54670f8af685a6388bca85cdf18bb07c6d86fa7b90d7
@@ -1,7 +1,7 @@
1
1
  PATH
2
2
  remote: .
3
3
  specs:
4
- resqutils (0.0.1)
4
+ resqutils (1.0.0)
5
5
  resque
6
6
 
7
7
  GEM
data/README.md CHANGED
@@ -4,12 +4,14 @@ This is a small library of useful modules and functions that can help dealing wi
4
4
 
5
5
  Currently:
6
6
 
7
+ * Job that kills stale workers
8
+ * Means to identify stale workers
7
9
  * Spec helper `:some_queue.should have_job_queued(class: FooJob)`
8
10
  * Methods to introspect queues, including the delayed queue, in your specs
9
11
  * Simple `resque:work` task wrapper to better handle exceptions in the worker
10
12
  * Marker interface to document jobs which should not be retried
11
13
 
12
- Maybe will have more stuff1
14
+ Maybe will have more stuff someday.
13
15
 
14
16
  ## To use
15
17
 
@@ -19,6 +21,30 @@ Add to your `Gemfile`:
19
21
  gem 'resqutils'
20
22
  ```
21
23
 
24
+ ## Stale Workers
25
+
26
+ It's possible (and, on Heroku, highly likely) that your jobs will appear to be running for "too long". Usually, this happens
27
+ when a worker exits without cleaning up after itself. Since Resque stores all state in Redis, and is process-based, it's
28
+ actually fairly easy to create this situation.
29
+
30
+ The good news is that, if your jobs are idempotent, you can just unregister the "stale" workers, which will kick-off the
31
+ failed handling (which is hopefully to restart your jobs).
32
+
33
+ You need a means of identifying these workers, and then killing them.
34
+
35
+ ```ruby
36
+ Resqutils::StaleWorkers.new.each do |worker|
37
+ # this worker is still considered running but has started over an hour ago
38
+ Resque.enqueue(WorkerKillerJob,worker.id)
39
+ end
40
+ ```
41
+
42
+ `Resqutils::StaleWorkers`'s default of an hour can be overridden either in the constructor or by setting the
43
+ `RESQUTILS_SECONDS_TO_BE_CONSIDERED_STALE` environment variable.
44
+
45
+ The queue that `WorkerKillerJob` will queue to is `worker_killer_job` by default, but can either be set during the `enqueue`
46
+ call or by setting the `RESQUTILS_WORKER_KILLER_JOB_QUEUE` environment variable.
47
+
22
48
  ## Spec Helpers
23
49
 
24
50
  ```ruby
@@ -35,6 +61,19 @@ end
35
61
 
36
62
  `require`ing the `resqutils/spec` will also set up the `have_job_queued` matcher, which is likely what you'll want to use.
37
63
 
64
+ ### Clearing Jobs
65
+
66
+ The most important part of using Resque in tests as making sure the queue has what you
67
+ think it has in it. To that end, you'll likely need `clear_queue` in a `setup` or
68
+ `before` block.
69
+
70
+ ```ruby
71
+ before do
72
+ clear_queue(MyImportantJob) # clears whatever queue this job is configured to use
73
+ clear_queue(:foobar) # clear the "foobar" queue
74
+ end
75
+ ```
76
+
38
77
  ### Checking that Jobs Were Queued
39
78
 
40
79
  ```ruby
@@ -48,6 +87,12 @@ end
48
87
 
49
88
  # foo_service_spec.rb
50
89
  describe FooService do
90
+ include Resqutils::Spec::ResqueHelpers
91
+
92
+ before do
93
+ clear_queue(FooJob) # Looks at what queue FooJob uses and clears before each test
94
+ end
95
+
51
96
  it "queues a job" do
52
97
  result = FooService.new.doit("blah")
53
98
 
@@ -71,6 +116,11 @@ end
71
116
  # foo_service_spec.rb
72
117
 
73
118
  describe FooService do
119
+ include Resqutils::Spec::ResqueHelpers
120
+
121
+ before do
122
+ clear_queue(:delayed) # Clears all delayed/scheduled queues
123
+ end
74
124
  it "queues a job" do
75
125
  result = FooService.new.doit("blah")
76
126
 
data/Rakefile CHANGED
@@ -5,9 +5,12 @@ rescue LoadError
5
5
  end
6
6
  require 'rubygems/package_task'
7
7
  require 'rspec/core/rake_task'
8
+ RSpec::Core::RakeTask.new(:spec)
8
9
 
9
10
  require 'rdoc/task'
10
11
 
12
+ include Rake::DSL
13
+
11
14
  RDoc::Task.new(:rdoc) do |rdoc|
12
15
  rdoc.rdoc_dir = 'rdoc'
13
16
  rdoc.title = 'ExtraExtra'
@@ -2,4 +2,6 @@ module Resqutils
2
2
  end
3
3
 
4
4
  require 'resqutils/do_not_auto_retry'
5
+ require 'resqutils/worker_killer_job'
6
+ require 'resqutils/stale_workers'
5
7
  require 'resqutils/version'
@@ -0,0 +1,59 @@
1
+ module Resqutils
2
+ # Vends stale workers that have been running "too long"
3
+ class StaleWorkers
4
+ include Enumerable
5
+
6
+ # Create a StaleWorkers instance.
7
+ #
8
+ # seconds_to_be_considered_stale:: if present, this is the number of seconds a worker will have to have been
9
+ # running to be considered stale
10
+ def initialize(seconds_to_be_considered_stale = 3600)
11
+ @seconds_to_be_considered_stale = seconds_to_be_considered_stale_from_env! || seconds_to_be_considered_stale
12
+ end
13
+
14
+ # Yield all currently stale workers. The yielded objects are Resque's representation, which is not
15
+ # well documented, however you can reasonably assume it will respond to #id, #queue, and #run_at
16
+ def each(&block)
17
+ if block.nil?
18
+ stale_workers.to_enum
19
+ else
20
+ stale_workers.each(&block)
21
+ end
22
+ end
23
+
24
+ private
25
+
26
+ def stale_workers
27
+ Resque.workers.map(&self.method(:worker_with_start_time)).select(&:stale?).map(&:worker)
28
+ end
29
+
30
+ def worker_with_start_time(worker)
31
+ WorkerWithStartTime.new(worker,@seconds_to_be_considered_stale)
32
+ end
33
+
34
+ def seconds_to_be_considered_stale_from_env!
35
+ seconds_to_be_considered_stale = String(ENV["RESQUTILS_SECONDS_TO_BE_CONSIDERED_STALE"])
36
+ if seconds_to_be_considered_stale.strip.length == 0
37
+ nil
38
+ elsif seconds_to_be_considered_stale.to_i == 0
39
+ raise "You set a stale value of 0 seconds, making all jobs stale; probably not what you want"
40
+ else
41
+ seconds_to_be_considered_stale.to_i
42
+ end
43
+ end
44
+
45
+ class WorkerWithStartTime
46
+ attr_reader :worker
47
+ def initialize(worker,seconds_to_be_considered_stale)
48
+ @worker = worker
49
+ @seconds_to_be_considered_stale = seconds_to_be_considered_stale
50
+ @start_time = Time.parse(worker.job["run_at"]) rescue nil
51
+ end
52
+
53
+ def stale?
54
+ return false if @start_time.nil?
55
+ @start_time <= (Time.now - @seconds_to_be_considered_stale)
56
+ end
57
+ end
58
+ end
59
+ end
@@ -1,3 +1,3 @@
1
1
  module Resqutils
2
- VERSION='0.0.1'
2
+ VERSION='1.0.0'
3
3
  end
@@ -0,0 +1,16 @@
1
+ module Resqutils
2
+ # Can be queued to kill a worker. By default, will be queued to the 'worker_killer_job' queue, however
3
+ # you can specify RESQUTILS_WORKER_KILLER_JOB_QUEUE in the environment to set an override. Of course, you can always
4
+ # forcibly enqueue it as needed.
5
+ class WorkerKillerJob
6
+ def self.queue
7
+ @queue ||= begin
8
+ queue = String(ENV["RESQUTILS_WORKER_KILLER_JOB_QUEUE"]).strip
9
+ queue.length == 0 ? :worker_killer_job : queue
10
+ end
11
+ end
12
+ def self.perform(worker_id)
13
+ Resque.workers.detect { |_| _.id == worker_id }.unregister_worker
14
+ end
15
+ end
16
+ end
@@ -1,3 +1,6 @@
1
+ require 'resque'
2
+ require 'resqutils'
3
+
1
4
  GEM_ROOT = File.expand_path(File.join(File.dirname(__FILE__),'..'))
2
5
  Dir["#{GEM_ROOT}/spec/support/**/*.rb"].sort.each {|f| require f}
3
6
 
@@ -0,0 +1,148 @@
1
+ require 'spec_helper'
2
+
3
+ describe Resqutils::StaleWorkers do
4
+ describe "each" do
5
+ let(:workers) {
6
+ [
7
+ worker,
8
+ worker(Time.now - 3601),
9
+ worker(Time.now - 5401),
10
+ worker,
11
+ worker(Time.now - 7201),
12
+ ]
13
+ }
14
+
15
+ before do
16
+ ENV.delete("RESQUTILS_SECONDS_TO_BE_CONSIDERED_STALE")
17
+ allow(Resque).to receive(:workers).and_return(workers)
18
+ end
19
+
20
+ context "default stale seconds" do
21
+
22
+ context "properly formed workers" do
23
+ subject(:stale_workers) { described_class.new }
24
+
25
+ context "with block" do
26
+ it "yields each stale worker" do
27
+ stale = []
28
+ stale_workers.each do |stale_worker|
29
+ stale << stale_worker
30
+ end
31
+ expect(stale.size).to eq(3)
32
+ expect(stale).to include(workers[1])
33
+ expect(stale).to include(workers[2])
34
+ expect(stale).to include(workers[4])
35
+ end
36
+ end
37
+ context "without block" do
38
+ it "returns an enumerator we can use" do
39
+ stale_worker_ids = stale_workers.each.map { |stale_worker|
40
+ stale_worker.id
41
+ }
42
+ expect(stale_worker_ids.size).to eq(3)
43
+ expect(stale_worker_ids).to include(workers[1].id)
44
+ expect(stale_worker_ids).to include(workers[2].id)
45
+ expect(stale_worker_ids).to include(workers[4].id)
46
+ end
47
+ end
48
+ end
49
+ context "some mangled workers" do
50
+
51
+ let(:workers) {
52
+ [
53
+ worker,
54
+ worker(Time.now - 3601),
55
+ worker("mangled time"),
56
+ "blah",
57
+ Object.new,
58
+ ]
59
+ }
60
+
61
+ subject(:stale_workers) { described_class.new }
62
+
63
+ it "yields each umangled stale worker" do
64
+ stale = []
65
+ stale_workers.each do |stale_worker|
66
+ stale << stale_worker
67
+ end
68
+ expect(stale.size).to eq(1)
69
+ expect(stale).to include(workers[1])
70
+ end
71
+ end
72
+ end
73
+ context "customized stale seconds" do
74
+ subject(:stale_workers) { described_class.new(7100) }
75
+
76
+ context "with block" do
77
+ it "yields each stale worker" do
78
+ stale = []
79
+ stale_workers.each do |stale_worker|
80
+ stale << stale_worker
81
+ end
82
+ expect(stale.size).to eq(1)
83
+ expect(stale).to include(workers[4])
84
+ end
85
+ end
86
+ context "without block" do
87
+ it "returns an enumerator we can use" do
88
+ stale_worker_ids = stale_workers.each.map { |stale_worker|
89
+ stale_worker.id
90
+ }
91
+ expect(stale_worker_ids.size).to eq(1)
92
+ expect(stale_worker_ids).to include(workers[4].id)
93
+ end
94
+ end
95
+ end
96
+ context "using the environment" do
97
+ context "with a sane value" do
98
+ before do
99
+ ENV["RESQUTILS_SECONDS_TO_BE_CONSIDERED_STALE"] = "5400"
100
+ end
101
+ subject(:stale_workers) { described_class.new(7100) }
102
+
103
+ context "with block" do
104
+ it "yields each stale worker" do
105
+ stale = []
106
+ stale_workers.each do |stale_worker|
107
+ stale << stale_worker
108
+ end
109
+ expect(stale.size).to eq(2)
110
+ expect(stale).to include(workers[2])
111
+ expect(stale).to include(workers[4])
112
+ end
113
+ end
114
+ context "without block" do
115
+ it "returns an enumerator we can use" do
116
+ stale_worker_ids = stale_workers.each.map { |stale_worker|
117
+ stale_worker.id
118
+ }
119
+ expect(stale_worker_ids.size).to eq(2)
120
+ expect(stale_worker_ids).to include(workers[2].id)
121
+ expect(stale_worker_ids).to include(workers[4].id)
122
+ end
123
+ end
124
+ end
125
+
126
+ context "erroneous values" do
127
+ it "does not like 0" do
128
+ ENV["RESQUTILS_SECONDS_TO_BE_CONSIDERED_STALE"] = "0"
129
+ expect {
130
+ described_class.new
131
+ }.to raise_error(/you set a stale value of 0 seconds, making all jobs stale.*probably not what you want/i)
132
+ end
133
+ it "does not like floats that are 0" do
134
+ ENV["RESQUTILS_SECONDS_TO_BE_CONSIDERED_STALE"] = "0.0000001"
135
+ expect {
136
+ described_class.new
137
+ }.to raise_error(/you set a stale value of 0 seconds, making all jobs stale.*probably not what you want/i)
138
+ end
139
+ end
140
+ end
141
+ end
142
+
143
+ def worker(run_at = Time.now)
144
+ double("resque worker", id: SecureRandom.uuid, job: { "queue" => "whatever",
145
+ "run_at" => run_at.to_s,
146
+ "payload" => { "class" => "Foo", "args" => [] } })
147
+ end
148
+ end
@@ -0,0 +1,62 @@
1
+ require 'spec_helper'
2
+ require 'securerandom'
3
+
4
+ describe Resqutils::WorkerKillerJob do
5
+ describe "queue" do
6
+ before do
7
+ ENV.delete("RESQUTILS_WORKER_KILLER_JOB_QUEUE")
8
+ # since we are memoizing it on the class
9
+ described_class.instance_variable_set("@queue",nil)
10
+ end
11
+
12
+ after do
13
+ ENV.delete("RESQUTILS_WORKER_KILLER_JOB_QUEUE")
14
+ end
15
+
16
+ it "uses the worker_killer_job queue by default" do
17
+ expect(Resque.queue_from_class(described_class)).to eq(:worker_killer_job)
18
+ end
19
+
20
+ it "can use a different queue if specified by the environment" do
21
+ ENV["RESQUTILS_WORKER_KILLER_JOB_QUEUE"] = "foobar"
22
+ expect(Resque.queue_from_class(described_class)).to eq("foobar")
23
+ end
24
+ end
25
+ describe "::perform" do
26
+ let(:workers) {
27
+ [
28
+ double(id: worker_id),
29
+ double(id: worker_id),
30
+ double(id: worker_id),
31
+ double(id: worker_id),
32
+ double(id: worker_id),
33
+ double(id: worker_id),
34
+ ]
35
+ }
36
+
37
+ let(:worker_to_kill) { workers[2] }
38
+
39
+ before do
40
+ allow(Resque).to receive(:workers).and_return(workers)
41
+ workers.each do |worker|
42
+ allow(worker).to receive(:unregister_worker)
43
+ end
44
+ described_class.perform(worker_to_kill.id)
45
+ end
46
+
47
+ it "unregisters the worker we want to kill" do
48
+ expect(worker_to_kill).to have_received(:unregister_worker)
49
+ end
50
+
51
+ it "doesn't touch the other workers" do
52
+ workers.reject { |_| _ == worker_to_kill }.each do |worker|
53
+ expect(worker).not_to have_received(:unregister_worker)
54
+ end
55
+ end
56
+
57
+ end
58
+
59
+ def worker_id
60
+ "#{SecureRandom.uuid}:#{rand(100)}:some_event"
61
+ end
62
+ end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: resqutils
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.0.1
4
+ version: 1.0.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Stitch Fix Engineering
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2014-09-19 00:00:00.000000000 Z
11
+ date: 2015-02-04 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: resque
@@ -72,10 +72,14 @@ files:
72
72
  - lib/resqutils/spec.rb
73
73
  - lib/resqutils/spec/resque_helpers.rb
74
74
  - lib/resqutils/spec/resque_matchers.rb
75
+ - lib/resqutils/stale_workers.rb
75
76
  - lib/resqutils/version.rb
77
+ - lib/resqutils/worker_killer_job.rb
76
78
  - lib/resqutils/worker_task.rb
77
79
  - resqutils.gemspec
78
80
  - spec/spec_helper.rb
81
+ - spec/stale_workers_spec.rb
82
+ - spec/worker_killer_job_spec.rb
79
83
  homepage: http://tech.stitchfix.com
80
84
  licenses: []
81
85
  metadata: {}
@@ -95,9 +99,11 @@ required_rubygems_version: !ruby/object:Gem::Requirement
95
99
  version: '0'
96
100
  requirements: []
97
101
  rubyforge_project: resqutils
98
- rubygems_version: 2.2.0.rc.1
102
+ rubygems_version: 2.4.2
99
103
  signing_key:
100
104
  specification_version: 4
101
105
  summary: Utilities for using Resque in a Rails app
102
106
  test_files:
103
107
  - spec/spec_helper.rb
108
+ - spec/stale_workers_spec.rb
109
+ - spec/worker_killer_job_spec.rb