sucker_punch 1.6.0 → 2.0.0.beta1

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: d6ccd098769b0b630161eb92038c13e662a58786
4
- data.tar.gz: 0a99540886529b095b99a745514b54cf49a75cf2
3
+ metadata.gz: fa1e2aa3c637a849298159655805260a18ce9315
4
+ data.tar.gz: 2a138d958c61e7c1e014ff7e9d682e8c9a5f78e4
5
5
  SHA512:
6
- metadata.gz: b9d74d6377d9abc828a43333460966ab7779499c75ece7fe78c68f9b1002572322da70754ed7032ccc963ceac797aa2cb7df0ac77a19b80e8ad13f559bb5001f
7
- data.tar.gz: 83a79b2a2ecdb619f781d8dcb2b2e50d6cea08d3a34e134b1c13f29fa129b7e4e92eb671bf4fe43b37788386690bf9273fa85f3ccb3a795536238e17d3a289e7
6
+ metadata.gz: 03d99e04ebd2fc6aaa07e91242673411727a5cbc50d350d3b83a9a2b28fb81ec90bc26eb7c4d83b3f840da0881928b784b3c07e077205ad1d41333d627dac5ed
7
+ data.tar.gz: 588214f4b4d233352bf06ae84ad6b2948e575e2b089c7b7fe9b39d9dceecccc4b9f189fd24a6945119b4ffb2cdbd2962529ba43779b213aef65b297f60790fca
data/.travis.yml CHANGED
@@ -1,12 +1,24 @@
1
1
  language: ruby
2
2
  rvm:
3
- - 1.9.3
4
- - jruby-19mode
5
3
  - rbx-2
6
4
  - 2.0.0
7
5
  - 2.1.0
6
+ - 2.2.0
7
+ - 2.3.0
8
8
  - ruby-head
9
+ - jruby-9.0.1.0
10
+ - jruby-9.0.3.0
11
+ - jruby-9.0.4.0
12
+ - jruby-head
13
+
14
+ jdk:
15
+ - oraclejdk8
16
+
9
17
  matrix:
10
18
  allow_failures:
11
- - rvm: jruby-19mode
12
19
  - rvm: rbx-2
20
+ - rvm: jruby-head
21
+
22
+ before_script:
23
+ - "echo $JAVA_OPTS"
24
+ - "export JAVA_OPTS=-Xmx1024m"
data/CHANGES.md CHANGED
@@ -1,3 +1,33 @@
1
+ 2.0.0.beta1
2
+ -------
3
+
4
+ - Refactor internals to use `concurrent-ruby`
5
+ - Yield more exception details to handler. The new syntax allows you to setup a
6
+ global exception with the following syntax:
7
+
8
+ ```ruby
9
+ # ex => The caught exception object
10
+ # klass => The job class
11
+ # args => An array of the args passed to the job
12
+
13
+ SuckerPunch.exception_handler = -> (ex, klass, args) { ExceptionNotifier.notify_exception(ex) }
14
+ ```
15
+
16
+ - Invoke asynchronous job via `perform_async` and `perform_in` class method (*backwards
17
+ incompatible change*):
18
+
19
+ ```ruby
20
+ LogJob.perform_async("login")
21
+ LogJob.perform_in(60, "login") # `perform` will be executed 60 sec. later
22
+ ```
23
+
24
+ - Drop support for Ruby `< 2.0`
25
+ - Allow shutdown timeout to be set:
26
+
27
+ ```ruby
28
+ SuckerPunch.shutdown_timeout = 15 # time in seconds
29
+ ```
30
+
1
31
  1.6.0
2
32
  --------
3
33
 
data/README.md CHANGED
@@ -1,20 +1,29 @@
1
+ # Notice: This README is for the `master` branch (`v2 beta`), not the latest stable branch
2
+
1
3
  # Sucker Punch
2
4
 
3
5
  [![Build Status](https://travis-ci.org/brandonhilkert/sucker_punch.png?branch=master)](https://travis-ci.org/brandonhilkert/sucker_punch)
4
6
  [![Code Climate](https://codeclimate.com/github/brandonhilkert/sucker_punch.png)](https://codeclimate.com/github/brandonhilkert/sucker_punch)
5
7
 
6
- Sucker Punch is a single-process Ruby asynchronous processing library. It's [girl_friday](https://github.com/mperham/girl_friday) and DSL sugar on top of [Celluloid](https://github.com/celluloid/celluloid/). With Celluloid's actor pattern, we can do asynchronous processing within a single process. This reduces costs of hosting on a service like Heroku along with the memory footprint of having to maintain additional jobs if hosting on a dedicated server. All queues can run within a single Rails/Sinatra process.
8
+ Sucker Punch is a single-process Ruby asynchronous processing library.
9
+ This reduces costs
10
+ of hosting on a service like Heroku along with the memory footprint of
11
+ having to maintain additional jobs if hosting on a dedicated server.
12
+ All queues can run within a single application (eg. Rails, Sinatra, etc.) process.
7
13
 
8
- Sucker Punch is perfect for asynchronous processes like emailing, data crunching, or social platform manipulation. No reason to hold up a user when you can do these things in the background within the same process as your web application...
14
+ Sucker Punch is perfect for asynchronous processes like emailing, data
15
+ crunching, or social platform manipulation. No reason to hold up a
16
+ user when you can do these things in the background within the same
17
+ process as your web application...
9
18
 
10
- Sucker Punch is built on top of [Celluloid
11
- Pools](https://github.com/celluloid/celluloid/wiki/Pools). Each job is setup as
19
+ Sucker Punch is built on top of [concurrent-ruby]
20
+ (https://github.com/ruby-concurrency/concurrent-ruby). Each job is setup as
12
21
  a pool, which equates to its own queue with individual workers working against
13
22
  the jobs. Unlike most other background processing libraries, Sucker
14
23
  Punch's jobs are stored in memory. The benefit to this is there is no
15
- additional infrastructure requirement (ie. database, redis, etc.). The downside
16
- is that if the web processes is restarted and there are jobs that haven't yet
17
- been processed, they will be lost. For this reason, Sucker Punch is generally
24
+ additional infrastructure requirement (ie. database, redis, etc.). However,
25
+ if the web processes are restarted with jobs remaining in the queue,
26
+ they will be lost. For this reason, Sucker Punch is generally
18
27
  recommended for jobs that are fast and non-mission critical (ie. logs, emails,
19
28
  etc.).
20
29
 
@@ -22,7 +31,7 @@ etc.).
22
31
 
23
32
  Add this line to your application's Gemfile:
24
33
 
25
- gem 'sucker_punch', '~> 1.0'
34
+ gem 'sucker_punch', '~> 2.0'
26
35
 
27
36
  And then execute:
28
37
 
@@ -37,7 +46,7 @@ Or install it yourself as:
37
46
  Each job acts as its own queue and should be a separate Ruby class that:
38
47
 
39
48
  * includes `SuckerPunch::Job`
40
- * defines the instance method `perform` that includes the code the job will run when enqueued
49
+ * defines the `perform` instance method that includes the code the job will run when enqueued
41
50
 
42
51
 
43
52
  ```Ruby
@@ -52,90 +61,94 @@ class LogJob
52
61
  end
53
62
  ```
54
63
 
55
- Synchronous:
64
+ #### Synchronous
56
65
 
57
66
  ```Ruby
58
67
  LogJob.new.perform("login")
59
68
  ```
60
69
 
61
- Asynchronous:
70
+ #### Asynchronous
62
71
 
63
72
  ```Ruby
64
- LogJob.new.async.perform("login") # => nil
73
+ LogJob.perform_async("login")
65
74
  ```
66
75
 
67
- Jobs interacting with `ActiveRecord` should take special precaution not to exhaust connections in the pool. This can be done with `ActiveRecord::Base.connection_pool.with_connection`, which ensures the connection is returned back to the pool when completed.
76
+ #### Configure the # of the Workers
68
77
 
69
- ```Ruby
70
- # app/jobs/awesome_job.rb
78
+ The default number of workers (threads) running against your job is `2`. If
79
+ you'd like to configure this manually, the number of workers can be
80
+ set on the job using the `workers` class method:
71
81
 
72
- class AwesomeJob
82
+ ```Ruby
83
+ class LogJob
73
84
  include SuckerPunch::Job
85
+ workers 4
74
86
 
75
- def perform(user_id)
76
- ActiveRecord::Base.connection_pool.with_connection do
77
- user = User.find(user_id)
78
- user.update_attributes(is_awesome: true)
79
- end
87
+ def perform(event)
88
+ Log.new(event).track
80
89
  end
81
90
  end
82
91
  ```
83
92
 
84
- We can create a job from within another job:
93
+ #### Executing Jobs in the Future
85
94
 
86
- ```Ruby
87
- class AwesomeJob
95
+ Many background processing libraries have methods to perform operations after a
96
+ certain amount of time and Sucker Punch is no different. Use the `perform_in`
97
+ with an argument of the number of seconds in the future you would like the job
98
+ to job to run.
99
+
100
+ ``` ruby
101
+ class DataJob
88
102
  include SuckerPunch::Job
89
103
 
90
- def perform(user_id)
91
- ActiveRecord::Base.connection_pool.with_connection do
92
- user = User.find(user_id)
93
- user.update_attributes(is_awesome: true)
94
- LogJob.new.async.perform("User #{user.id} became awesome!")
95
- end
104
+ def perform(data)
105
+ puts data
96
106
  end
97
107
  end
108
+
109
+ DataJob.perform_async("asdf") # immediately perform asynchronously
110
+ DataJob.perform_in(60, "asdf") # `perform` will be excuted 60 sec. later
98
111
  ```
99
112
 
100
- The number of workers can be set from the Job using the `workers` method:
113
+ #### `ActiveRecord` Connection Pool Connections
114
+
115
+ Jobs interacting with `ActiveRecord` should take special precaution not to
116
+ exhaust connections in the pool. This can be done
117
+ with `ActiveRecord::Base.connection_pool.with_connection`, which ensures
118
+ the connection is returned back to the pool when completed.
101
119
 
102
120
  ```Ruby
103
- class LogJob
121
+ # app/jobs/awesome_job.rb
122
+
123
+ class AwesomeJob
104
124
  include SuckerPunch::Job
105
- workers 4
106
125
 
107
- def perform(event)
108
- Log.new(event).track
126
+ def perform(user_id)
127
+ ActiveRecord::Base.connection_pool.with_connection do
128
+ user = User.find(user_id)
129
+ user.update(is_awesome: true)
130
+ end
109
131
  end
110
132
  end
111
133
  ```
112
134
 
113
- If the `workers` method is not set, the default is `2`.
114
-
115
- ## Perform In
116
-
117
- Many background processing libraries have methods to perform operations after a
118
- certain amount of time. Fortunately, timers are built-in to Celluloid, so you
119
- can take advantage of them with the `later` method:
135
+ We can create a job from within another job:
120
136
 
121
- ``` ruby
122
- class Job
137
+ ```Ruby
138
+ class AwesomeJob
123
139
  include SuckerPunch::Job
124
140
 
125
- def perform(data)
126
- puts data
127
- end
128
-
129
- def later(sec, data)
130
- after(sec) { perform(data) }
141
+ def perform(user_id)
142
+ ActiveRecord::Base.connection_pool.with_connection do
143
+ user = User.find(user_id)
144
+ user.update_attributes(is_awesome: true)
145
+ LogJob.perform_async("User #{user.id} became awesome!")
146
+ end
131
147
  end
132
148
  end
133
-
134
- Job.new.async.perform("asdf")
135
- Job.new.async.later(60, "asdf") # `perform` will be excuted 60 sec. later
136
149
  ```
137
150
 
138
- ## Logger
151
+ #### Logger
139
152
 
140
153
  ```Ruby
141
154
  SuckerPunch.logger = Logger.new('sucker_punch.log')
@@ -145,43 +158,57 @@ SuckerPunch.logger # => #<Logger:0x007fa1f28b83f0>
145
158
  _Note: If Sucker Punch is being used within a Rails application, Sucker Punch's logger
146
159
  is set to Rails.logger by default._
147
160
 
148
- ## Exceptions
161
+ #### Exceptions
149
162
 
150
163
  You can customize how to handle uncaught exceptions that are raised by your jobs.
151
164
 
152
- For example, using Rails and the ExceptionNotification gem, add a new initializer `config/initializers/sucker_punch.rb`:
165
+ For example, using Rails and the ExceptionNotification gem,
166
+ add a new initializer `config/initializers/sucker_punch.rb`:
153
167
 
154
168
  ```Ruby
155
- SuckerPunch.exception_handler { |ex| ExceptionNotifier.notify_exception(ex) }
169
+ # ex => The caught exception object
170
+ # klass => The job class
171
+ # args => An array of the args passed to the job
172
+
173
+ SuckerPunch.exception_handler = -> (ex, klass, args) { ExceptionNotifier.notify_exception(ex) }
156
174
  ```
157
175
 
158
176
  Or, using Airbrake:
159
177
 
160
178
  ```Ruby
161
- SuckerPunch.exception_handler { |ex| Airbrake.notify(ex) }
179
+ SuckerPunch.exception_handler = -> (ex, klass, args) { Airbrake.notify(ex) }
162
180
  ```
163
181
 
164
- Full job data can be reported like this:
182
+ #### Shutdown Timeout
165
183
 
166
- ```Ruby
167
- def perform(all, my, arguments)
168
- ... your code ...
169
- rescue StandardError
170
- Airbrake.error($!, [self.class.name, all, my, arguments].inspect)
171
- raise
172
- end
184
+ Sucker Punch goes through a series of checks to attempt to shut down the queues
185
+ and their threads. A "shutdown" command is issued to the queues, which gives
186
+ them notice but allows them to attempt to finish all remaining jobs.
187
+ Subsequently enqueued jobs are discarded at this time.
188
+
189
+ The default `shutdown_timeout` (the # of seconds to wait before forcefully
190
+ killing the threads) is 8 sec. This is to allow applications hosted on Heroku
191
+ to attempt to shutdown prior to the 10 sec. they give an application to
192
+ shutdown with some buffer.
193
+
194
+ To configure something other than the default 8 sec.:
195
+
196
+ ```ruby
197
+ SuckerPunch.shutdown_timeout = 15 # # of sec. to wait before killing threads
173
198
  ```
174
199
 
175
- ## Timeouts
200
+ #### Timeouts
176
201
 
177
- Using `Timeout` causes persistent connections to [randomly get corrupted](http://www.mikeperham.com/2015/05/08/timeout-rubys-most-dangerous-api).
178
- Do not use timeouts as control flow, use builtin connection timeouts.
202
+ Using `Timeout` causes persistent connections to
203
+ [randomly get corrupted](http://www.mikeperham.com/2015/05/08/timeout-rubys-most-dangerous-api).
204
+ Do not use timeouts as control flow, use built-in connection timeouts.
179
205
  If you decide to use Timeout, only use it as last resort to know something went very wrong and
180
206
  ideally restart the worker process after every timeout.
181
207
 
182
208
  ## Testing
183
209
 
184
- Requiring this library causes your jobs to run everything inline. So a call to the following will actually be SYNCHRONOUS:
210
+ Requiring this library causes your jobs to run everything inline.
211
+ So a call to the following will actually be SYNCHRONOUS:
185
212
 
186
213
  ```Ruby
187
214
  # spec/spec_helper.rb
@@ -189,7 +216,7 @@ require 'sucker_punch/testing/inline'
189
216
  ```
190
217
 
191
218
  ```Ruby
192
- Log.new.async.perform("login") # => Will be synchronous and block until job is finished
219
+ LogJob.perform_async("login") # => Will be synchronous and block until job is finished
193
220
  ```
194
221
 
195
222
  ## Rails
@@ -230,28 +257,10 @@ end
230
257
  ### Initializers for forking servers (Unicorn, Passenger, etc.)
231
258
 
232
259
  Previously, Sucker Punch required an initializer and that posed problems for
233
- Unicorn and Passenger and other servers that fork. Version 1 was rewritten to
260
+ servers that fork (ie. Unicorn and Passenger). Version 1 was rewritten to
234
261
  not require any special code to be executed after forking occurs. Please remove
235
262
  if you're using version `>= 1.0.0`
236
263
 
237
- ### Class naming
238
-
239
- Job classes are ultimately Celluloid Actor classes. As a result, class names
240
- are susceptible to being clobbered by Celluloid's internal classes. To ensure
241
- the intended application class is loaded, preface classes with `::`, or use
242
- names like `NotificationsMailer` or `UserMailer`. Example:
243
-
244
- ```Ruby
245
- class EmailJob
246
- include SuckerPunch::Job
247
-
248
- def perform(contact)
249
- @contact = contact
250
- ::Notifications.contact_form(@contact).deliver # => If you don't use :: in this case, the notifications class from Celluloid will be loaded
251
- end
252
- end
253
- ```
254
-
255
264
  ### Cleaning test data transactions
256
265
 
257
266
  If you're running tests in transactions (using Database Cleaner or a native solution), Sucker Punch jobs may have trouble finding database records that were created during test setup because the job class is running in a separate thread and the Transaction operates on a different thread so it clears out the data before the job can do its business. The best thing to do is cleanup data created for tests jobs through a truncation strategy by tagging the rspec tests as jobs and then specifying the strategy in `spec_helper` like below. And do not forget to turn off transactional fixtures (delete, comment or set it to `false`).
@@ -298,12 +307,6 @@ describe EmailJob, job: true do
298
307
  end
299
308
  ```
300
309
 
301
- ## Gem Name
302
-
303
- ...is awesome. But I can't take credit for it. Thanks to
304
- [@jmazzi](https://twitter.com/jmazzi) for his superior naming skills. If you're
305
- looking for a name for something, he is the one to go to.
306
-
307
310
  ## Contributing
308
311
 
309
312
  1. Fork it
data/Rakefile CHANGED
@@ -1,12 +1,15 @@
1
1
  require "bundler/gem_tasks"
2
- require 'rspec/core/rake_task'
2
+ require "rake/testtask"
3
3
 
4
- RSpec::Core::RakeTask.new('spec')
4
+ Rake::TestTask.new(:test) do |t|
5
+ t.libs << "test"
6
+ t.libs << "lib"
7
+ t.test_files = FileList['test/**/*_test.rb']
8
+ end
5
9
 
6
- # If you want to make this the default task
7
- task :default => :spec
8
- task :test => :spec
10
+ task :default => :test
9
11
 
10
12
  task :console do
11
13
  exec "irb -r sucker_punch -I ./lib"
12
- end
14
+ end
15
+
data/bin/load ADDED
@@ -0,0 +1,23 @@
1
+ #!/usr/bin/env ruby
2
+
3
+ require 'bundler'
4
+ Bundler.require(:default)
5
+
6
+ require_relative '../lib/sucker_punch'
7
+
8
+ class NoopJob
9
+ include SuckerPunch::Job
10
+
11
+ def perform(a)
12
+ end
13
+ end
14
+
15
+ puts "Enqueuing 1MM jobs..."
16
+
17
+ 1_000_000.times { NoopJob.perform_async(1) }
18
+
19
+ puts "Executing jobs..."
20
+
21
+ while SuckerPunch::Queue.all["NoopJob"]["jobs"]["enqueued"] > 0
22
+ sleep 0.01
23
+ end
data/bin/shutdown ADDED
@@ -0,0 +1,20 @@
1
+ #!/usr/bin/env ruby
2
+
3
+ require 'bundler'
4
+ Bundler.require(:default)
5
+
6
+ require_relative '../lib/sucker_punch'
7
+
8
+ class CountdownJob
9
+ include SuckerPunch::Job
10
+
11
+ def perform(i)
12
+ sleep 0.1
13
+ print "Executing job #{i}\n"
14
+ end
15
+ end
16
+
17
+ puts "Enqueuing 100 jobs..."
18
+
19
+ 100.times { |i| CountdownJob.perform_async(i) }
20
+ sleep 0.5
@@ -0,0 +1,18 @@
1
+ module SuckerPunch
2
+ module Job
3
+ def async
4
+ AsyncProxy.new(self)
5
+ end
6
+ end
7
+
8
+ class AsyncProxy
9
+ def initialize(job)
10
+ @job = job
11
+ end
12
+
13
+ def perform(*args)
14
+ @job.class.perform_async(*args)
15
+ end
16
+ end
17
+ end
18
+
@@ -1,9 +1,51 @@
1
- class String
2
- def underscore
3
- self.gsub(/::/, '/').
4
- gsub(/([A-Z]+)([A-Z][a-z])/,'\1_\2').
5
- gsub(/([a-z\d])([A-Z])/,'\1_\2').
6
- tr("-", "_").
7
- downcase
8
- end if !"".respond_to?(:underscore)
1
+ begin
2
+ require 'active_support/core_ext/class/attribute'
3
+ rescue LoadError
4
+
5
+ # A dumbed down version of ActiveSupport's
6
+ # Class#class_attribute helper.
7
+ class Class
8
+ def class_attribute(*attrs)
9
+ instance_writer = true
10
+
11
+ attrs.each do |name|
12
+ class_eval <<-RUBY, __FILE__, __LINE__ + 1
13
+ def self.#{name}() nil end
14
+ def self.#{name}?() !!#{name} end
15
+
16
+ def self.#{name}=(val)
17
+ singleton_class.class_eval do
18
+ define_method(:#{name}) { val }
19
+ end
20
+
21
+ if singleton_class?
22
+ class_eval do
23
+ def #{name}
24
+ defined?(@#{name}) ? @#{name} : singleton_class.#{name}
25
+ end
26
+ end
27
+ end
28
+ val
29
+ end
30
+
31
+ def #{name}
32
+ defined?(@#{name}) ? @#{name} : self.class.#{name}
33
+ end
34
+
35
+ def #{name}?
36
+ !!#{name}
37
+ end
38
+ RUBY
39
+
40
+ attr_writer name if instance_writer
41
+ end
42
+ end
43
+
44
+ private
45
+ def singleton_class?
46
+ ancestors.first != self
47
+ end
48
+ end
9
49
  end
50
+
51
+
@@ -0,0 +1,72 @@
1
+ module SuckerPunch
2
+ module Counter
3
+ module Utilities
4
+ def value
5
+ @counter.value
6
+ end
7
+
8
+ def increment
9
+ @counter.increment
10
+ end
11
+
12
+ def decrement
13
+ @counter.decrement
14
+ end
15
+ end
16
+
17
+ class Busy
18
+ attr_reader :counter
19
+
20
+ include Utilities
21
+
22
+ COUNTER = Concurrent::Map.new do |hash, name|
23
+ hash.compute_if_absent(name) { Concurrent::AtomicFixnum.new }
24
+ end
25
+
26
+ def self.clear
27
+ COUNTER.clear
28
+ end
29
+
30
+ def initialize(queue_name)
31
+ @counter = COUNTER[queue_name]
32
+ end
33
+ end
34
+
35
+ class Processed
36
+ attr_reader :counter
37
+
38
+ include Utilities
39
+
40
+ COUNTER = Concurrent::Map.new do |hash, name|
41
+ hash.compute_if_absent(name) { Concurrent::AtomicFixnum.new }
42
+ end
43
+
44
+ def self.clear
45
+ COUNTER.clear
46
+ end
47
+
48
+ def initialize(queue_name)
49
+ @counter = COUNTER[queue_name]
50
+ end
51
+ end
52
+
53
+ class Failed
54
+ attr_reader :counter
55
+
56
+ include Utilities
57
+
58
+ COUNTER = Concurrent::Map.new do |hash, name|
59
+ hash.compute_if_absent(name) { Concurrent::AtomicFixnum.new }
60
+ end
61
+
62
+ def self.clear
63
+ COUNTER.clear
64
+ end
65
+
66
+ def initialize(queue_name)
67
+ @counter = COUNTER[queue_name]
68
+ end
69
+ end
70
+ end
71
+ end
72
+