deferrer 0.0.2 → 0.1.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +50 -28
- data/deferrer.gemspec +3 -3
- data/example/client.rb +4 -6
- data/example/common.rb +2 -0
- data/example/runner.rb +7 -14
- data/lib/deferrer.rb +12 -3
- data/lib/deferrer/configuration.rb +4 -5
- data/lib/deferrer/json_encoding.rb +1 -7
- data/lib/deferrer/runner.rb +103 -0
- data/lib/deferrer/version.rb +1 -1
- data/spec/deferrer/runner_spec.rb +145 -0
- data/spec/spec_helper.rb +18 -0
- data/spec/support/helpers.rb +5 -0
- metadata +12 -10
- data/example/name_deferrer.rb +0 -9
- data/lib/deferrer/deferral.rb +0 -100
- data/spec/deferrer/deferral_spec.rb +0 -133
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 47517fb6ffc62ca2b600c64f1a0a5ae5e2379239
|
4
|
+
data.tar.gz: 06110ea8a1dbdf20002f5627798a33ac5fd550e8
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 47f4549a640efb122f2726e3fdf052c7f9998f748d978f608cd440f74c8b5b352a157918423a5001d4f5688c617ca30a3fa89256c5d879e80e4cbece5cade884
|
7
|
+
data.tar.gz: 1f330b0d1a8e6bae1a27dbb66fb0c69b5c0b83ca1818d4a3012ff2ef11738f34ff032753f55701d39548ce887cff9082a2f0a50cea7af87e620f0ed875f0ad3f
|
data/README.md
CHANGED
@@ -2,64 +2,86 @@
|
|
2
2
|
|
3
3
|
# Deferrer
|
4
4
|
|
5
|
-
|
6
|
-
|
5
|
+
Deferrer is a library for deferring work units for a time period or to a specific time. When time reaches, only the last work unit will be processed. Usually, the last work unit should be the one that summarizes all the previous ones. An example scenario would be sending live updates that happen *very* frequently and we want to limit them by sending an update every x seconds.
|
7
6
|
|
8
7
|
## Installation
|
9
8
|
|
10
9
|
Add this line to your application's Gemfile:
|
11
10
|
|
12
|
-
|
11
|
+
```
|
12
|
+
gem 'deferrer'
|
13
|
+
```
|
13
14
|
|
14
15
|
And then execute:
|
15
16
|
|
16
|
-
|
17
|
+
```
|
18
|
+
$ bundle install
|
19
|
+
```
|
17
20
|
|
18
21
|
Or install it yourself as:
|
19
22
|
|
20
|
-
|
21
|
-
|
23
|
+
```
|
24
|
+
$ gem install deferrer
|
25
|
+
```
|
22
26
|
|
23
27
|
|
24
28
|
## Usage
|
25
29
|
|
26
|
-
Configure redis
|
30
|
+
Configure deferrer (redis connection, logger)
|
31
|
+
|
32
|
+
```ruby
|
33
|
+
Deferrer.redis_config = { :host => "localhost", :port => 6379 }
|
34
|
+
Deferrer.logger = Logger.new(STDOUT)
|
35
|
+
```
|
36
|
+
|
37
|
+
|
38
|
+
Define deferrer worker that must respond to call method
|
39
|
+
|
40
|
+
```ruby
|
41
|
+
Deferrer.worker = lambda do |klass, *args|
|
42
|
+
# do some work
|
43
|
+
# Resque.enqueue(klass, *args)
|
44
|
+
end
|
45
|
+
```
|
27
46
|
|
28
|
-
|
47
|
+
Deferrer is usually used in combination with background processing tools like sidekiq and resque. If that's the case, Deferrer.worker can be light-weight and responsible only for pushing work to a background job.
|
29
48
|
|
30
49
|
|
31
|
-
|
50
|
+
Start a worker process.
|
32
51
|
|
33
|
-
|
34
|
-
|
35
|
-
puts "#{first_name} #{last_name}".upcase
|
36
|
-
end
|
37
|
-
end
|
52
|
+
```ruby
|
53
|
+
Deferrer.run(options = {})
|
38
54
|
|
55
|
+
# Following `options` are available:
|
56
|
+
# loop_frequency - sleep between loops, default to 0.1 seconds
|
57
|
+
# single_run - process items only for a single loop, useful for testing
|
58
|
+
# ignore_time - don't wait for time period to expire, useful for testing
|
59
|
+
```
|
39
60
|
|
40
|
-
Start a worker process. It needs to have redis configured and access to deferrer classes.
|
41
61
|
|
42
|
-
|
62
|
+
Defer some executions:
|
43
63
|
|
44
|
-
|
45
|
-
|
46
|
-
|
47
|
-
|
48
|
-
|
49
|
-
# single_run - process items only for a single loop, useful for testing
|
64
|
+
```ruby
|
65
|
+
Deferrer.defer_in(5, 'identifier', Worker, 'update 1')
|
66
|
+
Deferrer.defer_in(6, 'identifier', Worker, 'update 2')
|
67
|
+
Deferrer.defer_in(9, 'identifier', Worker, 'update 3')
|
68
|
+
```
|
50
69
|
|
51
70
|
|
52
|
-
|
71
|
+
It will stack all defered executions per identifier until first timeout expires (5 seconds) and then it will only execute the last update for the expired identifier, calling the deferrer worker:
|
53
72
|
|
54
|
-
|
55
|
-
|
56
|
-
|
73
|
+
```ruby
|
74
|
+
Deferrer.worker.call('Worker', 'update 3')
|
75
|
+
```
|
57
76
|
|
58
77
|
|
59
|
-
|
78
|
+
## Testing
|
60
79
|
|
61
|
-
|
80
|
+
For testing, two options of the `run` method are useful. `single_run` will run the loop only once and `ignore_time` will not wait for time period to expire but execute to job now.
|
62
81
|
|
82
|
+
```ruby
|
83
|
+
Deferrer.run(single_run: true, ignore_time: true)
|
84
|
+
```
|
63
85
|
|
64
86
|
|
65
87
|
## Contributing
|
data/deferrer.gemspec
CHANGED
@@ -8,8 +8,8 @@ Gem::Specification.new do |spec|
|
|
8
8
|
spec.version = Deferrer::VERSION
|
9
9
|
spec.authors = ["Dalibor Nasevic"]
|
10
10
|
spec.email = ["dalibor.nasevic@gmail.com"]
|
11
|
-
spec.description = %q{Defer
|
12
|
-
spec.summary = %q{Defer
|
11
|
+
spec.description = %q{Defer work units and process only the last one}
|
12
|
+
spec.summary = %q{Defer work units and process only the last work unit when time comes}
|
13
13
|
spec.homepage = ""
|
14
14
|
spec.license = "MIT"
|
15
15
|
|
@@ -22,5 +22,5 @@ Gem::Specification.new do |spec|
|
|
22
22
|
spec.add_dependency "multi_json"
|
23
23
|
spec.add_development_dependency "bundler", "~> 1.3"
|
24
24
|
spec.add_development_dependency "rake"
|
25
|
-
spec.add_development_dependency "rspec", "~>
|
25
|
+
spec.add_development_dependency "rspec", "~> 3.0.0"
|
26
26
|
end
|
data/example/client.rb
CHANGED
@@ -1,8 +1,6 @@
|
|
1
1
|
require 'deferrer'
|
2
|
-
require_relative './
|
2
|
+
require_relative './common'
|
3
3
|
|
4
|
-
|
5
|
-
|
6
|
-
|
7
|
-
Deferrer.defer_in(6, 'identifier', NameDeferrer, 'User', '2')
|
8
|
-
Deferrer.defer_in(9, 'identifier', NameDeferrer, 'User', '3')
|
4
|
+
1.upto(5) do |i|
|
5
|
+
Deferrer.defer_in(i + 3, 'identifier', "update #{i}")
|
6
|
+
end
|
data/example/common.rb
ADDED
data/example/runner.rb
CHANGED
@@ -1,21 +1,14 @@
|
|
1
1
|
require 'deferrer'
|
2
|
-
|
3
|
-
|
4
|
-
Deferrer.redis_config = { :host => "localhost", :port => 6379 }
|
5
|
-
|
6
|
-
class Logger
|
7
|
-
def self.info(message)
|
8
|
-
puts "INFO: #{message}"
|
9
|
-
end
|
10
|
-
|
11
|
-
def self.error(message)
|
12
|
-
puts "ERROR: #{message}"
|
13
|
-
end
|
14
|
-
end
|
2
|
+
require 'logger'
|
3
|
+
require_relative './common'
|
15
4
|
|
16
5
|
puts 'Runner started'
|
17
6
|
|
7
|
+
Deferrer.redis_config = { :host => "localhost", :port => 6379, :db => 15 }
|
8
|
+
Deferrer.logger = Logger.new(STDOUT)
|
9
|
+
Deferrer.worker = lambda do |*args|
|
10
|
+
p args
|
11
|
+
end
|
18
12
|
Deferrer.run({
|
19
13
|
:loop_frequency => 0.5,
|
20
|
-
:logger => Logger
|
21
14
|
})
|
data/lib/deferrer.rb
CHANGED
@@ -2,12 +2,21 @@ require 'redis'
|
|
2
2
|
require "deferrer/version"
|
3
3
|
|
4
4
|
module Deferrer
|
5
|
-
|
6
5
|
autoload :Configuration, 'deferrer/configuration'
|
7
6
|
autoload :JsonEncoding, 'deferrer/json_encoding'
|
8
|
-
autoload :
|
7
|
+
autoload :Runner, 'deferrer/runner'
|
8
|
+
autoload :Job, 'deferrer/job'
|
9
|
+
|
10
|
+
LIST_KEY = 'deferred_list'
|
11
|
+
ITEM_KEY_PREFIX = 'deferred'
|
9
12
|
|
10
13
|
extend Configuration
|
11
14
|
extend JsonEncoding
|
12
|
-
extend
|
15
|
+
extend Runner
|
16
|
+
|
17
|
+
class WorkerNotConfigured < NotImplementedError
|
18
|
+
def initialize
|
19
|
+
super("Deferrer worker not configured")
|
20
|
+
end
|
21
|
+
end
|
13
22
|
end
|
@@ -1,14 +1,13 @@
|
|
1
1
|
module Deferrer
|
2
2
|
module Configuration
|
3
3
|
|
4
|
+
attr_reader :redis
|
5
|
+
attr_accessor :logger
|
6
|
+
attr_accessor :worker
|
7
|
+
|
4
8
|
# Deferrer.redis_config = { :host => "localhost", :port => 6379 }
|
5
9
|
def redis_config=(config)
|
6
10
|
@redis = Redis.new(config)
|
7
11
|
end
|
8
|
-
|
9
|
-
# Returns the configured Redis instance
|
10
|
-
def redis
|
11
|
-
@redis
|
12
|
-
end
|
13
12
|
end
|
14
13
|
end
|
@@ -3,18 +3,12 @@ require 'multi_json'
|
|
3
3
|
module Deferrer
|
4
4
|
module JsonEncoding
|
5
5
|
|
6
|
-
class DecodeException < StandardError; end
|
7
|
-
|
8
6
|
def encode(item)
|
9
7
|
MultiJson.dump(item)
|
10
8
|
end
|
11
9
|
|
12
10
|
def decode(item)
|
13
|
-
|
14
|
-
MultiJson.load(item)
|
15
|
-
rescue ::MultiJson::DecodeError => e
|
16
|
-
raise DecodeException, e.message, e.backtrace
|
17
|
-
end
|
11
|
+
MultiJson.load(item)
|
18
12
|
end
|
19
13
|
end
|
20
14
|
end
|
@@ -0,0 +1,103 @@
|
|
1
|
+
module Deferrer
|
2
|
+
module Runner
|
3
|
+
def run(options = {})
|
4
|
+
loop_frequency = options.fetch(:loop_frequency, 0.1)
|
5
|
+
single_run = options.fetch(:single_run, false)
|
6
|
+
@ignore_time = options.fetch(:ignore_time, false)
|
7
|
+
|
8
|
+
raise WorkerNotConfigured unless worker
|
9
|
+
|
10
|
+
loop do
|
11
|
+
begin
|
12
|
+
while item = next_item
|
13
|
+
process_item(item)
|
14
|
+
end
|
15
|
+
rescue StandardError => e
|
16
|
+
log(:error, "Error: #{e.class}: #{e.message}")
|
17
|
+
rescue Exception => e
|
18
|
+
log(:error, "Error: #{e.class}: #{e.message}")
|
19
|
+
raise
|
20
|
+
end
|
21
|
+
|
22
|
+
break if single_run
|
23
|
+
sleep loop_frequency
|
24
|
+
end
|
25
|
+
end
|
26
|
+
|
27
|
+
def next_item
|
28
|
+
item = nil
|
29
|
+
decoded_item = nil
|
30
|
+
key = next_key
|
31
|
+
|
32
|
+
if key
|
33
|
+
item = redis.rpop(key)
|
34
|
+
if item
|
35
|
+
decoded_item = decode(item)
|
36
|
+
decoded_item['key'] = key
|
37
|
+
end
|
38
|
+
|
39
|
+
remove(key)
|
40
|
+
end
|
41
|
+
|
42
|
+
decoded_item
|
43
|
+
end
|
44
|
+
|
45
|
+
def defer_in(number_of_seconds_from_now, identifier, *args)
|
46
|
+
timestamp = Time.now + number_of_seconds_from_now
|
47
|
+
defer_at(timestamp, identifier, *args)
|
48
|
+
end
|
49
|
+
|
50
|
+
def defer_at(timestamp, identifier, *args)
|
51
|
+
key = item_key(identifier)
|
52
|
+
item = { 'args' => args }
|
53
|
+
|
54
|
+
push_item(key, item, timestamp)
|
55
|
+
end
|
56
|
+
|
57
|
+
private
|
58
|
+
def process_item(item)
|
59
|
+
log(:info, "Processing: #{item['key']}")
|
60
|
+
worker.call(*item['args'])
|
61
|
+
end
|
62
|
+
|
63
|
+
def item_key(identifier)
|
64
|
+
"#{ITEM_KEY_PREFIX}:#{identifier}"
|
65
|
+
end
|
66
|
+
|
67
|
+
def push_item(key, item, timestamp)
|
68
|
+
count = redis.rpush(key, encode(item))
|
69
|
+
|
70
|
+
# set score only on first update
|
71
|
+
if count == 1
|
72
|
+
score = calculate_score(timestamp)
|
73
|
+
redis.zadd(LIST_KEY, score, key)
|
74
|
+
end
|
75
|
+
end
|
76
|
+
|
77
|
+
def next_key
|
78
|
+
if @ignore_time
|
79
|
+
redis.zrange(LIST_KEY, 0, 1).first
|
80
|
+
else
|
81
|
+
score = calculate_score(Time.now)
|
82
|
+
redis.zrangebyscore(LIST_KEY, '-inf', score, :limit => [0, 1]).first
|
83
|
+
end
|
84
|
+
end
|
85
|
+
|
86
|
+
def calculate_score(timestamp)
|
87
|
+
timestamp.to_f
|
88
|
+
end
|
89
|
+
|
90
|
+
def remove(key)
|
91
|
+
redis.watch(key)
|
92
|
+
redis.multi do
|
93
|
+
redis.del(key)
|
94
|
+
redis.zrem(LIST_KEY, key)
|
95
|
+
end
|
96
|
+
redis.unwatch
|
97
|
+
end
|
98
|
+
|
99
|
+
def log(type, message)
|
100
|
+
logger.send(type, message) if logger
|
101
|
+
end
|
102
|
+
end
|
103
|
+
end
|
data/lib/deferrer/version.rb
CHANGED
@@ -0,0 +1,145 @@
|
|
1
|
+
require 'spec_helper'
|
2
|
+
require 'logger'
|
3
|
+
|
4
|
+
describe Deferrer::Runner do
|
5
|
+
let(:identifier) { 'some_identifier' }
|
6
|
+
let(:redis) { Deferrer.redis }
|
7
|
+
let(:list_key) { Deferrer::LIST_KEY }
|
8
|
+
let(:logger) { Logger.new(STDOUT) }
|
9
|
+
|
10
|
+
describe ".run" do
|
11
|
+
it "processes jobs" do
|
12
|
+
expect(Deferrer.worker).to receive(:call).with('TestWorker', 'test')
|
13
|
+
|
14
|
+
Deferrer.defer_in(-1, identifier, 'TestWorker', 'test')
|
15
|
+
Deferrer.run(single_run: true)
|
16
|
+
end
|
17
|
+
|
18
|
+
it "correctly sets arguments and converts symbols to strings for hashes" do
|
19
|
+
expect(Deferrer.worker).to receive(:call).with('TestWorker', 1, 'arg1', { "a" => "b"})
|
20
|
+
|
21
|
+
Deferrer.defer_in(-1, identifier, 'TestWorker', 1, 'arg1', { a: :b })
|
22
|
+
Deferrer.run(single_run: true)
|
23
|
+
end
|
24
|
+
|
25
|
+
it "rescues standard errors" do
|
26
|
+
allow(Deferrer).to receive(:next_item) { raise RuntimeError.new('error') }
|
27
|
+
|
28
|
+
Deferrer.defer_in(-1, identifier, 'TestWorker', 'test')
|
29
|
+
Deferrer.run(single_run: true)
|
30
|
+
end
|
31
|
+
|
32
|
+
it "rescues exceptions and logs error messages" do
|
33
|
+
expect(logger).to receive(:error).with("Error: Exception: error")
|
34
|
+
allow(Deferrer).to receive(:next_item) { raise Exception.new('error') }
|
35
|
+
|
36
|
+
Deferrer.logger = logger
|
37
|
+
Deferrer.defer_in(-1, identifier, 'TestWorker', 'test')
|
38
|
+
expect { Deferrer.run(single_run: true) }.to raise_error(Exception)
|
39
|
+
end
|
40
|
+
|
41
|
+
it "raises error if worker not configured" do
|
42
|
+
Deferrer.worker = nil
|
43
|
+
Deferrer.defer_in(-1, identifier, 'TestWorker', 'test')
|
44
|
+
|
45
|
+
expect { Deferrer.run(single_run: true) }.to raise_error(Deferrer::WorkerNotConfigured)
|
46
|
+
end
|
47
|
+
|
48
|
+
it "ignores time to wait and performs jobs" do
|
49
|
+
expect(Deferrer.worker).to receive(:call).with('Worker', { "c" => "d"})
|
50
|
+
|
51
|
+
Deferrer.defer_in(100, identifier, 'Worker', { a: :b })
|
52
|
+
Deferrer.defer_in(100, identifier, 'Worker', { c: :d })
|
53
|
+
|
54
|
+
Deferrer.run(single_run: true, ignore_time: true)
|
55
|
+
end
|
56
|
+
end
|
57
|
+
|
58
|
+
describe ".defer_at" do
|
59
|
+
it "deferrs at given time" do
|
60
|
+
Deferrer.defer_at(Time.now, identifier, 'TestWorker', 'test')
|
61
|
+
|
62
|
+
expect(redis.zrangebyscore(list_key, '-inf', Time.now.to_f, :limit => [0, 1]).first).not_to be_nil
|
63
|
+
expect(redis.exists(item_key(identifier))).to be_truthy
|
64
|
+
end
|
65
|
+
end
|
66
|
+
|
67
|
+
describe ".defer_in" do
|
68
|
+
it "defers in given interval" do
|
69
|
+
Deferrer.defer_in(1, identifier, 'TestWorker', 'test')
|
70
|
+
|
71
|
+
expect(redis.zrangebyscore(list_key, '-inf', (Time.now + 1).to_f, :limit => [0, 1]).first).not_to be_nil
|
72
|
+
expect(redis.exists(item_key(identifier))).to be_truthy
|
73
|
+
end
|
74
|
+
end
|
75
|
+
|
76
|
+
describe ".next_item" do
|
77
|
+
it "returns the next item" do
|
78
|
+
Deferrer.defer_at(Time.now, identifier, 'TestWorker', 'test')
|
79
|
+
|
80
|
+
item = Deferrer.next_item
|
81
|
+
|
82
|
+
expect(item['args']).to eq(['TestWorker', 'test'])
|
83
|
+
end
|
84
|
+
|
85
|
+
it "returns last update of an item" do
|
86
|
+
Deferrer.defer_at(Time.now - 3, identifier, 'TestWorker', 'update1')
|
87
|
+
Deferrer.defer_at(Time.now - 2, identifier, 'TestWorker', 'update2')
|
88
|
+
|
89
|
+
item = Deferrer.next_item
|
90
|
+
|
91
|
+
expect(item['args']).to eq(['TestWorker', 'update2'])
|
92
|
+
end
|
93
|
+
|
94
|
+
it "returns nil when no next item" do
|
95
|
+
expect(Deferrer.next_item).to be_nil
|
96
|
+
end
|
97
|
+
|
98
|
+
it "removes values from redis" do
|
99
|
+
Deferrer.defer_at(Time.now, identifier, 'TestWorker', 'test')
|
100
|
+
|
101
|
+
item = Deferrer.next_item
|
102
|
+
|
103
|
+
expect(redis.zrangebyscore(list_key, '-inf', Time.now.to_f, :limit => [0, 1]).first).to be_nil
|
104
|
+
expect(redis.exists(item_key(identifier))).to be_falsey
|
105
|
+
expect(Deferrer.next_item).to be_nil
|
106
|
+
end
|
107
|
+
|
108
|
+
it "doesn't block on empty lists" do
|
109
|
+
Deferrer.defer_in(-1, identifier, 'TestWorker', 'test')
|
110
|
+
redis.del(item_key(identifier))
|
111
|
+
|
112
|
+
expect(Deferrer.next_item).to be_nil
|
113
|
+
expect(redis.zrangebyscore(list_key, '-inf', 'inf', :limit => [0, 1]).first).to be_nil
|
114
|
+
end
|
115
|
+
end
|
116
|
+
|
117
|
+
describe ".logger" do
|
118
|
+
before :each do
|
119
|
+
Deferrer.logger = logger
|
120
|
+
end
|
121
|
+
|
122
|
+
it "logs info messages" do
|
123
|
+
expect(logger).to receive(:info).with("Processing: deferred:#{identifier}")
|
124
|
+
|
125
|
+
Deferrer.defer_in(-1, identifier, 'TestWorker', 'test')
|
126
|
+
Deferrer.run(single_run: true)
|
127
|
+
end
|
128
|
+
|
129
|
+
it "logs error messages" do
|
130
|
+
expect(logger).to receive(:error).with("Error: RuntimeError: error")
|
131
|
+
allow(Deferrer).to receive(:next_item) { raise RuntimeError.new('error') }
|
132
|
+
|
133
|
+
Deferrer.defer_in(-1, identifier, 'TestWorker', 'test')
|
134
|
+
Deferrer.run(single_run: true)
|
135
|
+
end
|
136
|
+
|
137
|
+
it "logs error messages on exceptions" do
|
138
|
+
expect(logger).to receive(:error).with("Error: Exception: error")
|
139
|
+
allow(Deferrer).to receive(:next_item) { raise Exception.new('error') }
|
140
|
+
|
141
|
+
Deferrer.defer_in(-1, identifier, 'TestWorker', 'test')
|
142
|
+
expect { Deferrer.run(single_run: true) }.to raise_error
|
143
|
+
end
|
144
|
+
end
|
145
|
+
end
|
data/spec/spec_helper.rb
CHANGED
@@ -1,3 +1,21 @@
|
|
1
1
|
require 'deferrer'
|
2
2
|
|
3
3
|
Deferrer.redis_config = { :host => "localhost", :port => 6379 }
|
4
|
+
|
5
|
+
|
6
|
+
ROOT = File.expand_path('../', File.dirname(__FILE__))
|
7
|
+
|
8
|
+
# Load support files
|
9
|
+
Dir["#{ROOT}/spec/support/**/*.rb"].each { |f| require f }
|
10
|
+
|
11
|
+
|
12
|
+
RSpec.configure do |config|
|
13
|
+
config.include Helpers
|
14
|
+
|
15
|
+
config.before :each do
|
16
|
+
Deferrer.redis.flushdb
|
17
|
+
Deferrer.logger = nil
|
18
|
+
Deferrer.worker = lambda { |klass, *args| }
|
19
|
+
end
|
20
|
+
end
|
21
|
+
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: deferrer
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.0
|
4
|
+
version: 0.1.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Dalibor Nasevic
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2014-07-
|
11
|
+
date: 2014-07-28 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: redis
|
@@ -72,15 +72,15 @@ dependencies:
|
|
72
72
|
requirements:
|
73
73
|
- - "~>"
|
74
74
|
- !ruby/object:Gem::Version
|
75
|
-
version:
|
75
|
+
version: 3.0.0
|
76
76
|
type: :development
|
77
77
|
prerelease: false
|
78
78
|
version_requirements: !ruby/object:Gem::Requirement
|
79
79
|
requirements:
|
80
80
|
- - "~>"
|
81
81
|
- !ruby/object:Gem::Version
|
82
|
-
version:
|
83
|
-
description: Defer
|
82
|
+
version: 3.0.0
|
83
|
+
description: Defer work units and process only the last one
|
84
84
|
email:
|
85
85
|
- dalibor.nasevic@gmail.com
|
86
86
|
executables: []
|
@@ -97,15 +97,16 @@ files:
|
|
97
97
|
- deferrer.gemspec
|
98
98
|
- example/README.md
|
99
99
|
- example/client.rb
|
100
|
-
- example/
|
100
|
+
- example/common.rb
|
101
101
|
- example/runner.rb
|
102
102
|
- lib/deferrer.rb
|
103
103
|
- lib/deferrer/configuration.rb
|
104
|
-
- lib/deferrer/deferral.rb
|
105
104
|
- lib/deferrer/json_encoding.rb
|
105
|
+
- lib/deferrer/runner.rb
|
106
106
|
- lib/deferrer/version.rb
|
107
|
-
- spec/deferrer/
|
107
|
+
- spec/deferrer/runner_spec.rb
|
108
108
|
- spec/spec_helper.rb
|
109
|
+
- spec/support/helpers.rb
|
109
110
|
homepage: ''
|
110
111
|
licenses:
|
111
112
|
- MIT
|
@@ -129,8 +130,9 @@ rubyforge_project:
|
|
129
130
|
rubygems_version: 2.2.0
|
130
131
|
signing_key:
|
131
132
|
specification_version: 4
|
132
|
-
summary: Defer
|
133
|
+
summary: Defer work units and process only the last work unit when time comes
|
133
134
|
test_files:
|
134
|
-
- spec/deferrer/
|
135
|
+
- spec/deferrer/runner_spec.rb
|
135
136
|
- spec/spec_helper.rb
|
137
|
+
- spec/support/helpers.rb
|
136
138
|
has_rdoc:
|
data/example/name_deferrer.rb
DELETED
data/lib/deferrer/deferral.rb
DELETED
@@ -1,100 +0,0 @@
|
|
1
|
-
module Deferrer
|
2
|
-
module Deferral
|
3
|
-
|
4
|
-
LIST_KEY = :deferred_list
|
5
|
-
|
6
|
-
def run(options = {})
|
7
|
-
loop_frequency = options.fetch(:loop_frequency, 0.1)
|
8
|
-
logger = options.fetch(:logger, nil)
|
9
|
-
before_each = options.fetch(:before_each, nil)
|
10
|
-
after_each = options.fetch(:after_each, nil)
|
11
|
-
single_run = options.fetch(:single_run, false)
|
12
|
-
|
13
|
-
loop do
|
14
|
-
while item = next_item
|
15
|
-
process_item(item, logger, before_each, after_each)
|
16
|
-
end
|
17
|
-
|
18
|
-
break if single_run
|
19
|
-
sleep loop_frequency
|
20
|
-
end
|
21
|
-
end
|
22
|
-
|
23
|
-
def next_item
|
24
|
-
item = nil
|
25
|
-
decoded_item = nil
|
26
|
-
score = calculate_score(Time.now)
|
27
|
-
|
28
|
-
key = redis.zrangebyscore(LIST_KEY, '-inf', score, :limit => [0, 1]).first
|
29
|
-
|
30
|
-
if key
|
31
|
-
item = redis.rpop(key)
|
32
|
-
if item
|
33
|
-
decoded_item = decode(item)
|
34
|
-
decoded_item['key'] = key
|
35
|
-
end
|
36
|
-
|
37
|
-
remove(key)
|
38
|
-
end
|
39
|
-
|
40
|
-
decoded_item
|
41
|
-
end
|
42
|
-
|
43
|
-
def constantize(klass_string)
|
44
|
-
klass_string.split('::').inject(Object) {|memo,name| memo = memo.const_get(name); memo}
|
45
|
-
end
|
46
|
-
|
47
|
-
def defer_in(number_of_seconds_from_now, identifier, klass, *args)
|
48
|
-
timestamp = Time.now + number_of_seconds_from_now
|
49
|
-
defer_at(timestamp, identifier, klass, *args)
|
50
|
-
end
|
51
|
-
|
52
|
-
def defer_at(timestamp, identifier, klass, *args)
|
53
|
-
item = build_item(klass, args)
|
54
|
-
key = item_key(identifier)
|
55
|
-
score = calculate_score(timestamp)
|
56
|
-
|
57
|
-
count = redis.rpush(key, encode(item))
|
58
|
-
|
59
|
-
# set score only on first update
|
60
|
-
if count == 1
|
61
|
-
redis.zadd(LIST_KEY, score, key)
|
62
|
-
end
|
63
|
-
end
|
64
|
-
|
65
|
-
def item_key(identifier)
|
66
|
-
"deferred:#{identifier}"
|
67
|
-
end
|
68
|
-
|
69
|
-
private
|
70
|
-
def process_item(item, logger, before_each, after_each)
|
71
|
-
before_each.call if before_each
|
72
|
-
klass = constantize(item['class'])
|
73
|
-
args = item['args']
|
74
|
-
|
75
|
-
logger.info("Executing: #{item['key']}") if logger
|
76
|
-
|
77
|
-
klass.send(:perform, *args)
|
78
|
-
after_each.call if after_each
|
79
|
-
rescue Exception => e
|
80
|
-
logger.error("Error: #{e.class}: #{e.message}") if logger
|
81
|
-
end
|
82
|
-
|
83
|
-
def build_item(klass, args)
|
84
|
-
{'class' => klass.to_s, 'args' => args}
|
85
|
-
end
|
86
|
-
|
87
|
-
def calculate_score(timestamp)
|
88
|
-
timestamp.to_f
|
89
|
-
end
|
90
|
-
|
91
|
-
def remove(key)
|
92
|
-
redis.watch(key)
|
93
|
-
redis.multi do
|
94
|
-
redis.del(key)
|
95
|
-
redis.zrem(LIST_KEY, key)
|
96
|
-
end
|
97
|
-
redis.unwatch
|
98
|
-
end
|
99
|
-
end
|
100
|
-
end
|
@@ -1,133 +0,0 @@
|
|
1
|
-
require 'spec_helper'
|
2
|
-
require 'timeout'
|
3
|
-
|
4
|
-
class CarDeferrer
|
5
|
-
def self.perform(car)
|
6
|
-
end
|
7
|
-
|
8
|
-
def self.callback
|
9
|
-
end
|
10
|
-
end
|
11
|
-
|
12
|
-
class Logger
|
13
|
-
def self.info(message)
|
14
|
-
end
|
15
|
-
|
16
|
-
def self.error(message)
|
17
|
-
end
|
18
|
-
end
|
19
|
-
|
20
|
-
class InvalidLogger
|
21
|
-
def self.error(message)
|
22
|
-
end
|
23
|
-
end
|
24
|
-
|
25
|
-
describe Deferrer::Deferral do
|
26
|
-
let(:car) { 'car' }
|
27
|
-
let(:car2) { 'car2' }
|
28
|
-
let(:identifier) { 'car1' }
|
29
|
-
let(:redis) { Deferrer.redis }
|
30
|
-
let(:list_key) { Deferrer::Deferral::LIST_KEY }
|
31
|
-
|
32
|
-
before :each do
|
33
|
-
redis.flushdb
|
34
|
-
end
|
35
|
-
|
36
|
-
describe "run" do
|
37
|
-
it "processes jobs" do
|
38
|
-
CarDeferrer.should_receive(:perform).with(car)
|
39
|
-
Deferrer.defer_in(-1, identifier, CarDeferrer, car)
|
40
|
-
Deferrer.run(single_run: true)
|
41
|
-
end
|
42
|
-
|
43
|
-
it "logs info messages if logger provided" do
|
44
|
-
Logger.should_receive(:info).with("Executing: deferred:#{identifier}")
|
45
|
-
Deferrer.defer_in(-1, identifier, CarDeferrer, car)
|
46
|
-
Deferrer.run(single_run: true, logger: Logger)
|
47
|
-
end
|
48
|
-
|
49
|
-
it "logs error messages if logger provided" do
|
50
|
-
InvalidLogger.should_receive(:error).with("Error: NoMethodError: undefined method `info' for InvalidLogger:Class")
|
51
|
-
Deferrer.defer_in(-1, identifier, CarDeferrer, car)
|
52
|
-
Deferrer.run(single_run: true, logger: InvalidLogger)
|
53
|
-
end
|
54
|
-
|
55
|
-
it "runs before callback" do
|
56
|
-
CarDeferrer.should_receive(:callback)
|
57
|
-
Deferrer.defer_in(-1, identifier, CarDeferrer, car)
|
58
|
-
Deferrer.run(single_run: true, before_each: Proc.new { CarDeferrer.callback })
|
59
|
-
end
|
60
|
-
|
61
|
-
it "runs after callback" do
|
62
|
-
CarDeferrer.should_receive(:callback)
|
63
|
-
Deferrer.defer_in(-1, identifier, CarDeferrer, car)
|
64
|
-
Deferrer.run(single_run: true, after_each: Proc.new { CarDeferrer.callback })
|
65
|
-
end
|
66
|
-
end
|
67
|
-
|
68
|
-
describe ".defer_at" do
|
69
|
-
it "deferrs at given time" do
|
70
|
-
Deferrer.defer_at(Time.now, identifier, CarDeferrer, car)
|
71
|
-
|
72
|
-
redis.zrangebyscore(list_key, '-inf', Time.now.to_f, :limit => [0, 1]).first.should_not be_nil
|
73
|
-
redis.exists(Deferrer.item_key(identifier)).should be_true
|
74
|
-
end
|
75
|
-
|
76
|
-
it "defers in given interval" do
|
77
|
-
Deferrer.defer_in(1, identifier, CarDeferrer, car)
|
78
|
-
|
79
|
-
redis.zrangebyscore(list_key, '-inf', (Time.now + 1).to_f, :limit => [0, 1]).first.should_not be_nil
|
80
|
-
redis.exists(Deferrer.item_key(identifier)).should be_true
|
81
|
-
end
|
82
|
-
end
|
83
|
-
|
84
|
-
describe ".next_item" do
|
85
|
-
it "returns the next item" do
|
86
|
-
Deferrer.defer_at(Time.now, identifier, CarDeferrer, car)
|
87
|
-
|
88
|
-
item = Deferrer.next_item
|
89
|
-
|
90
|
-
item['class'].should == CarDeferrer.to_s
|
91
|
-
item['args'].should == [car]
|
92
|
-
end
|
93
|
-
|
94
|
-
it "returns last update of an item" do
|
95
|
-
Deferrer.defer_at(Time.now - 3, identifier, CarDeferrer, car)
|
96
|
-
Deferrer.defer_at(Time.now - 2, identifier, CarDeferrer, car2)
|
97
|
-
|
98
|
-
item = Deferrer.next_item
|
99
|
-
|
100
|
-
item['class'].should == CarDeferrer.to_s
|
101
|
-
item['args'].should == [car2]
|
102
|
-
end
|
103
|
-
|
104
|
-
it "keep the old score value" do
|
105
|
-
Deferrer.defer_at(Time.now - 3, identifier, CarDeferrer, car)
|
106
|
-
Deferrer.defer_at(Time.now + 1, identifier, CarDeferrer, car2)
|
107
|
-
|
108
|
-
Deferrer.next_item.should_not be_nil
|
109
|
-
end
|
110
|
-
|
111
|
-
it "returns nil when no next item" do
|
112
|
-
Deferrer.next_item.should be_nil
|
113
|
-
end
|
114
|
-
|
115
|
-
it "removes values from redis" do
|
116
|
-
Deferrer.defer_at(Time.now, identifier, CarDeferrer, car)
|
117
|
-
|
118
|
-
item = Deferrer.next_item
|
119
|
-
|
120
|
-
redis.zrangebyscore(list_key, '-inf', Time.now.to_f, :limit => [0, 1]).first.should be_nil
|
121
|
-
redis.exists(Deferrer.item_key(identifier)).should be_false
|
122
|
-
Deferrer.next_item.should be_nil
|
123
|
-
end
|
124
|
-
|
125
|
-
it "doesn't block on empty lists" do
|
126
|
-
Deferrer.defer_in(-1, identifier, CarDeferrer, car)
|
127
|
-
redis.del Deferrer.item_key(identifier)
|
128
|
-
|
129
|
-
Timeout::timeout(2) { Deferrer.next_item.should be_nil }
|
130
|
-
redis.zrangebyscore(list_key, '-inf', 'inf', :limit => [0, 1]).first.should be_nil
|
131
|
-
end
|
132
|
-
end
|
133
|
-
end
|