perfectqueue 0.8.13 → 0.8.14
Sign up to get free protection for your applications and to get access to all the features.
- data/ChangeLog +6 -0
- data/README.md +12 -4
- data/lib/perfectqueue/version.rb +1 -1
- data/lib/perfectqueue/worker.rb +16 -3
- metadata +2 -2
data/ChangeLog
CHANGED
data/README.md
CHANGED
@@ -1,10 +1,17 @@
|
|
1
1
|
# PerfectQueue
|
2
2
|
|
3
3
|
PerfectQueue is a highly available distributed queue built on top of RDBMS.
|
4
|
-
|
5
|
-
PerfectQueue provides similar API to Amazon SQS. But unlike Amazon SQS, PerfectQueue never delivers finished tasks.
|
4
|
+
PerfectQueue provides similar API to Amazon SQS, while PerfectQueue focues on reliability and flexible scheduling rather than scalability.
|
6
5
|
|
7
|
-
|
6
|
+
PerfectQueue introduces following concepts:
|
7
|
+
|
8
|
+
* **At-least-once semantics:** Even if a worker node fails during processing a task, another worker takes over the task.
|
9
|
+
* **Multiuser-aware fair scheduling**: PerfectQueue schedules tasks submitted by users who have larger resource assignment.
|
10
|
+
* **Decider:** Decider is a simple mechanism to implement complex workflows on top of queues while keeping loose coupling.
|
11
|
+
* **Graceful and live restarting:** PerfectQueue continues processing of long-running tasks even during restarting.
|
12
|
+
* **Idempotent task submission:** All tasks have unique identifier and PerfectQueue prevents storing same task twice.
|
13
|
+
* Note: client applications should consider how to always generate a same string for a (semantically) same task.
|
14
|
+
* **Idempotent task processing support:** Workers can use a constant unique identifier to process tasks idempotently.
|
8
15
|
|
9
16
|
All you have to consider is implementing idempotent worker programs. PerfectQueue manages the other problems.
|
10
17
|
|
@@ -147,7 +154,7 @@ PerfectQueue::Worker.run(Dispatch) {
|
|
147
154
|
- **USR1:** graceful restart
|
148
155
|
- **HUP:** immediate restart
|
149
156
|
- **USR2:** reopen log files
|
150
|
-
- **INT:** detach process
|
157
|
+
- **INT:** detach process for live restarting
|
151
158
|
|
152
159
|
## Configuration
|
153
160
|
|
@@ -164,6 +171,7 @@ PerfectQueue::Worker.run(Dispatch) {
|
|
164
171
|
- **child_graceful_kill_limit:** threshold time to switch SIGTERM to SIGKILL (default: never)
|
165
172
|
- **child_heartbeat_interval:** interval to send heartbeat packets to a child process (default: 2 sec)
|
166
173
|
- **child_heartbeat_limit:** threshold time to detect freeze of a child process (default: 10.0 sec)
|
174
|
+
- **detach_wait:** sleep for this seconds before detaching process for live restarting (default: 10.0 sec)
|
167
175
|
|
168
176
|
## Backend types
|
169
177
|
|
data/lib/perfectqueue/version.rb
CHANGED
data/lib/perfectqueue/worker.rb
CHANGED
@@ -33,6 +33,7 @@ module PerfectQueue
|
|
33
33
|
@detach_wait = config[:detach_wait] || config['detach_wait'] || 10.0
|
34
34
|
|
35
35
|
@sv = Supervisor.new(runner, &block)
|
36
|
+
@detach = false
|
36
37
|
@finish_flag = BlockingFlag.new
|
37
38
|
end
|
38
39
|
|
@@ -46,14 +47,25 @@ module PerfectQueue
|
|
46
47
|
install_signal_handlers
|
47
48
|
|
48
49
|
begin
|
50
|
+
|
49
51
|
until @finish_flag.set?
|
50
52
|
pid, status = Process.waitpid2(@pid, Process::WNOHANG)
|
53
|
+
break if pid
|
51
54
|
@finish_flag.wait(1)
|
52
55
|
end
|
53
56
|
|
54
|
-
|
55
|
-
|
56
|
-
|
57
|
+
return if pid
|
58
|
+
|
59
|
+
if @detach
|
60
|
+
wait_time = Time.now + @detach_wait
|
61
|
+
while (w = wait_time - Time.now) > 0
|
62
|
+
sleep [1, w].max
|
63
|
+
pid, status = Process.waitpid2(@pid)
|
64
|
+
break if pid
|
65
|
+
end
|
66
|
+
|
67
|
+
else
|
68
|
+
# child process finished unexpectedly
|
57
69
|
end
|
58
70
|
|
59
71
|
rescue Errno::ECHILD
|
@@ -74,6 +86,7 @@ module PerfectQueue
|
|
74
86
|
|
75
87
|
def detach
|
76
88
|
send_signal(:INT)
|
89
|
+
@detach = true
|
77
90
|
@finish_flag.set!
|
78
91
|
end
|
79
92
|
|
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: perfectqueue
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.8.
|
4
|
+
version: 0.8.14
|
5
5
|
prerelease:
|
6
6
|
platform: ruby
|
7
7
|
authors:
|
@@ -9,7 +9,7 @@ authors:
|
|
9
9
|
autorequire:
|
10
10
|
bindir: bin
|
11
11
|
cert_chain: []
|
12
|
-
date: 2012-08-
|
12
|
+
date: 2012-08-16 00:00:00.000000000 Z
|
13
13
|
dependencies:
|
14
14
|
- !ruby/object:Gem::Dependency
|
15
15
|
name: sequel
|