perfectqueue 0.8.13 → 0.8.14

Sign up to get free protection for your applications and to get access to all the features.
data/ChangeLog CHANGED
@@ -1,4 +1,10 @@
1
1
 
2
+ == 2012-08-16 version 0.8.14
3
+
4
+ * Improved detach process routines
5
+ * Updated README.md
6
+
7
+
2
8
  == 2012-08-10 version 0.8.13
3
9
 
4
10
  * Force stop a processor when unknown exception occured during task heartbeat
data/README.md CHANGED
@@ -1,10 +1,17 @@
1
1
  # PerfectQueue
2
2
 
3
3
  PerfectQueue is a highly available distributed queue built on top of RDBMS.
4
- It provides at-least-once semantics; Even if a worker node fails during processing a task, the task is retried by another worker.
5
- PerfectQueue provides similar API to Amazon SQS. But unlike Amazon SQS, PerfectQueue never delivers finished tasks.
4
+ PerfectQueue provides similar API to Amazon SQS, while PerfectQueue focues on reliability and flexible scheduling rather than scalability.
6
5
 
7
- Since PerfectQueue also prevents storing a same task twice by using unique task identifier, client applications can retry to submit tasks until it succeeds.
6
+ PerfectQueue introduces following concepts:
7
+
8
+ * **At-least-once semantics:** Even if a worker node fails during processing a task, another worker takes over the task.
9
+ * **Multiuser-aware fair scheduling**: PerfectQueue schedules tasks submitted by users who have larger resource assignment.
10
+ * **Decider:** Decider is a simple mechanism to implement complex workflows on top of queues while keeping loose coupling.
11
+ * **Graceful and live restarting:** PerfectQueue continues processing of long-running tasks even during restarting.
12
+ * **Idempotent task submission:** All tasks have unique identifier and PerfectQueue prevents storing same task twice.
13
+ * Note: client applications should consider how to always generate a same string for a (semantically) same task.
14
+ * **Idempotent task processing support:** Workers can use a constant unique identifier to process tasks idempotently.
8
15
 
9
16
  All you have to consider is implementing idempotent worker programs. PerfectQueue manages the other problems.
10
17
 
@@ -147,7 +154,7 @@ PerfectQueue::Worker.run(Dispatch) {
147
154
  - **USR1:** graceful restart
148
155
  - **HUP:** immediate restart
149
156
  - **USR2:** reopen log files
150
- - **INT:** detach process
157
+ - **INT:** detach process for live restarting
151
158
 
152
159
  ## Configuration
153
160
 
@@ -164,6 +171,7 @@ PerfectQueue::Worker.run(Dispatch) {
164
171
  - **child_graceful_kill_limit:** threshold time to switch SIGTERM to SIGKILL (default: never)
165
172
  - **child_heartbeat_interval:** interval to send heartbeat packets to a child process (default: 2 sec)
166
173
  - **child_heartbeat_limit:** threshold time to detect freeze of a child process (default: 10.0 sec)
174
+ - **detach_wait:** sleep for this seconds before detaching process for live restarting (default: 10.0 sec)
167
175
 
168
176
  ## Backend types
169
177
 
@@ -1,3 +1,3 @@
1
1
  module PerfectQueue
2
- VERSION = "0.8.13"
2
+ VERSION = "0.8.14"
3
3
  end
@@ -33,6 +33,7 @@ module PerfectQueue
33
33
  @detach_wait = config[:detach_wait] || config['detach_wait'] || 10.0
34
34
 
35
35
  @sv = Supervisor.new(runner, &block)
36
+ @detach = false
36
37
  @finish_flag = BlockingFlag.new
37
38
  end
38
39
 
@@ -46,14 +47,25 @@ module PerfectQueue
46
47
  install_signal_handlers
47
48
 
48
49
  begin
50
+
49
51
  until @finish_flag.set?
50
52
  pid, status = Process.waitpid2(@pid, Process::WNOHANG)
53
+ break if pid
51
54
  @finish_flag.wait(1)
52
55
  end
53
56
 
54
- unless pid
55
- # child process is alive but detached
56
- sleep @detach_wait
57
+ return if pid
58
+
59
+ if @detach
60
+ wait_time = Time.now + @detach_wait
61
+ while (w = wait_time - Time.now) > 0
62
+ sleep [1, w].max
63
+ pid, status = Process.waitpid2(@pid)
64
+ break if pid
65
+ end
66
+
67
+ else
68
+ # child process finished unexpectedly
57
69
  end
58
70
 
59
71
  rescue Errno::ECHILD
@@ -74,6 +86,7 @@ module PerfectQueue
74
86
 
75
87
  def detach
76
88
  send_signal(:INT)
89
+ @detach = true
77
90
  @finish_flag.set!
78
91
  end
79
92
 
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: perfectqueue
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.8.13
4
+ version: 0.8.14
5
5
  prerelease:
6
6
  platform: ruby
7
7
  authors:
@@ -9,7 +9,7 @@ authors:
9
9
  autorequire:
10
10
  bindir: bin
11
11
  cert_chain: []
12
- date: 2012-08-10 00:00:00.000000000 Z
12
+ date: 2012-08-16 00:00:00.000000000 Z
13
13
  dependencies:
14
14
  - !ruby/object:Gem::Dependency
15
15
  name: sequel