perfectqueue 0.8.13 → 0.8.14

data/ChangeLog

@@ -1,4 +1,10 @@
 
+== 2012-08-16 version 0.8.14
+
+* Improved detach process routines
+* Updated README.md
+
+
 == 2012-08-10 version 0.8.13
 
 * Force stop a processor when unknown exception occured during task heartbeat

data/README.md

@@ -1,10 +1,17 @@
 # PerfectQueue
 
 PerfectQueue is a highly available distributed queue built on top of RDBMS.
-It provides at-least-once semantics; Even if a worker node fails during processing a task, the task is retried by another worker.
-PerfectQueue provides similar API to Amazon SQS. But unlike Amazon SQS, PerfectQueue never delivers finished tasks.
+PerfectQueue provides similar API to Amazon SQS, while PerfectQueue focues on reliability and flexible scheduling rather than scalability.
 
-Since PerfectQueue also prevents storing a same task twice by using unique task identifier, client applications can retry to submit tasks until it succeeds.
+PerfectQueue introduces following concepts:
+
+  * **At-least-once semantics:** Even if a worker node fails during processing a task, another worker takes over the task.
+  * **Multiuser-aware fair scheduling**: PerfectQueue schedules tasks submitted by users who have larger resource assignment.
+  * **Decider:** Decider is a simple mechanism to implement complex workflows on top of queues while keeping loose coupling.
+  * **Graceful and live restarting:** PerfectQueue continues processing of long-running tasks even during restarting.
+  * **Idempotent task submission:** All tasks have unique identifier and PerfectQueue prevents storing same task twice.
+    * Note: client applications should consider how to always generate a same string for a (semantically) same task.
+  * **Idempotent task processing support:** Workers can use a constant unique identifier to process tasks idempotently.
 
 All you have to consider is implementing idempotent worker programs. PerfectQueue manages the other problems.
 
@@ -147,7 +154,7 @@ PerfectQueue::Worker.run(Dispatch) {
 - **USR1:** graceful restart
 - **HUP:** immediate restart
 - **USR2:** reopen log files
-- **INT:** detach process
+- **INT:** detach process for live restarting
 
 ## Configuration
 
@@ -164,6 +171,7 @@ PerfectQueue::Worker.run(Dispatch) {
 - **child_graceful_kill_limit:** threshold time to switch SIGTERM to SIGKILL (default: never)
 - **child_heartbeat_interval:** interval to send heartbeat packets to a child process (default: 2 sec)
 - **child_heartbeat_limit:** threshold time to detect freeze of a child process (default: 10.0 sec)
+- **detach_wait:** sleep for this seconds before detaching process for live restarting (default: 10.0 sec)
 
 ## Backend types
 

data/lib/perfectqueue/version.rb

@@ -1,3 +1,3 @@
 module PerfectQueue
-  VERSION = "0.8.13"
+  VERSION = "0.8.14"
 end

data/lib/perfectqueue/worker.rb

@@ -33,6 +33,7 @@ module PerfectQueue
       @detach_wait = config[:detach_wait] || config['detach_wait'] || 10.0
 
       @sv = Supervisor.new(runner, &block)
+      @detach = false
       @finish_flag = BlockingFlag.new
     end
 
@@ -46,14 +47,25 @@ module PerfectQueue
       install_signal_handlers
 
       begin
+
         until @finish_flag.set?
           pid, status = Process.waitpid2(@pid, Process::WNOHANG)
+          break if pid
           @finish_flag.wait(1)
         end
 
-        unless pid
-          # child process is alive but detached
-          sleep @detach_wait
+        return if pid
+
+        if @detach
+          wait_time = Time.now + @detach_wait
+          while (w = wait_time - Time.now) > 0
+            sleep [1, w].max
+            pid, status = Process.waitpid2(@pid)
+            break if pid
+          end
+
+        else
+          # child process finished unexpectedly
         end
 
       rescue Errno::ECHILD
@@ -74,6 +86,7 @@ module PerfectQueue
 
     def detach
       send_signal(:INT)
+      @detach = true
       @finish_flag.set!
     end
 

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: perfectqueue
 version: !ruby/object:Gem::Version
-  version: 0.8.13
+  version: 0.8.14
   prerelease: 
 platform: ruby
 authors:
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2012-08-10 00:00:00.000000000 Z
+date: 2012-08-16 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: sequel