perfectqueue 0.7.32 → 0.8.0

data/.gitignore

@@ -0,0 +1,6 @@
+.bundle
+Gemfile.lock
+vendor/*
+coverage/*
+pkg/*
+spec/test.db

data/ChangeLog

@@ -1,65 +1,3 @@
-== 2016-07-22 version 0.7.32
-
-DON'T USE THIS VERSION UNLESS YOU KNOW WHAT THIS IS!
-This version is for migration v07 to v0.8.48.
-
-* Use PerfectQueue v0.8.48's timeout model (#41)
-
-
-== 2014-10-08 version 0.7.31
-
-* Upgraded sequal v3.48.0
-
-
-== 2014-04-24 version 0.7.30
-
-* RDBBackend: fixed NoMethodError for @url
-
-
-== 2014-04-24 version 0.7.29
-
-* RDBBackend: add MySQL's SSL support
-
-
-== 2014-02-18 version 0.7.28
-
-* RDBBackend: wrap blob data in Sequel::SQL::Blob
-
-
-== 2013-06-16 version 0.7.27
-
-* RDBBackend: #issue should not ignore erros excepting duplicate entry error
-
-
-== 2013-06-14 version 0.7.26
-
-* RDBBackend: #issue should not ignore erros excepting duplicate entry error
-
-
-== 2012-09-19 version 0.7.25
-
-* removed depenedency on aws-sdk which is optional
-
-
-== 2012-08-10 version 0.7.24
-
-* Kill running task if an unexpected error occured during extending timeout
-
-
-== 2012-08-03 version 0.7.23
-
-* Use free slot ratio instead of free slot amount to decide task priorities
-
-
-== 2012-07-19 version 0.7.22
-
-* Fixed RDBBackend SQL
-
-
-== 2012-07-19 version 0.7.21
-
-* Added max_running column for fair resource restriction
-
 
 == 2012-01-24 version 0.7.20
 

data/Gemfile

@@ -0,0 +1,3 @@
+source :rubygems
+
+gemspec

data/README.md

@@ -0,0 +1,239 @@
+# PerfectQueue
+
+PerfectQueue is a highly available distributed queue built on top of RDBMS.
+It provides at-least-once semantics; Even if a worker node fails during processing a task, the task is retried by another worker.
+PerfectQueue provides similar API to Amazon SQS. But unlike Amazon SQS, PerfectQueue never delivers finished tasks.
+
+Since PerfectQueue also prevents storing a same task twice by using unique task identifier, client applications can retry to submit tasks until it succeeds.
+
+All you have to consider is implementing idempotent worker programs. PerfectQueue manages the other problems.
+
+## API overview
+
+```
+# open a queue
+PerfectQueue.open(config, &block)  #=> #<Queue>
+
+# submit a task
+Queue#submit(task_id, type, data, options={})
+
+# poll a task
+# (you don't have to use this method directly. see following sections)
+Queue#poll  #=> #<AcquiredTask>
+
+# get data associated with a task
+AcquiredTask#data  #=> #<Hash>
+
+# finish a task
+AcquiredTask#finish!
+
+# retry a task
+AcquiredTask#retry!
+
+# create a task reference
+Queue#[](key)  #=> #<Task>
+
+# chack the existance of the task
+Task#exists?
+
+# request to cancel a task
+# (actual behavior depends on the worker program)
+Task#cancel_request!
+
+# force finish a task
+# be aware that worker programs can't detect it
+Task#force_finish!
+```
+
+### Error classes
+
+```
+TaskError
+
+##
+# Workers may get these errors:
+#
+
+CancelRequestedError < TaskError
+
+AlreadyFinishedError < TaskError
+
+PreemptedError < TaskError
+
+ProcessStopError < RuntimeError
+
+ImmediateProcessStopError < ProcessStopError
+
+GracefulProcessStopError < ProcessStopError
+
+##
+# Client or other situation:
+#
+
+ConfigError < RuntimeError
+
+NotFoundError < TaskError
+
+AlreadyExistsError < TaskError
+
+NotSupportedError < TaskError
+```
+
+
+###  Example
+
+```ruby
+# submit tasks
+PerfectQueue.open(config) {|queue|
+  data = {'key'=>"value"}
+  queue.submit("task-id", "type1", data)
+}
+```
+
+
+## Writing a worker application
+
+### 1. Implement PerfectQueue::Application::Base
+
+```ruby
+class TestHandler < PerfectQueue::Application::Base
+  # implement run method
+  def run
+    # do something ...
+    puts "acquired task: #{task.inspect}"
+
+    # call task.finish!, task.retry! or task.release!
+    task.finish!
+  end
+end
+```
+
+### 2. Implement PerfectQueue::Application::Dispatch
+
+```ruby
+class Dispatch < PerfectQueue::Application::Dispatch
+  # describe routing
+  route "type1" => TestHandler
+  route /^regexp-.*$/ => :TestHandler  # String or Regexp => Class or Symbol
+end
+```
+
+### 3. Run the worker
+
+In a launcher script or rake file:
+
+```ruby
+system('perfectqueue run -I. -rapp/workers/dispatch Dispatch')
+```
+
+or:
+
+```ruby
+require 'perfectqueue'
+require 'app/workers/dispatch'
+
+PerfectQueue::Worker.run(Dispatch) {
+  # this method is called when the worker process is restarted
+  raw = File.read('config/perfectqueue.yml')
+  yml = YAJL.load(raw)
+  yml[ENV['RAILS_ENV'] || 'development']
+}
+```
+
+### Signal handlers
+
+- **TERM**,**INT:** graceful shutdown
+- **QUIT:** immediate shutdown
+- **USR1:** graceful restart
+- **HUP:** immediate restart
+- **WINCH:** immediate binary replace
+- **CONT:** graceful binary replace
+- **USR2:** reopen log files
+
+## Configuration
+
+- **type:** backend type (required; see following sections)
+- **log:** log file path (default: use stderr)
+- **processors:** number of child processes (default: 1)
+- **processor_type:** type of processor ('process' or 'thread') (default: 'process')
+- **poll_interval:** interval to poll tasks in seconds (default: 1.0 sec)
+- **retention_time:** duration to retain finished tasks (default: 300 sec)
+- **task_heartbeat_interval:** interval to send heartbeat requests (default: 2 sec)
+- **alive_time:** duration to continue a heartbeat request (default: 300 sec)
+- **retry_wait:** duration to retry a retried task (default: 300 sec)
+- **child_kill_interval:** interval to send signals to a child process (default: 2.0 sec)
+- **child_graceful_kill_limit:** threshold time to switch SIGTERM to SIGKILL (default: never)
+- **child_heartbeat_interval:** interval to send heartbeat packets to a child process (default: 2 sec)
+- **child_heartbeat_limit:** threshold time to detect freeze of a child process (default: 10.0 sec)
+
+## Backend types
+
+### rdb\_compat
+
+additional configuration:
+
+- **url:** URL to the RDBMS (example: 'mysql://user:password@host:port/database')
+- **table:** name of the table to use
+
+### rdb
+
+Not implemented yet.
+
+
+## Command line management tool
+
+```
+Usage: perfectqueue [options] <command>
+
+commands:
+    list                             Show list of tasks
+    submit <key> <type> <data>       Submit a new task
+    cancel_request <key>             Cancel request
+    force_finish <key>               Force finish a task
+    run <class>                      Run a worker process
+    init                             Initialize a backend database
+
+options:
+    -e, --environment ENV            Framework environment (default: development)
+    -c, --config PATH.yml            Path to a configuration file (default: config/perfectqueue.yml)
+
+options for submit:
+    -u, --user USER                  Set user
+    -t, --time UNIXTIME              Set time to run the task
+
+options for run:
+    -I, --include PATH               Add $LOAD_PATH directory
+    -r, --require PATH               Require files before starting
+```
+
+
+### initializing a database
+
+    # assume that the config/perfectqueue.yml exists
+    $ perfectqueue init
+
+### submitting a task
+
+    $ perfectqueue submit k1 user_task '{"uid":1}' -u user_1
+
+### listing tasks
+
+    $ perfectqueue list
+                               key            type               user             status                   created_at                      timeout   data
+                                k1       user_task             user_1            waiting    2012-05-18 13:05:31 -0700    2012-05-18 14:27:36 -0700   {"uid"=>1, "type"=>"user_task"}
+                                k2       user_task             user_2            waiting    2012-05-18 13:35:33 -0700    2012-05-18 14:35:33 -0700   {"uid"=>2, "type"=>"user_task"}
+                                k3     system_task                               waiting    2012-05-18 14:04:02 -0700    2012-05-22 15:04:02 -0700   {"task_id"=>32, "type"=>"system_task"}
+    3 entries.
+
+### cancel a tasks
+
+    $ perfectqueue cancel_request k1
+
+### force finish a tasks
+
+    $ perfectqueue cancel_request k2
+
+### running a worker
+
+    $ perfectqueue run -I. -Ilib -rconfig/boot.rb -rapps/workers/task_dispatch.rb TaskDispatch
+

data/Rakefile

@@ -0,0 +1,19 @@
+require 'bundler'
+Bundler::GemHelper.install_tasks
+
+require 'rspec/core'
+require 'rspec/core/rake_task'
+
+RSpec::Core::RakeTask.new(:spec) do |t|
+  t.rspec_opts = ["-c", "-f progress", "-r ./spec/spec_helper.rb"]
+  t.pattern = 'spec/**/*_spec.rb'
+  t.verbose = true
+end
+
+task :coverage do |t|
+  ENV['SIMPLE_COV'] = '1'
+  Rake::Task["spec"].invoke
+end
+
+task :default => :build
+

data/lib/perfectqueue.rb

@@ -1,4 +1,68 @@
-require 'perfectqueue/engine'
-require 'perfectqueue/worker'
-require 'perfectqueue/backend'
-require 'perfectqueue/version'
+#
+# PerfectQueue
+#
+# Copyright (C) 2012 FURUHASHI Sadayuki
+#
+#    Licensed under the Apache License, Version 2.0 (the "License");
+#    you may not use this file except in compliance with the License.
+#    You may obtain a copy of the License at
+#
+#        http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS,
+#    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#    See the License for the specific language governing permissions and
+#    limitations under the License.
+#
+
+module PerfectQueue
+  require 'json'
+  require 'thread'  # Mutex, CoditionVariable
+
+  {
+    :Application => 'perfectqueue/application',
+    :Backend => 'perfectqueue/backend',
+    :BackendHelper => 'perfectqueue/backend',
+    :BlockingFlag => 'perfectqueue/blocking_flag',
+    :Client => 'perfectqueue/client',
+    :DaemonsLogger => 'perfectqueue/daemons_logger',
+    :Engine => 'perfectqueue/engine',
+    :Model => 'perfectqueue/model',
+    :Queue => 'perfectqueue/queue',
+    :Runner => 'perfectqueue/runner',
+    :Task => 'perfectqueue/task',
+    :TaskWithMetadata => 'perfectqueue/task',
+    :AcquiredTask => 'perfectqueue/task',
+    :TaskMetadata => 'perfectqueue/task_metadata',
+    :TaskMonitor => 'perfectqueue/task_monitor',
+    :TaskMetadataAccessors => 'perfectqueue/task_metadata',
+    :TaskStatus => 'perfectqueue/task_status',
+    :Worker => 'perfectqueue/worker',
+    :SignalQueue => 'perfectqueue/signal_queue',
+  }.each_pair {|k,v|
+    autoload k, File.expand_path(v, File.dirname(__FILE__))
+  }
+  [
+    'perfectqueue/multiprocess',
+    'perfectqueue/error',
+  ].each {|v|
+    require File.expand_path(v, File.dirname(__FILE__))
+  }
+
+  def self.open(config, &block)
+    c = Client.new(config)
+    begin
+      q = Queue.new(c)
+      if block
+        block.call(q)
+      else
+        c = nil
+        return q
+      end
+    ensure
+      c.close if c
+    end
+  end
+end
+

data/lib/perfectqueue/application.rb

@@ -0,0 +1,30 @@
+#
+# PerfectQueue
+#
+# Copyright (C) 2012 FURUHASHI Sadayuki
+#
+#    Licensed under the Apache License, Version 2.0 (the "License");
+#    you may not use this file except in compliance with the License.
+#    You may obtain a copy of the License at
+#
+#        http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS,
+#    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#    See the License for the specific language governing permissions and
+#    limitations under the License.
+#
+
+module PerfectQueue
+  module Application
+    {
+      :Dispatch => 'application/dispatch',
+      :Router => 'application/router',
+      :Base => 'application/base',
+    }.each_pair {|k,v|
+      autoload k, File.expand_path(v, File.dirname(__FILE__))
+    }
+  end
+end
+

data/lib/perfectqueue/application/base.rb

@@ -0,0 +1,27 @@
+#
+# PerfectQueue
+#
+# Copyright (C) 2012 FURUHASHI Sadayuki
+#
+#    Licensed under the Apache License, Version 2.0 (the "License");
+#    you may not use this file except in compliance with the License.
+#    You may obtain a copy of the License at
+#
+#        http://www.apache.org/licenses/LICENSE-2.0
+#
+#    Unless required by applicable law or agreed to in writing, software
+#    distributed under the License is distributed on an "AS IS" BASIS,
+#    WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+#    See the License for the specific language governing permissions and
+#    limitations under the License.
+#
+
+module PerfectQueue
+  module Application
+
+    class Base < Runner
+    end
+
+  end
+end
+