perfectqueue 0.7.0

data/ChangeLog

@@ -0,0 +1,16 @@
+
+== 2011-08-31 version 0.7.0
+
+* Backend uses logical delete instead of deleting tasks when task is finished
+  or canceled. It surely raises error when duplicated job id is submitted.
+
+
+== 2011-08-25 version 0.6.1
+
+* Supports SimpleDB Backend
+
+
+== 2011-08-23 version 0.6.0
+
+* First release
+

data/README.rdoc

@@ -0,0 +1,223 @@
+= PerfectQueue
+
+Highly available distributed queue.
+
+It provides exactly-once semantics unless backend database fails. Pushed tasks are surely retried by another worker node even if a worker fails. And it never delivers finished/canceled tasks.
+
+Backend database is pluggable. You can use any databases that supports CAS (compare-and-swap) operation. PerfectQueue supports RDBMS and Amazon SimpleDB for now.
+
+
+== Architecture
+
+PerfectQueue uses following database schema:
+
+  (
+    id:string        -- unique identifier of the task
+    data:blob        -- additional attributes of the task
+    created_at:int   -- unix time when the task is created (or null for canceled tasks)
+    timeout:int
+  )
+
+1. list: lists tasks whose timeout column is old enough.
+2. lock: updates timeout column of the first task
+3. run: executes a command
+   * if the task takes long time, updates the timeout column. this is repeated until the task is finished
+   * if the task takes more long time, kills the process
+4. remove: if it succeeded, removes the row from the backend database
+5. or leave: if it failed, leave the row and expect to be retried
+
+
+== Usage
+
+=== Submitting a task
+
+*Using* *command* *line:*
+
+  # RDBMS
+  $ perfectqueue \
+        --database mysql://user:password@localhost/mydb \
+        --table perfectqueue \
+        --push unique-key-id='{"any":"data"}'
+
+  # SimpleDB
+  $ perfectqueue \
+        --simpledb your-simpledb-domain-name \
+        -k AWS_KEY_ID \
+        -s AWS_SECRET_KEY \
+        --push unique-key-id='{"any":"data"}'
+
+*Using* *PerfectQueue* *library:*
+
+  require 'perfectqueue'
+  
+  # RDBMS
+  require 'perfectqueue/backend/rdb'
+  queue = PerfectQueue::Backend::RDBBackend.new(
+         'mysql://user:password@localhost/mydb', table='perfectqueue')
+
+  # SimpleDB
+  require 'perfectqueue/backend/simpledb'
+  queue = PerfectQueue::Backend::SimpleDBBackend.new(
+         'AWS_KEY_ID', 'AWS_SECRET_KEY', 'your-simpledb-domain-name')
+  
+  queue.submit('unique-key-id', '{"any":"data"}')
+
+
+Alternatively, you can insert a row into the backend database directly.
+
+*RDBMS:*
+
+  > CREATE TABLE IF NOT EXISTS perfectqueue (
+        id VARCHAR(256) NOT NULL,
+        timeout INT NOT NULL,
+        data BLOB NOT NULL,
+        created_at INT,
+        PRIMARY KEY (id)
+      );
+  > SET @now = UNIX_TIMESTAMP();
+  > INSERT INTO perfectqueue (id, timeout, data, created_at)
+      VALUES ('unique-task-id', @now, '{"any":"data"}', @now);
+
+*SimpleDB:*
+
+  require 'aws'  # gem install aws-sdk
+  queue = AWS::SimpleDB.new
+  domain = queue.domains['your-simpledb-domain-name']
+  
+  now = "%08x" % Time.now.to_i
+  domain.items['unique-task-id'].attributes.replace(
+      'timeout'=>now, 'data'=>'{"any":"data"}', 'created_at'=>now,
+      :unless=>'timeout')
+
+
+=== Canceling a queued task
+
+*Using* *command* *line:*
+
+  $ perfectqueue ... --cancel unique-key-id
+
+*Using* *PerfectQueue* *library:*
+
+  queue.cancel('unique-key-id')
+
+
+Alternatively, you can delete a row from the backend database directly.
+
+*RDBMS:*
+
+  > DELETE FROM perfectqueue WHERE id='unique-key-id';
+
+*SimpleDB:*
+
+  domain.items['unique-task-id'].delete
+
+
+=== Running worker node
+
+Use _perfectqueue_ command to execute a command.
+
+  Usage: perfectqueue [options] [-- <ARGV-for-exec-or-run>]
+  
+          --push ID=DATA               Push a task to the queue
+          --list                       Show queued tasks
+          --cancel ID                  Cancel a queued task
+          --configure PATH.yaml        Write configuration file
+  
+          --exec COMMAND               Execute command
+          --run SCRIPT.rb              Run method named 'run' defined in the script
+  
+      -f, --file PATH.yaml             Read configuration file
+      -C, --run-class                  Class name for --run (default: ::Run)
+      -t, --timeout SEC                Time for another worker to take over a task when this worker goes down (default: 600)
+      -b, --heartbeat-interval SEC     Threshold time to extend the timeout (heartbeat interval) (default: timeout * 3/4)
+      -x, --kill-timeout SEC           Threshold time to kill a task process (default: timeout * 10)
+      -X, --kill-interval SEC          Threshold time to retry killing a task process (default: 60)
+      -i, --poll-interval SEC          Polling interval (default: 1)
+      -r, --retry-wait SEC             Time to retry a task when it is failed (default: same as timeout)
+      -e, --expire SEC                 Threshold time to expire a task (default: 345600 (4days))
+  
+          --database URI               Use RDBMS for the backend database (e.g.: mysql://user:password@localhost/mydb)
+          --table NAME                 backend: name of the table (default: perfectqueue)
+          --simpledb DOMAIN            Use Amazon SimpleDB for the backend database (e.g.: --simpledb mydomain -k KEY_ID -s SEC_KEY)
+      -k, --key-id ID                  AWS Access Key ID
+      -s, --secret-key KEY             AWS Secret Access Key
+  
+      -w, --worker NUM                 Number of worker threads (default: 1)
+      -d, --daemon PIDFILE             Daemonize (default: foreground)
+      -o, --log PATH                   log file path
+      -v, --verbose                    verbose mode
+
+
+==== --exec
+
+Execute a command when a task is received. The the data column is passed to the stdin and the id column passed to the last argument. The command have to exit with status code 0 when it succeeded.
+
+*Example:*
+
+  #!/usr/bin/env ruby
+  
+  require 'json'
+  js = JSON.load(STDIN.read)
+  puts "received: id=#{ARGV.last} #{js.inspect}"
+  
+  #$ perfectqueue --database sqlite://test.db \
+     --exec ./this_file -- my cmd args
+
+When the kill timeout (-x, --kill-timeout) is elapsed, SIGTERM signal will be sent to the child process. The signal will be repeated every few seconds (-X, --kill-interval).
+
+
+==== --run
+
+This is same as 'exec' except that it creates a instance of a class named 'Run' defined in the file. The class should has 'initialize(task)', 'run' and 'kill' methods. You can get data column and id column of the task from the argument of the initialize method. It is assumed it succeeded if the method doesn't raise any errors.
+
+*Example:*
+
+  require 'json'
+  
+  class Run
+    def initialize(task)
+      @task = task
+    end
+  
+    def run
+      js = JSON.load(@task.data)
+      puts "received: id=#{@task.id} #{js.inspect}"
+    end
+  
+    def kill
+      puts "kill!"
+    end
+  end
+  
+  #$ perfectqueue --database sqlite://test.db \
+     --run ./this_file.rb -- my cmd args
+
+When the kill timeout (-x, --klill-timeout) is elapsed, Run#kill method will be called (if it is defined). It will be repeated every few seconds (-X, --kill-retry).
+
+
+==== --configure
+
+Write configuration file and exit. Written configuration file can be used with -f option:
+
+*Example:*
+
+  ## create myqueue.yaml file
+  $ perfectqueue --database mysql://root:my@localhost/mydb \
+    --run myrun.rb -- my cmd args \
+    --configure myqueue.yaml
+
+  ## run perfectqueue using the configuration file
+  $ perfectqueue -f myqueue.yaml
+
+
+==== --list
+
+Show queued tasks.
+
+*Example:*
+
+  $ perfectqueue --database sqlite://test.db --list
+                          id                 created_at                    timeout  data
+                       task1  2011-08-23 23:07:45 +0900  2011-08-23 23:07:45 +0900  {"attr1":"val1","attr":"val2"}
+  1 entries.
+

data/bin/perfectqueue

@@ -0,0 +1,6 @@
+#!/usr/bin/env ruby
+# -*- coding: utf-8 -*-
+require 'rubygems' unless defined?(gem)
+here = File.dirname(__FILE__)
+$LOAD_PATH << File.expand_path(File.join(here, '..', 'lib'))
+require 'perfectqueue/command/perfectqueue'

data/lib/perfectqueue.rb

@@ -0,0 +1,4 @@
+require 'perfectqueue/engine'
+require 'perfectqueue/worker'
+require 'perfectqueue/backend'
+require 'perfectqueue/version'

data/lib/perfectqueue/backend.rb

@@ -0,0 +1,51 @@
+
+module PerfectQueue
+
+
+class Task
+  def initialize(id, created_at, data)
+    @id = id
+    @created_at = created_at
+    @data = data
+  end
+
+  attr_reader :id, :created_at, :data
+end
+
+
+class CanceledError < RuntimeError
+end
+
+
+class Backend
+  # => list {|id,created_at,data,timeout| ... }
+  def list(&block)
+  end
+
+  # => token, task
+  def acquire(timeout, now=Time.now.to_i)
+  end
+
+  # => true (success) or false (canceled)
+  def finish(token, delete_timeout=3600, now=Time.now.to_i)
+  end
+
+  # => nil
+  def update(token, timeout)
+  end
+
+  # => true (success) or false (not found, canceled or finished)
+  def cancel(id, delete_timeout=3600, now=Time.now.to_i)
+  end
+
+  # => true (success) or nil (already exists)
+  def submit(id, data, time=Time.now.to_i)
+  end
+
+  def close
+  end
+end
+
+
+end
+

data/lib/perfectqueue/backend/rdb.rb

@@ -0,0 +1,119 @@
+
+module PerfectQueue
+
+
+class RDBBackend < Backend
+  def initialize(uri, table)
+    require 'sequel'
+    @uri = uri
+    @table = table
+    @db = Sequel.connect(@uri)
+    init_db(@uri.split(':',2)[0])
+  end
+
+  private
+  def init_db(type)
+    sql = ''
+    case type
+    when /mysql/i
+      sql << "CREATE TABLE IF NOT EXISTS `#{@table}` ("
+      sql << "  id VARCHAR(256) NOT NULL,"
+      sql << "  timeout INT NOT NULL,"
+      sql << "  data BLOB NOT NULL,"
+      sql << "  created_at INT,"
+      sql << "  PRIMARY KEY (id)"
+      sql << ") ENGINE=INNODB;"
+    else
+      sql << "CREATE TABLE IF NOT EXISTS `#{@table}` ("
+      sql << "  id VARCHAR(256) NOT NULL,"
+      sql << "  timeout INT NOT NULL,"
+      sql << "  data BLOB NOT NULL,"
+      sql << "  created_at INT,"
+      sql << "  PRIMARY KEY (id)"
+      sql << ");"
+    end
+    # TODO index
+    connect {
+      @db.run sql
+    }
+  end
+
+  def connect(&block)
+    begin
+      block.call
+    ensure
+      @db.disconnect
+    end
+  end
+
+  public
+  def list(&block)
+    @db.fetch("SELECT id, timeout, data, created_at FROM `#{@table}` WHERE created_at IS NOT NULL ORDER BY timeout ASC;") {|row|
+      yield row[:id], row[:created_at], row[:data], row[:timeout]
+    }
+  end
+
+  MAX_SELECT_ROW = 32
+
+  def acquire(timeout, now=Time.now.to_i)
+    connect {
+      while true
+        rows = 0
+        @db.fetch("SELECT id, timeout, data, created_at FROM `#{@table}` WHERE timeout <= ? ORDER BY timeout ASC LIMIT #{MAX_SELECT_ROW};", now) {|row|
+
+          unless row[:created_at]
+            # finished/canceled task
+            @db["DELETE FROM `#{@table}` WHERE id=?;", row[:id]].delete
+
+          else
+            n = @db["UPDATE `#{@table}` SET timeout=? WHERE id=? AND timeout=?;", timeout, row[:id], row[:timeout]].update
+            if n > 0
+              return row[:id], Task.new(row[:id], row[:created_at], row[:data])
+            end
+          end
+
+          rows += 1
+        }
+        if rows < MAX_SELECT_ROW
+          return nil
+        end
+      end
+    }
+  end
+
+  def finish(id, delete_timeout=3600, now=Time.now.to_i)
+    connect {
+      n = @db["UPDATE `#{@table}` SET timeout=?, created_at=NULL WHERE id=? AND created_at IS NOT NULL;", now+delete_timeout, id].update
+      return n > 0
+    }
+  end
+
+  def update(id, timeout)
+    connect {
+      n = @db["UPDATE `#{@table}` SET timeout=? WHERE id=? AND created_at IS NOT NULL;", timeout, id].update
+      if n <= 0
+        raise CanceledError, "Task id=#{id} is canceled."
+      end
+      return nil
+    }
+  end
+
+  def cancel(id, delete_timeout=3600, now=Time.now.to_i)
+    finish(id, delete_timeout, now)
+  end
+
+  def submit(id, data, time=Time.now.to_i)
+    connect {
+      begin
+        n = @db["INSERT INTO `#{@table}` (id, timeout, data, created_at) VALUES (?, ?, ?, ?);", id, time, data, time].insert
+        return true
+      rescue Sequel::DatabaseError
+        return nil
+      end
+    }
+  end
+end
+
+
+end
+

data/lib/perfectqueue/backend/simpledb.rb

@@ -0,0 +1,136 @@
+
+module PerfectQueue
+
+
+class SimpleDBBackend < Backend
+  def initialize(key_id, secret_key, domain)
+    gem "aws-sdk"
+    require 'aws'
+    @consistent_read = false
+
+    @db = AWS::SimpleDB.new(
+      :access_key_id => key_id,
+      :secret_access_key => secret_key)
+
+    @domain_name = domain
+    @domain = @db.domains[@domain_name]
+    unless @domain.exists?
+      @domain = @db.domains.create(@domain_name)
+    end
+  end
+
+  attr_accessor :consistent_read
+
+  def use_consistent_read(b=true)
+    @consistent_read = b
+    self
+  end
+
+  def list(&block)
+    @domain.items.each {|item|
+      id = item.name
+      attrs = item.data.attributes
+      salt = attrs['created_at'].first
+      if salt && !salt.empty?
+        created_at = int_decode(salt)
+        data = attrs['data'].first
+        timeout = int_decode(attrs['timeout'].first)
+        yield id, created_at, data, timeout
+      end
+    }
+  end
+
+  MAX_SELECT_ROW = 32
+
+  def acquire(timeout, now=Time.now.to_i)
+    while true
+      rows = 0
+      @domain.items.select('timeout', 'data', 'created_at',
+                          :where => "timeout <= '#{int_encode(now)}'",
+                          :order => [:timeout, :asc],
+                          :consistent_read => @consistent_read,
+                          :limit => MAX_SELECT_ROW) {|itemdata|
+        begin
+          id = itemdata.name
+          attrs = itemdata.attributes
+          salt = attrs['created_at'].first
+
+          if !salt || salt.empty?
+            # finished/canceled task
+            @domain.items[id].delete(:if=>{'created_at'=>''})
+
+          else
+            created_at = int_decode(salt)
+            @domain.items[id].attributes.replace('timeout'=>int_encode(timeout),
+                :if=>{'timeout'=>attrs['timeout'].first})
+
+            data = attrs['data'].first
+
+            return [id,salt], Task.new(id, created_at, data)
+          end
+
+        rescue AWS::SimpleDB::Errors::ConditionalCheckFailed, AWS::SimpleDB::Errors::AttributeDoesNotExist
+        end
+
+        rows += 1
+      }
+      if rows < MAX_SELECT_ROW
+        return nil
+      end
+    end
+  end
+
+  def finish(token, delete_timeout=3600, now=Time.now.to_i)
+    begin
+      id, salt = *token
+      @domain.items[id].attributes.replace('timeout'=>int_encode(now+delete_timeout), 'created_at'=>'',
+          :if=>{'created_at'=>salt})
+      return true
+    rescue AWS::SimpleDB::Errors::ConditionalCheckFailed, AWS::SimpleDB::Errors::AttributeDoesNotExist
+      return false
+    end
+  end
+
+  def update(token, timeout)
+    begin
+      id, salt = *token
+      @domain.items[id].attributes.replace('timeout'=>int_encode(timeout),
+          :if=>{'created_at'=>salt})
+    rescue AWS::SimpleDB::Errors::ConditionalCheckFailed, AWS::SimpleDB::Errors::AttributeDoesNotExist
+      raise CanceledError, "Task id=#{id} is canceled."
+    end
+    nil
+  end
+
+  def cancel(id, delete_timeout=3600, now=Time.now.to_i)
+    salt = @domain.items[id].attributes['created_at'].first
+    unless salt
+      return false
+    end
+    token = [id,salt]
+    finish(token, delete_timeout, now)
+  end
+
+  def submit(id, data, time=Time.now.to_i)
+    begin
+      @domain.items[id].attributes.replace('timeout'=>int_encode(time), 'created_at'=>int_encode(time), 'data'=>data,
+          :unless=>'timeout')
+      return true
+    rescue AWS::SimpleDB::Errors::ConditionalCheckFailed, AWS::SimpleDB::Errors::ExistsAndExpectedValue
+      return nil
+    end
+  end
+
+  private
+  def int_encode(num)
+    "%08x" % num
+  end
+
+  def int_decode(str)
+    str.to_i(16)
+  end
+end
+
+
+end
+