disbatch 0.0.4

data/LICENSE

@@ -0,0 +1,27 @@
+Copyright (c) 2011, Matthew Berg <mberg@synacor.com>
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without
+modification, are permitted provided that the following conditions are met:
+
+    * Redistributions of source code must retain the above copyright notice,
+      this list of conditions and the following disclaimer.
+
+    * Redistributions in binary form must reproduce the above copyright
+      notice, this list of conditions and the following disclaimer in the
+      documentation and/or other materials provided with the distribution.
+
+    * The names of its contributors may not be used to endorse or promote
+      products derived from this software without specific prior written
+      permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS"
+AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE
+IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE
+DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE
+FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL
+DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR
+SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER
+CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY,
+OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE
+OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.

data/bin/disbatchd

@@ -0,0 +1,74 @@
+#!/usr/bin/env ruby
+
+require 'rubygems'
+
+require 'disbatch'
+require 'eventmachine'
+require 'json'
+require 'trollop'
+
+opts = Trollop.options do
+	opt 'config', 'the disbatchd config file', :default => '/etc/disbatch/disbatchd.conf'
+	opt 'max', 'maximum number of threads per queue', :default => 10
+	opt 'plugins', 'path(s) to dispatch plugins', :multi => true
+	opt 'force',  'force node registration'
+end
+
+config_file = opts['config']
+config = File.exists?(config_file) ? JSON.parse(File.read(config_file)) : {}
+config = opts.merge(config)
+force = config['force'] || false
+
+node = Disbatch.node.register(force)
+max  = config['max']
+
+config.has_key?('plugins') && config['plugins'].each { |dir| Disbatch::Plugin.init_all(dir) }
+Disbatch::Plugin.init_all(File.dirname(File.expand_path(__FILE__)) + '/../lib/disbatch/plugin/**/*.rb')
+
+trap('TERM') { node.release; exit }
+trap('INT')  { node.release; exit }
+trap('QUIT') { node.release; exit }
+
+EventMachine::run do
+
+	queues = Disbatch::Queue.get_all
+
+	queues.each do |queue|
+
+		next unless Disbatch::Plugin[queue.plugin]
+
+		puts "Adding #{queue.plugin} runner for #{queue.id}"
+
+		tg = ThreadGroup.new
+
+		EventMachine::add_periodic_timer(0.1) do
+
+			next unless ((nt = tg.list.length) < max && (np = queue.length) > 0)
+
+			ft = max - nt
+
+			(np > ft ? ft : np).times do
+				t = Thread.new do
+					task = queue.pop
+
+					unless defined?(task) && !task.nil? && task.params 
+						puts 'Huh?  Got invalid task back.'
+						next
+					end
+
+					begin
+						task.execute!
+					rescue
+						puts 'Damn.  Execution failed.'
+						task.release
+					end
+				end
+
+				tg.add(t)
+			end
+
+		end
+
+	end
+
+end

data/disbatch_specification.txt

@@ -0,0 +1,205 @@
+Disbatch Execution Node Specification 1.0
+Matt Busigin <mbusigin@gmail.com>
+May, 2011
+
+Disbatch is a multi-platform, multi-language, elastic, distributed batch
+processing framework. This document outlines the protocol which a Disbatch
+Execution Node (DEN) must follow to correctly operate with Disbatch Command
+Nodes (DCN), as well as other DENs operating under the same
+network/database.
+
+OVERVIEW
+
+Disbatch is a multi-platform, multi-language elastic distributed batch
+processing framework. It heavily leverages MongoDB and JSON. The core
+components are MongoDB, N-number of DENs (Disbatch Execution Nodes), and
+potentially one or more DCNs (Disbatch Command Node).
+
+Execution is split among queues, which are of homogenous type. Each queue
+class is implemented as a plugin on the node. Nodes which poll for queues
+that don't have requisite classes implemented are ignored.
+
+MECHANICS
+
+Each DEN is partitioned into queues. Each queue contains a pool of threads,
+which are limited by the 'maxthreads' queue parameter on a per-node basis.
+
+On DEN startup, the node registers itself, pulls the list of queues from the
+MongoDB, which are created one-document per queue in the 'queues' collection.
+
+TASK LIFECYCLE
+
+A task is first created, generally on a DCN, either individually, or as a
+result of a batch operation. Individual Tasks are saved to the 'tasks'
+collection, with a reference back to the queue (by ObjectId) it sits under.
+All tasks have a number of attributes in common: id, ctime, mtime, queue,
+node, parameters, stdout, stderr, status.
+
+The mechanism for creating the tasks is up to the implementor of the DCN, or
+even through some other mechanism. It is their job to create tasks to this
+specification so that DENs may execute them, and a DCN can provide an
+interface into the data.
+
+A task is created, initialised with parameters, its status set to Created,
+and its node set to Unclaimed. Upon each Scheduling Interval, every DEN will
+seek out these Created and Unclaimed tasks, putting them into a Pending and
+Claimed state up until the per-node maximum thread threshold (Maxthreads) is
+saturated.
+
+When the DEN is ready, the Task is actually put into the Running status. The
+plugin itself may report back status at any interval it likes, and it may
+alter the task object any way it likes, but it is the responsibility of the
+DEN to assure the status is up to date after it is complete, whether the
+plugin has adjusted the status correctly or not.
+
+DISBATCH EXECUTION NODE RESPONSIBILITIES
+
+REQUIREMENTS
+
+Each DEN:
+
+        MUST have a unique Node ID
+
+        MUST register itself via the DEN Node Registration Protocol
+
+        MUST execute tasks via the DEN Task Execution Protocol
+
+NODE ID
+
+Node ID should be a short (1-8 characters), and follow the standard C identifier methdology:
+
+[a-zA-Z][a-zA-Z_0-9]*
+
+
+STARTUP & INITIALISATION
+
+Before a DEN starts processing tasks, it is obligated to clean up any tasks
+that were not put into a completed state.
+
+
+DEN NODE REGISTRATION PROTOCOL
+
+Upon startup, each node must register itself to the Nodes collection. The
+following elements must be included:
+
+        _id:            ObjectId()                              // Node ID,
+        heartbeat:      {...}                                   // Heartbeat object (see
+                                                                   below)
+        version:        "r-disbatch 1.21"                       // Engine version string
+        spec_version:   "1.0"                                   // DEN Specification
+        queuetypes      [ { class: "Synacor::Migration::IMAP2IMAP",
+                            version: '1.1a' }, .. ]             // Array of plugins enabled
+        
+
+
+HEARTBEAT OBJECT
+
+The heartbeat object includes status updates on both the engine, as well as any queues active.
+
+        datetime:       ISODate("2011-05-12T06:00:07Z")         // Heartbeat timestamp
+        pid:            1234                                    // System PID
+        queues:         [...]                                   // Queue heartbeat objects (see
+                                                                   below)
+
+QUEUE HEARTBEAT OBJECT
+
+All of this data is on a per-queue basis.
+
+        class:          'Synacor::Migration::IMAP2IMAP'         // Plugin class
+        queue:          ObjectId()                              // Queue reference
+        threads         50                                      // Thread count
+
+
+
+FULL NODE JSON OBJECT EXAMPLE
+
+{
+        _id:            'mig01',
+        version:        'Disbatch.pl v2.0.2',
+        den_version:    '1.0',
+        queuetypes:     [
+                                { 
+                                        class: 'Synacor::Migration::IMAP2IMAP',
+                                        version: '1.1a'
+                                },
+                                {
+                                        class: 'Synacor::Migration::Zimbra::UserImport',
+                                        version: '0.9'
+                                }
+                        ],
+        heartbeat:      {
+                                datetime: ISODate("2011-05-12T06:00:07Z"),
+                        
+                                pid:            34182,
+                                queues: [
+                                                {
+                                                        class: 'Synacor::Migration::IMAP2IMAP',
+                                                        queue: ObjectId(),
+                                                        threads: 50,
+                                                        idle: 15,
+                                                        processing: 35
+                                                }
+                                        ]
+                        }
+}
+
+
+QUEUE DOCUMENT SPECIFICATION
+
+The DENs will read from the "queues" collection, and can expect the following
+format:
+
+{
+        _id:            'insight1'                      // Queue ID,
+        ctime:          ISODate("2011-05-12T06:00:07Z"),// Create time
+        class:          'Synacor::Migration::IMAP2IMAP',// Class
+        description:    'Insight Primary Migration',    // Description
+        maxthreads:     50,                             // Maximum per-node concurrency
+
+        nodes_pin:      [ 'mig01', 'mig02' ],           // Pin queue to these nodes (and no-where else)
+        nodes_ignore:   [ 'mig03', 'mig04' ]            // OR: ignore on these nodes
+}
+
+The "nodes_pin" and "nodes_ignore" attributes are mutually exclusive.  If
+using pin, it is exclusive to the specified nodes.  If using ignore, it will
+run on all nodes except for those specified.  If both are erroneously
+provided, nodes_pin takes precedence over nodes_ignore.
+
+Queue IDs can either be automatically generated, or provided as a uniquely
+named user parameter.
+
+
+TASK DOCUMENT SPECIFICATION
+
+Each task the DEN pulls will be in the following format:
+
+{
+        _id:            ObjectId('238d892efadee'),
+        queue:          'insight1',
+        ctime:          ISODate("2011-05-12T06:00:07Z"),// Create time
+        mtime:          ISODate("2011-05-12T06:00:07Z"),// Modify time
+        parameters:     { username: 'foo@foo.com', host: 'md08.foo.com' }, // Parameters
+        stdout:         '',                             // Your task output goes here
+        stderr:         '',                             // Your task errors go here
+        status:         -2,                             // Task status code (described below)
+        node:           -1
+}
+
+TASK STATUS CODE
+
+        -4:             Blocked
+        -3:             Terminated
+        -2:             Created
+        -1:             Claimed
+        0:              Running
+        1:              Concluded
+
+
+TASK LIFECYCLE
+
+Each task is initialised with its node as -1 (unclaimed) and status as -2 (created).
+
+DENs can claim tasks from a queue one of two ways:
+
+        1. findAndModify()
+                where: queue: <queue> and n

data/doc/Disbatch/NoNodeError.html

@@ -0,0 +1,128 @@
+<!DOCTYPE html PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+  "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+  <head>
+    <meta http-equiv="Content-Type" content="text/html; charset=utf-8" />
+<title>
+  Exception: Disbatch::NoNodeError
+  
+    &mdash; Documentation by YARD 0.7.3
+  
+</title>
+
+  <link rel="stylesheet" href="../css/style.css" type="text/css" media="screen" charset="utf-8" />
+
+  <link rel="stylesheet" href="../css/common.css" type="text/css" media="screen" charset="utf-8" />
+
+<script type="text/javascript" charset="utf-8">
+  relpath = '..';
+  if (relpath != '') relpath += '/';
+</script>
+
+  <script type="text/javascript" charset="utf-8" src="../js/jquery.js"></script>
+
+  <script type="text/javascript" charset="utf-8" src="../js/app.js"></script>
+
+
+  </head>
+  <body>
+    <script type="text/javascript" charset="utf-8">
+      if (window.top.frames.main) document.body.className = 'frames';
+    </script>
+    
+    <div id="header">
+      <div id="menu">
+  
+    <a href="../_index.html">Index (N)</a> &raquo; 
+    <span class='title'><span class='object_link'><a href="../Disbatch.html" title="Disbatch (module)">Disbatch</a></span></span>
+     &raquo; 
+    <span class="title">NoNodeError</span>
+  
+  
+  <div class="noframes"><span class="title">(</span><a href="." target="_top">no frames</a><span class="title">)</span></div>
+</div>
+
+      <div id="search">
+  
+    <a id="class_list_link" href="#">Class List</a>
+  
+    <a id="method_list_link" href="#">Method List</a>
+  
+    <a id="file_list_link" href="#">File List</a>
+  
+</div>
+      <div class="clear"></div>
+    </div>
+    
+    <iframe id="search_frame"></iframe>
+    
+    <div id="content"><h1>Exception: Disbatch::NoNodeError
+  
+  
+  
+</h1>
+
+<dl class="box">
+  
+    <dt class="r1">Inherits:</dt>
+    <dd class="r1">
+      <span class="inheritName">RuntimeError</span>
+      
+        <ul class="fullTree">
+          <li>Object</li>
+          
+            <li class="next">RuntimeError</li>
+          
+            <li class="next">Disbatch::NoNodeError</li>
+          
+        </ul>
+        <a href="#" class="inheritanceTree">show all</a>
+      
+      </dd>
+    
+  
+  
+    
+  
+    
+  
+  
+  
+    <dt class="r2 last">Defined in:</dt>
+    <dd class="r2 last">lib/disbatch/errors.rb</dd>
+  
+</dl>
+<div class="clear"></div>
+
+<h2>Overview</h2><div class="docstring">
+  <div class="discussion">
+    <p>
+Raised when attempting to open an invalid node
+</p>
+
+
+  </div>
+</div>
+<div class="tags">
+  
+
+</div>
+
+  
+
+
+  
+
+
+  
+
+</div>
+    
+    <div id="footer">
+  Generated on Mon Oct 24 16:54:00 2011 by 
+  <a href="http://yardoc.org" title="Yay! A Ruby Documentation Tool" target="_parent">yard</a>
+  0.7.3 (ruby-1.8.7).
+</div>
+
+  </body>
+</html>