starfish 1.0.0

data/LICENSE

@@ -0,0 +1,10 @@
+Copyright (c) 2006, Lucas Carlson
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without modification, are permitted provided that the following conditions are met:
+
+Redistributions of source code must retain the above copyright notice, this list of conditions and the following disclaimer.
+Redistributions in binary form must reproduce the above copyright notice, this list of conditions and the following disclaimer in the documentation and/or other materials provided with the distribution.
+Neither the name of the Lucas Carlson nor the names of its contributors may be used to endorse or promote products derived from this software without specific prior written permission.
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED. IN NO EVENT SHALL THE COPYRIGHT OWNER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT, INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA, OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY, WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE) ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE POSSIBILITY OF SUCH DAMAGE.
+

data/README

@@ -0,0 +1,136 @@
+== Welcome to Starfish
+
+Starfish is a utility born from abstracting code from a highly popular website (http://mog.com) to make distributed programming ridiculously easy.
+
+== Download
+
+* gem install starfish
+* http://rubyforge.org/projects/starfish
+* svn co svn://rubyforge.org//var/svn/starfish
+
+== Usage
+
+Make a file and define the server and client. What follows is a simple example:
+
+    # foo.rb
+    class Foo
+      attr_reader :i
+      
+      def initialize
+        @i = 0
+      end
+      
+      def inc
+        logger.info "YAY it incremented by 1 up to #{@i}"
+        @i += 1
+      end
+    end
+    
+    server do |object|
+      object = Foo.new
+    end
+    
+    client do |object|
+      object.inc
+    end
+
+Notice that you need not specify any network code, nor learn a foreign API like DRb or Rinda. You need not even define logger, starfish takes care of it for you at runtime. You just tell it how to serve an object and how to process the object. To run this code, simply run:
+
+    starfish foo.rb
+
+This starts a server AND a client to start processing that object. The client runs in a continuous loop. If you call starfish foo.rb subsequent times from either the local machine or any machine on the local network, more clients will start processing the server concurrently.
+
+By default, the server will start a log file at /tmp/foo.rb.log. You can override this behaviour.
+
+    server :log => "/var/log/foo.log" do |object|
+      object = Foo.new
+    end
+
+The :log parameter is intelligent and can take various other formats including nil for no logging or instances of any kind of logger class you prefer to use.
+
+== MapReduce
+
+The crown jewel of Starfish is actually a library built on Starfish called MapReduce. Inspired by Google's MapReduce (http://en.wikipedia.org/wiki/MapReduce), it automates the process of divide and conquer for large data sets. Currently, Starfish's implementation only works with the ActiveRecord ORM from Ruby on Rails. The idea is that you have a very large data set that would be unmanageable to process otherwise. For example, let's say your database table has 30GB of information in it. It would be very difficult to keep 30GB of records in RAM at one time; even if you processed them serially, it would take a very long time. The idea is to have many clients grabbing chunks of the database at a time making much faster work, even on a single processor machine, than would otherwise be feasible.
+
+Here is the basic code that will get you up and running with MapReduce in Starfish.
+
+    # item.rb
+    ActiveRecord::Base.establish_connection(
+      :adapter  => "mysql",
+      :host     => "localhost",
+      :username => "root",
+      :password => "",
+      :database => "some_database"
+    )
+    
+    class Item < ActiveRecord::Base; end
+    
+    server do |map_reduce|
+      map_reduce.type = Item
+    end
+    
+    client do |item|
+      logger.info item.id
+    end
+
+Just like the previous example, just run:
+
+    starfish item.rb
+
+Starfish takes care of the rest. The code above does the following:
+
+* The server grabs all the items via: Item.find(:all)
+* Each of the clients grab an item from the collection
+* When there are no more items to be grabbed, everything shuts down
+
+However there are cases when you might want some fancier behaviour.
+
+    server do |map_reduce|
+      map_reduce.type = Item
+      map_reduce.conditions = ["some_important_flag = ?", 1]
+      map_reduce.vigilant = true
+      map_reduce.queue_size = 1000
+    end
+
+Here are three new options: conditions, vigilant, and queue_size. Conditions simply limits what is grabbed via ActiveRecord. It is equivalent to saying: Item.find(:all, :conditions => ["some_important_flag = ?", 1]). Vigilant simply keeps the clients alive and when Item.find(:all) runs out of items, it will vigilantly poll for newly created items periodically. Queue size specifies the number of items buffered in the server queue. In the case of the 30GB database table, you do not want to grab all 30GB at once, instead you grab 1000 at a time as needed by your clients.
+
+Another options you have when you use the vigilant option is map_reduce.empty_queue_wait which specifies how long to wait between polls for new information.
+
+There is yet a third main run-mode for MapReduce under ActiveRecord.
+
+    server do |map_reduce|
+      map_reduce.type = Item
+      map_reduce.rescan_when_complete = true
+    end
+
+The events follow like this when the rescan_when_complete option is enabled:
+
+* The server grabs all the items via: Item.find(:all)
+* Each of the clients grab an item from the collection
+* When there are no more items to be grabbed, the queue is re-filled via Item.find(:all) and the process starts again
+
+If you need to continuously check over data for integrity, this option will meet your demands.
+
+Another bonus you get while using MapReduce is automatic statistics. Simply call:
+
+    starfish item.rb stats
+
+And you will be returned a YAML hash of statistics.
+
+    --- 
+    time_spent_grabbing_objects: 0.090328
+    time_spent_processing_objects: 0.023431
+    time_began: 2006-08-16 09:30:32.926399 -07:00
+    num_queues_grabbed: 5
+    time_spent_grabbing_queues: 0.090328
+    num_objects_grabbed: 10
+
+== Examples
+
+See the examples/ directory.
+
+== Authors    
+* Lucas Carlson from MOG (mailto:lucas@rufy.com) - http://mog.com
+
+This library is released under the terms of the BSD.
+

data/Rakefile

@@ -0,0 +1,89 @@
+require 'rubygems'
+require 'rake'
+require 'rake/testtask'
+require 'rake/rdoctask'
+require 'rake/gempackagetask'
+require 'rake/contrib/rubyforgepublisher'
+require 'lib/starfish'
+
+PKG_VERSION = Starfish::VERSION
+
+PKG_FILES = FileList[
+    "lib/**/*", "bin/*", "test/**/*", "[A-Z]*", "Rakefile", "doc/**/*", "examples/**/*"
+]
+
+desc "Default Task"
+task :default => [ :test ]
+
+# Run the unit tests
+desc "Run all unit tests"
+Rake::TestTask.new("test") { |t|
+  t.libs << "lib"
+  t.pattern = 'test/*/*_test.rb'
+  t.verbose = true
+}
+
+# Make a console, useful when working on tests
+desc "Generate a test console"
+task :console do
+   verbose( false ) { sh "irb -I lib/ -r 'starfish'" }
+end
+
+# Genereate the RDoc documentation
+desc "Create documentation"
+Rake::RDocTask.new("doc") { |rdoc|
+  rdoc.title = "Starfish - ridiculously easy distributed programming with Ruby"
+  rdoc.rdoc_dir = 'doc'
+  rdoc.rdoc_files.include('README')
+  rdoc.rdoc_files.include('lib/**/*.rb')
+}
+
+# Genereate the package
+spec = Gem::Specification.new do |s|
+
+  #### Basic information.
+
+  s.name = 'starfish'
+  s.version = PKG_VERSION
+  s.summary = <<-EOF
+   Starfish is a utility to make distributed programming ridiculously easy.
+  EOF
+  s.description = <<-EOF
+   Starfish is a utility to make distributed programming ridiculously easy.
+  EOF
+
+  #### Which files are to be included in this gem?  Everything!  (Except CVS directories.)
+
+  s.files = PKG_FILES
+
+  #### Load-time details: library and application (you will need one or both).
+
+  s.bindir = 'bin'
+  s.executables = ['starfish']
+  s.require_path = 'lib'
+  s.autorequire = 'starfish'
+
+  #### Documentation and testing.
+
+  s.has_rdoc = true
+
+  #### Author and project details.
+
+  s.author = "Lucas Carlson"
+  s.email = "lucas@rufy.com"
+  s.homepage = "http://rubyforge.org/projects/starfish"
+end
+
+Rake::GemPackageTask.new(spec) do |pkg|
+  pkg.need_zip = true
+  pkg.need_tar = true
+end
+
+desc "Report code statistics (KLOCs, etc) from the application"
+task :stats do
+  require 'code_statistics'
+  CodeStatistics.new(
+    ["Library", "lib"],
+    ["Units", "test"]
+  ).to_s
+end

data/bin/starfish

@@ -0,0 +1,27 @@
+#!/usr/bin/env ruby
+$:.unshift(File.dirname(__FILE__)+'/../lib') unless
+  $:.include?(File.dirname(__FILE__)+'/../lib') || $:.include?(File.expand_path(File.dirname(__FILE__)+'/../lib'))
+
+require 'starfish'
+
+def server(options={}, &block)
+  Starfish.options = options
+  Starfish.server = block
+end
+
+def client(options={}, &block)
+  Starfish.options = options
+  Starfish.client = block
+end
+
+require ARGV.first
+
+if ARGV.include?('stop')
+  Starfish.new.stop
+elsif ARGV.include?('server')
+  Starfish.new.server
+elsif ARGV.include?('stats')
+  Starfish.new.stats
+else
+  Starfish.new.client
+end

data/doc/classes/MapReduce/ActiveRecord/Base/Client.html

@@ -0,0 +1,177 @@
+<?xml version="1.0" encoding="iso-8859-1"?>
+<!DOCTYPE html 
+     PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+     "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+<head>
+  <title>Class: MapReduce::ActiveRecord::Base::Client</title>
+  <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
+  <meta http-equiv="Content-Script-Type" content="text/javascript" />
+  <link rel="stylesheet" href="../../../.././rdoc-style.css" type="text/css" media="screen" />
+  <script type="text/javascript">
+  // <![CDATA[
+
+  function popupCode( url ) {
+    window.open(url, "Code", "resizable=yes,scrollbars=yes,toolbar=no,status=no,height=150,width=400")
+  }
+
+  function toggleCode( id ) {
+    if ( document.getElementById )
+      elem = document.getElementById( id );
+    else if ( document.all )
+      elem = eval( "document.all." + id );
+    else
+      return false;
+
+    elemStyle = elem.style;
+    
+    if ( elemStyle.display != "block" ) {
+      elemStyle.display = "block"
+    } else {
+      elemStyle.display = "none"
+    }
+
+    return true;
+  }
+  
+  // Make codeblocks hidden by default
+  document.writeln( "<style type=\"text/css\">div.method-source-code { display: none }</style>" )
+  
+  // ]]>
+  </script>
+
+</head>
+<body>
+
+
+
+    <div id="classHeader">
+        <table class="header-table">
+        <tr class="top-aligned-row">
+          <td><strong>Class</strong></td>
+          <td class="class-name-in-header">MapReduce::ActiveRecord::Base::Client</td>
+        </tr>
+        <tr class="top-aligned-row">
+            <td><strong>In:</strong></td>
+            <td>
+                <a href="../../../../files/lib/map_reduce/active_record_rb.html">
+                lib/map_reduce/active_record.rb
+                </a>
+        <br />
+            </td>
+        </tr>
+
+        <tr class="top-aligned-row">
+            <td><strong>Parent:</strong></td>
+            <td>
+                Object
+            </td>
+        </tr>
+        </table>
+    </div>
+  <!-- banner header -->
+
+  <div id="bodyContent">
+
+
+
+  <div id="contextContent">
+
+
+
+   </div>
+
+    <div id="method-list">
+      <h3 class="section-bar">Methods</h3>
+
+      <div class="name-list">
+      <a href="#M000017">each</a>&nbsp;&nbsp;
+      <a href="#M000018">logger</a>&nbsp;&nbsp;
+      <a href="#M000016">new</a>&nbsp;&nbsp;
+      </div>
+    </div>
+
+  </div>
+
+
+    <!-- if includes -->
+    <div id="includes">
+      <h3 class="section-bar">Included Modules</h3>
+
+      <div id="includes-list">
+        <span class="include-name">DRbUndumped</span>
+        <span class="include-name">Enumerable</span>
+      </div>
+    </div>
+
+    <div id="section">
+
+
+
+
+
+      
+
+
+    <!-- if method_list -->
+    <div id="methods">
+      <h3 class="section-bar">Public Class methods</h3>
+
+      <div id="method-M000016" class="method-detail">
+        <a name="M000016"></a>
+
+        <div class="method-heading">
+          <a href="Client.src/M000016.html" target="Code" class="method-signature"
+            onclick="popupCode('Client.src/M000016.html');return false;">
+          <span class="method-name">new</span><span class="method-args">(server_object)</span>
+          </a>
+        </div>
+      
+        <div class="method-description">
+        </div>
+      </div>
+
+      <h3 class="section-bar">Public Instance methods</h3>
+
+      <div id="method-M000017" class="method-detail">
+        <a name="M000017"></a>
+
+        <div class="method-heading">
+          <a href="Client.src/M000017.html" target="Code" class="method-signature"
+            onclick="popupCode('Client.src/M000017.html');return false;">
+          <span class="method-name">each</span><span class="method-args">() {|get_value_from(@server_object.get_id)| ...}</span>
+          </a>
+        </div>
+      
+        <div class="method-description">
+        </div>
+      </div>
+
+      <div id="method-M000018" class="method-detail">
+        <a name="M000018"></a>
+
+        <div class="method-heading">
+          <a href="Client.src/M000018.html" target="Code" class="method-signature"
+            onclick="popupCode('Client.src/M000018.html');return false;">
+          <span class="method-name">logger</span><span class="method-args">(*args)</span>
+          </a>
+        </div>
+      
+        <div class="method-description">
+        </div>
+      </div>
+
+
+    </div>
+
+
+  </div>
+
+
+<div id="validator-badges">
+  <p><small><a href="http://validator.w3.org/check/referer">[Validate]</a></small></p>
+</div>
+
+</body>
+</html>

data/doc/classes/MapReduce/ActiveRecord/Base/Client.src/M000016.html

@@ -0,0 +1,19 @@
+<?xml version="1.0" encoding="iso-8859-1"?>
+<!DOCTYPE html 
+     PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+     "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+
+<html>
+<head>
+  <title>new (MapReduce::ActiveRecord::Base::Client)</title>
+  <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
+  <link rel="stylesheet" href="../../../../.././rdoc-style.css" type="text/css" media="screen" />
+</head>
+<body class="standalone-code">
+  <pre><span class="ruby-comment cmt"># File lib/map_reduce/active_record.rb, line 14</span>
+        <span class="ruby-keyword kw">def</span> <span class="ruby-identifier">initialize</span>(<span class="ruby-identifier">server_object</span>)
+          <span class="ruby-ivar">@server_object</span> = <span class="ruby-identifier">server_object</span>
+          <span class="ruby-ivar">@type</span> = <span class="ruby-identifier">eval</span>(<span class="ruby-identifier">server_object</span>.<span class="ruby-identifier">type_to_s</span>)
+        <span class="ruby-keyword kw">end</span></pre>
+</body>
+</html>

data/doc/classes/MapReduce/ActiveRecord/Base/Client.src/M000017.html

@@ -0,0 +1,20 @@
+<?xml version="1.0" encoding="iso-8859-1"?>
+<!DOCTYPE html 
+     PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+     "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+
+<html>
+<head>
+  <title>each (MapReduce::ActiveRecord::Base::Client)</title>
+  <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
+  <link rel="stylesheet" href="../../../../.././rdoc-style.css" type="text/css" media="screen" />
+</head>
+<body class="standalone-code">
+  <pre><span class="ruby-comment cmt"># File lib/map_reduce/active_record.rb, line 19</span>
+        <span class="ruby-keyword kw">def</span> <span class="ruby-identifier">each</span>
+          <span class="ruby-ivar">@server_object</span>.<span class="ruby-identifier">limit</span>.<span class="ruby-identifier">times</span> <span class="ruby-keyword kw">do</span>
+            <span class="ruby-keyword kw">yield</span> <span class="ruby-identifier">get_value_from</span>(<span class="ruby-ivar">@server_object</span>.<span class="ruby-identifier">get_id</span>)
+          <span class="ruby-keyword kw">end</span>
+        <span class="ruby-keyword kw">end</span></pre>
+</body>
+</html>

data/doc/classes/MapReduce/ActiveRecord/Base/Client.src/M000018.html

@@ -0,0 +1,18 @@
+<?xml version="1.0" encoding="iso-8859-1"?>
+<!DOCTYPE html 
+     PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+     "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+
+<html>
+<head>
+  <title>logger (MapReduce::ActiveRecord::Base::Client)</title>
+  <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
+  <link rel="stylesheet" href="../../../../.././rdoc-style.css" type="text/css" media="screen" />
+</head>
+<body class="standalone-code">
+  <pre><span class="ruby-comment cmt"># File lib/map_reduce/active_record.rb, line 25</span>
+        <span class="ruby-keyword kw">def</span> <span class="ruby-identifier">logger</span>(<span class="ruby-operator">*</span><span class="ruby-identifier">args</span>)
+          <span class="ruby-ivar">@server_object</span>.<span class="ruby-identifier">logger</span>(<span class="ruby-operator">*</span><span class="ruby-identifier">args</span>)
+        <span class="ruby-keyword kw">end</span></pre>
+</body>
+</html>

data/doc/classes/MapReduce/ActiveRecord/Base.html

@@ -0,0 +1,170 @@
+<?xml version="1.0" encoding="iso-8859-1"?>
+<!DOCTYPE html 
+     PUBLIC "-//W3C//DTD XHTML 1.0 Transitional//EN"
+     "http://www.w3.org/TR/xhtml1/DTD/xhtml1-transitional.dtd">
+
+<html xmlns="http://www.w3.org/1999/xhtml" xml:lang="en" lang="en">
+<head>
+  <title>Module: MapReduce::ActiveRecord::Base</title>
+  <meta http-equiv="Content-Type" content="text/html; charset=iso-8859-1" />
+  <meta http-equiv="Content-Script-Type" content="text/javascript" />
+  <link rel="stylesheet" href="../../.././rdoc-style.css" type="text/css" media="screen" />
+  <script type="text/javascript">
+  // <![CDATA[
+
+  function popupCode( url ) {
+    window.open(url, "Code", "resizable=yes,scrollbars=yes,toolbar=no,status=no,height=150,width=400")
+  }
+
+  function toggleCode( id ) {
+    if ( document.getElementById )
+      elem = document.getElementById( id );
+    else if ( document.all )
+      elem = eval( "document.all." + id );
+    else
+      return false;
+
+    elemStyle = elem.style;
+    
+    if ( elemStyle.display != "block" ) {
+      elemStyle.display = "block"
+    } else {
+      elemStyle.display = "none"
+    }
+
+    return true;
+  }
+  
+  // Make codeblocks hidden by default
+  document.writeln( "<style type=\"text/css\">div.method-source-code { display: none }</style>" )
+  
+  // ]]>
+  </script>
+
+</head>
+<body>
+
+
+
+    <div id="classHeader">
+        <table class="header-table">
+        <tr class="top-aligned-row">
+          <td><strong>Module</strong></td>
+          <td class="class-name-in-header">MapReduce::ActiveRecord::Base</td>
+        </tr>
+        <tr class="top-aligned-row">
+            <td><strong>In:</strong></td>
+            <td>
+                <a href="../../../files/lib/map_reduce/active_record_rb.html">
+                lib/map_reduce/active_record.rb
+                </a>
+        <br />
+            </td>
+        </tr>
+
+        </table>
+    </div>
+  <!-- banner header -->
+
+  <div id="bodyContent">
+
+
+
+  <div id="contextContent">
+
+
+
+   </div>
+
+    <div id="method-list">
+      <h3 class="section-bar">Methods</h3>
+
+      <div class="name-list">
+      <a href="#M000015">get_id</a>&nbsp;&nbsp;
+      </div>
+    </div>
+
+  </div>
+
+
+    <!-- if includes -->
+
+    <div id="section">
+
+    <div id="class-list">
+      <h3 class="section-bar">Classes and Modules</h3>
+
+      Class <a href="Base/Client.html" class="link">MapReduce::ActiveRecord::Base::Client</a><br />
+
+    </div>
+
+
+
+
+    <div id="attribute-list">
+      <h3 class="section-bar">Attributes</h3>
+
+      <div class="name-list">
+        <table>
+        <tr class="top-aligned-row context-row">
+          <td class="context-item-name">empty_queue_wait</td>
+          <td class="context-item-value">&nbsp;[RW]&nbsp;</td>
+          <td class="context-item-desc"></td>
+        </tr>
+        <tr class="top-aligned-row context-row">
+          <td class="context-item-name">locked_queue_wait</td>
+          <td class="context-item-value">&nbsp;[RW]&nbsp;</td>
+          <td class="context-item-desc"></td>
+        </tr>
+        <tr class="top-aligned-row context-row">
+          <td class="context-item-name">queue_size</td>
+          <td class="context-item-value">&nbsp;[RW]&nbsp;</td>
+          <td class="context-item-desc"></td>
+        </tr>
+        <tr class="top-aligned-row context-row">
+          <td class="context-item-name">rescan_when_complete</td>
+          <td class="context-item-value">&nbsp;[RW]&nbsp;</td>
+          <td class="context-item-desc"></td>
+        </tr>
+        <tr class="top-aligned-row context-row">
+          <td class="context-item-name">vigilant</td>
+          <td class="context-item-value">&nbsp;[RW]&nbsp;</td>
+          <td class="context-item-desc"></td>
+        </tr>
+        </table>
+      </div>
+    </div>
+      
+
+
+    <!-- if method_list -->
+    <div id="methods">
+      <h3 class="section-bar">Public Instance methods</h3>
+
+      <div id="method-M000015" class="method-detail">
+        <a name="M000015"></a>
+
+        <div class="method-heading">
+          <a href="Base.src/M000015.html" target="Code" class="method-signature"
+            onclick="popupCode('Base.src/M000015.html');return false;">
+          <span class="method-name">get_id</span><span class="method-args">()</span>
+          </a>
+        </div>
+      
+        <div class="method-description">
+        </div>
+      </div>
+
+
+    </div>
+
+
+  </div>
+
+
+<div id="validator-badges">
+  <p><small><a href="http://validator.w3.org/check/referer">[Validate]</a></small></p>
+</div>
+
+</body>
+</html>