skynet 0.9.1 → 0.9.2

data/website/index.html

@@ -33,56 +33,96 @@
     <h1>skynet</h1>
     <div id="version" class="clickable" onclick='document.location = "http://rubyforge.org/projects/skynet"; return false'>
       <p>Get Version</p>
-      <a href="http://rubyforge.org/projects/skynet" class="numbers">0.9.1</a>
+      <a href="http://rubyforge.org/projects/skynet" class="numbers">0.9.2</a>
     </div>
-    <h1>&#x2192; &#8216;skynet&#8217;</h1>
+    <p><a href="http://rubyforge.org/projects/skynet/">Skynet RubyForge Home</a><br />
+<a href="doc/index.html">Documentation</a></p>
 
 
-	<h2>What</h2>
+	<h2>What is Skynet</h2>
+
+
+	<p>Skynet is an open source Ruby implementation of Google&#8217;s Map/Reduce framework, created at Geni.com. With Skynet, one can easily convert a time-consuming serial task, such as a computationally expensive Rails migration, into a distributed program running on many computers.</p>
+
+
+	<p>Skynet is an adaptive, self-upgrading, fault-tolerant, and fully distributed system with no single point of failure. It uses a &#8220;peer recovery&#8221; system where workers watch out for each other. If a worker dies or fails for any reason, another worker will notice and pick up that task. Skynet also has no special &#8216;master&#8217; servers, only workers which can act as a master for any task at any time. Even these master tasks can fail and will be picked up by other workers.</p>
 
 
 	<h2>Installing</h2>
 
 
+	<p>Skynet can be installed via RubyGems:</p>
+
+
 	<p><pre class='syntax'><span class="ident">sudo</span> <span class="ident">gem</span> <span class="ident">install</span> <span class="ident">skynet</span></pre></p>
 
 
-	<h2>The basics</h2>
+	<h2>More Info</h2>
 
 
-	<h2>Demonstration of usage</h2>
+	<p>For more info check out the <a href="doc/index.html"><span class="caps">README</span></a></p>
 
 
-	<h2>Forum</h2>
+	<h2>Contact</h2>
 
 
-	<p><a href="http://groups.google.com/group/skynet">http://groups.google.com/group/skynet</a></p>
+	<p>Comments are welcome. Send an email to &#8220;Adam Pisoni&#8221; apisoni at geni.com</p>
 
 
-	<p><span class="caps">TODO</span> &#8211; create Google Group &#8211; skynet</p>
+	<h2>Credits
+There are a number of people who either directly or indirectly worked on Skynet.
+John Beppu (wrote the original worker/manager code)
+Justin Balthrop
+Zack Parker
+Amos Elliston
+Zack Hobson
+Alan Braverman
+Mike Stangel
+Scott Steadman
+Andrew Arrow
+Jason Rojas</h2>
 
 
-	<h2>How to submit patches</h2>
+	<p>Skynet was inspired by and heavily influenced by Josh Carter and this blog post.
+http://multipart-mixed.com/software/simple_mapreduce_in_ruby.html</p>
 
 
-	<p>Read the <a href="http://drnicwilliams.com/2007/06/01/8-steps-for-fixing-other-peoples-code/">8 steps for fixing other people&#8217;s code</a> and for section <a href="http://drnicwilliams.com/2007/06/01/8-steps-for-fixing-other-peoples-code/#8b-google-groups">8b: Submit patch to Google Groups</a>, use the Google Group above.</p>
+	<p>Also by Starfish by Lucas Carlson
+http://tech.rufy.com/2006/08/mapreduce-for-ruby-ridiculously-easy.html
+http://rufy.com/starfish/doc/</p>
 
 
-	<p>The trunk repository is <code>svn://rubyforge.org/var/svn/skynet/trunk</code> for anonymous access.</p>
+	<h2><span class="caps">LICENSE</span>:</h2>
 
 
-	<h2>License</h2>
+	<p>(The <span class="caps">MIT</span> License)</p>
 
 
-	<p>This code is free to use under the terms of the <span class="caps">MIT</span> license.</p>
+	<p>Copyright&#169; 2007 Adam Pisoni, Geni.com</p>
 
 
-	<h2>Contact</h2>
+	<p>Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+&#8220;Software&#8221;), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:</p>
+
+
+	<p>The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.</p>
 
 
-	<p>Comments are welcome. Send an email to <a href="mailto:FIXME"><span class="caps">FIXME</span> full name</a> email via the <a href="http://groups.google.com/group/skynet">forum</a></p>
+	<p><span class="caps">THE SOFTWARE IS PROVIDED</span> &#8220;AS IS&#8221;, <span class="caps">WITHOUT WARRANTY OF ANY KIND</span>,
+<span class="caps">EXPRESS OR IMPLIED</span>, INCLUDING <span class="caps">BUT NOT LIMITED TO THE WARRANTIES OF</span>
+<span class="caps">MERCHANTABILITY</span>, FITNESS <span class="caps">FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT</span>.
+<span class="caps">IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY</span>
+<span class="caps">CLAIM</span>, DAMAGES <span class="caps">OR OTHER LIABILITY</span>, WHETHER <span class="caps">IN AN ACTION OF CONTRACT</span>,
+<span class="caps">TORT OR OTHERWISE</span>, ARISING <span class="caps">FROM</span>, OUT <span class="caps">OF OR IN CONNECTION WITH THE</span>
+<span class="caps">SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE</span>.</p>
     <p class="coda">
-      <a href="FIXME email">FIXME full name</a>, 16th December 2007<br>
+      Adam Pisoni, Geni.com 25th January 2008<br>
       Theme extended from <a href="http://rb2js.rubyforge.org/">Paul Battley</a>
     </p>
 </div>

data/website/index.txt

@@ -1,39 +1,69 @@
 h1. skynet
 
-h1. → 'skynet'
+<a href="http://rubyforge.org/projects/skynet/">Skynet RubyForge Home</a><br />
+<a href="doc/index.html">Documentation</a>
 
+h2. What is Skynet
 
-h2. What
+Skynet is an open source Ruby implementation of Google's Map/Reduce framework, created at Geni.com. With Skynet, one can easily convert a time-consuming serial task, such as a computationally expensive Rails migration, into a distributed program running on many computers.
 
+Skynet is an adaptive, self-upgrading, fault-tolerant, and fully distributed system with no single point of failure. It uses a "peer recovery" system where workers watch out for each other. If a worker dies or fails for any reason, another worker will notice and pick up that task. Skynet also has no special 'master' servers, only workers which can act as a master for any task at any time. Even these master tasks can fail and will be picked up by other workers.
 
 h2. Installing
 
-<pre syntax="ruby">sudo gem install skynet</pre>
-
-h2. The basics
-
-
-h2. Demonstration of usage
-
-
-
-h2. Forum
+Skynet can be installed via RubyGems:
 
-"http://groups.google.com/group/skynet":http://groups.google.com/group/skynet
-
-TODO - create Google Group - skynet
-
-h2. How to submit patches
-
-Read the "8 steps for fixing other people's code":http://drnicwilliams.com/2007/06/01/8-steps-for-fixing-other-peoples-code/ and for section "8b: Submit patch to Google Groups":http://drnicwilliams.com/2007/06/01/8-steps-for-fixing-other-peoples-code/#8b-google-groups, use the Google Group above.
-
-The trunk repository is <code>svn://rubyforge.org/var/svn/skynet/trunk</code> for anonymous access.
+<pre syntax="ruby">sudo gem install skynet</pre>
 
-h2. License
+h2. More Info
 
-This code is free to use under the terms of the MIT license. 
+For more info check out the <a href="doc/index.html">README</a>
 
 h2. Contact
 
-Comments are welcome. Send an email to "FIXME full name":mailto:FIXME email via the "forum":http://groups.google.com/group/skynet
-
+Comments are welcome. Send an email to "Adam Pisoni" apisoni at geni.com
+
+h2. Credits
+There are a number of people who either directly or indirectly worked on Skynet.
+John Beppu (wrote the original worker/manager code)
+Justin Balthrop
+Zack Parker
+Amos Elliston
+Zack Hobson
+Alan Braverman
+Mike Stangel
+Scott Steadman
+Andrew Arrow
+Jason Rojas
+
+Skynet was inspired by and heavily influenced by Josh Carter and this blog post.
+http://multipart-mixed.com/software/simple_mapreduce_in_ruby.html
+
+Also by Starfish by Lucas Carlson
+http://tech.rufy.com/2006/08/mapreduce-for-ruby-ridiculously-easy.html
+http://rufy.com/starfish/doc/
+
+h2. LICENSE:
+
+(The MIT License)
+
+Copyright (c) 2007 Adam Pisoni, Geni.com
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
+CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
+TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
+SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/website/template.rhtml

@@ -37,7 +37,7 @@
     </div>
     <%= body %>
     <p class="coda">
-      <a href="FIXME email">FIXME full name</a>, <%= modified.pretty %><br>
+      Adam Pisoni, Geni.com <%= modified.pretty %><br>
       Theme extended from <a href="http://rb2js.rubyforge.org/">Paul Battley</a>
     </p>
 </div>

metadata

@@ -1,10 +1,10 @@
 --- !ruby/object:Gem::Specification 
-rubygems_version: 0.9.0
+rubygems_version: 0.9.4
 specification_version: 1
 name: skynet
 version: !ruby/object:Gem::Version 
-  version: 0.9.1
-date: 2008-01-02 00:00:00 -08:00
+  version: 0.9.2
+date: 2008-01-28 00:00:00 -08:00
 summary: Skynet - A Ruby Map/Reduce Framework
 require_paths: 
 - lib
@@ -38,14 +38,14 @@ files:
 - app_generators/skynet_install/skynet_install_generator.rb
 - app_generators/skynet_install/templates/migration.rb
 - app_generators/skynet_install/templates/skynet
-- app_generators/skynet_install/templates/skynet_console
+- app_generators/skynet_install/templates/skynet_schema.sql
 - bin/skynet
-- bin/skynet_console
 - bin/skynet_install
 - bin/skynet_tuplespace_server
 - config/hoe.rb
 - config/requirements.rb
 - lib/skynet.rb
+- lib/skynet/mapreduce_helper.rb
 - lib/skynet/mapreduce_test.rb
 - lib/skynet/message_queue_adapters/message_queue_adapter.rb
 - lib/skynet/message_queue_adapters/mysql.rb
@@ -62,6 +62,7 @@ files:
 - lib/skynet/skynet_manager.rb
 - lib/skynet/skynet_message.rb
 - lib/skynet/skynet_message_queue.rb
+- lib/skynet/skynet_partitioners.rb
 - lib/skynet/skynet_ruby_extensions.rb
 - lib/skynet/skynet_task.rb
 - lib/skynet/skynet_tuplespace_server.rb
@@ -75,30 +76,37 @@ files:
 - script/generate
 - script/txt2html
 - setup.rb
-- sometest.rb
 - tasks/deployment.rake
 - tasks/environment.rake
 - tasks/website.rake
-- test/all_models_test.rb
-- test/mysql_message_queue_adaptor_test.rb
-- test/skynet_manager_test.rb
-- test/skynet_message_test.rb
+- test/test_active_record_extensions.rb
 - test/test_generator_helper.rb
 - test/test_helper.rb
+- test/test_mysql_message_queue_adapter.rb
 - test/test_skynet.rb
 - test/test_skynet_install_generator.rb
-- test/tuplespace_message_queue_test.rb
-- tmtags
+- test/test_skynet_job.rb
+- test/test_skynet_manager.rb
+- test/test_skynet_message.rb
+- test/test_skynet_task.rb
+- test/test_tuplespace_message_queue.rb
 - website/index.html
 - website/index.txt
 - website/javascripts/rounded_corners_lite.inc.js
 - website/stylesheets/screen.css
 - website/template.rhtml
 test_files: 
+- test/test_active_record_extensions.rb
 - test/test_generator_helper.rb
 - test/test_helper.rb
+- test/test_mysql_message_queue_adapter.rb
 - test/test_skynet.rb
 - test/test_skynet_install_generator.rb
+- test/test_skynet_job.rb
+- test/test_skynet_manager.rb
+- test/test_skynet_message.rb
+- test/test_skynet_task.rb
+- test/test_tuplespace_message_queue.rb
 rdoc_options: 
 - --main
 - README.txt
@@ -110,7 +118,6 @@ extra_rdoc_files:
 - website/index.txt
 executables: 
 - skynet
-- skynet_console
 - skynet_install
 - skynet_tuplespace_server
 extensions: []
@@ -127,3 +134,12 @@ dependencies:
       - !ruby/object:Gem::Version 
         version: "1"
     version: 
+- !ruby/object:Gem::Dependency 
+  name: rubigen
+  version_requirement: 
+  version_requirements: !ruby/object:Gem::Version::Requirement 
+    requirements: 
+    - - ">="
+      - !ruby/object:Gem::Version 
+        version: 1.1.1
+    version: 

data/app_generators/skynet_install/templates/skynet_console

@@ -1,16 +0,0 @@
-#!/usr/bin/env ruby
-
-files_to_require = []
-<% if in_rails %>
-files_to_require << File.dirname(__FILE__) + '/../config/boot'
-files_to_require << File.expand_path(File.dirname(__FILE__)) + '/../config/environment'
-files_to_require.each { |file| require file }
-<% end %>
-
-require 'rubygems'
-require 'skynet'                                     
-
-Skynet::CONFIG[:LAUNCHER_PATH]          = File.expand_path(__FILE__)
-Skynet::CONFIG[:SKYNET_LOG_LEVEL]       = Logger::ERROR
-
-Skynet::Console.start(files_to_require)

data/bin/skynet_console

@@ -1,9 +0,0 @@
-#!/usr/bin/env ruby
-
-require 'rubygems'
-require File.expand_path(File.dirname(__FILE__)) + '/../lib/skynet.rb'
-
-Skynet::CONFIG[:LAUNCHER_PATH]          = File.expand_path(__FILE__)
-Skynet::CONFIG[:SKYNET_LOG_LEVEL]       = Logger::ERROR
-
-Skynet::Console.start

data/sometest.rb

@@ -1,23 +0,0 @@
-class SomeTest
-  include SkynetDebugger
-  
-  def self.map(datas)
-    results = {}
-    datas.each do |data|
-      results[data] ||= 0
-      results[data] += 1
-    end                 
-    [results]      
-  end
-  
-  def self.reduce(datas)
-    results = {}
-    datas.each do |hashes|
-      hashes.each do |key,value|
-        results[key] ||= 0
-        results[key] += value
-      end
-    end
-    results
-  end
-end

data/test/all_models_test.rb

@@ -1,139 +0,0 @@
-require File.dirname(__FILE__) + '/../../functional_test_helper'
-require 'mapreduce_lib/all_models'
-
-module MapreduceLib
-  class AllModelsTest < Test::Unit::TestCase
-
-    fixtures :profiles, :profile_details
-
-    def test_map
-      eachmeth = lambda do |profile|
-        profile.claimed = false
-        profile.save        
-      end
-      data = [1,10,{:conditions => "profiles.claimed = 1"},"Profile",eachmeth]
-      
-      assert Profile.find(2).claimed, "claimed"
-
-      MapreduceLib::AllModels.map([data])
-      
-      assert !Profile.find(2).claimed, "not claimed"
-    end
-
-    def test_each_with_proc
-      Skynet.solo do
-        MapreduceLib::AllModels.find(:all, :conditions => "claimed = 1", :batch_size => 2, :model_class => "Profile", :limit => 5).each do |profile|
-          profile.suffix = 'z'
-          profile.save        
-        end
-      end      
-      assert_equal 'z', Profile.find(2).suffix
-    end
-
-    def test_big_batch
-      Skynet.solo do
-        MapreduceLib::AllModels.find(:all, :conditions => "claimed = 1", :batch_size => 200, :model_class => "Profile").each do |profile|
-          profile.suffix = 'z'
-          profile.save        
-        end
-      end      
-      assert_equal 'z', Profile.find(2).suffix
-    end
-
-    def test_joins
-      p = Profile.find(1)
-      p.details.zodiac_sign = 'hermit'
-      p.details.save
-      Skynet.solo do
-        MapreduceLib::AllModels.find(:all, :conditions => "profile_details.zodiac_sign='hermit'", :joins => "JOIN profile_details ON profiles.id = profile_details.profile_id", :batch_size => 2, :model_class => "Profile").each do |profile|
-          profile.suffix = 'z'
-          profile.save        
-        end
-      end      
-      assert_equal 'z', Profile.find(1).suffix
-    end
-
-    def test_stragglers
-      profiles = Profile.find(:all)
-      Skynet.solo do
-        MapreduceLib::AllModels.find(:all, :batch_size => profiles.size-1, :model_class => "Profile").each do |profile|
-          profile.suffix = 'z'
-          profile.save        
-        end
-      end                      
-      profiles.last.reload
-      assert_equal 'z', profiles.last.suffix      
-    end
-    
-    def test_small_limit
-      Skynet.solo do
-        MapreduceLib::AllModels.find(:all, :conditions => "claimed = 1", :batch_size => 2, :model_class => "Profile", :limit => 3).each do |profile|
-          profile.suffix = 'z'
-          profile.save        
-        end
-      end      
-      assert_equal 'z', Profile.find(2).suffix
-    end
-
-    def test_each_with_proc_exception
-      Skynet.solo(:SKYNET_LOG_LEVEL => Logger::FATAL) do
-        MapreduceLib::AllModels.find(:all, :conditions => "claimed = 1", :batch_size => 2, :model_class => "Profile").each do |profile|
-          raise "BUSTED" if profile.id == 6
-          profile.suffix = 'z'
-          profile.save        
-        end
-      end      
-      assert_equal 'z', Profile.find(2).suffix
-      assert_equal nil, Profile.find(6).suffix
-    end
-
-    def test_each_with_class_exception
-      Skynet.solo(:SKYNET_LOG_LEVEL => Logger::FATAL) do
-        MapreduceLib::AllModels.find(:all, :conditions => "claimed = 1", :batch_size => 2, :model_class => "Profile").each(MapreduceLib::AllModelsExTest)
-      end      
-      assert_equal 'k', Profile.find(2).suffix
-      assert_equal nil, Profile.find(8).suffix
-    end
-
-    def test_each_with_class
-      Skynet.solo do
-        MapreduceLib::AllModels.find(:all, :conditions => "claimed = 1", :batch_size => 2, :model_class => "Profile").each(self.class)
-      end      
-      assert_equal 'k', Profile.find(2).suffix
-    end
-    
-    def test_distributed_each
-      Skynet.solo do
-        Profile.distributed_find(:all, :conditions => "claimed = 1", :batch_size => 2, :model_class => "Profile").each do |profile|
-            profile.suffix = 'gg'
-            profile.save        
-        end
-        assert_equal 'gg', Profile.find(2).suffix
-        Profile.distributed_find(:all, :conditions => "claimed = 1", :batch_size => 2, :model_class => "Profile").each(self.class)
-        assert_equal 'k', Profile.find(2).suffix
-      end
-    end
-    
-    def test_distributed_each_with_symbol
-      Skynet.solo do
-        Profile.distributed_find(:all, :conditions => "claimed = 1", :batch_size => 2, :model_class => "Profile").each(:mark_modified)
-      end
-      assert ModifiedProfile.find(:first, :conditions => "profile_id = 2")
-    end
-        
-    def self.each(profile)
-      profile.suffix = 'k'
-      profile.save        
-    end
-    
-  end
-
-  class AllModelsExTest
-    def self.each(profile)
-      raise "BUSTED" if profile.id == 8
-      profile.suffix = 'k'
-      profile.save        
-    end
-  end
-end
-