skynet 0.9.1

data/History.txt

@@ -0,0 +1,4 @@
+== 0.0.1 2007-12-16
+
+* 1 major enhancement:
+  * Initial release

data/License.txt

@@ -0,0 +1,20 @@
+Copyright (c) 2007 Adam Pisoni, Geni.com
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/Manifest.txt

@@ -0,0 +1,65 @@
+History.txt
+License.txt
+Manifest.txt
+README.txt
+Rakefile
+app_generators/skynet_install/USAGE
+app_generators/skynet_install/skynet_install_generator.rb
+app_generators/skynet_install/templates/migration.rb
+app_generators/skynet_install/templates/skynet
+app_generators/skynet_install/templates/skynet_console
+bin/skynet
+bin/skynet_console
+bin/skynet_install
+bin/skynet_tuplespace_server
+config/hoe.rb
+config/requirements.rb
+lib/skynet.rb
+lib/skynet/mapreduce_test.rb
+lib/skynet/message_queue_adapters/message_queue_adapter.rb
+lib/skynet/message_queue_adapters/mysql.rb
+lib/skynet/message_queue_adapters/tuple_space.rb
+lib/skynet/skynet_active_record_extensions.rb
+lib/skynet/skynet_config.rb
+lib/skynet/skynet_console.rb
+lib/skynet/skynet_console_helper.rb
+lib/skynet/skynet_debugger.rb
+lib/skynet/skynet_guid_generator.rb
+lib/skynet/skynet_job.rb
+lib/skynet/skynet_launcher.rb
+lib/skynet/skynet_logger.rb
+lib/skynet/skynet_manager.rb
+lib/skynet/skynet_message.rb
+lib/skynet/skynet_message_queue.rb
+lib/skynet/skynet_ruby_extensions.rb
+lib/skynet/skynet_task.rb
+lib/skynet/skynet_tuplespace_server.rb
+lib/skynet/skynet_worker.rb
+lib/skynet/version.rb
+log/debug.log
+log/skynet.log
+log/skynet_tuplespace_server.log
+log/skynet_worker.pid
+script/destroy
+script/generate
+script/txt2html
+setup.rb
+sometest.rb
+tasks/deployment.rake
+tasks/environment.rake
+tasks/website.rake
+test/all_models_test.rb
+test/mysql_message_queue_adaptor_test.rb
+test/skynet_manager_test.rb
+test/skynet_message_test.rb
+test/test_generator_helper.rb
+test/test_helper.rb
+test/test_skynet.rb
+test/test_skynet_install_generator.rb
+test/tuplespace_message_queue_test.rb
+tmtags
+website/index.html
+website/index.txt
+website/javascripts/rounded_corners_lite.inc.js
+website/stylesheets/screen.css
+website/template.rhtml

data/README.txt

@@ -0,0 +1,100 @@
+Skynet
+	http://skynet.rubyforge.org/
+	by Adam Pisoni
+
+== DESCRIPTION:
+
+Skynet is an open source Ruby implementation of Google's Map/Reduce framework, created at Geni. With Skynet, one can easily convert a time-consuming serial task, such as a computationally expensive Rails migration, into a distributed program running on many computers.
+
+Skynet is an adaptive, self-upgrading, fault-tolerant, and fully distributed system with no single point of failure. It uses a "peer recovery" system where workers watch out for each other. If a worker dies or fails for any reason, another worker will notice and pick up that task. Skynet also has no special 'master' servers, only workers which can act as a master for any task at any time. Even these master tasks can fail and will be picked up by other workers.
+
+== INSTALLATION:
+
+Skynet can be installed via RubyGems:
+
+  $ sudo gem install skynet
+
+== GETTING STARTED
+Skynet works by putting "tasks" on a message queue which are picked up by skynet workers, who execute the tasks, then put their results back on the message queue.   Skynet works best when it runs with your code.  For example, you might have a rails app and want some code you've already written to run asynchronously or in a distributed way.   Skynet can run within your code by installing a skynet launcher into your app.   Running this skynet launcher within your app guarantees all skynet workers will have access to your code.   This will be covered later.
+
+Skynet currently supports 2 message queue systems, TupleSpace and Mysql.   By default, the TupleSpace queue is used as it is the easiest to set up, though it is less powerful and less scaleable for large installations.  
+
+== RUNING SKYNET FOR THE FIRST TIME
+Since Skynet is a distributed system, it requires you have a skynet message queue as well as any number of skynet workers running.  To start a skynet message queue and a small number of workers:
+
+  $ skynet
+
+This starts a skynet tuple space message queue and 4 workers.   You can now run the Skynet Console to play with skynet a little.
+
+  $ skynet_console
+
+Here are some commands you can run.  
+  > stats
+  > manager.worker_pids
+  > [1,2,3,1,1,4].mapreduce(Skynet::MapreduceTest)
+  
+That last command actually took whatever array you gave it and counted the number of times each element appeared in the array.  It's not a very useful task, but it shows how easy it is to use.
+
+== RUNING SKYNET IN YOUR APPLICATION
+
+To be really useful, you'll want to run skynet in your own application.   To do that run:
+
+  $ skynet_install [--rails] YOUR_APP_DIRECTORY
+
+If you pass --rails it will assume it is installing in a rails app.   Once it is installed in your application, you can run skynet with
+
+  $ ./script/skynet
+  $ ./script/skynet_console
+
+== USAGE:
+
+Skynet was designed to make doing easy things easy and hard things possible.   The easiest way to use skynet is to create a new class with a self.map class method.  You can optionally include self.reduce, self.reduce_partitioner, self.map_partitioner as well.  Each of those methods should expect a single array (regardless of what data you pass).   Then, simple create an array and call mapreduce on it passing your class name.   Skynet will figure out which methods your class supports and use them accordingly.
+
+== USING SKYNET IN RAILS
+
+Skynet includes an addition to ActiveRecord that is very powerful.   
+
+=== distributed_find
+  $ YourModel.distributed_find(:all).each(YourClass)
+or
+  $ YourModel.distributed_find(:all).each(:somemethod)
+  
+In the first example, a find is 'virtually' run with your model class, and the results are distributed to the skynet workers.  If you've implemented a self.map method in YourClass, the retrieved objects will be passed (as arrays) on all the workers.
+
+In the second example, once the objects of YourModel are distributed, each worker merely calls :somemethod against each object.
+
+=== send_later
+
+  $ model_object.send_later(:method,options,:save)
+  
+Sometimes you have a method you want to call on a model asynchronously.  Using :send_later you can call a method, pass it options, and decide whether you want Skynet to save that model or not once its done calling your method.
+
+== EXAMPLES:
+
+== LIMITATIONS:
+  
+
+== LICENSE:
+
+(The MIT License)
+
+Copyright (c) 2007 Adam Pisoni, Geni.com
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
+CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
+TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
+SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/Rakefile

@@ -0,0 +1,4 @@
+require 'config/requirements'
+require 'config/hoe' # setup Hoe + all gem configuration
+
+Dir['tasks/**/*.rake'].each { |rake| load rake }

data/app_generators/skynet_install/USAGE

@@ -0,0 +1,5 @@
+Description:
+  
+  
+Usage:
+  

data/app_generators/skynet_install/skynet_install_generator.rb

@@ -0,0 +1,84 @@
+class SkynetInstallGenerator < RubiGen::Base
+  
+  DEFAULT_SHEBANG = File.join(Config::CONFIG['bindir'],
+                              Config::CONFIG['ruby_install_name'])
+  
+  default_options :author             => nil
+  default_options :in_rails           => nil
+  default_options :include_migration  => false
+  
+  attr_reader :name
+  attr_reader :in_rails
+  attr_reader :include_migration
+  
+  def initialize(runtime_args, runtime_options = {})
+    super
+    usage if args.empty?
+    @destination_root = File.expand_path(args.shift)
+    @name = base_name
+    extract_options
+  end
+
+  def manifest
+    record do |m|
+      # Ensure appropriate folder(s) exists
+      m.directory 'script'
+      BASEDIRS.each { |path| m.directory path }
+      
+      # Create stubs
+      # m.template "template.rb",  "some_file_after_erb.rb"
+      m.template     "skynet",         "script/skynet", :assigns => {:rails => @rails}, :collision => :ask, :chmod => 0755, :shebang => options[:shebang]
+      m.template     "skynet_console", "script/skynet_console", :assigns => {:rails => @rails}, :collision => :ask, :chmod => 0755, :shebang => options[:shebang]
+      if include_migration
+        m.directory 'db/migrate'
+        m.migration_template "migration.rb", "db/migrate", :collision => :skip, :assigns => {
+              :migration_name => "CreateSkynetTables"
+          },  :migration_file_name => "create_skynet_tables"
+      end
+      # m.dependency "install_rubigen_scripts", [destination_root, 'skynet_install'], 
+      #   :shebang => options[:shebang], :collision => :force
+    end
+  end
+
+  protected
+    def banner
+      <<-EOS
+Creates a ...
+
+USAGE: #{spec.name} [--rails] directory (can be '.' for current)"
+EOS
+    end
+
+    def add_options!(opts)
+      opts.separator ''
+      opts.separator 'Options:'
+      # For each option below, place the default
+      # at the top of the file next to "default_options"
+      # opts.on("-a", "--author=\"Your Name\"", String,
+      #         "Some comment about this option",
+      #         "Default: none") { |options[:author]| }
+      opts.on("-v", "--version", "Show the #{File.basename($0)} version number and quit.")
+      opts.on("--include-migration", 
+             "Include mysql migration if you want to use mysql as your message queue") { |options[:include_migration]| }
+      opts.on("-r", "--rails",
+              "Install into rails app",
+              "Default: false") { |options[:rails]| }
+    end
+    
+    def extract_options
+      # for each option, extract it into a local variable (and create an "attr_reader :author" at the top)
+      # Templates can access these value via the attr_reader-generated methods, but not the
+      # raw instance variable value.
+      # @author = options[:author]
+      @in_rails       = options[:rails]
+      @include_migration = options[:include_migration]
+    end
+
+    # Installation skeleton.  Intermediate directories are automatically
+    # created so don't sweat their absence here.
+    BASEDIRS = %w(
+      log
+      script
+      db/migrate
+    )
+end

data/app_generators/skynet_install/templates/migration.rb

@@ -0,0 +1,60 @@
+class <%= migration_name %> < ActiveRecord::Migration
+  def self.up
+    create_table :skynet_worker_queues do |t|
+      t.column  :id,            "bigint unsigned primary key"
+      t.column  :created_on,    :timestamp
+      t.column  :updated_on,    :timestamp
+      t.column  :tasktype,      :string
+      t.column  :tasksubtype,   :string
+      t.column  :worker_id,     'bigint unsigned'
+      t.column  :hostname,      :string
+      t.column  :process_id,    :integer
+      t.column  :job_id,        'bigint unsigned'
+      t.column  :task_id,       'bigint unsigned'
+      t.column  :iteration,     :integer
+      t.column  :name,          :string
+      t.column  :map_or_reduce, :string
+      t.column  :started_at,    "decimal(16,4)"
+      t.column  :version,       :integer
+      t.column  :processed,     :integer
+      t.column  :timeout,       "decimal(16,4)"            
+    end                                 
+    create_table :skynet_message_queues do |t|
+      t.column  :id,            "bigint unsigned primary key"
+      t.column  :created_on,    :timestamp
+      t.column  :updated_on,    :timestamp
+      t.column  :tasktype,      :string
+      t.column  :task_id,       'bigint unsigned'
+      t.column  :job_id,        'bigint unsigned'
+      t.column  :raw_payload,   :text
+      t.column  :payload_type,  :string
+      t.column  :name,          :string
+      t.column  :expiry,        :integer
+      t.column  :expire_time,   "decimal(16,4)"
+      t.column  :iteration,     :integer
+      t.column  :version,       :integer
+      t.column  :timeout,       "decimal(16,4)"
+    end                                   
+    create_table :skynet_queue_temperature do |t|
+      t.column  :id,            "bigint unsigned primary key"
+      t.column  :updated_on,    :timestamp
+      t.column  :count,         :integer, :default => 0
+      t.column  :temperature,   "decimal(6,4) default 1"
+      t.column  :type,          :string
+    end
+    add_index :skynet_message_queues, [:job_id]
+    add_index :skynet_message_queues, :task_id
+    add_index :skynet_message_queues, [:tasktype,:payload_type,:expire_time], :name => "index_skynet_mqueue_for_take"
+    add_index :skynet_worker_queues, [:hostname, :process_id]
+    add_index :skynet_worker_queues, :worker_id, :unique=> true
+    execute "insert into skynet_queue_temperature (type) values ('master')"
+    execute "insert into skynet_queue_temperature (type) values ('any')"
+    execute "insert into skynet_queue_temperature (type) values ('task')"
+  end
+
+  def self.down
+    drop_table :skynet_worker_queues
+    drop_table :skynet_queue_temperature
+    drop_table :skynet_message_queues
+  end
+end

data/app_generators/skynet_install/templates/skynet

@@ -0,0 +1,33 @@
+#!/usr/bin/env ruby
+
+<% if in_rails %>
+require File.dirname(__FILE__) + '/../config/boot'
+require File.expand_path(File.dirname(__FILE__)) + '/../config/environment'
+<% end %>
+
+require 'skynet'         
+
+Skynet::CONFIG[:LAUNCHER_PATH]      = File.expand_path(__FILE__)
+Skynet::CONFIG[:SKYNET_LOG_FILE]    = Logger::ERROR
+Skynet::CONFIG[:SKYNET_LOG_LEVEL]   = Logger::ERROR
+<% if in_rails %>
+Skynet::CONFIG[:SKYNET_PIDS_FILE]   = File.expand_path("../log/skynet.pid"),
+Skynet::CONFIG[:SKYNET_LOG_FILE]    = File.expand_path("../log/skynet.log"),
+<% else %>
+Skynet::CONFIG[:SKYNET_PIDS_FILE]   = File.expand_path("#{RAILS_ROOT}/log/skynet_#{RAILS_ENV}.pid"),
+Skynet::CONFIG[:SKYNET_LOG_FILE]    = File.expand_path("#{RAILS_ROOT}/log/skynet_#{RAILS_ENV}.log"),
+<% end %>
+
+begin
+  mq = Skynet::MessageQueue.new
+rescue Skynet::ConnectionError
+  if Skynet::MessageQueue.adapter == :tuplespace
+    pid = fork do
+      exec("skynet_tuplespace_server start")
+    end
+    sleep 3
+  end
+end
+
+
+Skynet.new

data/app_generators/skynet_install/templates/skynet_console

@@ -0,0 +1,16 @@
+#!/usr/bin/env ruby
+
+files_to_require = []
+<% if in_rails %>
+files_to_require << File.dirname(__FILE__) + '/../config/boot'
+files_to_require << File.expand_path(File.dirname(__FILE__)) + '/../config/environment'
+files_to_require.each { |file| require file }
+<% end %>
+
+require 'rubygems'
+require 'skynet'                                     
+
+Skynet::CONFIG[:LAUNCHER_PATH]          = File.expand_path(__FILE__)
+Skynet::CONFIG[:SKYNET_LOG_LEVEL]       = Logger::ERROR
+
+Skynet::Console.start(files_to_require)

data/bin/skynet

@@ -0,0 +1,20 @@
+#!/usr/bin/env ruby
+
+require File.expand_path(File.dirname(__FILE__)) + '/../lib/skynet.rb'
+
+Skynet::CONFIG[:WORKER_CHECK_DELAY]     = 4
+Skynet::CONFIG[:LAUNCHER_PATH]          = File.expand_path(__FILE__)
+
+begin
+  mq = Skynet::MessageQueue.new
+rescue Skynet::ConnectionError
+  if Skynet::MessageQueue.adapter == :tuplespace
+    pid = fork do
+      exec("skynet_tuplespace_server start")
+    end
+    sleep 3
+  end
+end
+
+
+Skynet.new

data/bin/skynet_console

@@ -0,0 +1,9 @@
+#!/usr/bin/env ruby
+
+require 'rubygems'
+require File.expand_path(File.dirname(__FILE__)) + '/../lib/skynet.rb'
+
+Skynet::CONFIG[:LAUNCHER_PATH]          = File.expand_path(__FILE__)
+Skynet::CONFIG[:SKYNET_LOG_LEVEL]       = Logger::ERROR
+
+Skynet::Console.start

data/bin/skynet_install

@@ -0,0 +1,12 @@
+require 'rubygems'
+require 'rubigen'
+
+if %w(-v --version).include? ARGV.first
+  require 'skynet_install/version'
+  puts "#{File.basename($0)} #{SkynetInstall::VERSION::STRING}"
+  exit(0)
+end
+
+require 'rubigen/scripts/generate'
+RubiGen::Base.use_application_sources! :skynet_install
+RubiGen::Scripts::Generate.new.run(ARGV, :generator => 'skynet_install')