DTR 0.0.1 → 0.0.2

data/CHANGES

@@ -1,2 +1,3 @@
 = DTR Changelog
+* changed to rinda service model
 

data/README

@@ -4,20 +4,25 @@ Supporting DTR version: 0.0.x
 
 This package contains DTR, a distributed test runner program for decreasing build time.
 
-DTR works in two parts: DTR Client and DTR Server.
-DTR Server controls all tests running or setting up test env. DTR Clients is always 
-waiting for commands from DTR server to:
-* set up test env
-* run test and update report to DTR Server
-
 DTR has the following features:
 
-* Run tests in mutli-processes/machines
+* Run tests in mutli processes/machines.
+
+* Hot plug distributed runners.
+
+* Runtime injection, all tests run in same environment including TestRunner.
+
+DTR works in three parts: DTR Server, Runner and Test Suite Injection.
 
-* Setup test env before running test.
+* DTR Server is a Rinda Server which allows DRb services and clients to automatically find each other without knowing where they live.
 
-You may need to consider using sychronize tools (likes rsync(http://samba.anu.edu.au/rsync/))
-to synchronize project files between machines.
+* Runner is a DRb service host on distributed machines to run tests.
+
+* Test Suite Injection works by loading 'dtr/inject_with_svn.rb' with all test files.
+
+DTR only supports Subversion to update codebase now, you may need to consider using 
+sychronize tools (likes rsync(http://samba.anu.edu.au/rsync/)) to synchronize project 
+files between machines.
 
 == Download
 
@@ -60,46 +65,49 @@ If you wish to run the unit and functional tests that come with DTR:
 
 == Simple Example
 
-Once installed, you can run DTR client at your project directory as follows ...
+Once installed, you can run DTR server anywhere:
+
+  % dtr -s
 
-  % dtr --server druby://build_server_ip:1314 --name client_name --port 3344
-  (The default port of dtr server is '1314')
+After server started, you can run DTR runner at your project directory as follows:
+
+  % dtr --server_address dtr_server_ip -r runner_name
 
 Type "dtr --help" for an up-to-date option summary.
 Invoking <tt>dtr</tt> without any options causes dtr to run
-with option '--server druby://localhost:1314 --port 3344'
+with option '--server_address localhost'
+
+Most of time your project test suite need setup environment before run tests, you
+can set setup command by option '--setup', for example:
 
-In your Rakefile, you can custom a task to let DTR server run tests:
+  % dtr --server_address dtr_server_ip --r runner1 --setup "rake db:test:prepare"
+  
+At last, you need to add 'dtr/inject_with_svn.rb' into test_files of your test 
+task in your Rakefile, and for load it, you also need add dtr lib path:
   
-  require "rubygems"
   require 'dtr'
 
-  task :cruise do
-    tests = FileList["test/unit/**/*test.rb].to_a
-    fail "Failed" unless DTR.run(tests, ['druby://client_1_ip:3344', 'druby://localhost:3344'])
+  Rake::TestTask.new(:dtr_tests) do |t|
+    t.libs << DTR.lib_path
+    t.test_files = FileList['dtr/inject_with_svn.rb', 'test/*test.rb']
+    t.warning = false
+    t.verbose = false
   end
 
 == Run tests in multi-processes on one machine
 
 In your Rakefile:
 
-  task :dtr => ["start_clients", "db:test:prepare"] do
-    fail "Failed" unless DTR.run(FileList["test/unit/**/*test.rb].to_a, ['druby://localhost:3345', 'druby://localhost:3346'])
-  ensure
-    DTR::ClientServer.stop_all
-  end
-  
-  task :start_clients do
-    DTR::ClientServer.new(:client_name => "client 1", :port => "3345").start
-    puts "client 1 started"
-    DTR::ClientServer.new(:client_name => "client 2", :port => "3346").start
-    puts "client 2 started"
-    sleep(2)
+  require 'dtr/raketasks'
+
+  DTR::MPTask.new :test do |t|
+    t.test_files = FileList['test/*test.rb']
+    t.processes = 2 #default is 2
   end
 
 == Credits
 
-[<b>Josh Price</b>] For fixing tests packer.
+[<b>Josh Price</b>] For fixing tests packer in release 0.0.1.
 
 [<b>Wang Pengchao</b>] For share of lots of ideas.
 
@@ -126,22 +134,31 @@ DTR client is invoked from the command line using:
    % dtr [<em>options</em> ...]
 
 Options are:
-    -s, --server_uri SERVER_URI      DTR server uri
-    -n, --name NAME                  DTR client name, the default is string 'client' with a time stamp.
-    -o, --output_dir OUTPUT_DIR      the dir for DTR client to output logs, the default is 'dtr_tmp'.
-    -i, --setup COMMAND              Define the command for initializing test environment, the default is 'rake db:test:prepare'
-    -p, --port PORT                  Define DTR client port, the default is 3344
-    -t, --trace                      Turn on invoke/execute tracing.
-    -u, --stdout                     Print all log into stdout.
+    -s, --server                     Start DTR service server. There must be a DTR 
+                                     server running.
+    -r runner1_name,runner2_name     Start DTR test runner agent. The name should 
+                                     be uniq of all runners. Every runner will be 
+                                     start in different process. The daemons gem 
+                                     must be installed, when start multiple runners
+    -i, --setup COMMAND              Set the command for initializing test environment 
+                                     for test runner, the default is no setup command
+    -a, --server_address ADDRESS     Set server address. The default is 'localhost'. 
+                                     It should be domain name or ip address, for 
+                                     example: 192.168.0.1
+    -D, --daemon                     Start server/runners in daemon mode. The daemon 
+                                     gem must be installed
+    -S, --stop_server                Stop server run in daemon mode.
+    -R, --stop_runners               Stop runners run in daemon mode.
+    -d, --debug                      Output debug log
     -v, --version                    Show version
     -h, --help                       Show this help doc
 
 = Other stuff
 
 Author::  Li Xiao <swing1979@gmail.com>
-Requires:: Ruby 1.8.0 or later
+Requires:: Ruby 1.8.6 or later
 License::  Copyright 2007 by Li Xiao.
-           Released under an Apache License.  See the LICENSE file
+           Released under an Apache License 2.  See the LICENSE file
            included in the distribution.
 
 == Warranty

data/Rakefile

@@ -32,7 +32,7 @@ end
 
 $package_version = CURRENT_VERSION
 
-SRC_RB = FileList['lib/**/*.rb']
+SRC_RB = FileList['lib/**/*.rb', 'lib/**/*.rake']
 
 # The default task is run if rake is given no explicit arguments.
 
@@ -46,9 +46,9 @@ end
 
 # Common Abbreviations ...
 
-task :test_all => [:test_units, :test_functionals]
+task :test_all => [:test_units, :tf]
 task :tu => :test_units
-task :tf => :test_functionals
+task :tf => [:start_dtr_server, :start_dtr_runners, :test_functionals, :stop_dtr_runners, :stop_dtr_server]
 task :test => :test_units
 
 Rake::TestTask.new(:test_units) do |t|
@@ -63,6 +63,22 @@ Rake::TestTask.new(:test_functionals) do |t|
   t.verbose = false
 end
 
+task :start_dtr_server do
+  ruby "-I#{File.dirname(__FILE__) + "/lib"} #{File.dirname(__FILE__) + "/bin/dtr"} -s -D"
+end
+
+task :start_dtr_runners do
+  ruby "-I#{File.dirname(__FILE__) + "/lib"} #{File.dirname(__FILE__) + "/bin/dtr"} -r r1,r2,r3 -D"
+end
+
+task :stop_dtr_runners do
+  ruby "-I#{File.dirname(__FILE__) + "/lib"} #{File.dirname(__FILE__) + "/bin/dtr"} -R"
+end
+
+task :stop_dtr_server do
+  ruby "-I#{File.dirname(__FILE__) + "/lib"} #{File.dirname(__FILE__) + "/bin/dtr"} -S"
+end
+
 begin
   require 'rcov/rcovtask'
 
@@ -119,6 +135,7 @@ PKG_FILES = FileList[
   '[A-Z]*',
   'bin/**/*', 
   'lib/**/*.rb', 
+  'lib/**/*.rake', 
   'test/**/*.rb',
   'doc/**/*'
 ]
@@ -350,19 +367,63 @@ task :install_jamis_template do
   install "doc/jamis.rb", dest_dir, :verbose => true
 end
 
+require 'logger'
+
+def logger
+  logger = Logger.new STDOUT
+  logger.level = Logger::DEBUG
+  logger
+end
 
-task :run_dtr_lots do
+task :c1 do
   Dir.chdir('testdata') do
-    begin
-      FileUtils.makedirs 'run_dtr_lots'
-      test_files = Array.new(10) do |index|
-        test_file = "run_dtr_lots/test#{index}.rb"
-        FileUtils.copy_file 'a_test_case.rb', test_file
-        test_file
-      end
-      DTR.run test_files, ['druby://localhost:3344']
-    ensure
-      FileUtils.rm_rf 'run_dtr_lots'
-    end
+    DTR.launch_runners(['c1'], nil)
+  end
+end
+
+task :c3 do
+  Dir.chdir('testdata') do
+    DTR.launch_runners(['c1', 'c2', 'c3'], nil)
+  end
+end
+task :c10 do
+  Dir.chdir('testdata') do
+    DTR.launch_runners(['c1', 'c2', 'c3', 'c4', 'c5', 'c6', 'c7', 'c8', 'c9', 'c10'], nil)
   end
 end
+
+task :runners do
+  runners = DTR.runners
+  if runners.empty?
+    puts "No runner available!"
+  else
+    puts runners.collect{|r| r.name}.join(", ")
+  end
+end
+
+task :c2 do
+  DTROPTIONS[:names] = ['c1', 'c2']
+  DTROPTIONS[:setup] = nil
+  Dir.chdir('testdata') do
+    DTR.start_runners
+  end
+end
+
+task :server do
+  DTR.start_server
+end
+
+Rake::TestTask.new(:dtr) do |t|
+  t.libs << DTR.lib_path
+  t.test_files = FileList['dtr/inject_with_svn.rb', 'testdata/*.rb']
+  t.warning = true
+  t.verbose = false
+end
+
+require 'dtr/raketasks'
+
+DTR::MPTask.new :mt do |t|
+  t.test_files = FileList['testdata/*.rb']
+  t.processes = 2
+end
+

data/TODO

@@ -3,9 +3,19 @@
 Send suggestions for this list to swing1979@gmail.com
 
 === To Do
-  17. check thread safe 
   18. integrate with cc.rb
   19. error log trace has file path on client
-  20. rake task
   21. good example in README
+  29. test hot plug runner, it works bad on Windows
+  32. start runner by scenario_tests
+  33. error msg should not displayed when ctrl-C shutdown runners
 (moved DONE list to CHANGES file)
+  !20. rake task
+  !22. give warning message when there is no runner available
+  !24. make sure we don't need to stop drb service by ourselves, or do we need to close rinda bus?
+  !25. get svn revision
+  !27. ring server's log: seems no
+  !28. runner should give an info when loaded files not included 'dtr/inject' and 'dtr/service_provider'
+  !30. should update to HEAD when first time to init env
+  !31. load libs
+  

data/bin/dtr

@@ -1,4 +1,5 @@
-#!/usr/bin/env ruby -w
+require 'optparse'
+
 begin
   require 'dtr'
 rescue LoadError
@@ -6,59 +7,79 @@ rescue LoadError
   require 'dtr'
 end
 
-require 'optparse'
-require 'logger'
-
 opts = OptionParser.new do |opts|
-  opts.banner = "DTR client server usage: #{$0} [options]"
+  opts.banner = "DTR usage: #{$0} [options]"
   opts.separator ""
   opts.separator "Synopsis:"
-  opts.separator "dtr -s 'DTR server uri'"
+  opts.separator "dtr -s"
+  opts.separator "dtr -r runner1_name, runner2_name"
   opts.separator ""
   opts.separator "Options:"
 
-  opts.on("-s", "--server_uri SERVER_URI", "DTR server uri") do |server_uri|
-    DTROPTIONS[:server_uri] = server_uri.untaint
+  opts.on_tail("-s", "--server", "Start DTR service server. There must be a DTR server running.") do |server_uri|
+    DTROPTIONS[:start_server] = true
   end
   
-  opts.on("-n", "--name NAME", "DTR client server name, the default is string 'client' with a time stamp.") do |name|
-    DTROPTIONS[:client_name] = name.untaint
+  opts.on("-r runner1_name,runner2_name", Array, "Start DTR test runner agent. The name should be uniq of all runners. Every runner will be start in different process. The daemon gem must be installed, when start multiple runners") do |names|
+    DTROPTIONS[:names] = names.collect{|name| name.untaint}
   end
-
-  opts.on("-o", "--output_dir OUTPUT_DIR", "the dir for DTR client server to output logs, the default is 'dtr_tmp'.") do |output_dir|
-    DTROPTIONS[:tmp_dir] = output_dir.untaint
+  
+  opts.on("-a", "--server_address ADDRESS", "Set server address. The default is 'localhost'. It should be domain name or ip address, for example: 192.168.0.1") do |address|
+    DTR.broadcast_list = [address]
   end
 
-  opts.on("-i", "--setup COMMAND", "Define the command for initializing test environment, the default is 'rake db:test:prepare'") do |command|
+  opts.on("-i", "--setup COMMAND", "Set the command for initializing test environment for test runner, the default is no setup command") do |command|
     DTROPTIONS[:setup] = command.untaint
   end
-
-  opts.on("-p", "--port PORT", "Define DTR client server port, the default is 3344") do |port|
-    DTROPTIONS[:port] = port.untaint
+  
+  opts.on_tail("-D", "--daemon", "Start server/runners in daemon mode. The daemon gem must be installed") do
+    DTROPTIONS[:daemon] = true
   end
-
-  opts.on_tail("-t", "--trace", "Turn on invoke/execute tracing.") do 
-    DTROPTIONS[:log_level] = Logger::DEBUG
+  
+  opts.on_tail("-S", "--stop_server", "Stop server run in daemon mode.") do
+    DTR.stop_server_daemon_mode
   end
   
-  opts.on_tail("-u", "--stdout", "Print all log into stdout.") do 
-    DTROPTIONS[:output_model] = :stdout
+  opts.on_tail("-R", "--stop_runners", "Stop runners run in daemon mode.") do
+    DTR.stop_runners_daemon_mode
+  end
+  
+  opts.on_tail("-d", "--debug", "output debug log") do
+    $logger = true
   end
   
   opts.on_tail("-v", "--version", "Show version") do
     puts "dtr, version " + DTRVERSION
-    DTROPTIONS[:exit] = true
   end
 
   opts.on_tail("-h", "--help", "Show this help doc") do 
     puts opts
-    DTROPTIONS[:exit] = true
   end
 end
 
-begin
-  opts.parse!
-  DTR.start_client_server unless DTROPTIONS[:exit]
-rescue Exception => e
-  logger.error { e }
+opts.parse!
+
+if defined?($logger)
+  require 'logger'
+  $logger = Logger.new(STDOUT) 
+  $logger.level = Logger::DEBUG
+  def logger
+    $logger
+  end
+end
+
+if DTROPTIONS[:names]
+  if DTROPTIONS[:daemon]
+    DTR.start_runners_daemon_mode
+  else
+    DTR.start_runners
+  end
+end
+
+if DTROPTIONS[:start_server]
+  if DTROPTIONS[:daemon]
+    DTR.start_server_daemon_mode
+  else
+    DTR.start_server
+  end
 end

data/install.rb

@@ -31,7 +31,7 @@ def installBIN(from, opfile)
   File.open(from) do |ip|
     File.open(tmp_file, "w") do |op|
       ruby = File.join($realbindir, $ruby)
-      op.puts "#!#{ruby} -w"
+      op.puts "#!#{ruby}"
       op.write ip.read
     end
   end
@@ -72,7 +72,7 @@ File::chmod(0755, dtr_dest)
 
 # The library files
 
-files = Dir.chdir('lib') { Dir['**/*.rb'] }
+files = Dir.chdir('lib') { Dir['**/*.rb', '**/*.rake'] }
 
 for fn in files
   fn_dir = File.dirname(fn)
@@ -86,4 +86,3 @@ end
 # and the executable
 
 installBIN("bin/dtr", "dtr")
-installBIN("bin/dtrserve", "dtrserve")

data/lib/dtr.rb

@@ -1,88 +1,82 @@
-require "dtr/drb_dtr"
-require "fileutils"
-require "logger"
 
-DTRVERSION="0.0.1"
+DTRVERSION="0.0.2"
 DTROPTIONS = {
-  #dtr client
-  :client_name => "client(#{Time.now})",
-  :setup => "rake db:test:prepare",
-  :port => "3344",
-  :server_uri => "druby://localhost:1314",
-  #dtr server
-  :server_port => "1314",
-  :wait_a_moment => 5,
-  :output_model => :log,
-  :raise_on_no_alive_client => false,
-  #both
-  :tmp_dir => "tmp/dtr",
-  :log_level => Logger::INFO
+  #:setup => nil
+  #:names => ["DTR(#{Time.now})"]
 }
 
-unless defined?(logger)
-  def logger
-    return $logger if defined?($logger) && $logger
-    DTR.ensure_temp_dir_exist
-    $logger = DTROPTIONS[:output_model] == :stdout ? Logger.new(STDOUT) : Logger.new("#{DTROPTIONS[:tmp_dir]}/runtime.log", 2, 1024000)
-    $logger.level = DTROPTIONS[:log_level]
-    $logger
-  end
-end
-
 module DTR
 
-  def start_client_server(my_port=DTROPTIONS[:port], server_uri=DTROPTIONS[:server_uri])
-    my_uri = "druby://:" << my_port
-
-    DTROPTIONS[:tmp_dir] = DTROPTIONS[:tmp_dir] + '/' + DTROPTIONS[:client_name].gsub(/[^\d\w]+/, '_')
-    logger.info { "Initializing client server at '#{my_uri}'..." }
-    logger.info { "Working dir: #{Dir.pwd}" }
-    logger.info { "Listening server at '#{server_uri}'..." }
-    ensure_temp_dir_exist
-    server = DrbTestRunnerClientServer.new my_uri, server_uri
-    server.start
+  def start_server
+    require 'dtr/service_provider'
+    ServiceProvider.new.start
   end
-
-  def run(tests, clients, opts={})
-    opts =  {:my_uri => "druby://:#{DTROPTIONS[:server_port]}", :run_setup_before_start => false}.merge(opts)
-    server = DrbTestRunnerServer.new opts[:my_uri], tests, clients
-    server.start opts[:run_setup_before_start]
-    server.succeeded?
+  
+  def start_runners
+    launch_runners(DTROPTIONS[:names], DTROPTIONS[:setup])
   end
-
-  def ensure_temp_dir_exist
-    raise "Tmp dir(#{DTROPTIONS[:tmp_dir]}) is a file!" if File.file?(DTROPTIONS[:tmp_dir])
-    unless File.exist?(DTROPTIONS[:tmp_dir])
-      FileUtils.makedirs(DTROPTIONS[:tmp_dir])
+  
+  def launch_runners(names, setup=nil)
+    require 'dtr/runner'
+    names = names || "DTR(#{Time.now})"
+    DTR::RunnerProtectProcess.start(names, setup)
+  end
+  
+  def lib_path
+    File.expand_path(File.dirname(__FILE__))
+  end
+  
+  def broadcast_list=(list)
+    require 'dtr/service_provider'
+    ServiceProvider.broadcast_list = list
+  end
+  
+  def runners
+    require 'dtr/service_provider'
+    ServiceProvider.new.runners
+  end
+  
+  def stop_runners_daemon_mode
+    with_daemon_gem do
+      Daemons.run_proc "dtr_runners", :ARGV => ['stop'] 
     end
   end
-
-  module_function :start_client_server, :run, :ensure_temp_dir_exist
-
-  class ClientServer
-    @@servers = []
-    
-    def self.stop_all
-      logger.info { "client servers: #{@@servers.join(", ")}" }
-      @@servers.each do |server|
-        Process.kill "TERM", server
-        logger.info { "Killed #{server}" }
+  
+  def start_runners_daemon_mode
+    with_daemon_gem do
+      pwd = Dir.pwd
+      Daemons.run_proc "dtr_runners", :ARGV => ['start'], :backtrace => true, :log_output => true  do
+        Dir.chdir(pwd) do
+          start_runners
+        end
       end
     end
-    
-    def initialize(opts={})
-      @opts = DTROPTIONS.merge opts
-    end
-    
-    def start
-      @@servers << Process.fork do
-        DTROPTIONS.merge! @opts
-        trap("TERM") do
-          exit!
-        end
-        DTR.start_client_server
+  end
+  
+  def start_server_daemon_mode
+    with_daemon_gem do
+      Daemons.run_proc "dtr_server", :ARGV => ['start'] do
+        start_server
       end
     end
+    sleep(2)
+  end
+  
+  def stop_server_daemon_mode
+    with_daemon_gem do
+      Daemons.run_proc "dtr_server", :ARGV => ['stop']
+    end
+  end
+  
+  def with_daemon_gem
+    require "rubygems"
+    begin
+      require "daemons"
+    rescue LoadError
+      raise "The daemons gem must be installed"
+    end
+    yield
   end
   
+  module_function :start_server, :start_runners, :launch_runners, :lib_path, :broadcast_list=, :runners, :with_daemon_gem, :start_runners_daemon_mode, :stop_runners_daemon_mode, :start_server_daemon_mode, :stop_server_daemon_mode
 end