skynet 0.9.2 → 0.9.3

Sign up to get free protection for your applications and to get access to all the features.
Files changed (128) hide show
  1. data/History.txt +49 -0
  2. data/Manifest.txt +84 -6
  3. data/README.txt +75 -64
  4. data/app_generators/skynet_install/skynet_install_generator.rb +14 -8
  5. data/app_generators/skynet_install/templates/migration.rb +1 -24
  6. data/app_generators/skynet_install/templates/skynet_config.rb +50 -0
  7. data/app_generators/skynet_install/templates/skynet_initializer.rb +1 -0
  8. data/app_generators/skynet_install/templates/{skynet_schema.sql → skynet_mysql_schema.sql} +1 -24
  9. data/bin/skynet +37 -10
  10. data/bin/skynet_install +5 -5
  11. data/bin/skynet_tuplespace_server +27 -19
  12. data/examples/dgrep/README +70 -0
  13. data/examples/dgrep/config/skynet_config.rb +26 -0
  14. data/examples/dgrep/data/shakespeare/README +2 -0
  15. data/examples/dgrep/data/shakespeare/poetry/loverscomplaint +381 -0
  16. data/examples/dgrep/data/shakespeare/poetry/rapeoflucrece +2199 -0
  17. data/examples/dgrep/data/shakespeare/poetry/sonnets +2633 -0
  18. data/examples/dgrep/data/shakespeare/poetry/various +640 -0
  19. data/examples/dgrep/data/shakespeare/poetry/venusandadonis +1423 -0
  20. data/examples/dgrep/data/testfile1.txt +1 -0
  21. data/examples/dgrep/data/testfile2.txt +1 -0
  22. data/examples/dgrep/data/testfile3.txt +1 -0
  23. data/examples/dgrep/data/testfile4.txt +1 -0
  24. data/examples/dgrep/lib/dgrep.rb +59 -0
  25. data/examples/dgrep/lib/mapreduce_test.rb +32 -0
  26. data/examples/dgrep/lib/most_common_words.rb +45 -0
  27. data/examples/dgrep/script/dgrep +75 -0
  28. data/examples/rails_mysql_example/README +66 -0
  29. data/examples/rails_mysql_example/Rakefile +10 -0
  30. data/examples/rails_mysql_example/app/controllers/application.rb +10 -0
  31. data/examples/rails_mysql_example/app/helpers/application_helper.rb +3 -0
  32. data/examples/rails_mysql_example/app/models/user.rb +21 -0
  33. data/examples/rails_mysql_example/app/models/user_favorite.rb +5 -0
  34. data/examples/rails_mysql_example/app/models/user_mailer.rb +12 -0
  35. data/examples/rails_mysql_example/app/views/user_mailer/welcome.erb +5 -0
  36. data/examples/rails_mysql_example/config/boot.rb +109 -0
  37. data/examples/rails_mysql_example/config/database.yml +42 -0
  38. data/examples/rails_mysql_example/config/environment.rb +59 -0
  39. data/examples/rails_mysql_example/config/environments/development.rb +18 -0
  40. data/examples/rails_mysql_example/config/environments/production.rb +19 -0
  41. data/examples/rails_mysql_example/config/environments/test.rb +22 -0
  42. data/examples/rails_mysql_example/config/initializers/inflections.rb +10 -0
  43. data/examples/rails_mysql_example/config/initializers/mime_types.rb +5 -0
  44. data/examples/rails_mysql_example/config/initializers/skynet.rb +1 -0
  45. data/examples/rails_mysql_example/config/routes.rb +35 -0
  46. data/examples/rails_mysql_example/config/skynet_config.rb +36 -0
  47. data/examples/rails_mysql_example/db/migrate/001_create_skynet_tables.rb +43 -0
  48. data/examples/rails_mysql_example/db/migrate/002_create_users.rb +16 -0
  49. data/examples/rails_mysql_example/db/migrate/003_create_user_favorites.rb +14 -0
  50. data/examples/rails_mysql_example/db/schema.rb +85 -0
  51. data/examples/rails_mysql_example/db/skynet_mysql_schema.sql +33 -0
  52. data/examples/rails_mysql_example/doc/README_FOR_APP +2 -0
  53. data/examples/rails_mysql_example/lib/tasks/rails_mysql_example.rake +20 -0
  54. data/examples/rails_mysql_example/public/.htaccess +40 -0
  55. data/examples/rails_mysql_example/public/404.html +30 -0
  56. data/examples/rails_mysql_example/public/422.html +30 -0
  57. data/examples/rails_mysql_example/public/500.html +30 -0
  58. data/examples/rails_mysql_example/public/dispatch.cgi +10 -0
  59. data/examples/rails_mysql_example/public/dispatch.fcgi +24 -0
  60. data/examples/rails_mysql_example/public/dispatch.rb +10 -0
  61. data/{log/debug.log → examples/rails_mysql_example/public/favicon.ico} +0 -0
  62. data/examples/rails_mysql_example/public/images/rails.png +0 -0
  63. data/examples/rails_mysql_example/public/index.html +277 -0
  64. data/examples/rails_mysql_example/public/javascripts/application.js +2 -0
  65. data/examples/rails_mysql_example/public/javascripts/controls.js +963 -0
  66. data/examples/rails_mysql_example/public/javascripts/dragdrop.js +972 -0
  67. data/examples/rails_mysql_example/public/javascripts/effects.js +1120 -0
  68. data/examples/rails_mysql_example/public/javascripts/prototype.js +4225 -0
  69. data/examples/rails_mysql_example/public/robots.txt +5 -0
  70. data/examples/rails_mysql_example/script/about +3 -0
  71. data/examples/rails_mysql_example/script/console +3 -0
  72. data/examples/rails_mysql_example/script/destroy +3 -0
  73. data/examples/rails_mysql_example/script/generate +3 -0
  74. data/examples/rails_mysql_example/script/performance/benchmarker +3 -0
  75. data/examples/rails_mysql_example/script/performance/profiler +3 -0
  76. data/examples/rails_mysql_example/script/performance/request +3 -0
  77. data/examples/rails_mysql_example/script/plugin +3 -0
  78. data/examples/rails_mysql_example/script/process/inspector +3 -0
  79. data/examples/rails_mysql_example/script/process/reaper +3 -0
  80. data/examples/rails_mysql_example/script/process/spawner +3 -0
  81. data/examples/rails_mysql_example/script/runner +3 -0
  82. data/examples/rails_mysql_example/script/server +3 -0
  83. data/examples/rails_mysql_example/test/fixtures/user_favorites.yml +9 -0
  84. data/examples/rails_mysql_example/test/fixtures/users.yml +11 -0
  85. data/examples/rails_mysql_example/test/test_helper.rb +38 -0
  86. data/examples/rails_mysql_example/test/unit/user_favorite_test.rb +8 -0
  87. data/examples/rails_mysql_example/test/unit/user_test.rb +8 -0
  88. data/extras/README +7 -0
  89. data/extras/init.d/skynet +87 -0
  90. data/extras/nagios/check_skynet.sh +121 -0
  91. data/extras/rails/controllers/skynet_controller.rb +43 -0
  92. data/extras/rails/views/skynet/index.rhtml +137 -0
  93. data/lib/skynet.rb +59 -1
  94. data/lib/skynet/mapreduce_helper.rb +2 -2
  95. data/lib/skynet/mapreduce_test.rb +32 -1
  96. data/lib/skynet/message_queue_adapters/mysql.rb +422 -539
  97. data/lib/skynet/message_queue_adapters/tuple_space.rb +45 -71
  98. data/lib/skynet/skynet_active_record_extensions.rb +22 -11
  99. data/lib/skynet/skynet_config.rb +54 -20
  100. data/lib/skynet/skynet_console.rb +4 -1
  101. data/lib/skynet/skynet_console_helper.rb +5 -1
  102. data/lib/skynet/skynet_debugger.rb +58 -4
  103. data/lib/skynet/skynet_job.rb +61 -24
  104. data/lib/skynet/skynet_launcher.rb +29 -3
  105. data/lib/skynet/skynet_logger.rb +11 -1
  106. data/lib/skynet/skynet_manager.rb +403 -240
  107. data/lib/skynet/skynet_message.rb +1 -3
  108. data/lib/skynet/skynet_message_queue.rb +42 -19
  109. data/lib/skynet/skynet_partitioners.rb +19 -15
  110. data/lib/skynet/skynet_ruby_extensions.rb +18 -0
  111. data/lib/skynet/skynet_tuplespace_server.rb +17 -14
  112. data/lib/skynet/skynet_worker.rb +132 -98
  113. data/lib/skynet/version.rb +1 -1
  114. data/script/destroy +0 -0
  115. data/script/generate +0 -0
  116. data/script/txt2html +0 -0
  117. data/test/test_helper.rb +2 -0
  118. data/test/test_skynet.rb +13 -5
  119. data/test/test_skynet_manager.rb +24 -9
  120. data/test/test_skynet_task.rb +1 -1
  121. data/website/index.html +77 -29
  122. data/website/index.txt +53 -24
  123. data/website/stylesheets/screen.css +12 -12
  124. metadata +156 -66
  125. data/app_generators/skynet_install/templates/skynet +0 -46
  126. data/log/skynet.log +0 -29
  127. data/log/skynet_tuplespace_server.log +0 -7
  128. data/log/skynet_worker.pid +0 -1
@@ -0,0 +1,50 @@
1
+ # Skynet Configuration File. Should be in APP_ROOT/config/skynet_config.rb
2
+ # Start skynet from within your app root with
3
+ # > skynet start
4
+
5
+ require 'rubygems'
6
+ require 'skynet'
7
+
8
+ <% if in_rails -%>
9
+ # Load your rails app
10
+ if not defined?(RAILS_GEM_VERSION)
11
+ require File.expand_path(File.dirname(__FILE__)) + '/../config/environment'
12
+ end
13
+ <% end -%>
14
+
15
+ Skynet::CONFIG[:SKYNET_LOG_LEVEL] = Logger::ERROR
16
+ <% if in_rails -%>
17
+ Skynet::CONFIG[:APP_ROOT] = RAILS_ROOT
18
+ Skynet::CONFIG[:SKYNET_LOG_DIR] = File.expand_path(Skynet::CONFIG[:APP_ROOT]+ "/log")
19
+ Skynet::CONFIG[:SKYNET_PID_DIR] = File.expand_path(Skynet::CONFIG[:APP_ROOT] + "/log")
20
+ Skynet::CONFIG[:SKYNET_LOG_FILE] = "skynet_#{RAILS_ENV}.log"
21
+ Skynet::CONFIG[:SKYNET_PID_FILE] = "skynet_#{RAILS_ENV}.pid"
22
+ <% else -%>
23
+ Skynet::CONFIG[:SKYNET_LOG_DIR] = File.expand_path(File.dirname(__FILE__) + "/../log")
24
+ Skynet::CONFIG[:SKYNET_PID_DIR] = File.expand_path(File.dirname(__FILE__) + "/../log")
25
+ <% end -%>
26
+
27
+ <% if mysql -%>
28
+ <% if not in_rails -%>
29
+ # Use the mysql message queue adapter
30
+ Skynet::CONFIG[:MESSAGE_QUEUE_ADAPTER] = "Skynet::MessageQueueAdapter::Mysql"
31
+ Skynet::CONFIG[:MYSQL_HOST] = "localhost"
32
+ Skynet::CONFIG[:MYSQL_USERNAME] = "root"
33
+ Skynet::CONFIG[:MYSQL_PASSWORD] = ""
34
+ Skynet::CONFIG[:MYSQL_DATABASE] = "skynet"
35
+ <% else %>
36
+ # Use the mysql message queue adapter
37
+ Skynet::CONFIG[:MESSAGE_QUEUE_ADAPTER] = "Skynet::MessageQueueAdapter::Mysql"
38
+ <% end -%>
39
+ <% end -%>
40
+
41
+ # ==================================================================
42
+ # = Require any other libraries you want skynet to know about here =
43
+ # ==================================================================
44
+
45
+
46
+ # ===========================================
47
+ # = Set your own configuration options here =
48
+ # ===========================================
49
+ # You can also configure skynet with
50
+ # Skynet.configure(:SOME_CONFIG_OPTION => true, :SOME_OTHER_CONFIG => 3)
@@ -0,0 +1 @@
1
+ require RAILS_ROOT+'/config/skynet_config.rb'
@@ -7,7 +7,7 @@ CREATE TABLE skynet_message_queues (
7
7
  tasktype varchar(255) default NULL,
8
8
  task_id bigint(20) unsigned default NULL,
9
9
  job_id bigint(20) unsigned default NULL,
10
- raw_payload text,
10
+ raw_payload longtext,
11
11
  payload_type varchar(255) default NULL,
12
12
  name varchar(255) default NULL,
13
13
  expiry int(11) default NULL,
@@ -22,29 +22,6 @@ CREATE TABLE skynet_message_queues (
22
22
  KEY index_skynet_message_queues_on_task_id (task_id),
23
23
  KEY index_skynet_mqueue_for_take (queue_id,tasktype,payload_type,expire_time)
24
24
  ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
25
- CREATE TABLE skynet_worker_queues (
26
- id int(11) NOT NULL auto_increment,
27
- queue_id int(11) default '0',
28
- created_on datetime default NULL,
29
- updated_on datetime default NULL,
30
- tasktype varchar(255) default NULL,
31
- tasksubtype varchar(255) default NULL,
32
- worker_id bigint(20) unsigned default NULL,
33
- hostname varchar(255) default NULL,
34
- process_id int(11) default NULL,
35
- job_id bigint(20) unsigned default NULL,
36
- task_id bigint(20) unsigned default NULL,
37
- iteration int(11) default NULL,
38
- name varchar(255) default NULL,
39
- map_or_reduce varchar(255) default NULL,
40
- started_at decimal(16,4) default NULL,
41
- version int(11) default NULL,
42
- processed int(11) default NULL,
43
- timeout decimal(16,4) default NULL,
44
- PRIMARY KEY (id),
45
- UNIQUE KEY index_skynet_worker_queues_on_worker_id (worker_id),
46
- KEY index_skynet_worker_queues_on_hostname_and_process_id (hostname,process_id)
47
- ) ENGINE=InnoDB DEFAULT CHARSET=latin1;
48
25
  CREATE TABLE skynet_queue_temperature (
49
26
  id int(11) NOT NULL auto_increment,
50
27
  queue_id int(11) default '0',
data/bin/skynet CHANGED
@@ -27,18 +27,45 @@
27
27
  require 'rubygems'
28
28
  require File.expand_path(File.dirname(__FILE__)) + '/../lib/skynet.rb'
29
29
 
30
- Skynet::CONFIG[:WORKER_CHECK_DELAY] ||= 4
31
- Skynet::CONFIG[:LAUNCHER_PATH] = File.expand_path(__FILE__)
32
30
 
33
- begin
34
- mq = Skynet::MessageQueue.new
35
- rescue Skynet::ConnectionError
36
- if Skynet::MessageQueue.adapter == :tuplespace
37
- pid = fork do
38
- exec("skynet_tuplespace_server start")
31
+ if config_arg = ARGV.detect {|i| i =~ /--help/ }
32
+ Skynet.start
33
+ exit
34
+ end
35
+
36
+ conf_file = nil
37
+ if config_arg = ARGV.detect {|i| i =~ /--config/i }
38
+ conf_file = config_arg.split('=').last
39
+ ARGV.delete_if {|i| i =~ /--config/ }
40
+ else
41
+ ['config/skynet_config.rb', 'skynet_config.rb', "../config/skynet_config.rb" , "#{ENV['HOME']}/.skynet/config.rb"].each do |path|
42
+ if File.exists?(path)
43
+ conf_file = path
44
+ break
39
45
  end
40
- sleep 3
41
46
  end
42
47
  end
43
48
 
44
- Skynet.new
49
+ if conf_file
50
+ Skynet::CONFIG[:CONFIG_FILE] = File.expand_path(conf_file)
51
+ Skynet::CONFIG[:APP_ROOT] = File.dirname(Skynet::CONFIG[:CONFIG_FILE])
52
+ Skynet::CONFIG[:SKYNET_LOG_DIR] = File.expand_path(Skynet::CONFIG[:APP_ROOT]+ "/../log")
53
+ Skynet::CONFIG[:SKYNET_PID_DIR] = File.expand_path(Skynet::CONFIG[:APP_ROOT] + "/../log")
54
+ begin
55
+ require conf_file
56
+ rescue MissingSourceFile => e
57
+ puts "Specified config file #{conf_file} is missing."
58
+ exit
59
+ end
60
+ puts "using config file: #{File.expand_path(conf_file)}" unless conf_file == 'config/skynet_config.rb'
61
+ else
62
+ puts "No configuration file found.\nTry running skynet_install to create a config/skynet_config.rb or starting skynet with --config=CONFIG_FILE_LOCATION. Exiting."
63
+ exit
64
+ end
65
+
66
+ Skynet::CONFIG[:WORKER_CHECK_DELAY] ||= 4
67
+ Skynet::CONFIG[:LAUNCHER_PATH] = File.expand_path(__FILE__)
68
+
69
+ Skynet::CONFIG[:SYSTEM_RUNNER] = true
70
+
71
+ Skynet.start
@@ -1,15 +1,15 @@
1
1
  #!/usr/bin/env ruby
2
2
 
3
- # skynet_install is used to install skynet binaries into your application.
4
- # The value of using your own skynet binaries is that they can ensure your code
5
- # is available to all skynet workers.
3
+ # skynet_install is used to install skynet libraries into your application.
4
+ # Specifically, it installs a config/skynet_config.rb file
5
+ # Once you have a config/skynet_config.rb you can run skynet start from within your root_app_dir/
6
6
  # This is also how you can run skynet from within rails. (using --rails)
7
7
  #
8
- # USAGE: skynet_install [--rails] directory (can be '.' for current)"
8
+ # USAGE: skynet_install [--rails] directory/ (can be '.' for current)"
9
9
  #
10
10
  # Options:
11
11
  # -v, --version Show the skynet_install version number and quit.
12
- # --include-migration Include mysql migration if you want to use mysql as your message queue
12
+ # --mysql Include mysql migration if you want to use mysql as your message queue
13
13
  # -r, --rails Install into rails app
14
14
  # Default: false
15
15
  # General Options:
@@ -4,24 +4,25 @@
4
4
  # Skynet::CONFIG[:MESSAGE_QUEUE_ADAPTER] = "Skynet::MessageQueueAdapter::TupleSpace"
5
5
  #
6
6
  # Usage: skynet_tuplespace_server (start|stop|run) [options]
7
- # -t, --ontop TRUE Dont Daemonize
8
- # -p, --port PORT Port to listen on. default 7647
9
- # -o, --log LOGFILE Logfile to log to
10
- # -l, --loglevel LOGLEVEL Log level defaults to DEBUG
11
- # -d, --piddir PIDDIR Directory to put pidfile
12
- # -u, --drburi Drb URI What DRbURI to use
7
+ # -t, --ontop=TRUE Dont Daemonize
8
+ # -p, --port=PORT Port to listen on. default 7647
9
+ # -o, --logfile=LOGFILE Logfile to log to
10
+ # -l, --loglevel=LOGLEVEL Log level defaults to DEBUG
11
+ # -d, --piddir=PIDDIR Directory to put pidfile
12
+ # -u, --drburi=druby://DRB_URI What DRbURI to use
13
13
 
14
14
  require 'rubygems'
15
15
  require 'daemons'
16
16
  require 'pp'
17
17
 
18
- require File.expand_path(File.dirname(__FILE__)) + '/../lib/skynet.rb'
18
+ require File.expand_path(File.dirname(__FILE__)) + '/../lib/skynet.rb'
19
19
 
20
20
  options = {
21
- :port => 7647,
22
- :logfile => Skynet::CONFIG[:SKYNET_LOG_FILE],
23
- :loglevel => "DEBUG",
24
- :piddir => Skynet::CONFIG[:SKYNET_PID_DIR]
21
+ :port => 7647,
22
+ :logfile => Skynet::Config.pidfile_location,
23
+ :loglevel => "ERROR",
24
+ :piddir => Skynet::CONFIG[:SKYNET_PID_DIR],
25
+ :use_ringserver => Skynet::CONFIG[:TS_USE_RINGSERVER],
25
26
  }
26
27
 
27
28
  OptionParser.new do |opt|
@@ -29,26 +30,33 @@ OptionParser.new do |opt|
29
30
  opt.on('-t', '--ontop TRUE', 'Dont Daemonize') do |v|
30
31
  options[:ontop] = true if v.upcase == "TRUE" or v == "1"
31
32
  end
32
- opt.on('-p', '--port PORT', 'Port to listen on. default 7647') do |v|
33
+ opt.on('-p', '--port=PORT', 'Port to listen on. default 7647') do |v|
33
34
  options[:port] = v.to_i
34
35
  end
35
- opt.on('-o', '--log LOGFILE', 'Logfile to log to') do |v|
36
- options[:logfile] = v
36
+ opt.on('-o', '--logfile=LOGFILE', 'Logfile to log to') do |v|
37
+ options[:logfile] = File.expand_path(v)
37
38
  end
38
- opt.on('-l', '--loglevel LOGLEVEL', 'Log level defaults to DEBUG') do |v|
39
+ opt.on('-l', '--loglevel=LOGLEVEL', 'Log level defaults to DEBUG') do |v|
39
40
  options[:loglevel] = v
40
41
  end
41
- opt.on('-d', '--piddir PIDDIR', 'Directory to put pidfile') do |v|
42
+ opt.on('-d', '--piddir=PIDDIR', 'Directory to put pidfile') do |v|
42
43
  options[:piddir] = File.expand_path(v)
43
44
  end
44
- opt.on('-u', '--drburi Drb URI', 'What DRbURI to use') do |v|
45
+ opt.on('--use_ringserver=TRUE', 'Use ringserver?') do |v|
46
+ if v.downcase == "true" or v == "1"
47
+ options[:use_ringserver] = true
48
+ else
49
+ options[:use_ringserver] = false
50
+ end
51
+ end
52
+ opt.on('-u', '--drburi=druby://DRB_URI', 'What DRbURI to use') do |v|
45
53
  if v =~ %r{druby://}
46
54
  options[:drburi] = v
47
55
  else
48
56
  options[:drburi] = "druby://#{v}"
49
57
  end
50
58
  options[:drburi] =~ /druby:\/\/.+?:(\d*)/
51
- options[:port] = $1.to_i
59
+ options[:port] ||= $1.to_i
52
60
  end
53
61
 
54
62
  opt.parse!(ARGV)
@@ -62,5 +70,5 @@ Daemons.run_proc("skynet_tuplespace_server#{options[:port]}",
62
70
  # :monitor => true,
63
71
  :ontop => options[:ontop] || false
64
72
  }) do
65
- server = Skynet::Server.new(options)
73
+ server = Skynet::TuplespaceServer.start(options)
66
74
  end
@@ -0,0 +1,70 @@
1
+
2
+ EXAMPLE 1: Distributed Grep
3
+ ------------------------------
4
+ You have a list of files and want to see how many times each word shows up in those files.
5
+ All of the map_reduce code is in lib/mapreduce_test.rb
6
+
7
+ To
8
+ 1. Run Skynet
9
+ > ./script/skynet start
10
+
11
+ 2. Watch the logs in another terminal
12
+ > tail -f log/skynet.log
13
+
14
+ 3. Run dgrep providing it with some words and a directory
15
+ > ./script/dgrep -r love hate ./data
16
+
17
+ I've put all of shakespears poetry in the data directory. You can download all of Shakespeare's work
18
+ to add to the data at http://www.it.usyd.edu.au/~matty/Shakespeare/shakespeare.tar.gz
19
+
20
+ Thats it!!
21
+
22
+ EXAMPLE 2: Distributed Grep with distributed reducing
23
+ ------------------------------
24
+ Same example as above, but this time we want to aggregate our results across multiple partitioners. This would
25
+ be useful if you had a large reduce data set, or complex calculations in order to reduce. All the code is already
26
+ in dgrep.pl. Specifically, notice I've included a self.reduce_partition method (which just forwards to a pre-packaged
27
+ partitioner). This partitioner not only breaks up the data returned from the mappers, but also makes sure each reducer
28
+ gets all the same keys.
29
+
30
+ 1. Run dgrep passing --reducers=2 This tells Skynet to try and use up to 2 reducers (instead of the default of 1)
31
+ > ./script/dgrep -r --reducers=2 data file ./data
32
+
33
+ This sample application was created by creating in the following way
34
+
35
+ 1. Create new app
36
+ > mkdir simple_example
37
+ > mkdir data lib
38
+
39
+ 2. Install Skynet
40
+ > skynet_install
41
+
42
+ 3. Add Data Files to /data
43
+ These are any text files named testfileN.txt and so on
44
+
45
+ 4. Create dgrep.rb and drep script (found in lib/ and script/)
46
+
47
+ ------------------------------------
48
+
49
+ EXAMPLE 1: Most Common Words
50
+ ------------------------------
51
+ You have a list of files and want the 10 most common 5 or more letter words in all of those fules.
52
+
53
+ 1. Run Skynet
54
+ > ./script/skynet start
55
+
56
+ 2. Watch the logs in another terminal
57
+ > tail -f log/skynet.log
58
+
59
+ 3. Run dgrep providing it with the '-mcw' (most common words) command, some words and a directory
60
+ > ./script/dgrep -r -mcw ./data
61
+
62
+ To see what's going on check out lib/most_common_words.rb
63
+ ------------------------------------
64
+
65
+ If you'd like get a peak at what Skynet is doing, edit ./config/skynet_config.rb and set
66
+
67
+ Skynet::CONFIG[:SKYNET_LOG_LEVEL] = Logger::INFO # or Logger::INFO
68
+
69
+ and tail ./log/skynet.log
70
+
@@ -0,0 +1,26 @@
1
+ # Skynet Configuration File. Should be in APP_ROOT/config/skynet_config.rb
2
+ # Start skynet from within your app root with
3
+ # $ skynet start
4
+
5
+ require 'rubygems'
6
+ require 'skynet'
7
+
8
+ Skynet::CONFIG[:SKYNET_LOG_LEVEL] = Logger::ERROR
9
+ Skynet::CONFIG[:SKYNET_LOG_DIR] = File.expand_path(File.dirname(__FILE__) + "/../log")
10
+ Skynet::CONFIG[:SKYNET_PID_DIR] = File.expand_path(File.dirname(__FILE__) + "/../log")
11
+
12
+ # ==================================================================
13
+ # = Require any other libraries you want skynet to know about here =
14
+ # ==================================================================
15
+
16
+ # require all ruby files in lib/
17
+ Dir["#{File.dirname(__FILE__)}" + "/../lib/**/*.rb"].each do |file|
18
+ require file
19
+ end
20
+
21
+ # ===========================================
22
+ # = Set your own configuration options here =
23
+ # ===========================================
24
+ # You can also configure skynet with
25
+ # Skynet.configure(:SOME_CONFIG_OPTION => true, :SOME_OTHER_CONFIG => 3)
26
+
@@ -0,0 +1,2 @@
1
+ Download the complete text of shakespeare at
2
+ http://www.it.usyd.edu.au/~matty/Shakespeare/shakespeare.tar.gz
@@ -0,0 +1,381 @@
1
+ A LOVER'S COMPLAINT
2
+
3
+
4
+
5
+ FROM off a hill whose concave womb reworded
6
+ A plaintful story from a sistering vale,
7
+ My spirits to attend this double voice accorded,
8
+ And down I laid to list the sad-tuned tale;
9
+ Ere long espied a fickle maid full pale,
10
+ Tearing of papers, breaking rings a-twain,
11
+ Storming her world with sorrow's wind and rain.
12
+
13
+ Upon her head a platted hive of straw,
14
+ Which fortified her visage from the sun,
15
+ Whereon the thought might think sometime it saw
16
+ The carcass of beauty spent and done:
17
+ Time had not scythed all that youth begun,
18
+ Nor youth all quit; but, spite of heaven's fell rage,
19
+ Some beauty peep'd through lattice of sear'd age.
20
+
21
+ Oft did she heave her napkin to her eyne,
22
+ Which on it had conceited characters,
23
+ Laundering the silken figures in the brine
24
+ That season'd woe had pelleted in tears,
25
+ And often reading what contents it bears;
26
+ As often shrieking undistinguish'd woe,
27
+ In clamours of all size, both high and low.
28
+
29
+ Sometimes her levell'd eyes their carriage ride,
30
+ As they did battery to the spheres intend;
31
+ Sometime diverted their poor balls are tied
32
+ To the orbed earth; sometimes they do extend
33
+ Their view right on; anon their gazes lend
34
+ To every place at once, and, nowhere fix'd,
35
+ The mind and sight distractedly commix'd.
36
+
37
+ Her hair, nor loose nor tied in formal plat,
38
+ Proclaim'd in her a careless hand of pride
39
+ For some, untuck'd, descended her sheaved hat,
40
+ Hanging her pale and pined cheek beside;
41
+ Some in her threaden fillet still did bide,
42
+ And true to bondage would not break from thence,
43
+ Though slackly braided in loose negligence.
44
+
45
+ A thousand favours from a maund she drew
46
+ Of amber, crystal, and of beaded jet,
47
+ Which one by one she in a river threw,
48
+ Upon whose weeping margent she was set;
49
+ Like usury, applying wet to wet,
50
+ Or monarch's hands that let not bounty fall
51
+ Where want cries some, but where excess begs all.
52
+
53
+ Of folded schedules had she many a one,
54
+ Which she perused, sigh'd, tore, and gave the flood;
55
+ Crack'd many a ring of posied gold and bone
56
+ Bidding them find their sepulchres in mud;
57
+ Found yet moe letters sadly penn'd in blood,
58
+ With sleided silk feat and affectedly
59
+ Enswathed, and seal'd to curious secrecy.
60
+
61
+ These often bathed she in her fluxive eyes,
62
+ And often kiss'd, and often 'gan to tear:
63
+ Cried 'O false blood, thou register of lies,
64
+ What unapproved witness dost thou bear!
65
+ Ink would have seem'd more black and damned here!'
66
+ This said, in top of rage the lines she rents,
67
+ Big discontent so breaking their contents.
68
+
69
+ A reverend man that grazed his cattle nigh--
70
+ Sometime a blusterer, that the ruffle knew
71
+ Of court, of city, and had let go by
72
+ The swiftest hours, observed as they flew--
73
+ Towards this afflicted fancy fastly drew,
74
+ And, privileged by age, desires to know
75
+ In brief the grounds and motives of her woe.
76
+
77
+ So slides he down upon his grained bat,
78
+ And comely-distant sits he by her side;
79
+ When he again desires her, being sat,
80
+ Her grievance with his hearing to divide:
81
+ If that from him there may be aught applied
82
+ Which may her suffering ecstasy assuage,
83
+ 'Tis promised in the charity of age.
84
+
85
+ 'Father,' she says, 'though in me you behold
86
+ The injury of many a blasting hour,
87
+ Let it not tell your judgment I am old;
88
+ Not age, but sorrow, over me hath power:
89
+ I might as yet have been a spreading flower,
90
+ Fresh to myself, If I had self-applied
91
+ Love to myself and to no love beside.
92
+
93
+ 'But, woe is me! too early I attended
94
+ A youthful suit--it was to gain my grace--
95
+ Of one by nature's outwards so commended,
96
+ That maidens' eyes stuck over all his face:
97
+ Love lack'd a dwelling, and made him her place;
98
+ And when in his fair parts she did abide,
99
+ She was new lodged and newly deified.
100
+
101
+ 'His browny locks did hang in crooked curls;
102
+ And every light occasion of the wind
103
+ Upon his lips their silken parcels hurls.
104
+ What's sweet to do, to do will aptly find:
105
+ Each eye that saw him did enchant the mind,
106
+ For on his visage was in little drawn
107
+ What largeness thinks in Paradise was sawn.
108
+
109
+ 'Small show of man was yet upon his chin;
110
+ His phoenix down began but to appear
111
+ Like unshorn velvet on that termless skin
112
+ Whose bare out-bragg'd the web it seem'd to wear:
113
+ Yet show'd his visage by that cost more dear;
114
+ And nice affections wavering stood in doubt
115
+ If best were as it was, or best without.
116
+
117
+ 'His qualities were beauteous as his form,
118
+ For maiden-tongued he was, and thereof free;
119
+ Yet, if men moved him, was he such a storm
120
+ As oft 'twixt May and April is to see,
121
+ When winds breathe sweet, untidy though they be.
122
+ His rudeness so with his authorized youth
123
+ Did livery falseness in a pride of truth.
124
+
125
+ 'Well could he ride, and often men would say
126
+ 'That horse his mettle from his rider takes:
127
+ Proud of subjection, noble by the sway,
128
+ What rounds, what bounds, what course, what stop
129
+ he makes!'
130
+ And controversy hence a question takes,
131
+ Whether the horse by him became his deed,
132
+ Or he his manage by the well-doing steed.
133
+
134
+ 'But quickly on this side the verdict went:
135
+ His real habitude gave life and grace
136
+ To appertainings and to ornament,
137
+ Accomplish'd in himself, not in his case:
138
+ All aids, themselves made fairer by their place,
139
+ Came for additions; yet their purposed trim
140
+ Pieced not his grace, but were all graced by him.
141
+
142
+ 'So on the tip of his subduing tongue
143
+ All kinds of arguments and question deep,
144
+ All replication prompt, and reason strong,
145
+ For his advantage still did wake and sleep:
146
+ To make the weeper laugh, the laugher weep,
147
+ He had the dialect and different skill,
148
+ Catching all passions in his craft of will:
149
+
150
+ 'That he did in the general bosom reign
151
+ Of young, of old; and sexes both enchanted,
152
+ To dwell with him in thoughts, or to remain
153
+ In personal duty, following where he haunted:
154
+ Consents bewitch'd, ere he desire, have granted;
155
+ And dialogued for him what he would say,
156
+ Ask'd their own wills, and made their wills obey.
157
+
158
+ 'Many there were that did his picture get,
159
+ To serve their eyes, and in it put their mind;
160
+ Like fools that in th' imagination set
161
+ The goodly objects which abroad they find
162
+ Of lands and mansions, theirs in thought assign'd;
163
+ And labouring in moe pleasures to bestow them
164
+ Than the true gouty landlord which doth owe them:
165
+
166
+ 'So many have, that never touch'd his hand,
167
+ Sweetly supposed them mistress of his heart.
168
+ My woeful self, that did in freedom stand,
169
+ And was my own fee-simple, not in part,
170
+ What with his art in youth, and youth in art,
171
+ Threw my affections in his charmed power,
172
+ Reserved the stalk and gave him all my flower.
173
+
174
+ 'Yet did I not, as some my equals did,
175
+ Demand of him, nor being desired yielded;
176
+ Finding myself in honour so forbid,
177
+ With safest distance I mine honour shielded:
178
+ Experience for me many bulwarks builded
179
+ Of proofs new-bleeding, which remain'd the foil
180
+ Of this false jewel, and his amorous spoil.
181
+
182
+ 'But, ah, who ever shunn'd by precedent
183
+ The destined ill she must herself assay?
184
+ Or forced examples, 'gainst her own content,
185
+ To put the by-past perils in her way?
186
+ Counsel may stop awhile what will not stay;
187
+ For when we rage, advice is often seen
188
+ By blunting us to make our wits more keen.
189
+
190
+ 'Nor gives it satisfaction to our blood,
191
+ That we must curb it upon others' proof;
192
+ To be forbod the sweets that seem so good,
193
+ For fear of harms that preach in our behoof.
194
+ O appetite, from judgment stand aloof!
195
+ The one a palate hath that needs will taste,
196
+ Though Reason weep, and cry, 'It is thy last.'
197
+
198
+ 'For further I could say 'This man's untrue,'
199
+ And knew the patterns of his foul beguiling;
200
+ Heard where his plants in others' orchards grew,
201
+ Saw how deceits were gilded in his smiling;
202
+ Knew vows were ever brokers to defiling;
203
+ Thought characters and words merely but art,
204
+ And bastards of his foul adulterate heart.
205
+
206
+ 'And long upon these terms I held my city,
207
+ Till thus he gan besiege me: 'Gentle maid,
208
+ Have of my suffering youth some feeling pity,
209
+ And be not of my holy vows afraid:
210
+ That's to ye sworn to none was ever said;
211
+ For feasts of love I have been call'd unto,
212
+ Till now did ne'er invite, nor never woo.
213
+
214
+ ''All my offences that abroad you see
215
+ Are errors of the blood, none of the mind;
216
+ Love made them not: with acture they may be,
217
+ Where neither party is nor true nor kind:
218
+ They sought their shame that so their shame did find;
219
+ And so much less of shame in me remains,
220
+ By how much of me their reproach contains.
221
+
222
+ ''Among the many that mine eyes have seen,
223
+ Not one whose flame my heart so much as warm'd,
224
+ Or my affection put to the smallest teen,
225
+ Or any of my leisures ever charm'd:
226
+ Harm have I done to them, but ne'er was harm'd;
227
+ Kept hearts in liveries, but mine own was free,
228
+ And reign'd, commanding in his monarchy.
229
+
230
+ ''Look here, what tributes wounded fancies sent me,
231
+ Of paled pearls and rubies red as blood;
232
+ Figuring that they their passions likewise lent me
233
+ Of grief and blushes, aptly understood
234
+ In bloodless white and the encrimson'd mood;
235
+ Effects of terror and dear modesty,
236
+ Encamp'd in hearts, but fighting outwardly.
237
+
238
+ ''And, lo, behold these talents of their hair,
239
+ With twisted metal amorously impleach'd,
240
+ I have received from many a several fair,
241
+ Their kind acceptance weepingly beseech'd,
242
+ With the annexions of fair gems enrich'd,
243
+ And deep-brain'd sonnets that did amplify
244
+ Each stone's dear nature, worth, and quality.
245
+
246
+ ''The diamond,--why, 'twas beautiful and hard,
247
+ Whereto his invised properties did tend;
248
+ The deep-green emerald, in whose fresh regard
249
+ Weak sights their sickly radiance do amend;
250
+ The heaven-hued sapphire and the opal blend
251
+ With objects manifold: each several stone,
252
+ With wit well blazon'd, smiled or made some moan.
253
+
254
+ ''Lo, all these trophies of affections hot,
255
+ Of pensived and subdued desires the tender,
256
+ Nature hath charged me that I hoard them not,
257
+ But yield them up where I myself must render,
258
+ That is, to you, my origin and ender;
259
+ For these, of force, must your oblations be,
260
+ Since I their altar, you enpatron me.
261
+
262
+ ''O, then, advance of yours that phraseless hand,
263
+ Whose white weighs down the airy scale of praise;
264
+ Take all these similes to your own command,
265
+ Hallow'd with sighs that burning lungs did raise;
266
+ What me your minister, for you obeys,
267
+ Works under you; and to your audit comes
268
+ Their distract parcels in combined sums.
269
+
270
+ ''Lo, this device was sent me from a nun,
271
+ Or sister sanctified, of holiest note;
272
+ Which late her noble suit in court did shun,
273
+ Whose rarest havings made the blossoms dote;
274
+ For she was sought by spirits of richest coat,
275
+ But kept cold distance, and did thence remove,
276
+ To spend her living in eternal love.
277
+
278
+ ''But, O my sweet, what labour is't to leave
279
+ The thing we have not, mastering what not strives,
280
+ Playing the place which did no form receive,
281
+ Playing patient sports in unconstrained gyves?
282
+ She that her fame so to herself contrives,
283
+ The scars of battle 'scapeth by the flight,
284
+ And makes her absence valiant, not her might.
285
+
286
+ ''O, pardon me, in that my boast is true:
287
+ The accident which brought me to her eye
288
+ Upon the moment did her force subdue,
289
+ And now she would the caged cloister fly:
290
+ Religious love put out Religion's eye:
291
+ Not to be tempted, would she be immured,
292
+ And now, to tempt, all liberty procured.
293
+
294
+ ''How mighty then you are, O, hear me tell!
295
+ The broken bosoms that to me belong
296
+ Have emptied all their fountains in my well,
297
+ And mine I pour your ocean all among:
298
+ I strong o'er them, and you o'er me being strong,
299
+ Must for your victory us all congest,
300
+ As compound love to physic your cold breast.
301
+
302
+ ''My parts had power to charm a sacred nun,
303
+ Who, disciplined, ay, dieted in grace,
304
+ Believed her eyes when they to assail begun,
305
+ All vows and consecrations giving place:
306
+ O most potential love! vow, bond, nor space,
307
+ In thee hath neither sting, knot, nor confine,
308
+ For thou art all, and all things else are thine.
309
+
310
+ ''When thou impressest, what are precepts worth
311
+ Of stale example? When thou wilt inflame,
312
+ How coldly those impediments stand forth
313
+ Of wealth, of filial fear, law, kindred, fame!
314
+ Love's arms are peace, 'gainst rule, 'gainst sense,
315
+ 'gainst shame,
316
+ And sweetens, in the suffering pangs it bears,
317
+ The aloes of all forces, shocks, and fears.
318
+
319
+ ''Now all these hearts that do on mine depend,
320
+ Feeling it break, with bleeding groans they pine;
321
+ And supplicant their sighs to you extend,
322
+ To leave the battery that you make 'gainst mine,
323
+ Lending soft audience to my sweet design,
324
+ And credent soul to that strong-bonded oath
325
+ That shall prefer and undertake my troth.'
326
+
327
+ 'This said, his watery eyes he did dismount,
328
+ Whose sights till then were levell'd on my face;
329
+ Each cheek a river running from a fount
330
+ With brinish current downward flow'd apace:
331
+ O, how the channel to the stream gave grace!
332
+ Who glazed with crystal gate the glowing roses
333
+ That flame through water which their hue encloses.
334
+
335
+ 'O father, what a hell of witchcraft lies
336
+ In the small orb of one particular tear!
337
+ But with the inundation of the eyes
338
+ What rocky heart to water will not wear?
339
+ What breast so cold that is not warmed here?
340
+ O cleft effect! cold modesty, hot wrath,
341
+ Both fire from hence and chill extincture hath.
342
+
343
+ 'For, lo, his passion, but an art of craft,
344
+ Even there resolved my reason into tears;
345
+ There my white stole of chastity I daff'd,
346
+ Shook off my sober guards and civil fears;
347
+ Appear to him, as he to me appears,
348
+ All melting; though our drops this difference bore,
349
+ His poison'd me, and mine did him restore.
350
+
351
+ 'In him a plenitude of subtle matter,
352
+ Applied to cautels, all strange forms receives,
353
+ Of burning blushes, or of weeping water,
354
+ Or swooning paleness; and he takes and leaves,
355
+ In either's aptness, as it best deceives,
356
+ To blush at speeches rank to weep at woes,
357
+ Or to turn white and swoon at tragic shows.
358
+
359
+ 'That not a heart which in his level came
360
+ Could 'scape the hail of his all-hurting aim,
361
+ Showing fair nature is both kind and tame;
362
+ And, veil'd in them, did win whom he would maim:
363
+ Against the thing he sought he would exclaim;
364
+ When he most burn'd in heart-wish'd luxury,
365
+ He preach'd pure maid, and praised cold chastity.
366
+
367
+ 'Thus merely with the garment of a Grace
368
+ The naked and concealed fiend he cover'd;
369
+ That th' unexperient gave the tempter place,
370
+ Which like a cherubin above them hover'd.
371
+ Who, young and simple, would not be so lover'd?
372
+ Ay me! I fell; and yet do question make
373
+ What I should do again for such a sake.
374
+
375
+ 'O, that infected moisture of his eye,
376
+ O, that false fire which in his cheek so glow'd,
377
+ O, that forced thunder from his heart did fly,
378
+ O, that sad breath his spongy lungs bestow'd,
379
+ O, all that borrow'd motion seeming owed,
380
+ Would yet again betray the fore-betray'd,
381
+ And new pervert a reconciled maid!'