skynet 0.9.2 → 0.9.3
Sign up to get free protection for your applications and to get access to all the features.
- data/History.txt +49 -0
- data/Manifest.txt +84 -6
- data/README.txt +75 -64
- data/app_generators/skynet_install/skynet_install_generator.rb +14 -8
- data/app_generators/skynet_install/templates/migration.rb +1 -24
- data/app_generators/skynet_install/templates/skynet_config.rb +50 -0
- data/app_generators/skynet_install/templates/skynet_initializer.rb +1 -0
- data/app_generators/skynet_install/templates/{skynet_schema.sql → skynet_mysql_schema.sql} +1 -24
- data/bin/skynet +37 -10
- data/bin/skynet_install +5 -5
- data/bin/skynet_tuplespace_server +27 -19
- data/examples/dgrep/README +70 -0
- data/examples/dgrep/config/skynet_config.rb +26 -0
- data/examples/dgrep/data/shakespeare/README +2 -0
- data/examples/dgrep/data/shakespeare/poetry/loverscomplaint +381 -0
- data/examples/dgrep/data/shakespeare/poetry/rapeoflucrece +2199 -0
- data/examples/dgrep/data/shakespeare/poetry/sonnets +2633 -0
- data/examples/dgrep/data/shakespeare/poetry/various +640 -0
- data/examples/dgrep/data/shakespeare/poetry/venusandadonis +1423 -0
- data/examples/dgrep/data/testfile1.txt +1 -0
- data/examples/dgrep/data/testfile2.txt +1 -0
- data/examples/dgrep/data/testfile3.txt +1 -0
- data/examples/dgrep/data/testfile4.txt +1 -0
- data/examples/dgrep/lib/dgrep.rb +59 -0
- data/examples/dgrep/lib/mapreduce_test.rb +32 -0
- data/examples/dgrep/lib/most_common_words.rb +45 -0
- data/examples/dgrep/script/dgrep +75 -0
- data/examples/rails_mysql_example/README +66 -0
- data/examples/rails_mysql_example/Rakefile +10 -0
- data/examples/rails_mysql_example/app/controllers/application.rb +10 -0
- data/examples/rails_mysql_example/app/helpers/application_helper.rb +3 -0
- data/examples/rails_mysql_example/app/models/user.rb +21 -0
- data/examples/rails_mysql_example/app/models/user_favorite.rb +5 -0
- data/examples/rails_mysql_example/app/models/user_mailer.rb +12 -0
- data/examples/rails_mysql_example/app/views/user_mailer/welcome.erb +5 -0
- data/examples/rails_mysql_example/config/boot.rb +109 -0
- data/examples/rails_mysql_example/config/database.yml +42 -0
- data/examples/rails_mysql_example/config/environment.rb +59 -0
- data/examples/rails_mysql_example/config/environments/development.rb +18 -0
- data/examples/rails_mysql_example/config/environments/production.rb +19 -0
- data/examples/rails_mysql_example/config/environments/test.rb +22 -0
- data/examples/rails_mysql_example/config/initializers/inflections.rb +10 -0
- data/examples/rails_mysql_example/config/initializers/mime_types.rb +5 -0
- data/examples/rails_mysql_example/config/initializers/skynet.rb +1 -0
- data/examples/rails_mysql_example/config/routes.rb +35 -0
- data/examples/rails_mysql_example/config/skynet_config.rb +36 -0
- data/examples/rails_mysql_example/db/migrate/001_create_skynet_tables.rb +43 -0
- data/examples/rails_mysql_example/db/migrate/002_create_users.rb +16 -0
- data/examples/rails_mysql_example/db/migrate/003_create_user_favorites.rb +14 -0
- data/examples/rails_mysql_example/db/schema.rb +85 -0
- data/examples/rails_mysql_example/db/skynet_mysql_schema.sql +33 -0
- data/examples/rails_mysql_example/doc/README_FOR_APP +2 -0
- data/examples/rails_mysql_example/lib/tasks/rails_mysql_example.rake +20 -0
- data/examples/rails_mysql_example/public/.htaccess +40 -0
- data/examples/rails_mysql_example/public/404.html +30 -0
- data/examples/rails_mysql_example/public/422.html +30 -0
- data/examples/rails_mysql_example/public/500.html +30 -0
- data/examples/rails_mysql_example/public/dispatch.cgi +10 -0
- data/examples/rails_mysql_example/public/dispatch.fcgi +24 -0
- data/examples/rails_mysql_example/public/dispatch.rb +10 -0
- data/{log/debug.log → examples/rails_mysql_example/public/favicon.ico} +0 -0
- data/examples/rails_mysql_example/public/images/rails.png +0 -0
- data/examples/rails_mysql_example/public/index.html +277 -0
- data/examples/rails_mysql_example/public/javascripts/application.js +2 -0
- data/examples/rails_mysql_example/public/javascripts/controls.js +963 -0
- data/examples/rails_mysql_example/public/javascripts/dragdrop.js +972 -0
- data/examples/rails_mysql_example/public/javascripts/effects.js +1120 -0
- data/examples/rails_mysql_example/public/javascripts/prototype.js +4225 -0
- data/examples/rails_mysql_example/public/robots.txt +5 -0
- data/examples/rails_mysql_example/script/about +3 -0
- data/examples/rails_mysql_example/script/console +3 -0
- data/examples/rails_mysql_example/script/destroy +3 -0
- data/examples/rails_mysql_example/script/generate +3 -0
- data/examples/rails_mysql_example/script/performance/benchmarker +3 -0
- data/examples/rails_mysql_example/script/performance/profiler +3 -0
- data/examples/rails_mysql_example/script/performance/request +3 -0
- data/examples/rails_mysql_example/script/plugin +3 -0
- data/examples/rails_mysql_example/script/process/inspector +3 -0
- data/examples/rails_mysql_example/script/process/reaper +3 -0
- data/examples/rails_mysql_example/script/process/spawner +3 -0
- data/examples/rails_mysql_example/script/runner +3 -0
- data/examples/rails_mysql_example/script/server +3 -0
- data/examples/rails_mysql_example/test/fixtures/user_favorites.yml +9 -0
- data/examples/rails_mysql_example/test/fixtures/users.yml +11 -0
- data/examples/rails_mysql_example/test/test_helper.rb +38 -0
- data/examples/rails_mysql_example/test/unit/user_favorite_test.rb +8 -0
- data/examples/rails_mysql_example/test/unit/user_test.rb +8 -0
- data/extras/README +7 -0
- data/extras/init.d/skynet +87 -0
- data/extras/nagios/check_skynet.sh +121 -0
- data/extras/rails/controllers/skynet_controller.rb +43 -0
- data/extras/rails/views/skynet/index.rhtml +137 -0
- data/lib/skynet.rb +59 -1
- data/lib/skynet/mapreduce_helper.rb +2 -2
- data/lib/skynet/mapreduce_test.rb +32 -1
- data/lib/skynet/message_queue_adapters/mysql.rb +422 -539
- data/lib/skynet/message_queue_adapters/tuple_space.rb +45 -71
- data/lib/skynet/skynet_active_record_extensions.rb +22 -11
- data/lib/skynet/skynet_config.rb +54 -20
- data/lib/skynet/skynet_console.rb +4 -1
- data/lib/skynet/skynet_console_helper.rb +5 -1
- data/lib/skynet/skynet_debugger.rb +58 -4
- data/lib/skynet/skynet_job.rb +61 -24
- data/lib/skynet/skynet_launcher.rb +29 -3
- data/lib/skynet/skynet_logger.rb +11 -1
- data/lib/skynet/skynet_manager.rb +403 -240
- data/lib/skynet/skynet_message.rb +1 -3
- data/lib/skynet/skynet_message_queue.rb +42 -19
- data/lib/skynet/skynet_partitioners.rb +19 -15
- data/lib/skynet/skynet_ruby_extensions.rb +18 -0
- data/lib/skynet/skynet_tuplespace_server.rb +17 -14
- data/lib/skynet/skynet_worker.rb +132 -98
- data/lib/skynet/version.rb +1 -1
- data/script/destroy +0 -0
- data/script/generate +0 -0
- data/script/txt2html +0 -0
- data/test/test_helper.rb +2 -0
- data/test/test_skynet.rb +13 -5
- data/test/test_skynet_manager.rb +24 -9
- data/test/test_skynet_task.rb +1 -1
- data/website/index.html +77 -29
- data/website/index.txt +53 -24
- data/website/stylesheets/screen.css +12 -12
- metadata +156 -66
- data/app_generators/skynet_install/templates/skynet +0 -46
- data/log/skynet.log +0 -29
- data/log/skynet_tuplespace_server.log +0 -7
- data/log/skynet_worker.pid +0 -1
@@ -0,0 +1,50 @@
|
|
1
|
+
# Skynet Configuration File. Should be in APP_ROOT/config/skynet_config.rb
|
2
|
+
# Start skynet from within your app root with
|
3
|
+
# > skynet start
|
4
|
+
|
5
|
+
require 'rubygems'
|
6
|
+
require 'skynet'
|
7
|
+
|
8
|
+
<% if in_rails -%>
|
9
|
+
# Load your rails app
|
10
|
+
if not defined?(RAILS_GEM_VERSION)
|
11
|
+
require File.expand_path(File.dirname(__FILE__)) + '/../config/environment'
|
12
|
+
end
|
13
|
+
<% end -%>
|
14
|
+
|
15
|
+
Skynet::CONFIG[:SKYNET_LOG_LEVEL] = Logger::ERROR
|
16
|
+
<% if in_rails -%>
|
17
|
+
Skynet::CONFIG[:APP_ROOT] = RAILS_ROOT
|
18
|
+
Skynet::CONFIG[:SKYNET_LOG_DIR] = File.expand_path(Skynet::CONFIG[:APP_ROOT]+ "/log")
|
19
|
+
Skynet::CONFIG[:SKYNET_PID_DIR] = File.expand_path(Skynet::CONFIG[:APP_ROOT] + "/log")
|
20
|
+
Skynet::CONFIG[:SKYNET_LOG_FILE] = "skynet_#{RAILS_ENV}.log"
|
21
|
+
Skynet::CONFIG[:SKYNET_PID_FILE] = "skynet_#{RAILS_ENV}.pid"
|
22
|
+
<% else -%>
|
23
|
+
Skynet::CONFIG[:SKYNET_LOG_DIR] = File.expand_path(File.dirname(__FILE__) + "/../log")
|
24
|
+
Skynet::CONFIG[:SKYNET_PID_DIR] = File.expand_path(File.dirname(__FILE__) + "/../log")
|
25
|
+
<% end -%>
|
26
|
+
|
27
|
+
<% if mysql -%>
|
28
|
+
<% if not in_rails -%>
|
29
|
+
# Use the mysql message queue adapter
|
30
|
+
Skynet::CONFIG[:MESSAGE_QUEUE_ADAPTER] = "Skynet::MessageQueueAdapter::Mysql"
|
31
|
+
Skynet::CONFIG[:MYSQL_HOST] = "localhost"
|
32
|
+
Skynet::CONFIG[:MYSQL_USERNAME] = "root"
|
33
|
+
Skynet::CONFIG[:MYSQL_PASSWORD] = ""
|
34
|
+
Skynet::CONFIG[:MYSQL_DATABASE] = "skynet"
|
35
|
+
<% else %>
|
36
|
+
# Use the mysql message queue adapter
|
37
|
+
Skynet::CONFIG[:MESSAGE_QUEUE_ADAPTER] = "Skynet::MessageQueueAdapter::Mysql"
|
38
|
+
<% end -%>
|
39
|
+
<% end -%>
|
40
|
+
|
41
|
+
# ==================================================================
|
42
|
+
# = Require any other libraries you want skynet to know about here =
|
43
|
+
# ==================================================================
|
44
|
+
|
45
|
+
|
46
|
+
# ===========================================
|
47
|
+
# = Set your own configuration options here =
|
48
|
+
# ===========================================
|
49
|
+
# You can also configure skynet with
|
50
|
+
# Skynet.configure(:SOME_CONFIG_OPTION => true, :SOME_OTHER_CONFIG => 3)
|
@@ -0,0 +1 @@
|
|
1
|
+
require RAILS_ROOT+'/config/skynet_config.rb'
|
@@ -7,7 +7,7 @@ CREATE TABLE skynet_message_queues (
|
|
7
7
|
tasktype varchar(255) default NULL,
|
8
8
|
task_id bigint(20) unsigned default NULL,
|
9
9
|
job_id bigint(20) unsigned default NULL,
|
10
|
-
raw_payload
|
10
|
+
raw_payload longtext,
|
11
11
|
payload_type varchar(255) default NULL,
|
12
12
|
name varchar(255) default NULL,
|
13
13
|
expiry int(11) default NULL,
|
@@ -22,29 +22,6 @@ CREATE TABLE skynet_message_queues (
|
|
22
22
|
KEY index_skynet_message_queues_on_task_id (task_id),
|
23
23
|
KEY index_skynet_mqueue_for_take (queue_id,tasktype,payload_type,expire_time)
|
24
24
|
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
|
25
|
-
CREATE TABLE skynet_worker_queues (
|
26
|
-
id int(11) NOT NULL auto_increment,
|
27
|
-
queue_id int(11) default '0',
|
28
|
-
created_on datetime default NULL,
|
29
|
-
updated_on datetime default NULL,
|
30
|
-
tasktype varchar(255) default NULL,
|
31
|
-
tasksubtype varchar(255) default NULL,
|
32
|
-
worker_id bigint(20) unsigned default NULL,
|
33
|
-
hostname varchar(255) default NULL,
|
34
|
-
process_id int(11) default NULL,
|
35
|
-
job_id bigint(20) unsigned default NULL,
|
36
|
-
task_id bigint(20) unsigned default NULL,
|
37
|
-
iteration int(11) default NULL,
|
38
|
-
name varchar(255) default NULL,
|
39
|
-
map_or_reduce varchar(255) default NULL,
|
40
|
-
started_at decimal(16,4) default NULL,
|
41
|
-
version int(11) default NULL,
|
42
|
-
processed int(11) default NULL,
|
43
|
-
timeout decimal(16,4) default NULL,
|
44
|
-
PRIMARY KEY (id),
|
45
|
-
UNIQUE KEY index_skynet_worker_queues_on_worker_id (worker_id),
|
46
|
-
KEY index_skynet_worker_queues_on_hostname_and_process_id (hostname,process_id)
|
47
|
-
) ENGINE=InnoDB DEFAULT CHARSET=latin1;
|
48
25
|
CREATE TABLE skynet_queue_temperature (
|
49
26
|
id int(11) NOT NULL auto_increment,
|
50
27
|
queue_id int(11) default '0',
|
data/bin/skynet
CHANGED
@@ -27,18 +27,45 @@
|
|
27
27
|
require 'rubygems'
|
28
28
|
require File.expand_path(File.dirname(__FILE__)) + '/../lib/skynet.rb'
|
29
29
|
|
30
|
-
Skynet::CONFIG[:WORKER_CHECK_DELAY] ||= 4
|
31
|
-
Skynet::CONFIG[:LAUNCHER_PATH] = File.expand_path(__FILE__)
|
32
30
|
|
33
|
-
|
34
|
-
|
35
|
-
|
36
|
-
|
37
|
-
|
38
|
-
|
31
|
+
if config_arg = ARGV.detect {|i| i =~ /--help/ }
|
32
|
+
Skynet.start
|
33
|
+
exit
|
34
|
+
end
|
35
|
+
|
36
|
+
conf_file = nil
|
37
|
+
if config_arg = ARGV.detect {|i| i =~ /--config/i }
|
38
|
+
conf_file = config_arg.split('=').last
|
39
|
+
ARGV.delete_if {|i| i =~ /--config/ }
|
40
|
+
else
|
41
|
+
['config/skynet_config.rb', 'skynet_config.rb', "../config/skynet_config.rb" , "#{ENV['HOME']}/.skynet/config.rb"].each do |path|
|
42
|
+
if File.exists?(path)
|
43
|
+
conf_file = path
|
44
|
+
break
|
39
45
|
end
|
40
|
-
sleep 3
|
41
46
|
end
|
42
47
|
end
|
43
48
|
|
44
|
-
|
49
|
+
if conf_file
|
50
|
+
Skynet::CONFIG[:CONFIG_FILE] = File.expand_path(conf_file)
|
51
|
+
Skynet::CONFIG[:APP_ROOT] = File.dirname(Skynet::CONFIG[:CONFIG_FILE])
|
52
|
+
Skynet::CONFIG[:SKYNET_LOG_DIR] = File.expand_path(Skynet::CONFIG[:APP_ROOT]+ "/../log")
|
53
|
+
Skynet::CONFIG[:SKYNET_PID_DIR] = File.expand_path(Skynet::CONFIG[:APP_ROOT] + "/../log")
|
54
|
+
begin
|
55
|
+
require conf_file
|
56
|
+
rescue MissingSourceFile => e
|
57
|
+
puts "Specified config file #{conf_file} is missing."
|
58
|
+
exit
|
59
|
+
end
|
60
|
+
puts "using config file: #{File.expand_path(conf_file)}" unless conf_file == 'config/skynet_config.rb'
|
61
|
+
else
|
62
|
+
puts "No configuration file found.\nTry running skynet_install to create a config/skynet_config.rb or starting skynet with --config=CONFIG_FILE_LOCATION. Exiting."
|
63
|
+
exit
|
64
|
+
end
|
65
|
+
|
66
|
+
Skynet::CONFIG[:WORKER_CHECK_DELAY] ||= 4
|
67
|
+
Skynet::CONFIG[:LAUNCHER_PATH] = File.expand_path(__FILE__)
|
68
|
+
|
69
|
+
Skynet::CONFIG[:SYSTEM_RUNNER] = true
|
70
|
+
|
71
|
+
Skynet.start
|
data/bin/skynet_install
CHANGED
@@ -1,15 +1,15 @@
|
|
1
1
|
#!/usr/bin/env ruby
|
2
2
|
|
3
|
-
# skynet_install is used to install skynet
|
4
|
-
#
|
5
|
-
#
|
3
|
+
# skynet_install is used to install skynet libraries into your application.
|
4
|
+
# Specifically, it installs a config/skynet_config.rb file
|
5
|
+
# Once you have a config/skynet_config.rb you can run skynet start from within your root_app_dir/
|
6
6
|
# This is also how you can run skynet from within rails. (using --rails)
|
7
7
|
#
|
8
|
-
# USAGE: skynet_install [--rails] directory (can be '.' for current)"
|
8
|
+
# USAGE: skynet_install [--rails] directory/ (can be '.' for current)"
|
9
9
|
#
|
10
10
|
# Options:
|
11
11
|
# -v, --version Show the skynet_install version number and quit.
|
12
|
-
# --
|
12
|
+
# --mysql Include mysql migration if you want to use mysql as your message queue
|
13
13
|
# -r, --rails Install into rails app
|
14
14
|
# Default: false
|
15
15
|
# General Options:
|
@@ -4,24 +4,25 @@
|
|
4
4
|
# Skynet::CONFIG[:MESSAGE_QUEUE_ADAPTER] = "Skynet::MessageQueueAdapter::TupleSpace"
|
5
5
|
#
|
6
6
|
# Usage: skynet_tuplespace_server (start|stop|run) [options]
|
7
|
-
# -t, --ontop
|
8
|
-
# -p, --port
|
9
|
-
# -o, --
|
10
|
-
# -l, --loglevel
|
11
|
-
# -d, --piddir
|
12
|
-
# -u, --drburi
|
7
|
+
# -t, --ontop=TRUE Dont Daemonize
|
8
|
+
# -p, --port=PORT Port to listen on. default 7647
|
9
|
+
# -o, --logfile=LOGFILE Logfile to log to
|
10
|
+
# -l, --loglevel=LOGLEVEL Log level defaults to DEBUG
|
11
|
+
# -d, --piddir=PIDDIR Directory to put pidfile
|
12
|
+
# -u, --drburi=druby://DRB_URI What DRbURI to use
|
13
13
|
|
14
14
|
require 'rubygems'
|
15
15
|
require 'daemons'
|
16
16
|
require 'pp'
|
17
17
|
|
18
|
-
require File.expand_path(File.dirname(__FILE__)) + '/../lib/skynet.rb'
|
18
|
+
require File.expand_path(File.dirname(__FILE__)) + '/../lib/skynet.rb'
|
19
19
|
|
20
20
|
options = {
|
21
|
-
:port
|
22
|
-
:logfile
|
23
|
-
:loglevel
|
24
|
-
:piddir
|
21
|
+
:port => 7647,
|
22
|
+
:logfile => Skynet::Config.pidfile_location,
|
23
|
+
:loglevel => "ERROR",
|
24
|
+
:piddir => Skynet::CONFIG[:SKYNET_PID_DIR],
|
25
|
+
:use_ringserver => Skynet::CONFIG[:TS_USE_RINGSERVER],
|
25
26
|
}
|
26
27
|
|
27
28
|
OptionParser.new do |opt|
|
@@ -29,26 +30,33 @@ OptionParser.new do |opt|
|
|
29
30
|
opt.on('-t', '--ontop TRUE', 'Dont Daemonize') do |v|
|
30
31
|
options[:ontop] = true if v.upcase == "TRUE" or v == "1"
|
31
32
|
end
|
32
|
-
opt.on('-p', '--port
|
33
|
+
opt.on('-p', '--port=PORT', 'Port to listen on. default 7647') do |v|
|
33
34
|
options[:port] = v.to_i
|
34
35
|
end
|
35
|
-
opt.on('-o', '--
|
36
|
-
options[:logfile] = v
|
36
|
+
opt.on('-o', '--logfile=LOGFILE', 'Logfile to log to') do |v|
|
37
|
+
options[:logfile] = File.expand_path(v)
|
37
38
|
end
|
38
|
-
opt.on('-l', '--loglevel
|
39
|
+
opt.on('-l', '--loglevel=LOGLEVEL', 'Log level defaults to DEBUG') do |v|
|
39
40
|
options[:loglevel] = v
|
40
41
|
end
|
41
|
-
opt.on('-d', '--piddir
|
42
|
+
opt.on('-d', '--piddir=PIDDIR', 'Directory to put pidfile') do |v|
|
42
43
|
options[:piddir] = File.expand_path(v)
|
43
44
|
end
|
44
|
-
opt.on('
|
45
|
+
opt.on('--use_ringserver=TRUE', 'Use ringserver?') do |v|
|
46
|
+
if v.downcase == "true" or v == "1"
|
47
|
+
options[:use_ringserver] = true
|
48
|
+
else
|
49
|
+
options[:use_ringserver] = false
|
50
|
+
end
|
51
|
+
end
|
52
|
+
opt.on('-u', '--drburi=druby://DRB_URI', 'What DRbURI to use') do |v|
|
45
53
|
if v =~ %r{druby://}
|
46
54
|
options[:drburi] = v
|
47
55
|
else
|
48
56
|
options[:drburi] = "druby://#{v}"
|
49
57
|
end
|
50
58
|
options[:drburi] =~ /druby:\/\/.+?:(\d*)/
|
51
|
-
options[:port]
|
59
|
+
options[:port] ||= $1.to_i
|
52
60
|
end
|
53
61
|
|
54
62
|
opt.parse!(ARGV)
|
@@ -62,5 +70,5 @@ Daemons.run_proc("skynet_tuplespace_server#{options[:port]}",
|
|
62
70
|
# :monitor => true,
|
63
71
|
:ontop => options[:ontop] || false
|
64
72
|
}) do
|
65
|
-
server = Skynet::
|
73
|
+
server = Skynet::TuplespaceServer.start(options)
|
66
74
|
end
|
@@ -0,0 +1,70 @@
|
|
1
|
+
|
2
|
+
EXAMPLE 1: Distributed Grep
|
3
|
+
------------------------------
|
4
|
+
You have a list of files and want to see how many times each word shows up in those files.
|
5
|
+
All of the map_reduce code is in lib/mapreduce_test.rb
|
6
|
+
|
7
|
+
To
|
8
|
+
1. Run Skynet
|
9
|
+
> ./script/skynet start
|
10
|
+
|
11
|
+
2. Watch the logs in another terminal
|
12
|
+
> tail -f log/skynet.log
|
13
|
+
|
14
|
+
3. Run dgrep providing it with some words and a directory
|
15
|
+
> ./script/dgrep -r love hate ./data
|
16
|
+
|
17
|
+
I've put all of shakespears poetry in the data directory. You can download all of Shakespeare's work
|
18
|
+
to add to the data at http://www.it.usyd.edu.au/~matty/Shakespeare/shakespeare.tar.gz
|
19
|
+
|
20
|
+
Thats it!!
|
21
|
+
|
22
|
+
EXAMPLE 2: Distributed Grep with distributed reducing
|
23
|
+
------------------------------
|
24
|
+
Same example as above, but this time we want to aggregate our results across multiple partitioners. This would
|
25
|
+
be useful if you had a large reduce data set, or complex calculations in order to reduce. All the code is already
|
26
|
+
in dgrep.pl. Specifically, notice I've included a self.reduce_partition method (which just forwards to a pre-packaged
|
27
|
+
partitioner). This partitioner not only breaks up the data returned from the mappers, but also makes sure each reducer
|
28
|
+
gets all the same keys.
|
29
|
+
|
30
|
+
1. Run dgrep passing --reducers=2 This tells Skynet to try and use up to 2 reducers (instead of the default of 1)
|
31
|
+
> ./script/dgrep -r --reducers=2 data file ./data
|
32
|
+
|
33
|
+
This sample application was created by creating in the following way
|
34
|
+
|
35
|
+
1. Create new app
|
36
|
+
> mkdir simple_example
|
37
|
+
> mkdir data lib
|
38
|
+
|
39
|
+
2. Install Skynet
|
40
|
+
> skynet_install
|
41
|
+
|
42
|
+
3. Add Data Files to /data
|
43
|
+
These are any text files named testfileN.txt and so on
|
44
|
+
|
45
|
+
4. Create dgrep.rb and drep script (found in lib/ and script/)
|
46
|
+
|
47
|
+
------------------------------------
|
48
|
+
|
49
|
+
EXAMPLE 1: Most Common Words
|
50
|
+
------------------------------
|
51
|
+
You have a list of files and want the 10 most common 5 or more letter words in all of those fules.
|
52
|
+
|
53
|
+
1. Run Skynet
|
54
|
+
> ./script/skynet start
|
55
|
+
|
56
|
+
2. Watch the logs in another terminal
|
57
|
+
> tail -f log/skynet.log
|
58
|
+
|
59
|
+
3. Run dgrep providing it with the '-mcw' (most common words) command, some words and a directory
|
60
|
+
> ./script/dgrep -r -mcw ./data
|
61
|
+
|
62
|
+
To see what's going on check out lib/most_common_words.rb
|
63
|
+
------------------------------------
|
64
|
+
|
65
|
+
If you'd like get a peak at what Skynet is doing, edit ./config/skynet_config.rb and set
|
66
|
+
|
67
|
+
Skynet::CONFIG[:SKYNET_LOG_LEVEL] = Logger::INFO # or Logger::INFO
|
68
|
+
|
69
|
+
and tail ./log/skynet.log
|
70
|
+
|
@@ -0,0 +1,26 @@
|
|
1
|
+
# Skynet Configuration File. Should be in APP_ROOT/config/skynet_config.rb
|
2
|
+
# Start skynet from within your app root with
|
3
|
+
# $ skynet start
|
4
|
+
|
5
|
+
require 'rubygems'
|
6
|
+
require 'skynet'
|
7
|
+
|
8
|
+
Skynet::CONFIG[:SKYNET_LOG_LEVEL] = Logger::ERROR
|
9
|
+
Skynet::CONFIG[:SKYNET_LOG_DIR] = File.expand_path(File.dirname(__FILE__) + "/../log")
|
10
|
+
Skynet::CONFIG[:SKYNET_PID_DIR] = File.expand_path(File.dirname(__FILE__) + "/../log")
|
11
|
+
|
12
|
+
# ==================================================================
|
13
|
+
# = Require any other libraries you want skynet to know about here =
|
14
|
+
# ==================================================================
|
15
|
+
|
16
|
+
# require all ruby files in lib/
|
17
|
+
Dir["#{File.dirname(__FILE__)}" + "/../lib/**/*.rb"].each do |file|
|
18
|
+
require file
|
19
|
+
end
|
20
|
+
|
21
|
+
# ===========================================
|
22
|
+
# = Set your own configuration options here =
|
23
|
+
# ===========================================
|
24
|
+
# You can also configure skynet with
|
25
|
+
# Skynet.configure(:SOME_CONFIG_OPTION => true, :SOME_OTHER_CONFIG => 3)
|
26
|
+
|
@@ -0,0 +1,381 @@
|
|
1
|
+
A LOVER'S COMPLAINT
|
2
|
+
|
3
|
+
|
4
|
+
|
5
|
+
FROM off a hill whose concave womb reworded
|
6
|
+
A plaintful story from a sistering vale,
|
7
|
+
My spirits to attend this double voice accorded,
|
8
|
+
And down I laid to list the sad-tuned tale;
|
9
|
+
Ere long espied a fickle maid full pale,
|
10
|
+
Tearing of papers, breaking rings a-twain,
|
11
|
+
Storming her world with sorrow's wind and rain.
|
12
|
+
|
13
|
+
Upon her head a platted hive of straw,
|
14
|
+
Which fortified her visage from the sun,
|
15
|
+
Whereon the thought might think sometime it saw
|
16
|
+
The carcass of beauty spent and done:
|
17
|
+
Time had not scythed all that youth begun,
|
18
|
+
Nor youth all quit; but, spite of heaven's fell rage,
|
19
|
+
Some beauty peep'd through lattice of sear'd age.
|
20
|
+
|
21
|
+
Oft did she heave her napkin to her eyne,
|
22
|
+
Which on it had conceited characters,
|
23
|
+
Laundering the silken figures in the brine
|
24
|
+
That season'd woe had pelleted in tears,
|
25
|
+
And often reading what contents it bears;
|
26
|
+
As often shrieking undistinguish'd woe,
|
27
|
+
In clamours of all size, both high and low.
|
28
|
+
|
29
|
+
Sometimes her levell'd eyes their carriage ride,
|
30
|
+
As they did battery to the spheres intend;
|
31
|
+
Sometime diverted their poor balls are tied
|
32
|
+
To the orbed earth; sometimes they do extend
|
33
|
+
Their view right on; anon their gazes lend
|
34
|
+
To every place at once, and, nowhere fix'd,
|
35
|
+
The mind and sight distractedly commix'd.
|
36
|
+
|
37
|
+
Her hair, nor loose nor tied in formal plat,
|
38
|
+
Proclaim'd in her a careless hand of pride
|
39
|
+
For some, untuck'd, descended her sheaved hat,
|
40
|
+
Hanging her pale and pined cheek beside;
|
41
|
+
Some in her threaden fillet still did bide,
|
42
|
+
And true to bondage would not break from thence,
|
43
|
+
Though slackly braided in loose negligence.
|
44
|
+
|
45
|
+
A thousand favours from a maund she drew
|
46
|
+
Of amber, crystal, and of beaded jet,
|
47
|
+
Which one by one she in a river threw,
|
48
|
+
Upon whose weeping margent she was set;
|
49
|
+
Like usury, applying wet to wet,
|
50
|
+
Or monarch's hands that let not bounty fall
|
51
|
+
Where want cries some, but where excess begs all.
|
52
|
+
|
53
|
+
Of folded schedules had she many a one,
|
54
|
+
Which she perused, sigh'd, tore, and gave the flood;
|
55
|
+
Crack'd many a ring of posied gold and bone
|
56
|
+
Bidding them find their sepulchres in mud;
|
57
|
+
Found yet moe letters sadly penn'd in blood,
|
58
|
+
With sleided silk feat and affectedly
|
59
|
+
Enswathed, and seal'd to curious secrecy.
|
60
|
+
|
61
|
+
These often bathed she in her fluxive eyes,
|
62
|
+
And often kiss'd, and often 'gan to tear:
|
63
|
+
Cried 'O false blood, thou register of lies,
|
64
|
+
What unapproved witness dost thou bear!
|
65
|
+
Ink would have seem'd more black and damned here!'
|
66
|
+
This said, in top of rage the lines she rents,
|
67
|
+
Big discontent so breaking their contents.
|
68
|
+
|
69
|
+
A reverend man that grazed his cattle nigh--
|
70
|
+
Sometime a blusterer, that the ruffle knew
|
71
|
+
Of court, of city, and had let go by
|
72
|
+
The swiftest hours, observed as they flew--
|
73
|
+
Towards this afflicted fancy fastly drew,
|
74
|
+
And, privileged by age, desires to know
|
75
|
+
In brief the grounds and motives of her woe.
|
76
|
+
|
77
|
+
So slides he down upon his grained bat,
|
78
|
+
And comely-distant sits he by her side;
|
79
|
+
When he again desires her, being sat,
|
80
|
+
Her grievance with his hearing to divide:
|
81
|
+
If that from him there may be aught applied
|
82
|
+
Which may her suffering ecstasy assuage,
|
83
|
+
'Tis promised in the charity of age.
|
84
|
+
|
85
|
+
'Father,' she says, 'though in me you behold
|
86
|
+
The injury of many a blasting hour,
|
87
|
+
Let it not tell your judgment I am old;
|
88
|
+
Not age, but sorrow, over me hath power:
|
89
|
+
I might as yet have been a spreading flower,
|
90
|
+
Fresh to myself, If I had self-applied
|
91
|
+
Love to myself and to no love beside.
|
92
|
+
|
93
|
+
'But, woe is me! too early I attended
|
94
|
+
A youthful suit--it was to gain my grace--
|
95
|
+
Of one by nature's outwards so commended,
|
96
|
+
That maidens' eyes stuck over all his face:
|
97
|
+
Love lack'd a dwelling, and made him her place;
|
98
|
+
And when in his fair parts she did abide,
|
99
|
+
She was new lodged and newly deified.
|
100
|
+
|
101
|
+
'His browny locks did hang in crooked curls;
|
102
|
+
And every light occasion of the wind
|
103
|
+
Upon his lips their silken parcels hurls.
|
104
|
+
What's sweet to do, to do will aptly find:
|
105
|
+
Each eye that saw him did enchant the mind,
|
106
|
+
For on his visage was in little drawn
|
107
|
+
What largeness thinks in Paradise was sawn.
|
108
|
+
|
109
|
+
'Small show of man was yet upon his chin;
|
110
|
+
His phoenix down began but to appear
|
111
|
+
Like unshorn velvet on that termless skin
|
112
|
+
Whose bare out-bragg'd the web it seem'd to wear:
|
113
|
+
Yet show'd his visage by that cost more dear;
|
114
|
+
And nice affections wavering stood in doubt
|
115
|
+
If best were as it was, or best without.
|
116
|
+
|
117
|
+
'His qualities were beauteous as his form,
|
118
|
+
For maiden-tongued he was, and thereof free;
|
119
|
+
Yet, if men moved him, was he such a storm
|
120
|
+
As oft 'twixt May and April is to see,
|
121
|
+
When winds breathe sweet, untidy though they be.
|
122
|
+
His rudeness so with his authorized youth
|
123
|
+
Did livery falseness in a pride of truth.
|
124
|
+
|
125
|
+
'Well could he ride, and often men would say
|
126
|
+
'That horse his mettle from his rider takes:
|
127
|
+
Proud of subjection, noble by the sway,
|
128
|
+
What rounds, what bounds, what course, what stop
|
129
|
+
he makes!'
|
130
|
+
And controversy hence a question takes,
|
131
|
+
Whether the horse by him became his deed,
|
132
|
+
Or he his manage by the well-doing steed.
|
133
|
+
|
134
|
+
'But quickly on this side the verdict went:
|
135
|
+
His real habitude gave life and grace
|
136
|
+
To appertainings and to ornament,
|
137
|
+
Accomplish'd in himself, not in his case:
|
138
|
+
All aids, themselves made fairer by their place,
|
139
|
+
Came for additions; yet their purposed trim
|
140
|
+
Pieced not his grace, but were all graced by him.
|
141
|
+
|
142
|
+
'So on the tip of his subduing tongue
|
143
|
+
All kinds of arguments and question deep,
|
144
|
+
All replication prompt, and reason strong,
|
145
|
+
For his advantage still did wake and sleep:
|
146
|
+
To make the weeper laugh, the laugher weep,
|
147
|
+
He had the dialect and different skill,
|
148
|
+
Catching all passions in his craft of will:
|
149
|
+
|
150
|
+
'That he did in the general bosom reign
|
151
|
+
Of young, of old; and sexes both enchanted,
|
152
|
+
To dwell with him in thoughts, or to remain
|
153
|
+
In personal duty, following where he haunted:
|
154
|
+
Consents bewitch'd, ere he desire, have granted;
|
155
|
+
And dialogued for him what he would say,
|
156
|
+
Ask'd their own wills, and made their wills obey.
|
157
|
+
|
158
|
+
'Many there were that did his picture get,
|
159
|
+
To serve their eyes, and in it put their mind;
|
160
|
+
Like fools that in th' imagination set
|
161
|
+
The goodly objects which abroad they find
|
162
|
+
Of lands and mansions, theirs in thought assign'd;
|
163
|
+
And labouring in moe pleasures to bestow them
|
164
|
+
Than the true gouty landlord which doth owe them:
|
165
|
+
|
166
|
+
'So many have, that never touch'd his hand,
|
167
|
+
Sweetly supposed them mistress of his heart.
|
168
|
+
My woeful self, that did in freedom stand,
|
169
|
+
And was my own fee-simple, not in part,
|
170
|
+
What with his art in youth, and youth in art,
|
171
|
+
Threw my affections in his charmed power,
|
172
|
+
Reserved the stalk and gave him all my flower.
|
173
|
+
|
174
|
+
'Yet did I not, as some my equals did,
|
175
|
+
Demand of him, nor being desired yielded;
|
176
|
+
Finding myself in honour so forbid,
|
177
|
+
With safest distance I mine honour shielded:
|
178
|
+
Experience for me many bulwarks builded
|
179
|
+
Of proofs new-bleeding, which remain'd the foil
|
180
|
+
Of this false jewel, and his amorous spoil.
|
181
|
+
|
182
|
+
'But, ah, who ever shunn'd by precedent
|
183
|
+
The destined ill she must herself assay?
|
184
|
+
Or forced examples, 'gainst her own content,
|
185
|
+
To put the by-past perils in her way?
|
186
|
+
Counsel may stop awhile what will not stay;
|
187
|
+
For when we rage, advice is often seen
|
188
|
+
By blunting us to make our wits more keen.
|
189
|
+
|
190
|
+
'Nor gives it satisfaction to our blood,
|
191
|
+
That we must curb it upon others' proof;
|
192
|
+
To be forbod the sweets that seem so good,
|
193
|
+
For fear of harms that preach in our behoof.
|
194
|
+
O appetite, from judgment stand aloof!
|
195
|
+
The one a palate hath that needs will taste,
|
196
|
+
Though Reason weep, and cry, 'It is thy last.'
|
197
|
+
|
198
|
+
'For further I could say 'This man's untrue,'
|
199
|
+
And knew the patterns of his foul beguiling;
|
200
|
+
Heard where his plants in others' orchards grew,
|
201
|
+
Saw how deceits were gilded in his smiling;
|
202
|
+
Knew vows were ever brokers to defiling;
|
203
|
+
Thought characters and words merely but art,
|
204
|
+
And bastards of his foul adulterate heart.
|
205
|
+
|
206
|
+
'And long upon these terms I held my city,
|
207
|
+
Till thus he gan besiege me: 'Gentle maid,
|
208
|
+
Have of my suffering youth some feeling pity,
|
209
|
+
And be not of my holy vows afraid:
|
210
|
+
That's to ye sworn to none was ever said;
|
211
|
+
For feasts of love I have been call'd unto,
|
212
|
+
Till now did ne'er invite, nor never woo.
|
213
|
+
|
214
|
+
''All my offences that abroad you see
|
215
|
+
Are errors of the blood, none of the mind;
|
216
|
+
Love made them not: with acture they may be,
|
217
|
+
Where neither party is nor true nor kind:
|
218
|
+
They sought their shame that so their shame did find;
|
219
|
+
And so much less of shame in me remains,
|
220
|
+
By how much of me their reproach contains.
|
221
|
+
|
222
|
+
''Among the many that mine eyes have seen,
|
223
|
+
Not one whose flame my heart so much as warm'd,
|
224
|
+
Or my affection put to the smallest teen,
|
225
|
+
Or any of my leisures ever charm'd:
|
226
|
+
Harm have I done to them, but ne'er was harm'd;
|
227
|
+
Kept hearts in liveries, but mine own was free,
|
228
|
+
And reign'd, commanding in his monarchy.
|
229
|
+
|
230
|
+
''Look here, what tributes wounded fancies sent me,
|
231
|
+
Of paled pearls and rubies red as blood;
|
232
|
+
Figuring that they their passions likewise lent me
|
233
|
+
Of grief and blushes, aptly understood
|
234
|
+
In bloodless white and the encrimson'd mood;
|
235
|
+
Effects of terror and dear modesty,
|
236
|
+
Encamp'd in hearts, but fighting outwardly.
|
237
|
+
|
238
|
+
''And, lo, behold these talents of their hair,
|
239
|
+
With twisted metal amorously impleach'd,
|
240
|
+
I have received from many a several fair,
|
241
|
+
Their kind acceptance weepingly beseech'd,
|
242
|
+
With the annexions of fair gems enrich'd,
|
243
|
+
And deep-brain'd sonnets that did amplify
|
244
|
+
Each stone's dear nature, worth, and quality.
|
245
|
+
|
246
|
+
''The diamond,--why, 'twas beautiful and hard,
|
247
|
+
Whereto his invised properties did tend;
|
248
|
+
The deep-green emerald, in whose fresh regard
|
249
|
+
Weak sights their sickly radiance do amend;
|
250
|
+
The heaven-hued sapphire and the opal blend
|
251
|
+
With objects manifold: each several stone,
|
252
|
+
With wit well blazon'd, smiled or made some moan.
|
253
|
+
|
254
|
+
''Lo, all these trophies of affections hot,
|
255
|
+
Of pensived and subdued desires the tender,
|
256
|
+
Nature hath charged me that I hoard them not,
|
257
|
+
But yield them up where I myself must render,
|
258
|
+
That is, to you, my origin and ender;
|
259
|
+
For these, of force, must your oblations be,
|
260
|
+
Since I their altar, you enpatron me.
|
261
|
+
|
262
|
+
''O, then, advance of yours that phraseless hand,
|
263
|
+
Whose white weighs down the airy scale of praise;
|
264
|
+
Take all these similes to your own command,
|
265
|
+
Hallow'd with sighs that burning lungs did raise;
|
266
|
+
What me your minister, for you obeys,
|
267
|
+
Works under you; and to your audit comes
|
268
|
+
Their distract parcels in combined sums.
|
269
|
+
|
270
|
+
''Lo, this device was sent me from a nun,
|
271
|
+
Or sister sanctified, of holiest note;
|
272
|
+
Which late her noble suit in court did shun,
|
273
|
+
Whose rarest havings made the blossoms dote;
|
274
|
+
For she was sought by spirits of richest coat,
|
275
|
+
But kept cold distance, and did thence remove,
|
276
|
+
To spend her living in eternal love.
|
277
|
+
|
278
|
+
''But, O my sweet, what labour is't to leave
|
279
|
+
The thing we have not, mastering what not strives,
|
280
|
+
Playing the place which did no form receive,
|
281
|
+
Playing patient sports in unconstrained gyves?
|
282
|
+
She that her fame so to herself contrives,
|
283
|
+
The scars of battle 'scapeth by the flight,
|
284
|
+
And makes her absence valiant, not her might.
|
285
|
+
|
286
|
+
''O, pardon me, in that my boast is true:
|
287
|
+
The accident which brought me to her eye
|
288
|
+
Upon the moment did her force subdue,
|
289
|
+
And now she would the caged cloister fly:
|
290
|
+
Religious love put out Religion's eye:
|
291
|
+
Not to be tempted, would she be immured,
|
292
|
+
And now, to tempt, all liberty procured.
|
293
|
+
|
294
|
+
''How mighty then you are, O, hear me tell!
|
295
|
+
The broken bosoms that to me belong
|
296
|
+
Have emptied all their fountains in my well,
|
297
|
+
And mine I pour your ocean all among:
|
298
|
+
I strong o'er them, and you o'er me being strong,
|
299
|
+
Must for your victory us all congest,
|
300
|
+
As compound love to physic your cold breast.
|
301
|
+
|
302
|
+
''My parts had power to charm a sacred nun,
|
303
|
+
Who, disciplined, ay, dieted in grace,
|
304
|
+
Believed her eyes when they to assail begun,
|
305
|
+
All vows and consecrations giving place:
|
306
|
+
O most potential love! vow, bond, nor space,
|
307
|
+
In thee hath neither sting, knot, nor confine,
|
308
|
+
For thou art all, and all things else are thine.
|
309
|
+
|
310
|
+
''When thou impressest, what are precepts worth
|
311
|
+
Of stale example? When thou wilt inflame,
|
312
|
+
How coldly those impediments stand forth
|
313
|
+
Of wealth, of filial fear, law, kindred, fame!
|
314
|
+
Love's arms are peace, 'gainst rule, 'gainst sense,
|
315
|
+
'gainst shame,
|
316
|
+
And sweetens, in the suffering pangs it bears,
|
317
|
+
The aloes of all forces, shocks, and fears.
|
318
|
+
|
319
|
+
''Now all these hearts that do on mine depend,
|
320
|
+
Feeling it break, with bleeding groans they pine;
|
321
|
+
And supplicant their sighs to you extend,
|
322
|
+
To leave the battery that you make 'gainst mine,
|
323
|
+
Lending soft audience to my sweet design,
|
324
|
+
And credent soul to that strong-bonded oath
|
325
|
+
That shall prefer and undertake my troth.'
|
326
|
+
|
327
|
+
'This said, his watery eyes he did dismount,
|
328
|
+
Whose sights till then were levell'd on my face;
|
329
|
+
Each cheek a river running from a fount
|
330
|
+
With brinish current downward flow'd apace:
|
331
|
+
O, how the channel to the stream gave grace!
|
332
|
+
Who glazed with crystal gate the glowing roses
|
333
|
+
That flame through water which their hue encloses.
|
334
|
+
|
335
|
+
'O father, what a hell of witchcraft lies
|
336
|
+
In the small orb of one particular tear!
|
337
|
+
But with the inundation of the eyes
|
338
|
+
What rocky heart to water will not wear?
|
339
|
+
What breast so cold that is not warmed here?
|
340
|
+
O cleft effect! cold modesty, hot wrath,
|
341
|
+
Both fire from hence and chill extincture hath.
|
342
|
+
|
343
|
+
'For, lo, his passion, but an art of craft,
|
344
|
+
Even there resolved my reason into tears;
|
345
|
+
There my white stole of chastity I daff'd,
|
346
|
+
Shook off my sober guards and civil fears;
|
347
|
+
Appear to him, as he to me appears,
|
348
|
+
All melting; though our drops this difference bore,
|
349
|
+
His poison'd me, and mine did him restore.
|
350
|
+
|
351
|
+
'In him a plenitude of subtle matter,
|
352
|
+
Applied to cautels, all strange forms receives,
|
353
|
+
Of burning blushes, or of weeping water,
|
354
|
+
Or swooning paleness; and he takes and leaves,
|
355
|
+
In either's aptness, as it best deceives,
|
356
|
+
To blush at speeches rank to weep at woes,
|
357
|
+
Or to turn white and swoon at tragic shows.
|
358
|
+
|
359
|
+
'That not a heart which in his level came
|
360
|
+
Could 'scape the hail of his all-hurting aim,
|
361
|
+
Showing fair nature is both kind and tame;
|
362
|
+
And, veil'd in them, did win whom he would maim:
|
363
|
+
Against the thing he sought he would exclaim;
|
364
|
+
When he most burn'd in heart-wish'd luxury,
|
365
|
+
He preach'd pure maid, and praised cold chastity.
|
366
|
+
|
367
|
+
'Thus merely with the garment of a Grace
|
368
|
+
The naked and concealed fiend he cover'd;
|
369
|
+
That th' unexperient gave the tempter place,
|
370
|
+
Which like a cherubin above them hover'd.
|
371
|
+
Who, young and simple, would not be so lover'd?
|
372
|
+
Ay me! I fell; and yet do question make
|
373
|
+
What I should do again for such a sake.
|
374
|
+
|
375
|
+
'O, that infected moisture of his eye,
|
376
|
+
O, that false fire which in his cheek so glow'd,
|
377
|
+
O, that forced thunder from his heart did fly,
|
378
|
+
O, that sad breath his spongy lungs bestow'd,
|
379
|
+
O, all that borrow'd motion seeming owed,
|
380
|
+
Would yet again betray the fore-betray'd,
|
381
|
+
And new pervert a reconciled maid!'
|