ruby-cute 0.3 → 0.4

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: f54b6b52751a03c02f575ef8e544f399df99229a
4
- data.tar.gz: e2db4b6d5a93a21ec4d19cf238ec05606ee6ab97
3
+ metadata.gz: 07edfbc07f26d33908a645a9caf1b8fa786830e8
4
+ data.tar.gz: 95168f3f5d0b06290c526e713f8d2d3f9033e5e9
5
5
  SHA512:
6
- metadata.gz: 8a6a3d01f0eb79fa5d66955467529b92bd53add1f597bf88df283f388e53751f2e46980b55a0398f350671b02ae7b68d8f00409fd22defd91dccd9220c2a2e4e
7
- data.tar.gz: c4e54ceadd5e0aa271438b195942da1e4cc38bfe97d02db0394d787347c3f6a41be821d173e7cf6c83db2a7caa5b34cbe3d1141d58f6b2cde547a2f3c4f389e7
6
+ metadata.gz: 26efbf1b1a42f09c73ef2626fd7bf1420872c6efc04d0a389635dc865eec7edf6a0e6f0911d792ff1e26af7f6377d81d9402dd706ad50432c1ad027b6c066af9
7
+ data.tar.gz: e9210765ad54ff59a42f81e20c6232499b992cb2591daee8899cfbfdcb1a4fa7d78d741bdbe46f7f6d97fb71b6a08de14dec2bdd5169f5b440e3e1d977ef4478
data/.yardopts CHANGED
@@ -1,2 +1,4 @@
1
- lib/**/*.rb -
1
+ lib/**/*.rb
2
+ -
2
3
  examples/*.rb
4
+ examples/g5k-tutorial.md
data/README.md CHANGED
@@ -8,19 +8,18 @@ Grid'5000.
8
8
 
9
9
  ## Installation
10
10
 
11
+ To install latest release from [RubyGems](https://rubygems.org/gems/ruby-cute):
12
+
13
+ ```bash
14
+ $ gem install --user-install ruby-cute
15
+ ```
11
16
  From sources:
12
17
 
13
18
  ```bash
14
19
  $ git clone https://github.com/ruby-cute/ruby-cute
15
20
  $ cd ruby-cute
16
21
  $ gem build ruby-cute.gemspec
17
- $ gem install ruby-cute-*.gem
18
- ```
19
-
20
- In Grid'5000 the installation procedure goes as follows:
21
-
22
- ```bash
23
- $ gem install --user-install ruby-cute
22
+ $ gem install --user-install ruby-cute-*.gem
24
23
  ```
25
24
 
26
25
  Then, type the following for having ruby cute in your path (this is only necessary if you want to use interactive mode).
@@ -72,6 +71,8 @@ $ cute
72
71
  The variable *$g5k* is available which can be used to access the Grid'5000 API through the
73
72
  [G5K Module](http://www.rubydoc.info/github/ruby-cute/ruby-cute/master/Cute/G5K/API). For example,
74
73
  let's request the name of the sites available in Grid'5000.
74
+ (Before starting be sure to set up a configuration file for the module, please refer to
75
+ [G5K Module](http://www.rubydoc.info/github/ruby-cute/ruby-cute/master/Cute/G5K/API))
75
76
 
76
77
  ```bash
77
78
  [2] pry(main)> $g5k.site_uids()
data/Rakefile CHANGED
@@ -27,6 +27,7 @@ Rake::PackageTask::new("ruby-cute",get_version) do |p|
27
27
  p.package_files.include('ext/**/*')
28
28
  p.package_files.include('bin/**/*')
29
29
  p.package_files.include('test/**/*')
30
+ p.package_files.include('examples/**/*')
30
31
  p.package_files.include('Rakefile', 'COPYING','README', 'README.md')
31
32
  end
32
33
 
data/debian/changelog CHANGED
@@ -1,5 +1,5 @@
1
- ruby-cute (0.3) UNRELEASED; urgency=medium
1
+ ruby-cute (0.4) UNRELEASED; urgency=medium
2
2
 
3
3
  * Initial release.
4
4
 
5
- -- Lucas Nussbaum <lucas@debian.org> Tue, 10 Mar 2015 15:13:52 +0100
5
+ -- Lucas Nussbaum <lucas@debian.org> Fri, 22 Jan 2016 08:54:42 +0100
@@ -99,7 +99,10 @@ options[:network_mode] = 'classical'
99
99
  options[:network_interface] = nil
100
100
  options[:num_nodes] = 2
101
101
  options[:walltime] = "2:00:00"
102
-
102
+ options[:deb_repo] = 'http://distem.gforge.inria.fr/deb'
103
+ options[:cluster] = "nancy"
104
+ options[:env] = 'wheezy-x64-nfs'
105
+ options[:branch] = nil
103
106
  args = ARGV.dup
104
107
 
105
108
  optparse = OptionParser.new do |opts|
@@ -113,9 +116,16 @@ optparse = OptionParser.new do |opts|
113
116
  opts.separator ""
114
117
  opts.separator "Options:"
115
118
 
119
+ opts.on( '-f', '--node-list <nodelist_file>', 'File containing the list of nodes to install. If not specified, a job will be submitted using RubyCute') do |n|
120
+ options[:node_list] = n
121
+ end
122
+
116
123
  opts.on( '--nodes <number of nodes>', Integer,'Number of nodes to reserve on Grid5000') do |n|
117
124
  options[:num_nodes] = n.to_i
118
125
  end
126
+ opts.on( '--cluster <cluster name>', String,'Cluster name in Grid\'5000') do |n|
127
+ options[:cluster] = n
128
+ end
119
129
  opts.on( '-t', '--time <walltime>', 'Walltime for the reservation on Grid5000') do |n|
120
130
  options[:walltime] = n
121
131
  end
@@ -134,6 +144,12 @@ optparse = OptionParser.new do |opts|
134
144
  opts.on( '-D', '--distem-debug', 'Show distem output when executing distem commands (script, node init)' ) do
135
145
  options[:debug_distem] = true
136
146
  end
147
+ opts.on( '--env <environment>', String, 'Kadeploy environment to be used as a base' ) do |c|
148
+ options[:env] = c
149
+ end
150
+ opts.on( '--deploy', 'Redeploy the base image used for distem' ) do
151
+ options[:deploy] = true
152
+ end
137
153
  opts.on( '-k', '--ssh-key [<ssh_key_file>]', 'Path of the ssh private key to use' ) do |k|
138
154
  options[:ssh_key] = k
139
155
  end
@@ -155,6 +171,9 @@ optparse = OptionParser.new do |opts|
155
171
  opts.on( '-G', '--gerrit <gerrit_ref>', "Checkout a gerrit ref (e.g refs/changes/94/94/1) and rebuild a Debian package" ) do |n|
156
172
  options[:gerrit] = n
157
173
  end
174
+ opts.on( '-B', '--branch <branch_name>', "Checkout a specific branch and rebuild a Debian package" ) do |n|
175
+ options[:branch] = n
176
+ end
158
177
  opts.on( '-S', '--stealth-mode', 'Do not report usage statistics (Grid\'5000 only)' ) do |c|
159
178
  options[:stats] = false
160
179
  end
@@ -177,6 +196,19 @@ optparse = OptionParser.new do |opts|
177
196
  opts.on( '-i', '--network-interface <iface>', 'Define the root network interface for inter-pnode communication (use only with vxlan network mode)') do |iface|
178
197
  options[:network_interface] = iface
179
198
  end
199
+ opts.on( '--debian-version <version>', 'Choose the Debian version (squeeze, wheezy or jessie)') do |v|
200
+ case v
201
+ when 'squeeze'
202
+ options[:deb_repo] = 'http://distem.gforge.inria.fr/deb'
203
+ when 'wheezy'
204
+ options[:deb_repo] = 'http://distem.gforge.inria.fr/deb'
205
+ when 'jessie'
206
+ options[:deb_repo] = 'http://distem.gforge.inria.fr/deb-jessie'
207
+ else
208
+ puts 'Unsupported Debian version'
209
+ exit 1
210
+ end
211
+ end
180
212
  opts.on( '--verbose', 'Activate the verbose mode on Distem servers' ) do
181
213
  options[:verbose] = true
182
214
  end
@@ -200,6 +232,7 @@ begin
200
232
  optparse.parse!
201
233
  rescue OptionParser::InvalidOption => e
202
234
  logger.error(e.to_s)
235
+ exit 1
203
236
  end
204
237
 
205
238
  logger.level = Logger::INFO
@@ -209,23 +242,37 @@ g5k = Cute::G5K::API.new()
209
242
  g5k.logger = logger
210
243
  ## performing reservation with ruby-cute
211
244
 
212
- reserv_param = {:site => g5k.site, :nodes => options[:num_nodes], :name => 'distem',
213
- :walltime => options[:walltime], :env =>'wheezy-x64-nfs', :subnets => [22,1]}
245
+ reserv_param = {:site => g5k.site, :cluster => options[:cluster], :nodes => options[:num_nodes], :name => 'distem',
246
+ :walltime => options[:walltime], :env => options[:env], :subnets => [22,1]}
247
+
248
+ reserv_param[:keys] = options[:ssh_key] if options[:ssh_key]
214
249
 
215
- reserv_param[:keys] = options[:ssh_key] unless options[:ssh_key].nil?
250
+ nodelist = []
216
251
 
252
+ if options[:node_list]
253
+ File.open(options[:node_list], "r") do |f|
254
+ f.each_line do |line|
255
+ nodelist.push(line.chop)
256
+ end
257
+ end
258
+ end
217
259
 
218
- # we verify if we have already submit a job
219
- old_jobs = g5k.get_my_jobs(g5k.site).select{ |j| j["name"] == "distem"}
260
+ # if we pass a list of nodes
261
+ if nodelist.empty?
220
262
 
221
- job = old_jobs.empty? ? g5k.reserve(reserv_param) : old_jobs.first
263
+ # we verify if we have already submit a job
264
+ old_jobs = g5k.get_my_jobs(g5k.site).select{ |j| j["name"] == "distem"}
222
265
 
223
- nodelist = job.resources["cores"].uniq
224
- ## wait for the deploy
266
+ job = old_jobs.empty? ? g5k.reserve(reserv_param) : old_jobs.first
225
267
 
226
- g5k.wait_for_deploy(job)
268
+ nodelist = job.resources["cores"].uniq
269
+ ## wait for the deploy
270
+ reserv_param.delete(:nodes)
271
+ g5k.deploy(job,reserv_param) if options[:deploy]
227
272
 
228
- allnodes = nodelist.dup
273
+ g5k.wait_for_deploy(job)
274
+ end
275
+ allnodes = nodelist.dup
229
276
 
230
277
  # Generating ssh keys"
231
278
 
@@ -233,16 +280,28 @@ key_dir = Dir.mktmpdir("keys")
233
280
  system "ssh-keygen -P \'\' -f #{key_dir}/keys"
234
281
  logger.info "Keys generated in #{key_dir}"
235
282
 
283
+ # Generating ssh config"
284
+
285
+ ssh_conf = Tempfile.new('config')
286
+ File.open(ssh_conf.path,'w+') do |f|
287
+ f.puts "Host *"
288
+ f.puts "StrictHostKeyChecking no"
289
+ f.puts "UserKnownHostsFile=/dev/null "
290
+ end
291
+
292
+
236
293
  nodelist.each do |node|
237
294
 
238
295
  Net::SCP.start(node, "root") do |scp|
239
296
  logger.debug "Transfering key to #{node}"
240
- scp.upload! "#{key_dir}/keys.pub", "/root/.ssh/id_rsa.pub"
241
- scp.upload! "#{key_dir}/keys", "/root/.ssh/id_rsa"
297
+ scp.upload "#{key_dir}/keys.pub", "/root/.ssh/id_rsa.pub"
298
+ scp.upload "#{key_dir}/keys", "/root/.ssh/id_rsa"
299
+ scp.upload ssh_conf.path, "/root/.ssh/config"
242
300
  end
243
301
 
244
302
  end
245
303
 
304
+ ssh_conf.unlink
246
305
  # Deletating keys
247
306
  FileUtils.rm_rf(key_dir)
248
307
 
@@ -314,8 +373,7 @@ Net::SSH::Multi.start do |session|
314
373
  session.exec! "grep -q '#{rule}' /etc/security/limits.conf; true || echo '#{rule} #{ULIMIT_OPEN_FILES}' >> /etc/security/limits.conf"
315
374
 
316
375
  # setup Debian repo
317
- session.exec! "grep -q '#{DEB_REPOSITORY}' /etc/apt/sources.list || echo 'deb #{DEB_REPOSITORY} ./\ndeb-src #{DEB_REPOSITORY} ./' >> /etc/apt/sources.list"
318
-
376
+ session.exec! "grep -q '#{options[:deb_repo]}' /etc/apt/sources.list || echo 'deb #{options[:deb_repo]} ./\ndeb-src #{options[:deb_repo]} ./' >> /etc/apt/sources.list"
319
377
  logger.info "Updating debian packages list"
320
378
  session.exec! "apt-get update -q"
321
379
 
@@ -348,25 +406,39 @@ Net::SSH::Multi.start do |session|
348
406
  session.with(:corrd).exec! "git --git-dir=#{gitdir}/.git reset --hard #{options[:git]}"
349
407
  end
350
408
  end
351
- else # gerrit
409
+ if options[:branch]
410
+ logger.info("Setting up git repository from a specific branch:#{options[:branch]} on #{coordinator}")
411
+ session.with(:coord).exec! "cd #{gitdir} && git fetch origin #{options[:branch]}"
412
+ session.with(:coord).exec! "cd #{gitdir} && git checkout #{options[:branch]}"
413
+ end
414
+ else option[:gerrit] # gerrit
352
415
  logger.info("Setting up git repository from gerrit ref:#{options[:gerrit]} on #{coordinator}")
353
416
  session.with(:coord).exec! "git clone #{GERRIT_REPOSITORY} #{gitdir}"
354
417
  session.with(:coord).exec! "cd #{gitdir} && git fetch #{GERRIT_REPOSITORY} #{options[:gerrit]} && git checkout FETCH_HEAD"
355
418
  end
419
+ # Checking out a specific commit from Distem repository
420
+
421
+
356
422
  logger.info("Building debian package of distem on #{coordinator}")
357
423
  session.with(:coord).exec! "rm -f #{TMP_DIR}/*.deb #{TMP_DIR}/*.changes #{TMP_DIR}/*.dsc #{TMP_DIR}/*.tar.gz"
358
424
  session.with(:coord).exec! "cd #{gitdir}; rake snapshot"
359
425
 
360
- debarchivefile = session.with(:coord).exec! "find #{TMP_DIR} -maxdepth 1 -name *.deb"
426
+ res = session.with(:coord).exec! "find #{TMP_DIR} -maxdepth 1 -name distem*.deb"
361
427
 
362
- session.with(:coord).exec! "Copying generated debian package #{File.basename(debarchivefile)}"
428
+ debarchivefile = res[coordinator][:stdout]
429
+
430
+ logger.info("Copying generated debian package #{File.basename(debarchivefile)}")
363
431
 
364
432
  #exec(session,"cp #{debarchivefile} #{TMP_DIR}",:coord,true)
365
433
  nodelist.each {|node| session.with(:coord).exec! "scp -o StrictHostKeyChecking=no #{debarchivefile} root@#{node}:#{TMP_DIR}"}
366
434
  logger.info("Installing generated debian package #{File.basename(debarchivefile)}")
367
- depends = session.with(:coord).exec! "dpkg -I #{debarchivefile} | grep 'Depends:'"
435
+ res = session.with(:coord).exec! "dpkg -I #{debarchivefile} | grep 'Depends:'"
436
+
437
+ raw_deps = res[coordinator][:stdout].sub('Depends:','').split(",")
438
+
439
+ depends = ""
440
+ raw_deps.each{ |str| depends+=str.split(" ")[0]; depends+="," unless str==raw_deps.last}
368
441
 
369
- depends = depends.split("|").last.gsub!(' ','')
370
442
  session.exec! "dpkg --ignore-depends #{depends} -i #{TMP_DIR}/#{File.basename(debarchivefile)}"
371
443
 
372
444
  session.exec! "DEBIAN_FRONTEND=noninteractive apt-get install -q -y --force-yes -f"
@@ -388,7 +460,7 @@ Net::SSH::Multi.start do |session|
388
460
  end
389
461
  options[:gempackages].each do |gempkg|
390
462
  logger.info "Installing additional gem package '#{gempkg}'"
391
- session.with(:coord).exec! "export http_proxy=#{HTTP_PROXY}; gem install #{gempkg}"
463
+ session.exec! "export http_proxy=#{HTTP_PROXY}; gem install #{gempkg}"
392
464
  end
393
465
 
394
466
  if options[:init_pnodes] or ARGV[0]
@@ -426,8 +498,7 @@ Net::SSH::Multi.start do |session|
426
498
  begin
427
499
  Timeout.timeout(10) do
428
500
  root_iface_opt = options[:network_interface] ? "--network-interface #{options[:network_interface]}" : ''
429
- distem_cmd = "LANG=C distemd #{options[:verbose] ? '--verbose' : ''} --network-mode "+
430
- "#{options[:network_mode]} #{root_iface_opt} -d &>#{File.join(PATH_DISTEMD_LOGS,'distemd.log')}&"
501
+ distem_cmd = "LANG=C distemd #{options[:verbose] ? '--verbose' : ''} -d &>#{File.join(PATH_DISTEMD_LOGS,'distemd.log')}&"
431
502
 
432
503
  session.with(:coord).exec! distem_cmd
433
504
 
@@ -475,7 +546,7 @@ Net::SSH::Multi.start do |session|
475
546
  if ARGV[0]
476
547
  begin
477
548
  File.open(ARGV[0], 'r') do |f|
478
- filename = session.with(:coord).exec!('tempfile')[coordinator][0]
549
+ filename = session.with(:coord).exec!('tempfile')[coordinator][:stdout]
479
550
  logger.info("Copying script file in '#{filename}' on #{coordinator}")
480
551
  `scp #{ARGV[0]} root@#{coordinator}:#{filename}`
481
552
  session.with(:coord).exec! "chmod +x #{filename}"
@@ -484,7 +555,7 @@ Net::SSH::Multi.start do |session|
484
555
  argv_dup = ARGV.dup
485
556
  argv_dup.shift
486
557
  script_args = (argv_dup.length > 0 ? argv_dup.join(" ") : "")
487
- session.with(:coord).exec! "export #{VAR_DISTEM_NODES}='#{(nodelist + [coordinator]) * "\n"}'; export #{VAR_DISTEM_COORD}='#{coordinator}';#{filename} #{script_args}"
558
+ #session.with(:coord).exec! "export #{VAR_DISTEM_NODES}='#{(nodelist + [coordinator]) * "\n"}'; export #{VAR_DISTEM_COORD}='#{coordinator}';#{filename} #{script_args}"
488
559
  session.with(:coord).exec! "rm #{filename}"
489
560
  end
490
561
  logger.info 'Script execution done'
@@ -496,21 +567,21 @@ Net::SSH::Multi.start do |session|
496
567
  logger.info("Coordinator: #{coordinator}") if options[:init_pnodes] or ARGV[0]
497
568
 
498
569
  # Send stats record to stats server
499
- if g5k? and options[:stats]
500
- begin
501
- stats = {
502
- :time => Time.now.to_i,
503
- :site => Socket.gethostname,
504
- :user => ENV['USER'],
505
- :oar => ENV['OAR_JOB_ID'].to_i,
506
- :nodes => allnodes,
507
- :params => args * ' ',
508
- :length => (Time.now - $startt).to_i
509
- }
510
- sock = TCPSocket.open(STATS_SERV, STATS_PORT)
511
- sock.send(stats.to_yaml,0)
512
- sock.close
513
- rescue SocketError, Errno::ECONNREFUSED, Errno::ECONNRESET, Errno::ETIMEDOUT, Errno::EHOSTUNREACH
514
- end
515
- end
570
+ if g5k? and options[:stats]
571
+ begin
572
+ stats = {
573
+ :time => Time.now.to_i,
574
+ :site => Socket.gethostname,
575
+ :user => g5k.g5k_user,
576
+ :oar => job['uid'],
577
+ :nodes => allnodes,
578
+ :params => args * ' ',
579
+ :length => (Time.now - $startt).to_i
580
+ }
581
+ sock = TCPSocket.open(STATS_SERV, STATS_PORT)
582
+ sock.send(stats.to_yaml,0)
583
+ sock.close
584
+ rescue SocketError, Errno::ECONNREFUSED, Errno::ECONNRESET, Errno::ETIMEDOUT, Errno::EHOSTUNREACH
585
+ end
586
+ end
516
587
  end
@@ -0,0 +1,999 @@
1
+ # @title Grid'5000 tutorial
2
+ # Grid'5000 tutorial
3
+
4
+
5
+ This tutorial aims at showing how **Ruby-Cute** can be used to
6
+ help the scripting of an experiment in the context of the Grid'5000 testbed.
7
+ The programming language used, as you would expect, is {https://www.ruby-lang.org/en/ Ruby}.
8
+ We will use a powerful console debugger called {http://pryrepl.org/ Pry} which offers
9
+ several functionalities that can be used for the step-by-step scripting of complex experiments.
10
+
11
+ In this tutorial the first two sections are dedicated to the installation and basic use of **Ruby-Cute**.
12
+ You can skip those sections if you are already acquainted with **Ruby-Cute**.
13
+ The other sections show how to use **Ruby-cute** for scripting complex experiments.
14
+ This is shown through three examples:
15
+
16
+ 1. **Infiniband performance test**: the experiment will illustrate how to perform a reservation,
17
+ the execution of commands in a reserved node and it explains several `pry` commands that will help you with the writing of your experiment.
18
+ 2. **Running NAS benchmarks in Grid’5000**: you will get acquainted with parallel execution using simple SSH or {http://taktuk.gforge.inria.fr/ TakTuk}.
19
+ 3. **Performing network measures within a reserved VLAN**: you will learn how to reserve a routed VLAN
20
+ and to query the G5K metrology {https://www.grid5000.fr/mediawiki/index.php/API API}.
21
+ For this particular experiment we will query the {https://www.grid5000.fr/mediawiki/index.php/Monitoring Kwapi} service.
22
+
23
+ The aforementioned experiments are independent, you can perform them in any order.
24
+ However, you may need some concepts that are explained only in specific sections.
25
+
26
+ ## Installing and preparing Ruby cute
27
+
28
+ The installation procedure is shown in {file:README.md Ruby-Cute install}.
29
+ After this step you will normally have `ruby-cute` and `pry` gems installed.
30
+
31
+ Before using **Ruby-Cute** you have to create the following file:
32
+
33
+ $ cat > ~/.grid5000_api.yml << EOF
34
+ $ uri: https://api.grid5000.fr/
35
+ $ version: 3.0
36
+ $ EOF
37
+
38
+ ## Getting acquainted with the pry console
39
+
40
+ After instaling `ruby-cute` and `pry` gems you can lunch a pry console
41
+ with **ruby-cute** loaded by typing:
42
+
43
+ $ cute
44
+
45
+ Which will open a `pry` console:
46
+
47
+ [1] pry(main)>
48
+
49
+ In this console, we can evaluate Ruby code, execute shell commands, consult
50
+ documentation, explore classes and more.
51
+ The variable *$g5k* is available which can be used to access the Grid'5000 API through the
52
+ [G5K Module](http://www.rubydoc.info/github/ruby-cute/ruby-cute/master/Cute/G5K/API). For example,
53
+ let's request the name of the sites available in Grid'5000.
54
+
55
+ [2] pry(main)> $g5k.site_uids()
56
+ => ["grenoble", "lille", "luxembourg", "lyon", "nancy", "nantes", "reims", "rennes", "sophia", "toulouse"]
57
+
58
+ We can consult the name of the clusters available in a specific site.
59
+
60
+ [3] pry(main)> $g5k.cluster_uids("grenoble")
61
+ => ["adonis", "edel", "genepi"]
62
+
63
+ It is possible to execute shell commands, however all commands have to be prefixed with a dot ".". For example we could generate a pair of SSH keys using:
64
+
65
+ [7] pry(main)> .ssh-keygen -b 1024 -N "" -t rsa -f ~/my_ssh_jobkey
66
+
67
+ Another advantage is the possibility of exploring the loaded Ruby modules.
68
+ Let's explore the [Cute](http://www.rubydoc.info/github/ruby-cute/ruby-cute/master/Cute) module.
69
+
70
+ [8] pry(main)> cd Cute
71
+ [9] pry(Cute):1> ls
72
+ constants: Bash Execute G5K TakTuk VERSION
73
+ locals: _ __ _dir_ _ex_ _file_ _in_ _out_ _pry_
74
+
75
+ We can see that the [Cute](http://www.rubydoc.info/github/ruby-cute/ruby-cute/master/Cute) module
76
+ is composed of other helpful modules such as:
77
+ [G5K](http://www.rubydoc.info/github/ruby-cute/ruby-cute/master/Cute/G5K/API),
78
+ [TakTuk](http://www.rubydoc.info/github/ruby-cute/ruby-cute/master/Cute/TakTuk), etc.
79
+ To quit the Cute namespace type:
80
+
81
+ [10] pry(main)> cd
82
+
83
+ Let's explore the methods defined in the
84
+ [G5K Module](http://www.rubydoc.info/github/ruby-cute/ruby-cute/master/Cute/G5K/API),
85
+ so you can observe which methods can be used with `$g5k` variable.
86
+
87
+ [11] pry(main)> ls Cute::G5K::API
88
+ Object.methods: yaml_tag
89
+ Cute::G5K::API#methods:
90
+ check_deployment deploy environments get_job get_subnets get_vlan_nodes nodes_status reserve site_status wait_for_deploy
91
+ cluster_uids deploy_status g5k_user get_jobs get_switch logger release rest site_uids wait_for_job
92
+ clusters environment_uids get_deployments get_my_jobs get_switches logger= release_all site sites
93
+
94
+ We can access the respective YARD documentation of a given method by typing:
95
+
96
+ [12] pry(main)> show-doc Cute::G5K::API#deploy
97
+
98
+ In the following section we will see how
99
+ `pry` can be used to setup an experiment step by step using **Ruby-Cute**.
100
+
101
+ Pry can be customized by creating the file `.pryrc`. We will create this
102
+ file with the following content in order to choose our prefered editor:
103
+
104
+ $ cat > ~/.pryrc << EOF
105
+ Pry.config.editor = "emacs"
106
+ EOF
107
+
108
+ ## First experiment: Infiniband performance test
109
+
110
+ Here, we will use **Ruby-cute** to carry out an experiment.
111
+ In this experiment, we will ask for two nodes equipped with infiniband and
112
+ then, we will perform some performance test using a network benchmark called *NETPIPE*.
113
+ For this particular experiment we have the following requirements:
114
+
115
+ - A pair of SSH keys
116
+ - Use of standard environment (no deploy)
117
+ - Two nodes connected with infiniband (10G or 20G)
118
+ - MPI benchmark NETPIPE
119
+ - A MPI runtime (OpenMPI or MPICH)
120
+
121
+ We will do it interactively using `pry`.
122
+ Let's create a directory for keeping all the scripts that we will write throughout the tutorial.
123
+
124
+ $ mkdir ruby-cute-tutorial
125
+
126
+ Then, we execute the `pry` console form this directory:
127
+
128
+ $ cd ruby-cute-tutorial
129
+ $ cute
130
+
131
+ First, let's find the sites that offer Infiniband interconnection.
132
+ For that we will write a small script from `pry` console using the command edit.
133
+
134
+
135
+ [13] pry(main)> edit -n find_infiniband.rb
136
+
137
+ This will open a new file with your prefered editor. Here we will put the following
138
+ ruby script:
139
+
140
+ sites = $g5k.site_uids
141
+
142
+ sites_infiniband = []
143
+
144
+ sites.each do |site|
145
+ sites_infiniband.push(site) unless $g5k.get_switches(site).select{ |t| t["model"] == "Infiniband" }.empty?
146
+ end
147
+
148
+ Then, we execute it using the `play` command which will execute line by line this script in the context of a Pry session.
149
+
150
+ [21] pry(main)> play find_infiniband.rb
151
+ => ["grenoble", "lille", "luxembourg", "lyon", "nancy", "nantes", "reims", "rennes", "sophia"]
152
+
153
+ We can observe that the variable `sites_infiniband` is now defined, telling us that Grenoble and Nancy sites offer Infiniband interconnection.
154
+
155
+ [22] pry(main)> sites_infiniband
156
+ => ["grenoble", "nancy"]
157
+
158
+ Then, create a pair of SSH keys (Necessary for OARSSH):
159
+
160
+ $ ssh-keygen -b 1024 -N "" -t rsa -f ~/my_ssh_jobkey
161
+
162
+ We send the generated keys to the chosen site:
163
+
164
+ [22] pry(main)> .scp ~/my_ssh* nancy:~/
165
+
166
+ Now that we have found the sites, let's submit a job. You can use between Grenoble and Nancy sites. If you
167
+ take a look at {https://www.grid5000.fr/mediawiki/index.php/Status Monika} you will see that in Nancy we should use the OAR property 'ib20g' and in Grenoble we should use 'ib10g'.
168
+ Given that the MPI bench uses just one MPI process, we will need in realty just one core of a given machine.
169
+ We will use OAR syntax to ask for two cores in two different nodes with ib10g in Grenoble.
170
+
171
+ [23] pry(main)> job = $g5k.reserve(:site => "grenoble", :resources => "{ib10g='YES'}/nodes=2/core=1",:walltime => '01:00:00', :keys => "~/my_ssh_jobkey" )
172
+ 2015-12-04 14:07:31.370 => Reserving resources: {ib20g='YES'}/nodes=2/core=1,walltime=01:00 (type: ) (in grenoble)
173
+ 2015-12-04 14:07:41.358 => Waiting for reservation 692665
174
+ 2015-12-04 14:07:41.444 => Reservation 692665 should be available at 2015-12-04 14:07:34 +0100 (0 s)
175
+ 2015-12-04 14:07:41.444 => Reservation 692665 ready
176
+
177
+ A hash is returned containing all the information about the job that we have just submitted.
178
+
179
+ [58] pry(main)> job
180
+ => {"uid"=>692665,
181
+ "user_uid"=>"cruizsanabria",
182
+ "user"=>"cruizsanabria",
183
+ "walltime"=>3600,
184
+ "queue"=>"default",
185
+ "state"=>"running",
186
+ "project"=>"default",
187
+ "name"=>"rubyCute job",
188
+ "types"=>[],
189
+ "mode"=>"PASSIVE",
190
+ "command"=>"sleep 3600",
191
+ "submitted_at"=>1449234452,
192
+ "scheduled_at"=>1449234454,
193
+ "started_at"=>1449234454,
194
+ "message"=>"FIFO scheduling OK",
195
+ "properties"=>"(maintenance = 'NO') AND production = 'NO'",
196
+ "directory"=>"/home/cruizsanabria",
197
+ "events"=>[],
198
+ "links"=>
199
+ [{"rel"=>"self", "href"=>"/sid/sites/nancy/jobs/692665", "type"=>"application/vnd.grid5000.item+json"},
200
+ {"rel"=>"parent", "href"=>"/sid/sites/nancy", "type"=>"application/vnd.grid5000.item+json"}],
201
+ "resources_by_type"=>{"cores"=>["graphene-67.nancy.grid5000.fr", "graphene-45.nancy.grid5000.fr"]},
202
+ "assigned_nodes"=>["graphene-67.nancy.grid5000.fr", "graphene-45.nancy.grid5000.fr"]}
203
+
204
+ An important information is the nodes that has been assigned, let's put this information in another variable:
205
+
206
+ [60] pry(main)> nodes = job["assigned_nodes"]
207
+ => ["graphene-67.nancy.grid5000.fr", "graphene-45.nancy.grid5000.fr"]
208
+
209
+ Then, we create a file with the name of the reserved machines:
210
+
211
+ [62] pry(main)> machine_file = Tempfile.open('machine_file')
212
+ => #<File:/tmp/machine_file20151204-28888-1ll3brs>
213
+
214
+ [64] pry(main)> nodes.each{ |node| machine_file.puts node }
215
+ => ["graphene-67.nancy.grid5000.fr", "graphene-45.nancy.grid5000.fr"]
216
+
217
+ [66] pry(main)> machine_file.close
218
+
219
+
220
+ We will need to setup SSH options for OAR, we can do it with the {Cute::OARSSHopts OARSSHopts} class helper provided by ruby-cute:
221
+
222
+ [6] pry(main)> grid5000_opt = Cute::OARSSHopts.new(:keys => "~/my_ssh_jobkey")
223
+ => {:user=>"oar", :keys=>"~/my_ssh_jobkey", :port=>6667}
224
+
225
+ Now, we can communicate using SSH with our nodes. Let's send the machinefile using SCP.
226
+ From a `pry` console let's load the SCP module to transfer files:
227
+
228
+ [12] pry(main)> require 'net/scp'
229
+
230
+ Then, copy-paste the following code in pry console:
231
+
232
+ Net::SCP.start(nodes.first, "oar", grid5000_opt) do |scp|
233
+ scp.upload! machine_file.path, "/tmp/machine_file"
234
+ end
235
+
236
+ The previous code will sent the machine file into the first node.
237
+ We can check this by performing an SSH connection into the node.
238
+ Here to illustrate the use of temporary files, let's type the following:
239
+
240
+ [6] pry(main)> edit -t
241
+
242
+ and copy-paste the following code:
243
+
244
+ Net::SSH.start(nodes.first, "oar", grid5000_opt) do |ssh|
245
+ puts ssh.exec("cat /tmp/machine_file")
246
+ end
247
+
248
+ If we save and quit the editor, the code will be evaluated in Pry context.
249
+ Which will generate the following output:
250
+
251
+ [12] pry(main)> edit -t
252
+ #<Net::SSH::Connection::Channel:0x00000001247150>
253
+ graphene-80.nancy.grid5000.fr
254
+ graphene-81.nancy.grid5000.fr
255
+ => nil
256
+
257
+ We confirmed the existence of the file in the first reserved node.
258
+ Now let's download, compile and execute
259
+ the benchmark. Create a Ruby file called netpipe:
260
+
261
+ [12] pry(main)> edit -n netpipe.rb
262
+
263
+ With the following content:
264
+
265
+ Net::SSH.start(nodes.first, "oar", grid5000_opt) do |ssh|
266
+ netpipe_url = "http://pkgs.fedoraproject.org/repo/pkgs/NetPIPE/NetPIPE-3.7.1.tar.gz/5f720541387be065afdefc81d438b712/NetPIPE-3.7.1.tar.gz"
267
+ ssh.exec!("mkdir -p netpipe_exp")
268
+ ssh.exec!("wget -O ~/netpipe_exp/NetPIPE.tar.gz #{netpipe_url}")
269
+ ssh.exec!("cd netpipe_exp && tar -zvxf NetPIPE.tar.gz")
270
+ ssh.exec!("cd netpipe_exp/NetPIPE-3.7.1 && make mpi")
271
+ ssh.exec("mpirun --mca plm_rsh_agent \"oarsh\" -machinefile /tmp/machine_file ~/netpipe_exp/NetPIPE-3.7.1/NPmpi")
272
+ end
273
+
274
+ Then, execute the created script:
275
+
276
+ [16] pry(main)> play netpipe.rb
277
+ #<Net::SSH::Connection::Channel:0x000000021679f0>
278
+ Permission denied (publickey,keyboard-interactive).
279
+ --------------------------------------------------------------------------
280
+ A daemon (pid 4615) died unexpectedly with status 255 while attempting
281
+ to launch so we are aborting.
282
+
283
+ There may be more information reported by the environment (see above).
284
+
285
+ This may be because the daemon was unable to find all the needed shared
286
+ libraries on the remote node. You may set your LD_LIBRARY_PATH to have the
287
+ location of the shared libraries on the remote nodes and this will
288
+ automatically be forwarded to the remote nodes.
289
+ --------------------------------------------------------------------------
290
+ --------------------------------------------------------------------------
291
+ mpirun noticed that the job aborted, but has no info as to the process
292
+ that caused that situation.
293
+ --------------------------------------------------------------------------
294
+ mpirun: clean termination accomplished
295
+
296
+ => nil
297
+
298
+ We got an error related to the SSH keys and it is due to the fact that `oarsh` cannot not find the appropriate key files.
299
+ We can fix this problem by prefixing the `mpirun` command with `export OAR_JOB_KEY_FILE=~/my_ssh_jobkey`.
300
+ Now the code will look like this:
301
+
302
+ Net::SSH.start(nodes.first, "oar", grid5000_opt) do |ssh|
303
+ netpipe_url = "http://pkgs.fedoraproject.org/repo/pkgs/NetPIPE/NetPIPE-3.7.1.tar.gz/5f720541387be065afdefc81d438b712/NetPIPE-3.7.1.tar.gz"
304
+ ssh.exec!("mkdir -p netpipe_exp")
305
+ ssh.exec!("wget -O ~/netpipe_exp/NetPIPE.tar.gz #{netpipe_url}")
306
+ ssh.exec!("cd netpipe_exp && tar -zvxf NetPIPE.tar.gz")
307
+ ssh.exec!("cd netpipe_exp/NetPIPE-3.7.1 && make mpi")
308
+ ssh.exec("export OAR_JOB_KEY_FILE=~/my_ssh_jobkey;mpirun --mca plm_rsh_agent \"oarsh\" -machinefile /tmp/machine_file ~/netpipe_exp/NetPIPE-3.7.1/NPmpi")
309
+ end
310
+
311
+ After running the script, it will show the output of the benchmark in the `pry` console:
312
+
313
+ [34] pry(main)> play netpipe.rb
314
+ #<Net::SSH::Connection::Channel:0x00000002edc6d0>
315
+ 0: adonis-9
316
+ 1: adonis-10
317
+ Now starting the main loop
318
+ 0: 1 bytes 32103 times --> 4.58 Mbps in 1.67 usec
319
+ 1: 2 bytes 59994 times --> 9.22 Mbps in 1.65 usec
320
+ 2: 3 bytes 60440 times --> 13.79 Mbps in 1.66 usec
321
+ 3: 4 bytes 40180 times --> 18.34 Mbps in 1.66 usec
322
+ 4: 6 bytes 45076 times --> 27.07 Mbps in 1.69 usec
323
+ 5: 8 bytes 29563 times --> 36.16 Mbps in 1.69 usec
324
+ 6: 12 bytes 37023 times --> 53.84 Mbps in 1.70 usec
325
+ 7: 13 bytes 24500 times --> 57.97 Mbps in 1.71 usec
326
+ 8: 16 bytes 26977 times --> 71.61 Mbps in 1.70 usec
327
+ 9: 19 bytes 32995 times --> 84.65 Mbps in 1.71 usec
328
+ 10: 21 bytes 36882 times --> 93.27 Mbps in 1.72 usec
329
+ 11: 24 bytes 38808 times --> 106.69 Mbps in 1.72 usec
330
+ 12: 27 bytes 41271 times --> 119.77 Mbps in 1.72 usec
331
+
332
+ We can modify slightly the previous script to write the result into a file.
333
+ We need to use `ssh.exec!` to capture the output of the commands.
334
+
335
+ Net::SSH.start(nodes.first, "oar", grid5000_opt) do |ssh|
336
+ netpipe_url = "http://pkgs.fedoraproject.org/repo/pkgs/NetPIPE/NetPIPE-3.7.1.tar.gz/5f720541387be065afdefc81d438b712/NetPIPE-3.7.1.tar.gz"
337
+ ssh.exec!("mkdir -p netpipe_exp")
338
+ ssh.exec!("wget -O ~/netpipe_exp/NetPIPE.tar.gz #{netpipe_url}")
339
+ ssh.exec!("cd netpipe_exp && tar -zvxf NetPIPE.tar.gz")
340
+ ssh.exec!("cd netpipe_exp/NetPIPE-3.7.1 && make mpi")
341
+
342
+ File.open("output_netpipe.txt", 'w') do |f|
343
+ f.puts ssh.exec!("OAR_JOB_KEY_FILE=~/my_ssh_jobkey; mpirun --mca plm_rsh_agent \"oarsh\" -machinefile /tmp/machine_file ~/netpipe_exp/NetPIPE-3.7.1/NPmpi")
344
+ end
345
+ end
346
+
347
+ We can check the results by doing:
348
+
349
+ [16] pry(main)> .cat output_netpipe.txt
350
+ 0: adonis-9
351
+ 1: adonis-10
352
+ Now starting the main loop
353
+ 0: 1 bytes 31441 times --> 4.62 Mbps in 1.65 usec
354
+ 1: 2 bytes 60550 times --> 9.24 Mbps in 1.65 usec
355
+ 2: 3 bytes 60580 times --> 13.87 Mbps in 1.65 usec
356
+ 3: 4 bytes 40404 times --> 18.39 Mbps in 1.66 usec
357
+ 4: 6 bytes 45183 times --> 27.22 Mbps in 1.68 usec
358
+ 5: 8 bytes 29729 times --> 36.17 Mbps in 1.69 usec
359
+ 6: 12 bytes 37039 times --> 54.01 Mbps in 1.70 usec
360
+ 7: 13 bytes 24578 times --> 58.40 Mbps in 1.70 usec
361
+ 8: 16 bytes 27177 times --> 71.87 Mbps in 1.70 usec
362
+ 9: 19 bytes 33116 times --> 85.00 Mbps in 1.71 usec
363
+
364
+ At the end of the experiment you can use the command `hist` to see what you have done so far.
365
+ This can help you to assemble everything together in a whole script.
366
+
367
+ [22] pry(main)> hist
368
+ 1: edit -n find_infiniband.rb
369
+ 3: play find_infiniband.rb
370
+ 4: sites_infiniband
371
+ 5: .ls ~/my_ssh*
372
+ 6: .scp ~/my_ssh* nancy:~/
373
+ 7: job = $g5k.reserve(:site => "nancy", :resources => "{ib20g='YES'}/nodes=2/core=1",:walltime => '01:00:00', :keys => "~/my_ssh_jobkey" )
374
+ 8: nodes = job["assigned_nodes"]
375
+ 9: machine_file = Tempfile.open('machine_file')
376
+ 10: nodes.each{ |node| machine_file.puts node }
377
+ 11: machine_file.close
378
+ 12: grid5000_opt = OARSSHopts.new(:keys => "~/my_ssh_jobkey")
379
+ 13: require 'net/scp'
380
+ 14: Net::SCP.start(nodes.first, "oar", grid5000_opt) do |scp|
381
+ 15: scp.upload! machine_file.path, "/tmp/machine_file"
382
+ 16: end
383
+ 17: edit -n netpipe.rb
384
+ 18: play netpipe.rb
385
+ 19: edit -n netpipe.rb
386
+ 20: play netpipe.rb
387
+
388
+ ## Running NAS benchmarks in Grid'5000: getting acquainted with parallel command execution
389
+
390
+ In this experiment, we will run the NAS benchmark in Grid'5000 and we will script a scalability test for one of the benchmarks.
391
+ This experiment has the following requirements:
392
+
393
+ - 4 or 2 nodes from any Grid'5000 sites
394
+ - Use of standard environment (no deploy)
395
+ - NAS MPI behchmark
396
+ - A MPI runtime (OpenMPI or MPICH)
397
+
398
+ If you have not created a directory for the tutorial, create it and execute the `pry` console from there:
399
+
400
+ $ mkdir ruby-cute-tutorial
401
+ $ cd ruby-cute-tutorial
402
+ $ cute
403
+
404
+ First, let's find the necessary nodes for our experiment. As resources in Grid'5000 could be very busy, we are going
405
+ to script a loop that will explore all Grid'5000 sites and find the first site that can provide us with the required nodes.
406
+ Open an editor form `pry` console:
407
+
408
+ [5] pry(main)> edit -n find_nodes.rb
409
+
410
+ and type the following code:
411
+
412
+ sites = $g5k.site_uids
413
+ job = {}
414
+
415
+ sites.each do |site|
416
+ job = $g5k.reserve(:site => site, :nodes => 4, :wait => false, :walltime => "01:00:00")
417
+ begin
418
+ job = $g5k.wait_for_job(job, :wait_time => 60)
419
+ puts "Nodes assigned #{job['assigned_nodes']}"
420
+ break
421
+ rescue Cute::G5K::EventTimeout
422
+ puts "We waited too long in site #{site} let's release the job and try in another site"
423
+ $g5k.release(job)
424
+ end
425
+ end
426
+
427
+ Here, we use the method {Cute::G5K::API#site_uids site_uids} for getting all available sites.
428
+ Then, a job is submitted using the method {Cute::G5K::API#reserve reserve}.
429
+ We ask for 4 nodes in a given site and we set the parameter `wait` to false which makes the method to return immediately.
430
+ Then, we use {Cute::G5K::API#wait_for_job wait_for_job} to set a timeout. If the timeout is reached a {Cute::G5K::EventTimeout Timeout} exception will be triggered
431
+ that we catch in order to consequently release the submitted job and try to submit it in another site.
432
+ Let's execute the script using `play` command:
433
+
434
+ [8] pry(main)> play find_nodes.rb
435
+ 2015-12-08 16:50:35.582 => Reserving resources: /nodes=4,walltime=01:00 (type: ) (in grenoble)
436
+ 2015-12-08 16:50:36.465 => Waiting for reservation 1702197
437
+ 2015-12-08 16:50:41.587 => Reservation 1702197 should be available at 2015-12-08 16:50:37 +0100 (0 s)
438
+ 2015-12-08 16:50:41.587 => Reservation 1702197 ready
439
+ Nodes assigned ["edel-10.grenoble.grid5000.fr", "edel-11.grenoble.grid5000.fr", "edel-12.grenoble.grid5000.fr", "edel-13.grenoble.grid5000.fr"]
440
+ => nil
441
+
442
+ The variable `job` is updated in Pry context. When no keys are specified, the option *-allow_classic_ssh* is activated
443
+ which enables the access via default SSH to the reserved machines. You can verify it by doing:
444
+
445
+ [11] pry(main)> .ssh edel-11.grenoble.grid5000.fr "hostname"
446
+ Warning: Permanently added 'edel-11.grenoble.grid5000.fr,172.16.17.11' (RSA) to the list of known hosts.
447
+ edel-11.grenoble.grid5000.fr
448
+
449
+ Let's explore the available modules for the parallel execution of commands in several remote machines.
450
+ The following example shows how to use the {Cute::TakTuk TakTuk} module.
451
+
452
+ nodes = job["assigned_nodes"]
453
+ Cute::TakTuk.start(nodes) do |tak|
454
+ tak.exec("hostname")
455
+ end
456
+
457
+ Which generates as output:
458
+
459
+ edel-10.grenoble.grid5000.fr/output/0:edel-10.grenoble.grid5000.fr
460
+ edel-12.grenoble.grid5000.fr/output/0:edel-12.grenoble.grid5000.fr
461
+ edel-13.grenoble.grid5000.fr/output/0:edel-13.grenoble.grid5000.fr
462
+ edel-10.grenoble.grid5000.fr/status/0:0
463
+ edel-11.grenoble.grid5000.fr/output/0:edel-11.grenoble.grid5000.fr
464
+ edel-12.grenoble.grid5000.fr/status/0:0
465
+ edel-13.grenoble.grid5000.fr/status/0:0
466
+ edel-11.grenoble.grid5000.fr/status/0:0
467
+
468
+ The following example shows how to use the {Net::SSH::Multi Net::SSH::Multi} module.
469
+
470
+ Net::SSH::Multi.start do |session|
471
+ nodes.each{ |node| session.use node }
472
+ session.exec("hostname")
473
+ end
474
+
475
+ If we type that into pry console we will get:
476
+
477
+ [edel-10.grenoble.grid5000.fr] edel-10.grenoble.grid5000.fr
478
+ [edel-11.grenoble.grid5000.fr] edel-11.grenoble.grid5000.fr
479
+ [edel-12.grenoble.grid5000.fr] edel-12.grenoble.grid5000.fr
480
+ [edel-13.grenoble.grid5000.fr] edel-13.grenoble.grid5000.fr
481
+
482
+ It is possible to capture the output of the executed command by adding **!** to exec method.
483
+ For example, let's find the number of cores available in the reserved machines:
484
+
485
+ results = {}
486
+ Net::SSH::Multi.start do |session|
487
+ nodes.each{ |node| session.use node }
488
+ results = session.exec!("nproc")
489
+ end
490
+
491
+ The {Net::SSH::Multi::SessionActions#exec! exec!} method will return a Hash that looks like this:
492
+
493
+ [27] pry(main)> results
494
+ => {"edel-10.grenoble.grid5000.fr"=>{:stdout=>"8", :status=>0},
495
+ "edel-11.grenoble.grid5000.fr"=>{:stdout=>"8", :status=>0},
496
+ "edel-12.grenoble.grid5000.fr"=>{:stdout=>"8", :status=>0},
497
+ "edel-13.grenoble.grid5000.fr"=>{:stdout=>"8", :status=>0}}
498
+
499
+ Where the Hash keys are the names of the machines and the values correspond to the output of the commands.
500
+ Then, we can easily get the total number of cores by typing:
501
+
502
+ [11] pry(main)> num_cores = results.values.inject(0){ |sum, item| sum+item[:stdout].to_i}
503
+ => 32
504
+
505
+ Another way to do that is to use the information given by the G5K API regarding the submitted job:
506
+
507
+ [36] pry(main)> job["resources_by_type"]["cores"].length
508
+ => 32
509
+
510
+ Let's create a machine file that we will need later on for our experiments:
511
+
512
+ machine_file = Tempfile.open('machine_file')
513
+ nodes.each{ |node| machine_file.puts node }
514
+ machine_file.close
515
+
516
+ After creating the machine file, we need to send it to the other machines.
517
+ Additionally, we need to download and compile the benchmark.
518
+ Let's write a small script that help us to perform the aforementioned tasks.
519
+ Open the editor in pry console:
520
+
521
+ [17] pry(main)> edit -n NAS-expe.rb
522
+
523
+ Then, type:
524
+
525
+ SOURCE_NAS = "http://public.rennes.grid5000.fr/~cruizsanabria/NPB3.3.tar"
526
+
527
+ `wget #{SOURCE_NAS} -O /tmp/NAS.tar`
528
+
529
+ Cute::TakTuk.start(nodes) do |tak|
530
+
531
+ tak.put(machine_file.path, "machine_file")
532
+ tak.put("/tmp/NAS.tar", "/tmp/NAS.tar")
533
+
534
+ tak.exec!("cd /tmp/; tar -xvf NAS.tar")
535
+ puts tak.exec!("make lu NPROCS=#{num_cores} CLASS=A MPIF77=mpif77 -C /tmp/NPB3.3/NPB3.3-MPI/")
536
+
537
+ end
538
+
539
+ We can observe in the previous snippet of code that {Cute::TakTuk TakTuk} module can be used to
540
+ transfer files to several remote nodes. {Cute::TakTuk::TakTuk#put put} and {Cute::TakTuk::TakTuk#exec exec} methods
541
+ can be used in the same block. Finally, execute the script:
542
+
543
+ [102] pry(main)> play NAS-expe.rb
544
+
545
+ We can check if each node has the generated binary and the machine file:
546
+
547
+ Net::SSH::Multi.start do |session|
548
+ nodes.each{ |node| session.use node }
549
+ session.exec("ls /tmp/NPB3.3/NPB3.3-MPI/bin/")
550
+ session.exec("ls ~/machine*")
551
+ end
552
+
553
+ After typing it into `pry` console we will get something like:
554
+
555
+ [genepi-27.grenoble.grid5000.fr] lu.A.32
556
+ [genepi-29.grenoble.grid5000.fr] lu.A.32
557
+ [genepi-29.grenoble.grid5000.fr] /home/cruizsanabria/machine_file
558
+ [genepi-19.grenoble.grid5000.fr] lu.A.32
559
+ [genepi-19.grenoble.grid5000.fr] /home/cruizsanabria/machine_file
560
+ [genepi-2.grenoble.grid5000.fr] lu.A.32
561
+ [genepi-2.grenoble.grid5000.fr] /home/cruizsanabria/machine_file
562
+ [genepi-27.grenoble.grid5000.fr] /home/cruizsanabria/machine_file
563
+
564
+ Which confirms the presence of both files on the nodes.
565
+ We can get the path of the binary by typing the following into `pry` console.
566
+
567
+ Net::SSH::Multi.start do |session|
568
+ nodes.each{ |node| session.use node }
569
+ results = session.exec!("find /tmp/ -name lu.A.32")
570
+ end
571
+
572
+ We will get some errors caused by the `find` command:
573
+
574
+ [32] pry(main)> results
575
+ => {"genepi-27.grenoble.grid5000.fr"=>{:stdout=>"/tmp/NPB3.3/NPB3.3-MPI/bin/lu.A.32", :stderr=>": Permission denied", :status=>1},
576
+ "genepi-29.grenoble.grid5000.fr"=>{:stdout=>"/tmp/NPB3.3/NPB3.3-MPI/bin/lu.A.32", :stderr=>": Permission denied", :status=>1},
577
+ "genepi-19.grenoble.grid5000.fr"=>{:stderr=>": Permission denied", :stdout=>"/tmp/NPB3.3/NPB3.3-MPI/bin/lu.A.32", :status=>1},
578
+ "genepi-2.grenoble.grid5000.fr"=>{:stdout=>"/tmp/NPB3.3/NPB3.3-MPI/bin/lu.A.32", :stderr=>": Permission denied", :status=>1}}
579
+
580
+
581
+ Then, we can assign this to a new variable:
582
+
583
+ [33] pry(main)> lu_path = results.values.first[:stdout]
584
+ => "/tmp/NPB3.3/NPB3.3-MPI/bin/lu.A.32"
585
+
586
+ The setup of the experiment is done. It is time to execute the benchmark by typing the following into `pry` console.
587
+
588
+ Net::SSH.start(nodes.first,"cruizsanabria") do |ssh|
589
+ results = ssh.exec!("mpirun --mca btl self,sm,tcp -np 32 --machinefile machine_file #{lu_path}")
590
+ end
591
+
592
+ Let's now perform a scalability test of the LU application for 2, 4, 8, 16, 32 processes. Open the editor:
593
+
594
+ [100] pry(main)> edit -n scalability_NAS.rb
595
+
596
+ And copy-paste the following script:
597
+
598
+ num_cores = [2,4,8,16,32]
599
+
600
+ Cute::TakTuk.start(nodes) do |tak|
601
+
602
+ num_cores.each do |cores|
603
+ puts tak.exec!("make lu NPROCS=#{cores} CLASS=A MPIF77=mpif77 -C /tmp/NPB3.3/NPB3.3-MPI/")
604
+ end
605
+
606
+ results = tak.exec!("find /tmp/ -name lu.A.*")
607
+ end
608
+
609
+ binaries = results.values.first[:output].split("\n")
610
+
611
+ expe_res = {}
612
+
613
+ Net::SSH.start(nodes.first,"cruizsanabria") do |ssh|
614
+ binaries.each do |binary|
615
+ processes = /A\.(\d*)/.match(binary)[1]
616
+ expe_res[processes]= {}
617
+ result = ssh.exec!("mpirun --mca btl self,sm,tcp -np #{processes} --machinefile machine_file #{binary}")
618
+ expe_res[processes][:output]= result
619
+ expe_res[processes][:time] =result.split("\n").select{ |t| t["Time in"]}.first
620
+ end
621
+ end
622
+
623
+ Then, we execute it:
624
+
625
+ [102] pry(main)> play scalability_NAS.rb
626
+
627
+ It will take approximately 2 ~ 3 minutes to run. After finishing a new Hash will
628
+ be defined called *expe_res* that we can use to print the results:
629
+
630
+ num_cores.each{ |cores| puts "#{cores} cores: #{expe_res[cores.to_s][:time]}"}
631
+
632
+ It will generate:
633
+
634
+ [107] pry(main)> num_cores.each{ |cores| puts "#{cores} cores: #{expe_res[cores.to_s][:time]}"}
635
+ 2 cores: Time in seconds = 42.93
636
+ 4 cores: Time in seconds = 26.50
637
+ 8 cores: Time in seconds = 12.39
638
+ 16 cores: Time in seconds = 7.01
639
+ 32 cores: Time in seconds = 6.00
640
+
641
+ Finally, we can use the command `hist` to try to assemble all we have done so far into a script.
642
+ Once finished, we could release the job:
643
+
644
+ [34] pry(main)> $g5k.release(job)
645
+ => ""
646
+
647
+ ## Performing network measurements within a reserved VLAN
648
+
649
+ In this experiment, we will perform network measurements between two nodes located in different Grid'5000 sites.
650
+ The network measurements will be carried out in an isolated VLAN.
651
+ We will first reserved two nodes located in two different Grid'5000 sites in deploy mode and we will ask for two routed VLANs.
652
+ Once the nodes are ready an environment will be deployed and the application iperf will be install in all nodes.
653
+ Then, we will perform some network measurements among the nodes.
654
+ Finally, we will query the KWAPI using the G5K metrology API to get the network traffic generated during our experiment.
655
+
656
+ This experiment has the following requirements:
657
+
658
+ - Two nodes in two different G5K sites
659
+ - Environment deployment
660
+ - VLAN reservation
661
+ - Iperf application
662
+ - Access to Network traffic data.
663
+
664
+ If you have not created a directory for the tutorial, create it and execute the `pry` console from there:
665
+
666
+ $ mkdir ruby-cute-tutorial
667
+ $ cd ruby-cute-tutorial
668
+ $ cute
669
+
670
+ Let's create a small script that will help us with the reservation of nodes.
671
+ Open the `pry` editor:
672
+
673
+ [35] pry(main)> edit -n multisite.rb
674
+
675
+ and type:
676
+
677
+ jobs = {}
678
+ threads = []
679
+ ["nancy","rennes"].each do |site|
680
+
681
+ threads.push<< Thread.new do
682
+
683
+ jobs[site] = job = $g5k.reserve(:site => site, :nodes => 1,
684
+ :env => 'jessie-x64-min',
685
+ :vlan => :routed)
686
+ end
687
+ end
688
+
689
+ threads.each{ |t| t.join}
690
+
691
+
692
+ In the script, we have chosen Nancy and Rennes sites. You are encouraged to try other sites as the number of routed VLANs is limited in each site.
693
+ For the purpose of this tutorial you have to choose a site where Kwapi is available: Grenoble, Nancy, Rennes, Lyon, Nantes.
694
+ We use the method {Cute::G5K::API#reserve reserve} with parameter *env* for specifying the environment we want to deploy.
695
+ This will automatically submit a deploy job and it will deploy the specified environment.
696
+ The parameter *vlan* will additionally reserve a VLAN and pass it to Kadeploy to setup the VLAN.
697
+ After executing this small script we got:
698
+
699
+ [36] pry(main)> play multisite.rb
700
+ 2016-01-20 12:48:15.010 => Reserving resources: {type='kavlan'}/vlan=1+/nodes=1,walltime=01:00 (type: deploy) (in nancy)
701
+ 2016-01-20 12:48:15.010 => Reserving resources: {type='kavlan'}/vlan=1+/nodes=1,walltime=01:00 (type: deploy) (in rennes)
702
+ 2016-01-20 12:48:16.145 => Waiting for reservation 740698
703
+ 2016-01-20 12:48:16.246 => Waiting for reservation 802917
704
+ 2016-01-20 12:48:21.270 => Reservation 740698 should be available at 2016-01-20 12:48:17 +0100 (0 s)
705
+ 2016-01-20 12:48:26.344 => Reservation 740698 should be available at 2016-01-20 12:48:17 +0100 (0 s)
706
+ 2016-01-20 12:48:26.404 => Reservation 802917 should be available at 2016-01-20 12:48:13 +0100 (0 s)
707
+ 2016-01-20 12:48:26.404 => Reservation 802917 ready
708
+ 2016-01-20 12:48:26.541 => Found VLAN with uid = 4
709
+ 2016-01-20 12:48:26.541 => Creating deployment
710
+ 2016-01-20 12:48:27.256 => Waiting for 1 deployment
711
+ 2016-01-20 12:48:31.296 => Waiting for 1 deployment
712
+ 2016-01-20 12:48:31.406 => Reservation 740698 should be available at 2016-01-20 12:48:17 +0100 (0 s)
713
+ 2016-01-20 12:48:31.406 => Reservation 740698 ready
714
+ 2016-01-20 12:48:31.469 => Found VLAN with uid = 4
715
+ 2016-01-20 12:48:31.469 => Creating deployment
716
+ 2016-01-20 12:48:31.869 => Waiting for 1 deployment
717
+ 2016-01-20 12:48:35.414 => Waiting for 1 deployment
718
+
719
+ At the end of the process the variable `jobs` will be defined and it will contain the jobs' information in each site.
720
+ In this variable, we can find information related with the deployment.
721
+
722
+ [44] pry(main)> jobs["nancy"]["deploy"]
723
+ => [{"created_at"=>1450439620,
724
+ "environment"=>"jessie-x64-min",
725
+ "key"=>"https://api.grid5000.fr/sid/sites/nancy/files/cruizsanabria-key-84f3f1dbb1279bc1bddcd618e26c960307d653c5",
726
+ "nodes"=>["graphite-4.nancy.grid5000.fr"],
727
+ "result"=>{"graphite-4.nancy.grid5000.fr"=>{"macro"=>nil, "micro"=>nil, "state"=>"OK"}},
728
+ "site_uid"=>"nancy",
729
+ "status"=>"terminated",
730
+ "uid"=>"D-b026879e-b185-4e20-8bc5-ea0842a6954b",
731
+ "updated_at"=>1450439860,
732
+ "user_uid"=>"cruizsanabria",
733
+ "vlan"=>14,
734
+ "links"=>
735
+ [{"rel"=>"self", "href"=>"/sid/sites/nancy/deployments/D-b026879e-b185-4e20-8bc5-ea0842a6954b", "type"=>"application/vnd.grid5000.item+json"},
736
+ {"rel"=>"parent", "href"=>"/sid/sites/nancy", "type"=>"application/vnd.grid5000.item+json"}]}]
737
+
738
+ Some important information are: the status of the whole process and the state per node.
739
+ We can use this information to check if the deployment have finished successfully in all nodes.
740
+ This data structure is used by the method {Cute::G5K::API#check_deployment check_deployment}.
741
+ Let's check the documentation of this method:
742
+
743
+ [16] pry(main)> show-doc Cute::G5K::API#check_deployment
744
+
745
+ From: /home/cruizsanabria/Repositories/ruby-cute/lib/cute/g5k_api.rb @ line 1198:
746
+ Owner: Cute::G5K::API
747
+ Visibility: public
748
+ Signature: check_deployment(deploy_info)
749
+ Number of lines: 10
750
+
751
+ It returns an array of machines that did not deploy successfully
752
+ = Example
753
+ It can be used to try a new deploy:
754
+
755
+ badnodes = g5k.check_deployment(job["deploy"].last)
756
+ g5k.deploy(job,:nodes => badnodes, :env => 'wheezy-x64-base')
757
+ g5k.wait_for_deploy(job)
758
+
759
+ return [Array] machines that did not deploy successfully
760
+ param deploy_info [Hash] deployment structure information
761
+
762
+ We can use this method with the jobs we have just submitted
763
+ (The output will be probably long, so you will need to scroll up to see what it is shown here):
764
+
765
+ [47] pry(main)> jobs.each{ |site,job| puts "all nodes OK in site: #{site}" if $g5k.check_deployment(job["deploy"].last).empty?}
766
+ all nodes OK in site: rennes
767
+ all nodes OK in site: nancy
768
+
769
+ Now, the reserved nodes are in a VLAN; within this VLAN a DHCP server will assign new IP addresses to the nodes.
770
+ You can configure your own if you want (please refer to {https://www.grid5000.fr/mediawiki/index.php/Network_isolation_on_Grid%275000 KVLAN tutorial}
771
+ if you want to know more). We can get the new assigned names by doing:
772
+
773
+ nodes = []
774
+ jobs.each{ |site,job| nodes.push($g5k.get_vlan_nodes(job))}
775
+
776
+ After putting that into `pry` we will get something like this:
777
+
778
+ [50] pry(main)> nodes
779
+ => [["paranoia-6-kavlan-16.rennes.grid5000.fr"], ["graphite-4-kavlan-14.nancy.grid5000.fr"]]
780
+
781
+ [51] pry(main)> nodes.flatten
782
+ => ["paranoia-6-kavlan-16.rennes.grid5000.fr", "graphite-4-kavlan-14.nancy.grid5000.fr"]
783
+
784
+ Now, let's install `iperf` application in order to perform our network measurements.
785
+ Copy-paste the following code into `pry`:
786
+
787
+ nodes = nodes.flatten
788
+
789
+ Net::SSH::Multi.start do |session|
790
+ nodes.each{ |node| session.use("root@#{node}") }
791
+ session.exec!("apt-get update")
792
+ session.exec("DEBIAN_FRONTEND=noninteractive apt-get install -q -y iperf")
793
+ end
794
+
795
+ You should get something like this:
796
+
797
+ [paranoia-6-kavlan-16.rennes.grid5000.fr] Reading package lists...
798
+ [paranoia-6-kavlan-16.rennes.grid5000.fr] Building dependency tree...
799
+ [graphite-4-kavlan-14.nancy.grid5000.fr] Reading package lists...
800
+ [graphite-4-kavlan-14.nancy.grid5000.fr] Building dependency tree...
801
+ [paranoia-6-kavlan-16.rennes.grid5000.fr]
802
+ [paranoia-6-kavlan-16.rennes.grid5000.fr] Reading state information...
803
+ [paranoia-6-kavlan-16.rennes.grid5000.fr] The following NEW packages will be installed:
804
+ [paranoia-6-kavlan-16.rennes.grid5000.fr] iperf
805
+ [paranoia-6-kavlan-16.rennes.grid5000.fr] 0 upgraded, 1 newly installed, 0 to remove and 8 not upgraded.
806
+ [paranoia-6-kavlan-16.rennes.grid5000.fr] Need to get 51.4 kB of archives.
807
+ [paranoia-6-kavlan-16.rennes.grid5000.fr] After this operation, 179 kB of additional disk space will be used.
808
+ [paranoia-6-kavlan-16.rennes.grid5000.fr] Get:1 http://ftp.debian.org/debian/ jessie/main iperf amd64 2.0.5+dfsg1-2 [51.4 kB]
809
+ [graphite-4-kavlan-14.nancy.grid5000.fr]
810
+ [graphite-4-kavlan-14.nancy.grid5000.fr] Reading state information...
811
+ [graphite-4-kavlan-14.nancy.grid5000.fr] The following NEW packages will be installed:
812
+ [graphite-4-kavlan-14.nancy.grid5000.fr] iperf
813
+
814
+
815
+ You can check if the application has been successfully installed,
816
+ by typing the following into the `pry` console:
817
+
818
+ Net::SSH::Multi.start do |session|
819
+ nodes.each{ |node| session.use("root@#{node}") }
820
+ session.exec("iperf --version")
821
+ end
822
+
823
+ Which will generate:
824
+
825
+ [65] pry(main)* end
826
+ [paranoia-6-kavlan-16.rennes.grid5000.fr] iperf version 2.0.5 (08 Jul 2010) pthreads
827
+ [graphite-4-kavlan-14.nancy.grid5000.fr] iperf version 2.0.5 (08 Jul 2010) pthreads
828
+ => nil
829
+
830
+ Let's perform some iperf tests, let's write a small script.
831
+ Open the editor:
832
+
833
+ [76] pry(main)> edit -n iperf_test.rb
834
+
835
+ and type:
836
+
837
+ results = {}
838
+
839
+ Net::SSH::Multi.start do |session|
840
+
841
+ session.group :server do
842
+ session.use("root@#{nodes[0]}")
843
+ end
844
+
845
+ session.group :client do
846
+ session.use("root@#{nodes[1]}")
847
+ end
848
+
849
+ session.with(:server).exec("iperf -s &")
850
+
851
+ #bandwith
852
+
853
+ results[:bandwidth]= session.with(:client).exec!("iperf -c #{nodes[0]}")
854
+
855
+ # bi-directional bandwidth measurement
856
+
857
+ results[:bidi]= session.with(:client).exec!("iperf -c #{nodes[0]} -r")
858
+
859
+ # TCP windows size
860
+ results[:window]= session.with(:client).exec!("iperf -c #{nodes[0]} -w 2000")
861
+
862
+ # shutdown server
863
+
864
+ session.with(:server).exec("skill iperf")
865
+ end
866
+
867
+ Then, if we execute it with the `play` command:
868
+
869
+
870
+ [77] pry(main)> play iperf_test.rb
871
+ [paranoia-6-kavlan-16.rennes.grid5000.fr] ------------------------------------------------------------
872
+ [paranoia-6-kavlan-16.rennes.grid5000.fr] Server listening on TCP port 5001
873
+ [paranoia-6-kavlan-16.rennes.grid5000.fr] TCP window size: 85.3 KByte (default)
874
+ [paranoia-6-kavlan-16.rennes.grid5000.fr] ------------------------------------------------------------
875
+ [paranoia-6-kavlan-16.rennes.grid5000.fr] [ 4] local 10.27.204.71 port 5001 connected with 10.19.200.240 port 32769
876
+ [paranoia-6-kavlan-16.rennes.grid5000.fr] [ ID] Interval Transfer Bandwidth
877
+ [paranoia-6-kavlan-16.rennes.grid5000.fr] [ 4] 0.0-10.0 sec 1.12 GBytes 957 Mbits/sec
878
+ [paranoia-6-kavlan-16.rennes.grid5000.fr] [ 5] local 10.27.204.71 port 5001 connected with 10.19.200.240 port 32770
879
+ [paranoia-6-kavlan-16.rennes.grid5000.fr] [ 5] 0.0-10.0 sec 1.10 GBytes 947 Mbits/sec
880
+ [paranoia-6-kavlan-16.rennes.grid5000.fr] ------------------------------------------------------------
881
+ [paranoia-6-kavlan-16.rennes.grid5000.fr] Client connecting to 10.19.200.240, TCP port 5001
882
+ [paranoia-6-kavlan-16.rennes.grid5000.fr] TCP window size: 85.0 KByte (default)
883
+ [paranoia-6-kavlan-16.rennes.grid5000.fr] ------------------------------------------------------------
884
+ [paranoia-6-kavlan-16.rennes.grid5000.fr] [ 5] local 10.27.204.71 port 47604 connected with 10.19.200.240 port 5001
885
+ [paranoia-6-kavlan-16.rennes.grid5000.fr] [ 5] 0.0-10.0 sec 1.12 GBytes 958 Mbits/sec
886
+ [paranoia-6-kavlan-16.rennes.grid5000.fr] [ 4] local 10.27.204.71 port 5001 connected with 10.19.200.240 port 32771
887
+ [paranoia-6-kavlan-16.rennes.grid5000.fr] [ 4] 0.0-10.7 sec 2.25 MBytes 1.77 Mbits/sec
888
+
889
+
890
+ The variable `results` will be defined which contains the results for each test.
891
+ Let's print the results. Type the following into the `pry` console:
892
+
893
+ results.each do |test, res|
894
+ puts "Results of test: #{test}"
895
+ res.each { |node,r| puts r[:stdout]}
896
+ end
897
+
898
+ Which will give us:
899
+
900
+ Results of test: bandwidth
901
+ [ ID] Interval Transfer Bandwidth
902
+ [ 3] 0.0-10.0 sec 1.12 GBytes 958 Mbits/sec
903
+ Results of test: bidi
904
+ [ 4] 0.0-10.0 sec 1.12 GBytes 957 Mbits/sec
905
+ Results of test: window
906
+ [ ID] Interval Transfer Bandwidth
907
+ [ 3] 0.0-10.7 sec 2.25 MBytes 1.77 Mbits/sec
908
+
909
+
910
+ Now let's look at the network traffic that we have generated during our experiment using KWAPI.
911
+ **Ruby-cute** offers the {Cute::G5K::API#get_metric get_metric} method to consult the G5K Metrology API.
912
+ In order to carry out a query and get the values of a specific probe,
913
+ we have to know the time interval of the values and the name of the probe.
914
+ Let's get the values for the metric `network_in`.
915
+ We could get all the names of the probes specific to this metric by typing:
916
+
917
+ probes = $g5k.get_metric("rennes",:metric => "network_in").uids
918
+
919
+ If you type that in `pry` you will get:
920
+
921
+ [13] pry(main)> probes
922
+ => ["parasilo-11-eth0",
923
+ "parasilo-11-eth1",
924
+ "paravance-48-eth1",
925
+ "paravance-48-eth0",
926
+ "paravance-2-eth0",
927
+ "paravance-2-eth1",
928
+ "paranoia-4",
929
+ "paranoia-5",
930
+ "paranoia-6",
931
+ "paranoia-7",
932
+ "paravance-72-eth0",
933
+
934
+ In order to choose the right probes, we need to get the real names of the machines
935
+ and not the ones assigned by the VLAN. We can consult the job information:
936
+
937
+ nodes_normal = []
938
+ jobs.each{ |site,job| nodes_normal.push(job["assigned_nodes"])}
939
+
940
+ Which will give us an Array of Arrays that we can flatten by doing:
941
+
942
+ [97] pry(main)> nodes_normal.flatten!
943
+ => [["paranoia-6.rennes.grid5000.fr"], ["graphite-4.nancy.grid5000.fr"]]
944
+
945
+ As we are going to fetch the data for Rennes (First node). We could do:
946
+
947
+ [68] pry(main)> probe_expe = probes.select{ |p| p[nodes_normal[0].split(".")[0]]}
948
+
949
+ So, at this point we already have the probe we want to request.
950
+ Next step is to get the start time of the interval, we can choose for example, the time at which deployments have finished:
951
+
952
+ deploy_end = []
953
+ jobs.each{ |site,job| deploy_end.push(job["deploy"].last["updated_at"])}
954
+
955
+ Therefore, we could choose the maximum timestamp from the ones returned:
956
+
957
+ start = deploy_end.max
958
+
959
+ Now, we can proceed by performing the query:
960
+
961
+ $g5k.get_metric("rennes",:metric => "network_in",:query => {:from => start, :to => start+3600, :only => probe_expe.first})
962
+
963
+ An Array is returned. We can then open an editor and write a small script that will write these values into a file
964
+
965
+ [33] pry(main)> edit -n get_results.rb
966
+
967
+ type:
968
+
969
+ raw_data = $g5k.get_metric("rennes",:metric => "network_in",
970
+ :query => {:from => start, :to => start+3600, :only => probe_expe.first})
971
+
972
+ network_in = raw_data.map{ |r| r["values"]}.flatten
973
+ time = raw_data.map{ |r| r["timestamps"]}.flatten
974
+
975
+ values = Hash[time.zip(network_in)]
976
+
977
+ File.open("network_in-values.txt",'w+') do |f|
978
+ f.puts("time\t bytes")
979
+ values.each{ |k,v| f.puts("#{k}\t#{v}")}
980
+ end
981
+
982
+ and execute it with:
983
+
984
+ pry(main)> play get_results.rb
985
+ => {1453293153.732378=>4498405298908,
986
+ 1453293155.183099=>4498405298908,
987
+ 1453293156.582115=>4498405298908,
988
+ 1453293157.924968=>4498405298908,
989
+ 1453293159.28666=>4498405298908,
990
+ 1453293160.655534=>4498405298908,
991
+ 1453293161.998718=>4498405299219,
992
+
993
+ ## Conclusions
994
+
995
+ This tutorial has shown how the scripting of complex experiment can be done using the Ruby scripting language.
996
+ We saw that in the context of Grid'5000,
997
+ **Ruby-Cute** offers useful methods for accessing the platform's services and executing commands in parallel.
998
+ The aim of this tutorial was to give you some ideas for coding your experiments using **Ruby-Cute**
999
+ and we hope it will be useful for your experiments.