ruby-cute 0.4 → 0.5
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/debian/changelog +6 -0
- data/examples/g5k-tutorial.md +48 -14
- data/lib/cute/g5k_api.rb +70 -47
- data/lib/cute/taktuk.rb +60 -2
- data/lib/cute/version.rb +1 -1
- data/spec/g5k_api_check_spec.rb +63 -0
- data/spec/g5k_api_spec.rb +12 -1
- metadata +5 -3
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 1bcde24cec062bee7628aff6089b9a146c46be33
|
4
|
+
data.tar.gz: 4398bc96b38dca6b342118828649b766dcb1550b
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 97d07be46381ca5ccca6c531eba4f8bbef52038ea0cce3d0acba250cd8d7505c5ef8f8ff7311b19d979cff04bf7c9f4f96b9c2c4175ad0852ae8454dcfaf2639
|
7
|
+
data.tar.gz: 960f749e351f01142c235620df0c7cef5d210dfbf620ac2900939c3a04f7e997c639315f9941d5b6e7a79e0b4427474f37839a0aefa3d5f85976dc78df6b5052
|
data/debian/changelog
CHANGED
data/examples/g5k-tutorial.md
CHANGED
@@ -109,7 +109,10 @@ file with the following content in order to choose our prefered editor:
|
|
109
109
|
|
110
110
|
Here, we will use **Ruby-cute** to carry out an experiment.
|
111
111
|
In this experiment, we will ask for two nodes equipped with infiniband and
|
112
|
-
then, we will perform some performance
|
112
|
+
then, we will perform some performance tests using a network benchmark called
|
113
|
+
[NETPIPE](http://bitspjoule.org/netpipe/).
|
114
|
+
NETPIPE performs simple ping-pong tests, bouncing messages of increasing size between two processes.
|
115
|
+
Message sizes are chosen at regular intervals, and with slight perturbations, to provide a complete test of the communication system.
|
113
116
|
For this particular experiment we have the following requirements:
|
114
117
|
|
115
118
|
- A pair of SSH keys
|
@@ -148,7 +151,6 @@ ruby script:
|
|
148
151
|
Then, we execute it using the `play` command which will execute line by line this script in the context of a Pry session.
|
149
152
|
|
150
153
|
[21] pry(main)> play find_infiniband.rb
|
151
|
-
=> ["grenoble", "lille", "luxembourg", "lyon", "nancy", "nantes", "reims", "rennes", "sophia"]
|
152
154
|
|
153
155
|
We can observe that the variable `sites_infiniband` is now defined, telling us that Grenoble and Nancy sites offer Infiniband interconnection.
|
154
156
|
|
@@ -157,19 +159,20 @@ We can observe that the variable `sites_infiniband` is now defined, telling us t
|
|
157
159
|
|
158
160
|
Then, create a pair of SSH keys (Necessary for OARSSH):
|
159
161
|
|
160
|
-
|
162
|
+
[23] pry(main)> .ssh-keygen -b 1024 -N "" -t rsa -f ~/my_ssh_jobkey
|
161
163
|
|
162
|
-
We send the generated keys to the chosen site
|
164
|
+
We send the generated keys to the chosen site (ssh configuration has be set up for the following command to work,
|
165
|
+
see [SSH Configuration](https://www.grid5000.fr/mediawiki/index.php/SSH_and_Grid%275000) for more information):
|
163
166
|
|
164
|
-
[
|
167
|
+
[24] pry(main)> .scp ~/my_ssh* nancy:~/
|
165
168
|
|
166
169
|
Now that we have found the sites, let's submit a job. You can use between Grenoble and Nancy sites. If you
|
167
170
|
take a look at {https://www.grid5000.fr/mediawiki/index.php/Status Monika} you will see that in Nancy we should use the OAR property 'ib20g' and in Grenoble we should use 'ib10g'.
|
168
171
|
Given that the MPI bench uses just one MPI process, we will need in realty just one core of a given machine.
|
169
172
|
We will use OAR syntax to ask for two cores in two different nodes with ib10g in Grenoble.
|
170
173
|
|
171
|
-
[
|
172
|
-
2015-12-04 14:07:31.370 => Reserving resources: {ib20g='YES'}/nodes=2/core=1,walltime=01:00 (type: ) (in
|
174
|
+
[25] pry(main)> job = $g5k.reserve(:site => "nancy", :resources => "{ib20g='YES'}/nodes=2/core=1",:walltime => '01:00:00', :keys => "~/my_ssh_jobkey" )
|
175
|
+
2015-12-04 14:07:31.370 => Reserving resources: {ib20g='YES'}/nodes=2/core=1,walltime=01:00 (type: ) (in nancy)
|
173
176
|
2015-12-04 14:07:41.358 => Waiting for reservation 692665
|
174
177
|
2015-12-04 14:07:41.444 => Reservation 692665 should be available at 2015-12-04 14:07:34 +0100 (0 s)
|
175
178
|
2015-12-04 14:07:41.444 => Reservation 692665 ready
|
@@ -329,6 +332,20 @@ After running the script, it will show the output of the benchmark in the `pry`
|
|
329
332
|
11: 24 bytes 38808 times --> 106.69 Mbps in 1.72 usec
|
330
333
|
12: 27 bytes 41271 times --> 119.77 Mbps in 1.72 usec
|
331
334
|
|
335
|
+
The latency is given by the last column for a 1 byte message; the maximum throughput is given by the last line.
|
336
|
+
We can try to performn the same test without using infiniband, in order to observe the difference in bandwidth and latency:
|
337
|
+
|
338
|
+
Net::SSH.start(nodes.first, "oar", grid5000_opt) do |ssh|
|
339
|
+
netpipe_url = "http://pkgs.fedoraproject.org/repo/pkgs/NetPIPE/NetPIPE-3.7.1.tar.gz/5f720541387be065afdefc81d438b712/NetPIPE-3.7.1.tar.gz"
|
340
|
+
ssh.exec!("mkdir -p netpipe_exp")
|
341
|
+
ssh.exec!("wget -O ~/netpipe_exp/NetPIPE.tar.gz #{netpipe_url}")
|
342
|
+
ssh.exec!("cd netpipe_exp && tar -zvxf NetPIPE.tar.gz")
|
343
|
+
ssh.exec!("cd netpipe_exp/NetPIPE-3.7.1 && make mpi")
|
344
|
+
mpi_command = "export OAR_JOB_KEY_FILE=~/my_ssh_jobkey;"
|
345
|
+
mpi_command+= "mpirun --mca plm_rsh_agent \"oarsh\" --mca btl self,sm,tcp -machinefile /tmp/machine_file ~/netpipe_exp/NetPIPE-3.7.1/NPmpi"
|
346
|
+
ssh.exec(mpi_command)
|
347
|
+
end
|
348
|
+
|
332
349
|
We can modify slightly the previous script to write the result into a file.
|
333
350
|
We need to use `ssh.exec!` to capture the output of the commands.
|
334
351
|
|
@@ -361,6 +378,13 @@ We can check the results by doing:
|
|
361
378
|
8: 16 bytes 27177 times --> 71.87 Mbps in 1.70 usec
|
362
379
|
9: 19 bytes 33116 times --> 85.00 Mbps in 1.71 usec
|
363
380
|
|
381
|
+
|
382
|
+
|
383
|
+
Once finished, we could release the job:
|
384
|
+
|
385
|
+
[34] pry(main)> $g5k.release(job)
|
386
|
+
=> ""
|
387
|
+
|
364
388
|
At the end of the experiment you can use the command `hist` to see what you have done so far.
|
365
389
|
This can help you to assemble everything together in a whole script.
|
366
390
|
|
@@ -387,7 +411,12 @@ This can help you to assemble everything together in a whole script.
|
|
387
411
|
|
388
412
|
## Running NAS benchmarks in Grid'5000: getting acquainted with parallel command execution
|
389
413
|
|
390
|
-
In this experiment, we will run the NAS
|
414
|
+
In this experiment, we will run the NAS benchmarks in Grid'5000 and we will script a scalability test for one of the benchmarks.
|
415
|
+
The NAS Parallel Benchmarks (NPB) are a set of benchmarks targeting performance evaluation of highly parallel supercomputers.
|
416
|
+
These benchmarks gather parallel kernels and three simluated applications.
|
417
|
+
They mimic the workload of large scale computational fluid dynamic applications.
|
418
|
+
The objective of this tutorial is to perform a scalability test of the NAS benchmarks. We are going to study how the
|
419
|
+
number of computing units used during the computation reduce the execution time of the application.
|
391
420
|
This experiment has the following requirements:
|
392
421
|
|
393
422
|
- 4 or 2 nodes from any Grid'5000 sites
|
@@ -413,7 +442,7 @@ and type the following code:
|
|
413
442
|
job = {}
|
414
443
|
|
415
444
|
sites.each do |site|
|
416
|
-
job = $g5k.reserve(:site => site, :nodes => 4, :wait => false, :walltime => "01:00:00")
|
445
|
+
job = $g5k.reserve(:site => site, :cluster => 1, :nodes => 4, :wait => false, :walltime => "01:00:00")
|
417
446
|
begin
|
418
447
|
job = $g5k.wait_for_job(job, :wait_time => 60)
|
419
448
|
puts "Nodes assigned #{job['assigned_nodes']}"
|
@@ -583,9 +612,9 @@ Then, we can assign this to a new variable:
|
|
583
612
|
[33] pry(main)> lu_path = results.values.first[:stdout]
|
584
613
|
=> "/tmp/NPB3.3/NPB3.3-MPI/bin/lu.A.32"
|
585
614
|
|
586
|
-
The setup of the experiment is done. It is time to execute the benchmark by typing the following into `pry` console
|
615
|
+
The setup of the experiment is done. It is time to execute the benchmark by typing the following into `pry` console:
|
587
616
|
|
588
|
-
Net::SSH.start(nodes.first
|
617
|
+
Net::SSH.start(nodes.first) do |ssh|
|
589
618
|
results = ssh.exec!("mpirun --mca btl self,sm,tcp -np 32 --machinefile machine_file #{lu_path}")
|
590
619
|
end
|
591
620
|
|
@@ -610,7 +639,7 @@ And copy-paste the following script:
|
|
610
639
|
|
611
640
|
expe_res = {}
|
612
641
|
|
613
|
-
Net::SSH.start(nodes.first
|
642
|
+
Net::SSH.start(nodes.first) do |ssh|
|
614
643
|
binaries.each do |binary|
|
615
644
|
processes = /A\.(\d*)/.match(binary)[1]
|
616
645
|
expe_res[processes]= {}
|
@@ -649,7 +678,7 @@ Once finished, we could release the job:
|
|
649
678
|
In this experiment, we will perform network measurements between two nodes located in different Grid'5000 sites.
|
650
679
|
The network measurements will be carried out in an isolated VLAN.
|
651
680
|
We will first reserved two nodes located in two different Grid'5000 sites in deploy mode and we will ask for two routed VLANs.
|
652
|
-
Once the nodes are ready an environment will be deployed and the application iperf will be install in all nodes.
|
681
|
+
Once the nodes are ready, an environment will be deployed and the application iperf will be install in all nodes.
|
653
682
|
Then, we will perform some network measurements among the nodes.
|
654
683
|
Finally, we will query the KWAPI using the G5K metrology API to get the network traffic generated during our experiment.
|
655
684
|
|
@@ -916,6 +945,7 @@ We could get all the names of the probes specific to this metric by typing:
|
|
916
945
|
|
917
946
|
probes = $g5k.get_metric("rennes",:metric => "network_in").uids
|
918
947
|
|
948
|
+
Please replace the first parameter with the site you have used in the experiment.
|
919
949
|
If you type that in `pry` you will get:
|
920
950
|
|
921
951
|
[13] pry(main)> probes
|
@@ -944,7 +974,7 @@ Which will give us an Array of Arrays that we can flatten by doing:
|
|
944
974
|
|
945
975
|
As we are going to fetch the data for Rennes (First node). We could do:
|
946
976
|
|
947
|
-
[68] pry(main)> probe_expe = probes.select{ |p| p
|
977
|
+
[68] pry(main)> probe_expe = probes.select{ |p| p == nodes_normal[0].split(".")[0] }
|
948
978
|
|
949
979
|
So, at this point we already have the probe we want to request.
|
950
980
|
Next step is to get the start time of the interval, we can choose for example, the time at which deployments have finished:
|
@@ -990,6 +1020,10 @@ and execute it with:
|
|
990
1020
|
1453293160.655534=>4498405298908,
|
991
1021
|
1453293161.998718=>4498405299219,
|
992
1022
|
|
1023
|
+
We can release the nodes:
|
1024
|
+
|
1025
|
+
[58] pry(main)> jobs.values.each{ |j| $g5k.release(j)}
|
1026
|
+
|
993
1027
|
## Conclusions
|
994
1028
|
|
995
1029
|
This tutorial has shown how the scripting of complex experiment can be done using the Ruby scripting language.
|
data/lib/cute/g5k_api.rb
CHANGED
@@ -200,7 +200,7 @@ module Cute
|
|
200
200
|
def initialize(uri,api_version,user,pass,on_error)
|
201
201
|
@user = user
|
202
202
|
@pass = pass
|
203
|
-
@api_version = api_version.nil? ? "
|
203
|
+
@api_version = api_version.nil? ? "stable" : api_version
|
204
204
|
if (user.nil? or pass.nil?)
|
205
205
|
@endpoint = uri # Inside Grid'5000
|
206
206
|
else
|
@@ -289,15 +289,7 @@ module Cute
|
|
289
289
|
|
290
290
|
# Handling G5k API errors
|
291
291
|
case e.http_code
|
292
|
-
|
293
|
-
# This part deals with bug: https://intranet.grid5000.fr/bugzilla/show_bug.cgi?id=5912
|
294
|
-
# Grid'5000 returns 500 error code even though the error was generated by a bad request
|
295
|
-
http_body = JSON.parse("{#{e.http_body.split("\n").select{ |x| x.include?("code")}.first}}")
|
296
|
-
if http_body["code"] == 400
|
297
|
-
raise BadRequest.new("Bad request", e)
|
298
|
-
else
|
299
|
-
raise RequestFailed.new("Service internal error", e)
|
300
|
-
end
|
292
|
+
|
301
293
|
when 400
|
302
294
|
raise BadRequest.new("Bad request", e)
|
303
295
|
when 404
|
@@ -308,7 +300,7 @@ module Cute
|
|
308
300
|
if @on_error == :ignore
|
309
301
|
return nil
|
310
302
|
else
|
311
|
-
raise RequestFailed.new("
|
303
|
+
raise RequestFailed.new("Grid5000 API internal error", e)
|
312
304
|
end
|
313
305
|
end
|
314
306
|
end
|
@@ -320,14 +312,14 @@ module Cute
|
|
320
312
|
# To simplify the use of the module, it is better to create a file with the following information:
|
321
313
|
#
|
322
314
|
# $ cat > ~/.grid5000_api.yml << EOF
|
323
|
-
#
|
324
|
-
#
|
325
|
-
#
|
326
|
-
#
|
327
|
-
#
|
315
|
+
# uri: https://api.grid5000.fr/
|
316
|
+
# username: user
|
317
|
+
# password: **********
|
318
|
+
# version: sid
|
319
|
+
# EOF
|
328
320
|
#
|
329
321
|
# The *username* and *password* are not necessary if you are using the module from inside Grid'5000.
|
330
|
-
# You can take a look at the {Cute::G5K::API#initialize G5K::API constructor} to see more details
|
322
|
+
# You can take a look at the {Cute::G5K::API#initialize G5K::API constructor} to see more details of
|
331
323
|
# this configuration.
|
332
324
|
#
|
333
325
|
# = Getting started
|
@@ -339,8 +331,8 @@ module Cute
|
|
339
331
|
# In the presence of error, {Cute::G5K::API G5K::API} raises exceptions (see {Cute::G5K::Error G5K exceptions}),
|
340
332
|
# that you can handle to decide the workflow of your experiment
|
341
333
|
# (see {Cute::G5K::API#wait_for_deploy wait_for_deploy} and {Cute::G5K::API#wait_for_deploy wait_for_job}).
|
342
|
-
#
|
343
|
-
#
|
334
|
+
# In the following example it is shown how {Cute::G5K::API G5K::API} is used. The example represents
|
335
|
+
# the reservation of 3 nodes in Nancy site for 1 hour:
|
344
336
|
#
|
345
337
|
# require 'cute'
|
346
338
|
#
|
@@ -354,7 +346,7 @@ module Cute
|
|
354
346
|
#
|
355
347
|
# $ ruby example.rb
|
356
348
|
#
|
357
|
-
# The execution will block until you got the reservation. Then, you can interact with the nodes you reserved the way you used to or
|
349
|
+
# The execution will block until you got the reservation. Then, you can interact with the nodes you reserved in the way you used to or
|
358
350
|
# add more code to the previous script for controlling your experiment with Ruby-Cute as shown in this
|
359
351
|
# {http://www.rubydoc.info/github/ruby-cute/ruby-cute/master/file/examples/g5k_exp_virt.rb example}.
|
360
352
|
# We have just used the method {Cute::G5K::API#reserve reserve} that allow us to reserve resources in Grid'5000.
|
@@ -372,7 +364,7 @@ module Cute
|
|
372
364
|
#
|
373
365
|
# puts "Assigned nodes : #{job['assigned_nodes']}"
|
374
366
|
#
|
375
|
-
#
|
367
|
+
# By default your public ssh key '~/.ssh/id_rsa.pub' will be copied on the deployed machines,
|
376
368
|
# you can specify another path for your keys with the option *:keys*.
|
377
369
|
# In order to deploy your own environment, you have to put the tar file that contains the operating system you want to deploy and
|
378
370
|
# the environment description file, under the public directory of a given site.
|
@@ -393,7 +385,7 @@ module Cute
|
|
393
385
|
# puts "Log in into the nodes using the following hostnames: #{g5k.get_vlan_nodes(job)}"
|
394
386
|
#
|
395
387
|
# If you do not want that the method {Cute::G5K::API#reserve reserve} perform the deployment for you, you have to use the option :type => :deploy.
|
396
|
-
# This can be useful when deploying different environments in your reserved nodes
|
388
|
+
# This can be useful when deploying different environments in your reserved nodes, for example: deploying the environments for a small HPC cluster.
|
397
389
|
# You have to use the method {Cute::G5K::API#deploy deploy} for performing the deploy.
|
398
390
|
# This method do not block by default, that is why you have to use the method {Cute::G5K::API#wait_for_deploy wait_for_deploy} in order to block the execution
|
399
391
|
# until the deployment is done.
|
@@ -480,7 +472,7 @@ module Cute
|
|
480
472
|
#
|
481
473
|
# g5k = Cute::G5K::API.new(:conf_file =>"config file path")
|
482
474
|
#
|
483
|
-
# You can specify other
|
475
|
+
# You can specify other parameters to use:
|
484
476
|
#
|
485
477
|
# g5k = Cute::G5K::API.new(:uri => "https://api.grid5000.fr", :version => "sid")
|
486
478
|
#
|
@@ -509,8 +501,8 @@ module Cute
|
|
509
501
|
config = YAML.load(File.open(params[:conf_file],'r')) unless params[:conf_file].nil?
|
510
502
|
@user = params[:username] || config["username"]
|
511
503
|
@pass = params[:password] || config["password"]
|
512
|
-
@uri = params[:uri] || config["uri"]
|
513
|
-
@api_version = params[:version] || config["version"] || "
|
504
|
+
@uri = params[:uri] || config["uri"] || "https://api.grid5000.fr/"
|
505
|
+
@api_version = params[:version] || config["version"] || "stable"
|
514
506
|
@logger = nil
|
515
507
|
|
516
508
|
begin
|
@@ -629,13 +621,18 @@ module Cute
|
|
629
621
|
# if a uid is provided only the jobs owned by the user are shown.
|
630
622
|
# @param site [String] a valid Grid'5000 site name
|
631
623
|
# @param uid [String] user name in Grid'5000
|
632
|
-
# @param
|
633
|
-
def get_jobs(site, uid = nil,
|
634
|
-
|
635
|
-
|
636
|
-
|
637
|
-
|
638
|
-
|
624
|
+
# @param states [Array] or [String] jobs state: running, waiting (multiple states can be specified)
|
625
|
+
def get_jobs(site, uid = nil, states = nil)
|
626
|
+
|
627
|
+
parameters = []
|
628
|
+
if states then
|
629
|
+
states = [states] if states.is_a?(String)
|
630
|
+
parameters.push("state=#{states.join(",")}")
|
631
|
+
end
|
632
|
+
parameters.push("user=#{uid}") if uid
|
633
|
+
parameters.push("limit=25") if (states.nil? and uid.nil?)
|
634
|
+
|
635
|
+
jobs = @g5k_connection.get_json(api_uri("/sites/#{site}/jobs?#{parameters.join("&")}")).items
|
639
636
|
jobs.map{ |j| @g5k_connection.get_json(j.rel_self)}
|
640
637
|
# This request sometime is could take a little long when all jobs are requested
|
641
638
|
# The API return by default 50 the limit was set to 25 (e.g., 23 seconds).
|
@@ -718,13 +715,18 @@ module Cute
|
|
718
715
|
# By default it only shows the jobs in state *running*.
|
719
716
|
# You can specify another state like this:
|
720
717
|
#
|
721
|
-
# =
|
722
|
-
# get_my_jobs("nancy",
|
718
|
+
# = Examples
|
719
|
+
# get_my_jobs("nancy", "waiting")
|
720
|
+
# Getting several states:
|
721
|
+
# get_my_jobs("nancy", ["waiting","running"])
|
723
722
|
# Valid states are specified in {https://api.grid5000.fr/doc/4.0/reference/spec.html Grid'5000 API spec}
|
724
723
|
# @return [Array] all my submitted jobs to a given site and their associated deployments.
|
725
724
|
# @param site [String] a valid Grid'5000 site name
|
726
|
-
|
727
|
-
|
725
|
+
# @param states [String/Array] possible job state values (waiting, launching, running, hold, error, terminated)
|
726
|
+
def get_my_jobs(site, states = "running")
|
727
|
+
|
728
|
+
# raise ArgumentError,"States parameter should be an Array" unless states.is_a?(Array)
|
729
|
+
jobs = get_jobs(site, g5k_user, states)
|
728
730
|
deployments = get_deployments(site, g5k_user)
|
729
731
|
# filtering deployments only the job in state running make sense
|
730
732
|
jobs.map{ |j| j["deploy"] = deployments.select{ |d| d["created_at"] > j["started_at"]} if j["state"] == "running"}
|
@@ -877,7 +879,7 @@ module Cute
|
|
877
879
|
# @option opts [Numeric] :nodes Number of nodes to reserve
|
878
880
|
# @option opts [String] :walltime Walltime of the reservation
|
879
881
|
# @option opts [String] :site Grid'5000 site
|
880
|
-
# @option opts [Symbol] :type Type of reservation: :deploy, :
|
882
|
+
# @option opts [Symbol] :type Type of reservation: :deploy, :allow_classic_ssh
|
881
883
|
# @option opts [String] :name Reservation name
|
882
884
|
# @option opts [String] :cmd The command to execute when the job starts (e.g. ./my-script.sh).
|
883
885
|
# @option opts [String] :cluster Valid Grid'5000 cluster
|
@@ -902,7 +904,7 @@ module Cute
|
|
902
904
|
nodes = opts.fetch(:nodes, 1)
|
903
905
|
walltime = opts.fetch(:walltime, '01:00:00')
|
904
906
|
site = opts[:site]
|
905
|
-
type = opts
|
907
|
+
type = opts.fetch(:type, [])
|
906
908
|
name = opts.fetch(:name, 'rubyCute job')
|
907
909
|
command = opts[:cmd]
|
908
910
|
opts[:wait] = true if opts[:wait].nil?
|
@@ -914,7 +916,8 @@ module Cute
|
|
914
916
|
properties = opts[:properties]
|
915
917
|
reservation = opts[:reservation]
|
916
918
|
resources = opts.fetch(:resources, "")
|
917
|
-
type = :deploy if opts[:env]
|
919
|
+
type = [:deploy] if opts[:env]
|
920
|
+
type = [type] if type.is_a?(Symbol)
|
918
921
|
keys = opts[:keys]
|
919
922
|
queue = opts[:queue]
|
920
923
|
|
@@ -936,14 +939,17 @@ module Cute
|
|
936
939
|
walltime = walltime.to_time
|
937
940
|
|
938
941
|
command = "sleep #{secs}" if command.nil?
|
939
|
-
type = type.to_sym unless type.nil?
|
940
942
|
|
941
943
|
if resources == ""
|
942
944
|
resources = "/switch=#{switches}" unless switches.nil?
|
943
945
|
resources += "/nodes=#{nodes}"
|
944
946
|
resources += "/cpu=#{cpus}" unless cpus.nil?
|
945
947
|
resources += "/core=#{cores}" unless cores.nil?
|
946
|
-
|
948
|
+
|
949
|
+
if cluster
|
950
|
+
resources = (cluster.is_a?(Fixnum) ? "/cluster=#{cluster}" : "{cluster='#{cluster}'}") + resources
|
951
|
+
end
|
952
|
+
|
947
953
|
resources = "{type='#{vlan}'}/vlan=1+" + resources unless vlan.nil?
|
948
954
|
resources = "slash_#{subnets[0]}=#{subnets[1]}+" + resources unless subnets.nil?
|
949
955
|
end
|
@@ -959,10 +965,11 @@ module Cute
|
|
959
965
|
info "Reserving resources: #{resources} (type: #{type}) (in #{site})"
|
960
966
|
|
961
967
|
payload['properties'] = properties unless properties.nil?
|
962
|
-
payload['types'] =
|
968
|
+
payload['types'] = type.map{ |t| t.to_s} unless type.nil?
|
969
|
+
type.map!{|t| t.to_sym} unless type.nil?
|
963
970
|
payload['queue'] = queue if queue
|
964
971
|
|
965
|
-
|
972
|
+
unless type.include?(:deploy)
|
966
973
|
if opts[:keys]
|
967
974
|
payload['import-job-key-from-file'] = [ File.expand_path(keys) ]
|
968
975
|
else
|
@@ -1030,7 +1037,8 @@ module Cute
|
|
1030
1037
|
# }
|
1031
1038
|
#
|
1032
1039
|
# @param job [G5KJSON] as described in {Cute::G5K::G5KJSON job}
|
1033
|
-
# @param
|
1040
|
+
# @param [Hash] opts Options
|
1041
|
+
# @option opts [Numeric] :wait_time Number of seconds to wait before triggering a timeout
|
1034
1042
|
def wait_for_job(job,opts={})
|
1035
1043
|
opts[:wait_time] = 36000 if opts[:wait_time].nil?
|
1036
1044
|
jid = job['uid']
|
@@ -1073,10 +1081,15 @@ module Cute
|
|
1073
1081
|
#
|
1074
1082
|
# deploy(job, :nodes => ["genepi-2.grid5000.fr"], :env => "wheezy-x64-xen", :keys => "~/my_key")
|
1075
1083
|
#
|
1084
|
+
# The parameter *:keys* [String] can be a string specifying the path of the key (as the previous case)
|
1085
|
+
# or the contents of the public ssh key as the example given below:
|
1086
|
+
#
|
1087
|
+
# deploy(job,:env => "jessie-x64-big", :keys => File.read("/tmp/test_key/test_key.pub"))
|
1088
|
+
#
|
1076
1089
|
# @param job [G5KJSON] as described in {Cute::G5K::G5KJSON job}
|
1077
1090
|
# @param [Hash] opts Deploy options
|
1078
1091
|
# @option opts [String] :env {http://kadeploy3.gforge.inria.fr/ Kadeploy} environment to deploy
|
1079
|
-
# @option opts [
|
1092
|
+
# @option opts [Array] :nodes Specifies the nodes to deploy on
|
1080
1093
|
# @option opts [String] :keys Specifies the SSH keys to copy for the deployment
|
1081
1094
|
# @option opts [Boolean] :wait Whether or not to wait until the deployment is done (default is false)
|
1082
1095
|
# @return [G5KJSON] a job with deploy information as described in {Cute::G5K::G5KJSON job}
|
@@ -1102,9 +1115,19 @@ module Cute
|
|
1102
1115
|
|
1103
1116
|
if opts[:keys].nil? then
|
1104
1117
|
public_key_path = File.expand_path("~/.ssh/id_rsa.pub")
|
1105
|
-
|
1118
|
+
if File.exist?(public_key_path) then
|
1119
|
+
public_key_file = File.read(public_key_path)
|
1120
|
+
else
|
1121
|
+
raise ArgumentError, "No public ssh key found"
|
1122
|
+
end
|
1123
|
+
|
1106
1124
|
else
|
1107
|
-
|
1125
|
+
# We check if the string passed contains the ssh public key
|
1126
|
+
if (opts[:keys].length < 300 && (opts[:keys] =~ /^ssh.*/).nil?)
|
1127
|
+
public_key_file = File.read("#{File.expand_path(opts[:keys])}.pub").chop
|
1128
|
+
else
|
1129
|
+
public_key_file = opts[:keys]
|
1130
|
+
end
|
1108
1131
|
end
|
1109
1132
|
|
1110
1133
|
payload = {
|
data/lib/cute/taktuk.rb
CHANGED
@@ -19,8 +19,66 @@ module Cute
|
|
19
19
|
# end
|
20
20
|
# puts results
|
21
21
|
#
|
22
|
-
#
|
23
|
-
#
|
22
|
+
# = Understanding {TakTuk::TakTuk#exec exec} and {TakTuk::TakTuk#exec! exec!}
|
23
|
+
#
|
24
|
+
# This section explains the differences between {TakTuk::TakTuk#exec exec} and {TakTuk::TakTuk#exec! exec!} with
|
25
|
+
# several examples.
|
26
|
+
# == Example 1
|
27
|
+
#
|
28
|
+
# Cute::TakTuk.start(['host1','host2','host3'],:user => "root") do |tak|
|
29
|
+
# tak.exec("df")
|
30
|
+
# tak.exec("ls -l")
|
31
|
+
# tak.exec("sleep 20")
|
32
|
+
# tak.exec("tar xvf file.tar")
|
33
|
+
# end
|
34
|
+
#
|
35
|
+
# In the previous example all the commands will be executed concurrently on each host*.
|
36
|
+
# This will be equivalent to execute the following sequence in bash:
|
37
|
+
#
|
38
|
+
# $ df &
|
39
|
+
# $ ls -l &
|
40
|
+
# $ sleep 20 &
|
41
|
+
# $ tar xvf file.tar &
|
42
|
+
# $ wait
|
43
|
+
#
|
44
|
+
# The {Cute::TakTuk#start start} method waits for all commands, it performs a {TakTuk::TakTuk#loop loop()} implicitly.
|
45
|
+
# This implicit {TakTuk::TakTuk#loop loop()} has the same behaviour as the 'wait' command in bash.
|
46
|
+
# == Example 2
|
47
|
+
# Cute::TakTuk.start(['host1','host2','host3'],:user => "root") do |tak|
|
48
|
+
# tak.exec("df")
|
49
|
+
# tak.exec("ls -l")
|
50
|
+
# tak.loop()
|
51
|
+
# tak.exec("sleep 20")
|
52
|
+
# tak.exec("tar xvf file.tar")
|
53
|
+
# end
|
54
|
+
# This will execute the two first comamnds concurrently and then the remaining commands concurrently.
|
55
|
+
# It is equivalent to execute the following sequence in bash:
|
56
|
+
# $ df &
|
57
|
+
# $ ls -l &
|
58
|
+
# $ wait
|
59
|
+
# $ sleep 20 &
|
60
|
+
# $ tar xvf file.tar &
|
61
|
+
# $ wait
|
62
|
+
# == Example 3
|
63
|
+
# Cute::TakTuk.start(['host1','host2','host3'],:user => "root") do |tak|
|
64
|
+
# tak.exec("df")
|
65
|
+
# tak.exec("ls -l")
|
66
|
+
# tak.exec!("sleep 20")
|
67
|
+
# tak.exec("tar xvf file.tar")
|
68
|
+
# end
|
69
|
+
#
|
70
|
+
# Notice that we use now the {TakTuk::TakTuk#exec! exec!} method
|
71
|
+
# which will wait for the previous commands and then it will block until the command finishes.
|
72
|
+
# It is equivalent to execute the following sequence in bash:
|
73
|
+
# $ df &
|
74
|
+
# $ ls -l &
|
75
|
+
# $ wait
|
76
|
+
# $ sleep 20 &
|
77
|
+
# $ wait
|
78
|
+
# $ tar xvf file.tar &
|
79
|
+
# $ wait
|
80
|
+
# You can go directly to the documentation of the mentioned methods {TakTuk::TakTuk#exec exec},
|
81
|
+
# {TakTuk::TakTuk#exec! exec!} and other useful methods such as: {TakTuk::TakTuk#put put}, {TakTuk::TakTuk#input input}, etc.
|
24
82
|
# @see http://taktuk.gforge.inria.fr/.
|
25
83
|
# @see TakTuk::TakTuk TakTuk Class for more documentation.
|
26
84
|
module TakTuk
|
data/lib/cute/version.rb
CHANGED
@@ -0,0 +1,63 @@
|
|
1
|
+
require 'spec_helper'
|
2
|
+
|
3
|
+
#These tests will check the API Grid'5000 using all the methods that perform a GET request
|
4
|
+
|
5
|
+
describe Cute::G5K::API do
|
6
|
+
|
7
|
+
subject { g5k = ENV['TEST_REAL'].nil?? Cute::G5K::API.new(:user => "test") : Cute::G5K::API.new() }
|
8
|
+
|
9
|
+
before :each do
|
10
|
+
if ENV['TEST_REAL']
|
11
|
+
WebMock.disable!
|
12
|
+
end
|
13
|
+
end
|
14
|
+
|
15
|
+
it "checks initialization of G5K::API" do
|
16
|
+
expect(subject.rest).to be_an_instance_of(Cute::G5K::G5KRest)
|
17
|
+
end
|
18
|
+
|
19
|
+
it "returns an array with the clusters ids in nancy" do
|
20
|
+
clusters = subject.cluster_uids("nancy")
|
21
|
+
expect(clusters.length).to be > 0
|
22
|
+
end
|
23
|
+
|
24
|
+
it "returns an array with the clusters ids in grenoble" do
|
25
|
+
clusters = subject.cluster_uids("grenoble")
|
26
|
+
expect(clusters.length).to be > 0
|
27
|
+
end
|
28
|
+
|
29
|
+
it "returns an array with the clusters ids in lyon" do
|
30
|
+
clusters = subject.cluster_uids("lyon")
|
31
|
+
expect(clusters.length).to be > 0
|
32
|
+
end
|
33
|
+
|
34
|
+
it "returns an array with the clusters ids in lille" do
|
35
|
+
clusters = subject.cluster_uids("lille")
|
36
|
+
expect(clusters.length).to be > 0
|
37
|
+
end
|
38
|
+
|
39
|
+
it "returns a JSON Hash with the status of a site grenoble" do
|
40
|
+
expect(subject.site_status("grenoble")).to be_an_instance_of(Cute::G5K::G5KJSON)
|
41
|
+
end
|
42
|
+
|
43
|
+
it "returns a JSON Hash with the status of a site nancy" do
|
44
|
+
expect(subject.site_status("nancy")).to be_an_instance_of(Cute::G5K::G5KJSON)
|
45
|
+
end
|
46
|
+
|
47
|
+
it "return my_jobs in lille site" do
|
48
|
+
expect(subject.get_my_jobs("lille")).to be_an_instance_of(Array)
|
49
|
+
end
|
50
|
+
|
51
|
+
it "returns all deployments in nancy" do
|
52
|
+
expect(subject.get_deployments("nancy")).to be_an_instance_of(Cute::G5K::G5KArray)
|
53
|
+
end
|
54
|
+
|
55
|
+
it "returns all deployments in lille" do
|
56
|
+
expect(subject.get_deployments("lille")).to be_an_instance_of(Cute::G5K::G5KArray)
|
57
|
+
end
|
58
|
+
|
59
|
+
it "returns all deployments in grenoble" do
|
60
|
+
expect(subject.get_deployments("grenoble")).to be_an_instance_of(Cute::G5K::G5KArray)
|
61
|
+
end
|
62
|
+
|
63
|
+
end
|
data/spec/g5k_api_spec.rb
CHANGED
@@ -13,12 +13,13 @@ describe Cute::G5K::API do
|
|
13
13
|
index = ((day/31.to_f)*sites.length).to_i
|
14
14
|
index = 1 if index == 0
|
15
15
|
@rand_site = sites[index]
|
16
|
-
@env =
|
16
|
+
@env = "jessie-x64-big" # This could change but there is no way to get the available environments
|
17
17
|
if ENV['TEST_REAL']
|
18
18
|
WebMock.disable!
|
19
19
|
puts "Testing in real Grid'5000 using site: #{@rand_site}"
|
20
20
|
puts "Warning G5K_USER environment variable has to be defined for some tests" if ENV['G5K_USER'].nil?
|
21
21
|
end
|
22
|
+
|
22
23
|
end
|
23
24
|
|
24
25
|
it "checks initialization of G5K::API" do
|
@@ -56,6 +57,10 @@ describe Cute::G5K::API do
|
|
56
57
|
expect(subject.get_my_jobs(@rand_site)).to be_an_instance_of(Array)
|
57
58
|
end
|
58
59
|
|
60
|
+
it "return my jobs in different states" do
|
61
|
+
expect(subject.get_my_jobs(@rand_site,["running","terminated"]).length).to be > 1
|
62
|
+
end
|
63
|
+
|
59
64
|
it "returns all deployments" do
|
60
65
|
expect(subject.get_deployments(@rand_site)).to be_an_instance_of(Cute::G5K::G5KArray)
|
61
66
|
end
|
@@ -104,6 +109,12 @@ describe Cute::G5K::API do
|
|
104
109
|
subject.release(job)
|
105
110
|
end
|
106
111
|
|
112
|
+
it "reserves besteffort" do
|
113
|
+
job = subject.reserve(:site => @rand_site, :type => [:besteffort,:deploy])
|
114
|
+
expect(job).to be_an_instance_of(Cute::G5K::G5KJSON)
|
115
|
+
subject.release(job)
|
116
|
+
end
|
117
|
+
|
107
118
|
|
108
119
|
it "reserves with vlan and get vlan hostnames" do
|
109
120
|
job = subject.reserve(:site => @rand_site, :nodes => 1, :type => :deploy, :vlan => :routed)
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: ruby-cute
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: '0.
|
4
|
+
version: '0.5'
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Algorille team
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2016-
|
11
|
+
date: 2016-05-03 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: bundler
|
@@ -207,6 +207,7 @@ files:
|
|
207
207
|
- lib/cute/version.rb
|
208
208
|
- ruby-cute.gemspec
|
209
209
|
- spec/extensions_spec.rb
|
210
|
+
- spec/g5k_api_check_spec.rb
|
210
211
|
- spec/g5k_api_spec.rb
|
211
212
|
- spec/spec_helper.rb
|
212
213
|
- spec/taktuk_spec.rb
|
@@ -232,12 +233,13 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
232
233
|
version: 1.3.6
|
233
234
|
requirements: []
|
234
235
|
rubyforge_project:
|
235
|
-
rubygems_version: 2.
|
236
|
+
rubygems_version: 2.2.2
|
236
237
|
signing_key:
|
237
238
|
specification_version: 4
|
238
239
|
summary: Critically Useful Tools for Experiments
|
239
240
|
test_files:
|
240
241
|
- spec/extensions_spec.rb
|
242
|
+
- spec/g5k_api_check_spec.rb
|
241
243
|
- spec/g5k_api_spec.rb
|
242
244
|
- spec/spec_helper.rb
|
243
245
|
- spec/taktuk_spec.rb
|