claudius 0.0.3 → 0.0.4

data/README.md

@@ -11,11 +11,66 @@ Install it as:
 
 If you want export execution tree to image you need [Graphivz](http://www.graphviz.org).
 
+## Keywords
+
+* **experiment** - central part of claudius. Method defines a new experiment.  Parameters: 
+    * **experiment_name**
+    * **body** - block of code describing performed experiment
+    
+    You may call the following methods on returned experiment object:
+
+    * **run** - method starts previously defined experiment, 
+    * **export_tree(path = 'execution_tree_path')** - method creates an execution graph of experiment and save it as an image
+
+* **define_providers** - method takes as a parameter description of machines used in experiment. In experiments You are allowed to use 2 different types of machines
+
+    * **manual** - those machines have been created before experiment start, You should possess appropriate credentials such as: ip address, login, password etc. 
+    
+    * **cloud** -  if You like to perform your experiment in cloud, You should firstly provide necessary information to authenticate to Your cloud provider ( e.g. AWS). After successfully authentication, please define instances which You are going to use in your experiment by create_instances method.
+    
+* **foreach**  - keyword is similar to  ruby ‘each’ method, but in contrast to ordinary ‘each’, you may provide adjectives describing how parameters should be process
+	Currently supported adjectives are. 
+    * **asynchronously** - each parameter is processed in separate thread. Execution of instructions after foreach block is pursued when each loop is finished.
+    * **safely** - experiment is continued even if some exception occurs during loop execution, 
+* **on** - specify on which machine instructions should be executed, by default it is localhost. It takes instance name as a parameter.
+
+* **before**, **after** - keywords are used to construct metrics 
+
+* **concurrent**  - if elements (such as execute or foreach) are in common concurrent block, they are executed each in separate thread.
+
+* **execute** - method takes as a parameter block of code (which contain ssh methods calls), end perform instructions sequentially
+
+* **ssh** - specify instruction(s) which are going to be invoked on machine. As a parameter takes a string representing shell command.
+
+## Credentials
+
+In order to authenticate in AWS services (at other cloud providers also) you are obligated to provide some credentials data, which are: aws_access_key_id, aws_secret_access_key, and so on. It is recomended to store  configuration in json file and refer to them, when they are require. Sample config file looks like one below:
+
+
+``` javascript
+{
+   "provider" : "AWS",
+   "region" : "eu-west-1",
+   "aws_access_key_id" : "XXXXXXXXXXXXXXXXXXXX",
+   "aws_secret_access_key" : "XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX",
+   "key_name" : "My-irleand-key",
+   "groups" : "My-irleand-group"
+   "path_to_pem_file" : "./pems/my-irleand-key.pem"
+}
+```
+
+After creation, You could load your file, assign it to variable and use freely as dictionary.
+
+``` ruby
+config = load_config('./user_config.json')
+aws_key = config['aws_access_key_id']
+```
+
+
 ## Usage
 
 Check doc [here](http://blostic.github.io/claudius).
 
-
 ## Documentation
 
 Documemtation is autogenerated from examples by [Groc](https://github.com/nevir/groc).
@@ -31,7 +86,6 @@ Generate to GitHub page:
 
     groc --gh examples/* README.md
 
-
 ## Contributing
 
 1. Fork it

data/claudius.gemspec

@@ -4,11 +4,16 @@ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
 
 Gem::Specification.new do |spec|
   spec.name          = "claudius"
-  spec.version       = "0.0.3"
+  spec.version       = "0.0.4"
   spec.authors       = ["radk0s", "blost"]
-  spec.email         = ["rachamot@gmail.com"]
-  spec.description   = "Write a gem description"
-  spec.summary       = "Write a gem summary"
+  spec.email         = ["rachamot@gmail.com", "piotr.skibiak@gmail.com"]
+  spec.description   = 'Claudius is an easy-to-use domain specific language for cloud experiments. ' +
+      'It has been designed to speed up process of building distributed experiments and highly reduce time of ' +
+      'remote machines configuration. To avoid vendor lock-in, Claudius was build on top of fog.io library, ' +
+      'which enables flexible and powerful way to manage machine instances at various cloud providers. Remote ' +
+      'commands execution is based on  SSH protocol (SSH-2). DLS allow users to generate readable execution graph, ' +
+      'which is extremely useful for experiment flow verification and help avoid wasting your money.'
+  spec.summary       = "DSL for cloud experiments"
   spec.homepage      = "https://github.com/blostic/claudius"
   spec.license       = "MIT"
   spec.files         = `git ls-files`.split($/)
@@ -18,7 +23,7 @@ Gem::Specification.new do |spec|
 
   spec.add_development_dependency "bundler", "~> 1.3"
   spec.add_development_dependency "rake"
-  spec.add_runtime_dependency "fog", ">= 1.22.0"
+  spec.add_runtime_dependency "fog", "1.22.0"
   spec.add_runtime_dependency "net-ssh", "2.9.1"
   spec.add_runtime_dependency "graph", "2.7.0"
   spec.add_runtime_dependency "awesome_print", "1.2.0"

data/examples/distributed_app.rb

@@ -0,0 +1,47 @@
+
+# Distributed app
+# ---------
+# Set up distributed key-value store Etcd using Docker.
+#
+
+require "claudius"
+config = load_config('./user_config.json')
+$discovery_token = "https://discovery.etcd.io/c4c7919104cbf432d2a92036cddd8d33"
+
+execution_tree = experiment 'Hello' do
+  define_providers do
+    cloud('aws',  :provider => config['provider'],
+          :region =>config['region'],
+          :endpoint => 'https://ec2.eu-west-1.amazonaws.com/',
+          :aws_access_key_id => config['aws_access_key_id'],
+          :aws_secret_access_key => config['aws_secret_access_key'])
+    .create_instances(
+        ['t1.micro=>in1'],
+                      :username => 'ubuntu',
+                      :private_key_path =>config['path_to_pem_file'],
+                      :key_name => config['key_name'],
+                      :groups => config['groups'])
+  end
+  foreach ['in1', 'in2'] do |instance_name|
+      on instance_name do
+        before "set up docker" do
+          # ssh "curl -sSL https://get.docker.io/ubuntu/ | sudo sh"
+          # ssh "sudo apt-get install git -y"
+          # ssh "wget https://github.com/coreos/etcd/archive/v0.4.6.tar.gz"
+          # ssh "tar xzvf v0.4.6.tar.gz"
+          # ssh "mv etcd-0.4.6 etcd"
+          # ssh "rm v0.4.6.tar.gz"
+        end
+        execute do
+          # ssh "sudo docker build  -t etcd etcd"
+          # ssh "sudo docker run --name etcd1 -d -p 4001:4001 -p 7001:7001 -v /data/etcd1:/etcd-data etcd -name #{instance_name} -peer-addr #{getOtherInstances(instance_name).hosts}:7001 -addr #{getInstance(instance_name).host}:4001 -discovery %s" % $discovery_token
+        end
+        after "clean up docker containers" do
+          # ssh "docker stop $(docker ps -a -q)"
+          # ssh "docker rm $(docker ps -a -q)"
+        end
+      end
+  end
+end
+
+ap execution_tree.run

data/examples/distributed_montage.rb

@@ -0,0 +1,84 @@
+require "claudius"
+require "json"
+
+config = load_config('./user_config.json')
+
+execution_tree = experiment 'Helloo' do
+  define_providers do
+    cloud('aws',  :provider => config['provider'],
+          :region =>config['region'],
+          :endpoint => 'https://ec2.eu-west-1.amazonaws.com/',
+          :aws_access_key_id => config['aws_access_key_id'],
+          :aws_secret_access_key => config['aws_secret_access_key'])
+    .create_instances(
+        ['m3.medium=>in1', 'm3.large=>in2'],
+        :username => 'ubuntu',
+        :private_key_path =>config['path_to_pem_file'],
+        :key_name => config['key_name'],
+        :image_id => config['image_id'],
+        :groups => config['groups'])
+  end
+  foreach ['in1', 'in2'] do |instance_name|
+    on instance_name do
+      execute do
+        ssh "yes | sudo apt-get update"
+        ssh "yes | sudo apt-get install build-essential"
+
+        ssh "yes | sudo apt-get install python-software-properties python g++ make"
+        ssh "yes | sudo add-apt-repository ppa:chris-lea/node.js"
+        ssh "yes | sudo apt-get update"
+        ssh "yes | sudo apt-get install nodejs"
+
+        ssh "yes | sudo apt-get install git-core"
+
+        ssh "yes | sudo apt-get -y install build-essential zlib1g-dev libssl-dev libreadline6-dev libyaml-dev"
+        ssh "yes | sudo apt-get install ruby2.0 ruby2.0-dev build-essential libxml2-dev libxslt1-dev"
+        ssh "yes | sudo apt-get install rabbitmq-server"
+        ssh "yes | sudo apt-get install redis-server"
+
+        ssh "echo 'Git:'"
+        ssh "git --version"
+        ssh "echo 'Ruby'"
+        ssh "ruby --version"
+        ssh "echo 'Nodejs'"
+        ssh "nodejs --version"
+        ssh "echo 'Redis'"
+        ssh "redis-server --version"
+
+        ssh "curl -O http://pegasus.isi.edu/montage/Montage_v3.3_patched_4.tar.gz"
+        ssh "tar zxvf Montage_v3.3_patched_4.tar.gz"
+        ssh "cd Montage_v3.3_patched_4 && make"
+
+        ssh "curl -O https://dl.dropboxusercontent.com/u/81819/hyperflow-amqp-executor.gem"
+        ssh "sudo gem2.0 install --no-ri --no-rdoc hyperflow-amqp-executor.gem"
+
+        ssh "wget https://github.com/dice-cyfronet/hyperflow/archive/v1.0.0-beta-6.tar.gz"
+        ssh "tar zxvf v1.0.0-beta-6.tar.gz"
+        ssh "mv hyperflow-1.0.0-beta-6 hyperflow"
+        ssh "cd hyperflow && npm install"
+      end
+    end
+  end
+  on 'in1' do
+    execute do
+      ssh "cd hyperflow && echo 'amqp_url: amqp://localhost\nstorage: local\nthreads: <%= Executor::cpu_count %>' > executor_config.yml"
+      ssh "export PATH=$PATH:~/Montage_v3.3_patched_4/bin; nohup hyperflow-amqp-executor $(pwd)/hyperflow/executor_config.yml > amqp.out 2>&1 &"
+    end
+  end
+  on 'in2' do
+    execute do
+      ssh "mkdir data"
+      ssh "cd data && wget https://gist.github.com/kfigiela/9075623/raw/dacb862176e9d576c1b23f6a243f9fa318c74bce/bootstrap.sh"
+      ssh "cd data && chmod +x bootstrap.sh"
+      ssh "export PATH=$PATH:~/Montage_v3.3_patched_4/bin && ./data/bootstrap.sh 0.25"
+      ssh "cd hyperflow && echo \"var AMQP_URL=process.env.AMQP_URL?process.env.AMQP_URL:'amqp://#{getInstance('in1').host}:5672';exports.amqp_url=AMQP_URL;exports.options={'storage':'local','workdir':'/home/ubuntu/0.25/input'}\" > functions/amqpCommand.config.js"
+      ssh "nodejs hyperflow/scripts/dax_convert_amqp.js 0.25/workdir/dag.xml > 0.25/workdir/dag.json"
+      ssh "nohup nodejs hyperflow/scripts/runwf.js -f 0.25/workdir/dag.json -s > amqp.out 2>&1 &"
+    end
+  end
+end
+
+result = execution_tree.run
+ap result
+
+puts result.to_json

data/examples/montage.rb

@@ -0,0 +1,88 @@
+require "claudius"
+require "json"
+
+config = load_config('./user_config.json')
+
+execution_tree = experiment 'Helloo' do
+  define_providers do
+    cloud('aws',  :provider => config['provider'],
+          :region =>config['region'],
+          :endpoint => 'https://ec2.eu-west-1.amazonaws.com/',
+          :aws_access_key_id => config['aws_access_key_id'],
+          :aws_secret_access_key => config['aws_secret_access_key'])
+    .create_instances(
+        ['m3.medium=>in2', 'm3.large=>in3', 'c3.large=>in4', 'c3.xlarge=>in5'],
+        :username => 'ubuntu',
+        :private_key_path =>config['path_to_pem_file'],
+        :key_name => config['key_name'],
+        :image_id => config['image_id'],
+        :groups => config['groups'])
+  end
+  foreach ['in2', 'in3', 'in4', 'in5'] do |instance_name|
+    on instance_name do
+      before "set up docker" do
+        ssh "yes | sudo apt-get update"
+        ssh "yes | sudo apt-get install build-essential"
+
+        ssh "yes | sudo apt-get install python-software-properties python g++ make"
+        ssh "yes | sudo add-apt-repository ppa:chris-lea/node.js"
+        ssh "yes | sudo apt-get update"
+        ssh "yes | sudo apt-get install nodejs"
+
+        ssh "yes | sudo apt-get install git-core"
+
+        ssh "yes | sudo apt-get -y install build-essential zlib1g-dev libssl-dev libreadline6-dev libyaml-dev"
+        ssh "yes | sudo apt-get install ruby2.0 ruby2.0-dev build-essential libxml2-dev libxslt1-dev"
+        ssh "yes | sudo apt-get install rabbitmq-server"
+        ssh "yes | sudo apt-get install redis-server"
+
+        ssh "echo 'Git:'"
+        ssh "git --version"
+        ssh "echo 'Ruby'"
+        ssh "ruby --version"
+        ssh "echo 'Nodejs'"
+        ssh "nodejs --version"
+        ssh "echo 'Redis'"
+        ssh "redis-server --version"
+
+        ssh "curl -O http://pegasus.isi.edu/montage/Montage_v3.3_patched_4.tar.gz"
+        ssh "tar zxvf Montage_v3.3_patched_4.tar.gz"
+        ssh "cd Montage_v3.3_patched_4 && make"
+
+        ssh "curl -O https://dl.dropboxusercontent.com/u/81819/hyperflow-amqp-executor.gem"
+        ssh "sudo gem2.0 install --no-ri --no-rdoc hyperflow-amqp-executor.gem"
+
+        ssh "wget https://github.com/dice-cyfronet/hyperflow/archive/v1.0.0-beta-6.tar.gz"
+        ssh "tar zxvf v1.0.0-beta-6.tar.gz"
+        ssh "mv hyperflow-1.0.0-beta-6 hyperflow"
+        ssh "cd hyperflow && npm install"
+
+        ssh "cd hyperflow && echo 'amqp_url: amqp://localhost\nstorage: local\nthreads: <%= Executor::cpu_count %>' > executor_config.yml"
+
+        ssh "mkdir data"
+        ssh "cd data && wget https://gist.github.com/kfigiela/9075623/raw/dacb862176e9d576c1b23f6a243f9fa318c74bce/bootstrap.sh"
+        ssh "cd data && chmod +x bootstrap.sh"
+
+      end
+      foreach [1, 2, 3] do |a|
+        foreach [0.1, 0.25, 0.4] do |size|
+          execute do
+
+            ssh "export PATH=$PATH:~/Montage_v3.3_patched_4/bin && ./data/bootstrap.sh #{size}#{a}"
+
+            ssh "cd hyperflow && echo \"var AMQP_URL=process.env.AMQP_URL?process.env.AMQP_URL:'amqp://localhost:5672';exports.amqp_url=AMQP_URL;exports.options={'storage':'local','workdir':'/home/ubuntu/#{size}#{a}/input'}\" > functions/amqpCommand.config.js"
+
+            ssh "export PATH=$PATH:~/Montage_v3.3_patched_4/bin; nohup hyperflow-amqp-executor $(pwd)/hyperflow/executor_config.yml > amqp.out 2>&1 &"
+            ssh "nodejs hyperflow/scripts/dax_convert_amqp.js #{size}#{a}/workdir/dag.xml > #{size}#{a}/workdir/dag.json"
+            # ssh "nohup nodejs hyperflow/scripts/runwf.js -f #{size}/workdir/dag.json -s > amqp.out 2>&1 &"
+          end
+        end
+      end
+    end
+  end
+end
+
+result = execution_tree.run
+ap result
+
+puts result.to_json

data/examples/ping.rb

@@ -13,14 +13,14 @@ execution_tree = experiment 'Hello' do
           :endpoint => 'https://ec2.eu-west-1.amazonaws.com/',
           :aws_access_key_id => config['aws_access_key_id'],
           :aws_secret_access_key => config['aws_secret_access_key'])
-    .create_instances(['t1.micro=>in1'],
+    .create_instances(['t2.micro=>in1'],
                       :username => 'ubuntu',
-                      :private_key_path =>'./Piotr-key-pair-irleand.pem',
-                      :key_name => 'Piotr-key-pair-irleand',
-                      :groups => ['Piotr-irleand'])
-    manual('kali', '172.16.0.109', 'root', :password => 'toor')
+                      :private_key_path =>config['path_to_pem_file'],
+                      :key_name => config['key_name'],
+                      :image_id => config['image_id'],
+                      :groups => config['groups'])
   end
-  foreach ['kali', 'in1'] do |instance_name|
+  foreach ['in1'] do |instance_name|
     concurrent do
       on 'localhost' do
         execute do

data/lib/remote_execution/provider.rb

@@ -3,6 +3,7 @@ require 'cloud_provider.rb'
 require 'json'
 
 $virtual_machines = Hash.new
+$vms_manager
 
 class MachineManager
   attr_accessor :cloud_providers
@@ -40,6 +41,12 @@ class MachineManager
     end
   end
 
+  def destroy_machines
+    cloud_providers.each do |provider|
+      provider.destroy
+    end
+  end
+
 end
 
 def load_config( filename )
@@ -49,7 +56,7 @@ def load_config( filename )
 end
 
 def define_providers (&block)
-  MachineManager.new &block
+  $vms_manager = MachineManager.new &block
 end
 
 def getInstance(name)

data/lib/tree_building/claudius.rb

@@ -95,6 +95,7 @@ class Experiment
 
   def run
     @root.run(nil)
+    # $vms_manager.destroy_machines
   end
 
   def export_tree(path = 'execution_tree')

data/lib/tree_building/execution_node.rb

@@ -17,7 +17,6 @@ class ExecutionNode < Node
             :before => 0,
             :exec => [],
         }
-
   end
 
   def draw_block(graph)

data/lib/tree_building/node.rb

@@ -25,7 +25,6 @@ class Node
             :name => @name,
             :exec => []
         }
-
   end
 
   def run(instance)

data/test/test_local_execution.rb

@@ -8,11 +8,12 @@ class Test_local_execution < Test::Unit::TestCase
   end
 
   def test_simple
-    experiment 'Hello' do
+    helloExp = experiment 'Hello' do
       execute do
         ssh 'mkdir Hello123456'
       end
     end
+    helloExp.run()
     result = `ls -la`
     assert_equal(true, result.include?('Hello123456'), 'Test should create a folder in current directory')
   end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: claudius
 version: !ruby/object:Gem::Version
-  version: 0.0.3
+  version: 0.0.4
   prerelease: 
 platform: ruby
 authors:
@@ -10,7 +10,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2014-06-07 00:00:00.000000000 Z
+date: 2014-12-17 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
@@ -49,7 +49,7 @@ dependencies:
   requirement: !ruby/object:Gem::Requirement
     none: false
     requirements:
-    - - ! '>='
+    - - '='
       - !ruby/object:Gem::Version
         version: 1.22.0
   type: :runtime
@@ -57,7 +57,7 @@ dependencies:
   version_requirements: !ruby/object:Gem::Requirement
     none: false
     requirements:
-    - - ! '>='
+    - - '='
       - !ruby/object:Gem::Version
         version: 1.22.0
 - !ruby/object:Gem::Dependency
@@ -108,9 +108,17 @@ dependencies:
     - - '='
       - !ruby/object:Gem::Version
         version: 1.2.0
-description: Write a gem description
+description: Claudius is an easy-to-use domain specific language for cloud experiments.
+  It has been designed to speed up process of building distributed experiments and
+  highly reduce time of remote machines configuration. To avoid vendor lock-in, Claudius
+  was build on top of fog.io library, which enables flexible and powerful way to manage
+  machine instances at various cloud providers. Remote commands execution is based
+  on  SSH protocol (SSH-2). DLS allow users to generate readable execution graph,
+  which is extremely useful for experiment flow verification and help avoid wasting
+  your money.
 email:
 - rachamot@gmail.com
+- piotr.skibiak@gmail.com
 executables: []
 extensions: []
 extra_rdoc_files: []
@@ -122,6 +130,9 @@ files:
 - Rakefile
 - claudius.gemspec
 - examples/_
+- examples/distributed_app.rb
+- examples/distributed_montage.rb
+- examples/montage.rb
 - examples/on_instance.rb
 - examples/ping.rb
 - examples/simple_program.rb
@@ -163,7 +174,7 @@ rubyforge_project:
 rubygems_version: 1.8.23
 signing_key: 
 specification_version: 3
-summary: Write a gem summary
+summary: DSL for cloud experiments
 test_files:
 - test/execution_test.rb
 - test/gem_usage_test.rb