kameleon-builder 2.1.1 → 2.1.3

data/docs/source/grid5000_tutorial.rst

@@ -202,6 +202,7 @@ The recipe looks like this:
 
 .. literalinclude:: debian7.yaml
    :lines: 69-125
+   :language: yaml
 
 The previous recipe build a debian wheezy using qemu.
 It looks verbose but normally you as user you wont see it.
@@ -223,9 +224,9 @@ That you can try out by executing::
      $ sudo qemu-system-x86_64 -enable-kvm builds/debian7/debian7.qcow2
 
 
-
+--------------------------------
 Customizing a software appliance
-================================
+--------------------------------
 
 Now, lets customize a given template in order to create a software appliance that have OpenMPI, Taktuk and tools necessary to compile source code.
 Kameleon allows us to extend a given template. We will use this for adding the necessary software. Type the following::
@@ -279,27 +280,11 @@ For building execute::
 
 Then, you can follow the same steps as before to try it out and verify that the software was installed.
 Now, let's make things a little more complicated. We will now compile and install TAU in our system.
-So, for that let's create a step file that will look like this::
-
-     - get_tau:
-       - exec_in: cd /tmp/
-       - exec_in: wget  -q http://www.cs.uoregon.edu/research/tau/tau_releases/tau-2.22.2.tar.gz
-       - exec_in: wget -q http://www.cs.uoregon.edu/research/tau/pdt_releases/pdt-3.19.tar.gz
+So, for that let's create a step file that will look like this:
 
-     - pdt_install:
-       - exec_in: cd /tmp/
-       - exec_in: tar -xzf pdt-3.19.tar.gz
-       - exec_in: cd /tmp/pdtoolkit-3.19
-       - exec_in: ./configure -prefix=/usr/local/pdt-install
-       - exec_in: make clean install
-
-     - tau_install:
-       - exec_in: cd /tmp/
-       - exec_in: tar -xzf tau-2.22.2.tar.gz
-       - exec_in: cd /tmp/tau-2.22.2
-       - exec_in: ./configure -prefix=/usr/local/tau-install -pdt=/usr/local/pdt-install/ -mpiinc=/usr/local/openmpi-install/include -mpilib=/usr/local/openmpi-install/lib
-       - exec_in: make install
 
+.. literalinclude:: tau_install.yaml
+   :language: yaml
 
 You have to put it under the directory *steps/setup/* and you can call it tau_install.
 In order to use it in your recipe, modify it as follows::
@@ -371,7 +356,7 @@ If you carry out the building again you will see that now everything goes smooth
 Again Kameleon will use the checkpoint system to avoid starting from scratch.
 
 ---------------------------------
-Creating a Grid'5000 environment.
+Creating a Grid'5000 environment
 ---------------------------------
 
 Now, let's use the extend and export functionalities for creating a Grid'5000 environment.
@@ -459,6 +444,7 @@ deploy the image using kadeploy::
 With luck the image will be deployed on baremetal after some few minutes.
 
 
+------------------------------
 Playing with Kameleon contexts
 ------------------------------
 
@@ -471,7 +457,7 @@ provided by Grid'5000. This can be done by using Kameleon contexts.
 The idea is to re-utilize the same recipe we have written before.
 
 Kameleon already provides a recipe for interacting with Grid'5000 where
-the configuration of the context is as follows:
+the configuration of the contexts is as follows:
 
 * Local context: it is the user's machine.
 
@@ -491,7 +477,7 @@ we call it for instance debian_customized_g5k.yaml.
 This recipe will look like this:
 
 .. literalinclude:: debian_customized_g5k.yaml
-
+   :language: yaml
 
 But there will be a problem with the installation of TAU. Because
 we download the tarball directly from its web site which is an
@@ -500,6 +486,7 @@ using a web proxy.
 To solve this we have to modify the step *tau_install* like this:
 
 .. literalinclude:: tau_install_g5k.yaml
+   :language: yaml
 
 Here, we change the context for performing the operation of download.
 For now on, it will be the local context that is going to download the
@@ -511,3 +498,28 @@ With those changes we will be able to build a G5k environment with
 our already tested configuration. The recipe saves
 the environment on the Kameleon workdir on the frontend.
 Thus the environment is accessible to be deployed the number of times needed.
+
+-------------
+Atlas example
+-------------
+
+Here, a more complicated example, where we install the benchmark HPL which
+is used to benchmark and rank supercomputers for the TOP500 list:
+
+.. literalinclude:: atlas_debian_g5k.yaml
+   :language: yaml
+
+We have to add to the *steps/setup* directory the following files *install_atlas.yaml* and *install_hpl.yaml* for installing atlas and hpl respectively,
+Atlas:
+
+.. literalinclude:: install_atlas.yaml
+   :language: yaml
+
+HPL:
+
+.. literalinclude:: install_hpl.yaml
+   :language: yaml
+
+
+.. note::
+   The building of this appliance could take around half an hour.

data/docs/source/install_atlas.yaml

@@ -0,0 +1,25 @@
+
+- download_tarball:
+    - exec_local: |
+               cd /tmp/
+               wget -q $$repository/$$version/atlas$$version.tar.bz2/download -O atlas$$version.tar.bz2
+    - local2in:
+        - /tmp/atlas$$version.tar.bz2
+        - /root/atlas$$version.tar.bz2
+
+- deactivating_cpu_throttling:
+    - exec_in: cpufreq-selector -g performance
+
+- install:
+    - exec_in: cd /root/
+    - exec_in: bash -c "bunzip2 -c atlas$$version.tar.bz2 | tar xfm - "
+    - exec_in: |
+               mv ATLAS ATLAS$$version                          # get unique dir name
+               mkdir ATLAS$$version/Linux_test                         # create BLDdir
+               cd ATLAS$$version/Linux_test/
+               ../configure -b 64 -D c -DPentiumCPS=2400
+               make build
+               make check
+               make ptcheck ## this is for parallel tests
+               make time ## this is interactive
+               make install

data/docs/source/install_hpl.yaml

@@ -0,0 +1,24 @@
+
+- download_tarball:
+    - exec_local: |
+               cd /tmp/
+               wget -q $$repository/hpl-$$version.tar.gz
+    - local2in:
+        - /tmp/hpl-$$version.tar.gz
+        - /root/hpl-$$version.tar.gz
+
+
+- install:
+    - exec_in: cd /root/
+    - exec_in: |
+               tar -xvf hpl-$$version.tar.gz
+               mv hpl-$$version hpl
+
+    - local2in:
+        - $$hpl_makefile
+        - /root/hpl/Make.Linux
+
+    - exec_in: |
+               rm /usr/lib/libf2c.so
+               ln -s /usr/lib/libf2c.a /usr/lib/libf2c.so
+               make -C /root/hpl/ arch=Linux

data/docs/source/installation.rst

@@ -4,13 +4,8 @@
 Installation
 ------------
 
-The easiest way to install and test Kameleon if you
-are using a Debian based linux distribution is by using the deb package available at `kameleon_deb`_.
-
-.. _kameleon_deb: http://kameleon.imag.fr/pkg/kameleon_2.1.0+20140612204940-1_amd64.deb
-
-.. note::
-    On debian based distribution be sure to install the ``ruby-dev`` package first
+Gem package
+-----------
 
 To install the latest release from `RubyGems`_:
 
@@ -18,20 +13,39 @@ To install the latest release from `RubyGems`_:
 
 ::
 
-    gem install kameleon-builder --pre
+    gem install kameleon-builder
 
 Or from source::
 
-    git clone https://github.com/oar-team/kameleon.git
-    cd kameleon
+    git clone https://github.com/oar-team/kameleon.git && cd kameleon
     gem build kameleon-builder.gemspec
-    gem install kameleon-builder-<version>.gem
+    gem install kameleon-builder-*.gem
+
+On debian based distribution be sure to install the ``ruby-dev`` package first
+
+
+Dist packages
+-------------
+
+These packages contain Kameleon and all its dependencies (Ruby, polipo and all
+gems with their native extensions already compiled). These packages are made
+with `omnibus project`_.
 
+.. _`omnibus project`: https://github.com/opscode/omnibus-ruby
 
-Any troubles?
-~~~~~~~~~~~~~
-If you got an error message like this one::
++-------------------------------+------------------------+--------------------------------------------+----------------------------------+
+|                               |        Platform        |                  Download                  |               MD5                |
++-------------------------------+------------------------+--------------------------------------------+----------------------------------+
+| .. image:: _static/debian.png | **Debian 7 64bit**     | `kameleon_2.1.3-omnibus-1_amd64.deb`_      | dc69d6386b1acb5b9434e8c186ad962c |
+|   :align: center              |                        |                                            |                                  |
++-------------------------------+------------------------+--------------------------------------------+----------------------------------+
+| .. image:: _static/ubuntu.png | **Ubuntu 12.04 64bit** | `kameleon_2.1.3-omnibus-1_amd64.deb`_      | dc69d6386b1acb5b9434e8c186ad962c |
+|   :align: center              |                        |                                            |                                  |
++-------------------------------+------------------------+--------------------------------------------+----------------------------------+
+| .. image:: _static/centos.png | **CentOS 6.5 64bit**   | `kameleon-2.1.3_omnibus-1.el6.x86_64.rpm`_ | 7bc5cee07249f5d4c316e5ea885a2949 |
+|   :align: center              |                        |                                            |                                  |
++-------------------------------+------------------------+--------------------------------------------+----------------------------------+
 
-    ``/usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require': cannot load such file -- mkmf (LoadError)``
 
-It's because you need the ``ruby-dev`` package to fit the dependancies.
+.. _`kameleon_2.1.3-omnibus-1_amd64.deb`: http://kameleon.imag.fr/pkg/kameleon_2.1.3-omnibus-1_amd64.deb
+.. _`kameleon-2.1.3_omnibus-1.el6.x86_64.rpm`: http://kameleon.imag.fr/pkg/kameleon-2.1.3_omnibus-1.el6.x86_64.rpm

data/docs/source/persistent_cache.rst

@@ -22,13 +22,13 @@ generated binary using the option ``--proxy_path``. To use, you just have to
 add the option ``--cache`` as an argument of the build command.
 For example::
 
-  kameleon build debian_test -b /tmp/kameleon/ --cache
+  kameleon build my_recipe.yaml -b /tmp/kameleon/ --cache
 
 This will create a tar file in the build directory ``/tmp/kameleon`` called
-``debian_test-cache.tar``.  In order to use this generated cache file in
+``my_recipe-cache.tar``.  In order to use this generated cache file in
 another build, we have just to use the options ``--from_cache`` as follows::
 
-   kameleon build debian_test -b /tmp/kameleon/ --from_cache /tmp/debian_test-cache.tar
+   kameleon build my_recipe.yaml -b /tmp/kameleon/ --from_cache /tmp/my_recipe-cache.tar
 
 
 .. [1] http://www.pps.univ-paris-diderot.fr/~jch/software/polipo/

data/docs/source/recipe.rst

@@ -2,7 +2,7 @@
 Recipe
 ------
 
-Kameleon compute YAML files, named  *recipes*, that describes how you will
+Kameleon reads YAML files, named  *recipes*, that describes how you will
 build your appliance. These files are stored in the root of your :ref:`workspace`.
 A recipe is a hierarchical structure of `Section`_, `Step`_, `Microstep`_ and
 :ref:`commands`. Here is an overview of this structure:
@@ -22,108 +22,22 @@ A recipe is a hierarchical structure of `Section`_, `Step`_, `Microstep`_ and
 The recipe also contains set of `Global variables`_ declaration and some
 imports like :ref:`aliases` and :ref:`checkpoint`.
 
-This is an example of a recipe:
+Here is an example of a recipe:
 
-.. code-block:: yaml
-
-    ---
-    # Loads some helpful aliases
-    aliases: defaults.yaml
-    # Enables qcow2 checkpoint
-    checkpoint: qcow2.yaml
-    #== Global variables use by Kameleon engine and the steps
-    global:
-      ## User varibales : used by the recipe
-      cachedir: /var/cache/kameleon
-      rootfs: $$kameleon_cwd/rootfs
-      user_name: kameleon
-      arch: amd64
-
-      nbd_device: /dev/nbd10
-      container: "$${kameleon_recipe_name}_temp.qcow2"
-
-      distrib: debian
-      release: wheezy
-
-      ## System variables. Required by kameleon engine
-      # Include specific steps
-      include_steps: [$$distrib/$$release, $$distrib]
-
-      # Shell session from where we launch exec_out commands. There is often a
-      # local bash session, but it can be a remote shell on other machines or on
-      # any shell. (eg. bash, chroot, fakechroot, ssh, tmux, lxc...)
-      out_context:
-        cmd: bash
-        workdir: $$kameleon_cwd
-
-      # Shell session that allows us to connect to the building machine in order to
-      # configure it and setup additional programs
-      default_env: "USER=root HOME=/root LC_ALL=POSIX"
-      in_context:
-        cmd: $$default_env chroot $$rootfs bash
-        workdir: /
-
-    #== Bootstrap the new system and create the 'in_context'
-    bootstrap:
-      - debootstrap:
-        - release: $$release
-        - arch: $$arch
-        - repository: http://ftp.fr.debian.org/debian/
-        - rootfs_archive: $$cachedir/$$distrib/$$release/$$arch/debootstrap.tar.gz
-      - prepare_appliance_with_nbd:
-        - mountdir: $$rootfs
-        - image_size: 2G
-        - filesystem_type: ext4
-        - rootfs_archive: $$cachedir/$$distrib/$$release/$$arch/debootstrap.tar.gz
-      - start_chroot:
-        - rootfs: $$rootfs
-
-
-    #== Install and configuration steps
-    # WARNING: this part should be independante from the build context (whenever
-    # possible...)
-    setup:
-      # Install
-      - software_install:
-        - packages: >
-            debian-keyring ntp zip unzip rsync sudo less vim bash-completion
-      - kernel_install:
-        - arch: $$arch
-      # Configuration
-      - system_config:
-        - locales: fr_FR en_US
-        - lang: fr_FR.UTF-8
-        - timezone: UTC
-      - keyboard_config:
-        - layout: "fr,us"
-      - network_config:
-        - hostname: kameleon-$$distrib
-      - create_user:
-        - name: $$user_name
-        - group: admin
-        - password: $$user_name
-
-    #== Export the generated appliance in the format of your choice
-    export:
-      - save_appliance_from_nbd:
-        - filename: "$${kameleon_recipe_name}"
-        - save_as_qcow2
-        # - save_as_tgz
-        # - save_as_raw
-        # - save_as_vmdk
-        # - save_as_vdi
+.. literalinclude:: debian7.yaml
+   :language: yaml
 
 Section
 -------
 
-Each section is a list of steps. Currently, there is 3 sections:
+Each section is a group of steps. Currently, there are 3 sections:
 
 bootstrap
     This section contains the bootstrap of the new system and create the *in*
     context (see :ref:`context`).
 
 setup
-    This one is dedicated to the install and configuration steps.
+    It is dedicated to install and configuration steps.
 
 export
     Export the generated appliance in the format of your choice.

data/docs/source/tau_install.yaml

@@ -0,0 +1,19 @@
+- tau_version: "2.22.2"
+- pdt_version: "3.19"
+- get_tau:
+  - exec_in: cd /tmp/
+  - exec_in: wget  -q http://www.cs.uoregon.edu/research/tau/tau_releases/tau-$$tau_version.tar.gz
+  - exec_in: wget -q http://www.cs.uoregon.edu/research/tau/pdt_releases/pdt-$$pdt_version.tar.gz
+- pdt_install:
+  - exec_in: cd /tmp/
+  - exec_in: tar -xzf pdt-$$pdt_version.tar.gz
+  - exec_in: cd /tmp/pdtoolkit-$$pdt_version
+  - exec_in: ./configure -prefix=/usr/local/pdt-install
+  - exec_in: make clean install
+
+- tau_install:
+  - exec_in: cd /tmp/
+  - exec_in: tar -xzf tau-$$tau_version.tar.gz
+  - exec_in: cd /tmp/tau-$$tau_version
+  - exec_in: ./configure -prefix=/usr/local/tau-install -pdt=/usr/local/pdt-install/ -mpiinc=/usr/local/openmpi-install/include -mpilib=/usr/local/openmpi-install/lib
+  - exec_in: make install

data/docs/source/tau_install_g5k.yaml

@@ -1,24 +1,25 @@
+- tau_version: "2.22.2"
+- pdt_version: "3.19"
 - get_tau:
   - exec_local: cd /tmp/
-  - exec_local: wget  -q http://www.cs.uoregon.edu/research/tau/tau_releases/tau-2.22.2.tar.gz
-  - exec_local: wget -q http://www.cs.uoregon.edu/research/tau/pdt_releases/pdt-3.19.tar.gz
+  - exec_local: wget  -q http://www.cs.uoregon.edu/research/tau/tau_releases/tau-$$tau_version.tar.gz
+  - exec_local: wget -q http://www.cs.uoregon.edu/research/tau/pdt_releases/pdt-$$pdt_version.tar.gz
   - local2in:
-      - /tmp/tau-2.22.2.tar.gz
-      - /tmp/tau-2.22.2.tar.gz
+      - /tmp/tau-$$tau_version.tar.gz
+      - /tmp/tau-$$tau_version.tar.gz
   - local2in:
-      - /tmp/pdt-3.19.tar.gz
-      - /tmp/pdt-3.19.tar.gz
+      - /tmp/pdt-$$pdt_version.tar.gz
+      - /tmp/pdt-$$pdt_version.tar.gz
 - pdt_install:
   - exec_in: cd /tmp/
-  - exec_in: tar -xzf pdt-3.19.tar.gz
-  - exec_in: cd /tmp/pdtoolkit-3.19
+  - exec_in: tar -xzf pdt-$$pdt_version.tar.gz
+  - exec_in: cd /tmp/pdtoolkit-$$pdt_version
   - exec_in: ./configure -prefix=/usr/local/pdt-install
   - exec_in: make clean install
 
 - tau_install:
   - exec_in: cd /tmp/
-  - exec_in: tar -xzf tau-2.22.2.tar.gz
-  - exec_in: cd /tmp/tau-2.22.2
-#  - exec_in: ./configure -prefix=/usr/local/tau-install -pdt=/usr/local/pdt-install/ -mpiinc=/usr/local/openmpi-install/include -mpilib=/usr/local/openmpi-install/lib
-  - exec_in: ./configure -prefix=/usr/local/tau-install -pdt=/usr/local/pdt-install/ -mpiinc=/usr/include/openmpi/ -mpilib=/usr/lib/openmpi/
+  - exec_in: tar -xzf tau-$$tau_version.tar.gz
+  - exec_in: cd /tmp/tau-$$tau_version
+  - exec_in: ./configure -prefix=/usr/local/tau-install -pdt=/usr/local/pdt-install/ -mpiinc=/usr/local/openmpi-install/include -mpilib=/usr/local/openmpi-install/lib
   - exec_in: make install

data/docs/source/workspace.rst

@@ -4,8 +4,8 @@
 Workspace
 ---------
 
-The workspaces are the folders containing your Kameleon recipes and builds. When
-you use ``kameleon new`` a workspace is created if it does not exists. A
+The workspaces are the folders containing your Kameleon recipes and builds.
+When you use ``kameleon new`` the current directory is the workspace. A
 workspace may contains several recipes.
 
 *Be careful*: All the *steps are shared between recipes within a workspace*. So

data/lib/kameleon/cli.rb

@@ -26,7 +26,7 @@ module Kameleon
       templates_path = Kameleon.env.templates_path
       template_path = File.join(templates_path, template_name) + '.yaml'
       begin
-        template_recipe = RecipeTemplate.new(template_path)
+        template_recipe = RecipeTemplate.new(template_path, :strict => false)
       rescue
         raise TemplateNotFound, "Template '#{template_name}' not found. " \
                                 "To see all templates, run the command "\
@@ -49,7 +49,7 @@ module Kameleon
       templates_path = Kameleon.env.templates_path
       template_path = File.join(templates_path, template_name) + '.yaml'
       begin
-        template_recipe = RecipeTemplate.new(template_path)
+        template_recipe = RecipeTemplate.new(template_path, :strict => false)
       rescue
         raise TemplateNotFound, "Template '#{template_name}' not found. " \
                                 "To see all templates, run the command "\
@@ -71,7 +71,7 @@ module Kameleon
       templates_hash = []
       Kameleon.templates_files.each do |f|
         begin
-        recipe = RecipeTemplate.new(f)
+        recipe = RecipeTemplate.new(f, :strict => false)
         templates_hash.push({
           "name" => recipe.name,
           "description" => recipe.metainfo['description'],