kameleon-builder 2.1.0 → 2.1.1

data/CHANGELOG.rst

@@ -0,0 +1,87 @@
+Kameleon CHANGELOG
+==================
+
+version 2.1.0
+-------------
+
+Released on May 12th 2014
+
+- [core] Fixed psych yaml parsing (#1)
+- [core] Changed option ``--no-no-color`` to ``--color``
+- [core] Saved the contexts state files in their WORKDIR (#3)
+- [core] Set context in/out/local cmd to /bin/bash by default (#5)
+- [core] Made global section non mandatory
+- [core] Made writing embedded step in recipe possible (#12)
+- [core] Improved the readability of logs and the progress bar
+- [core] Moved aliases and checkpoints folders to steps
+- [core] Removed the ``recipes`` folder and the ``workspace`` (#2)
+- [core] Make a safe copy with ``kameleon new`` command
+- [core] Added a simple extend recipe feature (#11)
+- [core] Introduced the keyword "@base" in the extended recipes (#11)
+- [core] Don't log identifier of microstep during build process
+- [core] Added ``kameleon import`` command (#11)
+- [core] Added ``--clean`` option to ``kameleon build`` command
+- [core] Added the lazy context initialization (#10)
+- [core] Set the variable ``KAMELEON_WORKDIR`` for all contexts
+- [core] Used ``KAMELEON_WORKDIR`` when working with PIPE
+- [core] Added persistent cache feature to Kameleon, So far it is caching just packages comming from the network using Polipo
+- [template] Added new templates :
+
+  - archlinux
+  - archlinux-desktop
+  - debian-testing
+  - debian7
+  - debian7-desktop
+  - debian7-oar-dev
+  - fedora-rawhide
+  - fedora20
+  - fedora20-desktop
+  - ubuntu-12.04
+  - ubuntu-12.04-desktop
+  - ubuntu-14.04
+  - ubuntu-14.04-desktop
+  - vagrant-debian7
+- [template] Installed the extlinux bootloader depending on distributions
+- [template] New way to bootstrap fedora using Liveos image
+- [template] Installed linux kernel and extlinux bootloader from bootstrap section
+- [template] Used parted instead of sfdisk
+- [template] Added save_as_qed step
+- [template] Removed insecure ssh key before any export
+- [template] Added shell auto-completion for bash, zsh and fish shell
+- [template] Default user group is sudo
+- [template] Added a new qemu/kvm template with full-snapshot support
+- [template] Ability to add user in multiple groups (with usermod -G)
+- [template] Improved I/O performance with qemu/kvm
+- [template] Removed force-unsafe-io for dpkg to avoid corrupted filesystem
+- [template] Used qemu by default instead of chroot
+- [template] Added option to disable debootstrap cache
+- [template] Refactor qcow2 backing file checkpoints
+- [template] Make QEMU checkpoint more robust and avoid disk corruption
+- [template] Major revision of steps to make it easier to use in different templates
+- [template] Rename steps for more semantic consistency
+- [template] Making the 'save_appliance' step not dependent on the state of the machine
+- [template] Enabled cache for arch_bootstrap
+- [template] Added openssh in arch-bootstrap and enabled sshd.service/dhcp.service
+- [template] Added user 'nobody' to allow sshd  to run in the archlinux virtual machine
+- [template] Enabled checkpoints (backing-file) only in the "setup" stage
+- [template] Fixed .ssh and authorized_keys permissions
+- [template] Avoid crash of in_context when we send a shutdown command to the virtual machine
+- [template] Exclude special files with rsync (proc/dev...) when copying rootfs to the disk
+- [template] Force stop qemu if still running
+- [template] Make debian-chroot depreciated
+- [template] Refactor archlinux template to use it with qemu/kvm
+- [template] Improved the LiveOS fedora bootstrap step to get the system running with qemu/kvm
+- [template] Refactor fedora20/debian8 templates to use them with qemu/kvm
+- [template] Set timezone to UTC by default
+- [template] Used ProxyCommand to improve the debian7-g5k recipe
+- [aliases] Updated write_file and append_file aliases to support double quotes
+- [aliases] Defined new aliases for unmounting devices
+- [docs] More documentation
+
+
+version 2.0.0
+=============
+
+Released on February 17th 2014
+
+Initial public release of kameleon 2

data/docs/Makefile

@@ -65,7 +65,7 @@ singlehtml:
 	@echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml."
 
 publish: html
-	rsync -avh -e "ssh" $(BUILDDIR)/html/ kameleon.website:~/www
+	rsync -avh --exclude 'pkg' -e "ssh" $(BUILDDIR)/html/ kameleon.website:~/www
 
 
 pickle:

data/docs/source/context.rst

@@ -7,7 +7,9 @@ Context
 To understand how Kameleon works you have to get the *context* notion. A context
 is an execution environnement with its variables (like $PATH, $TERM,...), its
 tools (debootstrap, yum, ...) and all its specifics (filesystem, local/remote,
-...). It also manage the connections to your context and make it easy and reliable. 
+...).
+
+It also manage the connections to your context and make it easy and reliable.
 When you build an appliance you deal with 3 contexts:
 
 - The *local* context which is the Kameleon execution environnement

data/docs/source/debian7.yaml

@@ -0,0 +1,128 @@
+#==============================================================================
+# vim: softtabstop=2 shiftwidth=2 expandtab fenc=utf-8 cc=81 tw=80
+#==============================================================================
+#
+# DESCRIPTION: Debian 7 (Wheezy) base system
+#
+#==============================================================================
+
+---
+# Loads some helpful aliases
+aliases: defaults.yaml
+# Enables qemu checkpoint
+checkpoint: qemu.yaml
+#== Global variables use by Kameleon engine and the steps
+global:
+  ## User varibales : used by the recipe
+  user_name: kameleon
+  user_password: $$user_name
+
+  # Distribution
+  distrib: debian
+  release: wheezy
+  arch: amd64
+
+  ## QEMU options
+  qemu_enable_kvm: true
+  qemu_cpu: 2
+  qemu_memory_size: 512
+  qemu_monitor_port: 10023
+  qemu_ssh_port: 55423
+  qemu_arch: x86_64
+
+  ## Disk options
+  nbd_device: /dev/nbd1
+  image_disk: $$kameleon_cwd/base_$$kameleon_recipe_name.qcow2
+  image_size: 10G
+  filesystem_type: ext4
+
+  # rootfs options
+  rootfs: $$kameleon_cwd/rootfs
+  rootfs_download_path: /var/cache/kameleon/$$distrib/$$release/$$arch/rootfs
+
+  ## System variables. Required by kameleon engine
+  # Include specific steps
+  include_steps:
+    - $$distrib/$$release
+    - $$distrib
+
+  # Apt options
+  apt_repository: http://ftp.debian.org/debian/
+  apt_enable_contrib: true
+  apt_enable_nonfree: true
+  apt_install_recommends: false
+
+  # Shell session from where we launch exec_out commands. There is often a
+  # local bash session, but it can be a remote shell on other machines or on
+  # any shell. (eg. bash, chroot, fakechroot, ssh, tmux, lxc...)
+  out_context:
+    cmd: bash
+    workdir: $$kameleon_cwd
+
+  # Shell session that allows us to connect to the building machine in order to
+  # configure it and setup additional programs
+  ssh_config_file: $$kameleon_cwd/ssh_config
+  in_context:
+    cmd: LC_ALL=POSIX ssh -F $$ssh_config_file $$kameleon_recipe_name -t /bin/bash
+    workdir: /
+
+#== Bootstrap the new system and create the 'in_context'
+bootstrap:
+  - debootstrap:
+    - include_pkg: >
+        ifupdown locales libui-dialog-perl dialog isc-dhcp-client netbase
+        net-tools iproute acpid openssh-server pciutils extlinux
+        linux-image-$$arch
+    - release: $$release
+    - arch: $$arch
+    - repository: $$apt_repository
+    - enable_cache: true
+  - initialize_disk_qemu
+  - prepare_qemu
+  - install_bootloader
+  - start_qemu
+
+#== Install and configuration steps
+# WARNING: this part should be independante from the out context (whenever
+# possible...)
+setup:
+  # Install
+  - configure_apt:
+    - repository: $$apt_repository
+    - enable_contrib_repo: $$apt_enable_contrib
+    - enable_nonfree_repo: $$apt_enable_nonfree
+    - install_recommends: $$apt_install_recommends
+  - upgrade_system:
+    - dist_upgrade: true
+  - install_software:
+    - packages: >
+        debian-keyring ntp zip unzip rsync sudo less vim bash-completion
+  - configure_kernel:
+    - arch: $$arch
+  # Configuration
+  - configure_system:
+    - locales: POSIX C en_US fr_FR de_DE
+    - lang: en_US.UTF-8
+    - timezone: UTC
+  - configure_keyboard:
+    - layout: "us,fr,de"
+  - configure_network:
+    - hostname: kameleon-$$distrib
+  - create_group:
+    - name: admin
+  - create_user:
+    - name: $$user_name
+    - groups: sudo admin
+    - password: $$user_password
+
+#== Export the generated appliance in the format of your choice
+export:
+  - save_appliance:
+    - input: $$image_disk
+    - output: $$kameleon_cwd/$$kameleon_recipe_name
+    - save_as_qcow2
+    # - save_as_qed
+    # - save_as_tgz
+    # - save_as_raw
+    # - save_as_vmdk
+    # - save_as_vdi

data/docs/source/debian_customized.yaml

@@ -0,0 +1,28 @@
+#==============================================================================
+# vim: softtabstop=2 shiftwidth=2 expandtab fenc=utf-8 cc=81 tw=80
+#==============================================================================
+#
+# DESCRIPTION: <MY RECIPE DESCRIPTION>
+#
+#==============================================================================
+
+---
+extend: debian7
+
+global:
+    # You can see the base template `debian7.yaml` to know the
+    # variables that you can override
+    # g5k_user: cruizsanabria  # CHANGE ME
+    # g5k_site: grenoble  # CHANGE ME
+
+bootstrap:
+  - @base
+
+setup:
+  - @base
+  - install_software:
+    - packages: >
+        g++ make taktuk openssh-server openmpi-bin openmpi-common openmpi-dev
+#  - tau_install
+export:
+  - @base

data/docs/source/debian_customized_g5k.yaml

@@ -0,0 +1,21 @@
+
+---
+extend: debian7-g5k
+
+global:
+    # You can see the base template `debian7.yaml` to know the
+    # variables that you can override
+    g5k_user: cruizsanabria  # CHANGE ME
+    g5k_site: grenoble  # CHANGE ME
+
+bootstrap:
+  - @base
+
+setup:
+  - @base
+  - install_software:
+    - packages: >
+        g++ make taktuk openssh-server openmpi-bin openmpi-common openmpi-dev
+  - tau_install
+export:
+  - @base

data/docs/source/grid5000_tutorial.rst

@@ -2,27 +2,27 @@
 Grid'5000 Tutorial
 ==================
 
-This tutorial will introduce Kameleon a tool to build software appliances that can be
-deployed on different infrastructures such as: virtualization, cloud computing, baremetal, etc.
+This tutorial will introduce Kameleon, a tool to build software appliances.
+With Kameleon it is possible to generate appliances that can be deployed on different virtualization hypervisors or on baremetal.
+It targets an important activity in Grid'5000 which is the customization of the experimental environments.
 
 ---------------
 Kameleon basics
 ---------------
 
-First of all, let's see all the syntax flavors that Kameleon have to offer.
-From this point, we assume that kameleon have been installed and it's already working
-in your system, otherwise will refer to[1].
-Kameleon can be seen as a shell sequencier which will boost your shell scripts.
+First of all, let's see all the syntax flavors that *Kameleon* has to offer.
+From this point, we assume that *Kameleon* have been installed and it's already working
+in your system, otherwise go to :ref:`installation` section.
+Kameleon can be seen as a shell sequencer which will boost your shell scripts.
 It is based on the execution of shell scripts but it provides some syntax sugar that makes
-the work with shell scripts less painfull.
-
-We will start with the basics
+the work with shell scripts less painful.
+Let's start with the basics
 
 Kameleon Hello world
 ~~~~~~~~~~~~~~~~~~~~
 
-Everything we want to build have to be specified by a recipe. Kameleon will read this recipe
-and it will execute the appropiate actions. Let's create a hello world recipe for kameleon.
+Everything we want to build have to be specified by a recipe. Kameleon reads this recipe
+and executes the appropriate actions. Let's create a hello world recipe using Kameleon.
 Open a text editor and write the following::
 
      setup:
@@ -31,10 +31,10 @@ Open a text editor and write the following::
          - exec_local: echo "Hello world"
      # The end
 
-save the privious file as a YAML file. For instance hello_world.yaml.
+Save the previous file as a YAML file. For instance, hello_world.yaml.
 
 .. note::
-    Be sure of respecting the YAML syntax `yaml`_.
+    Be sure of respecting the YAML syntax and indentation `yaml`_.
 
 .. _yaml: http://www.yaml.org/
 
@@ -66,45 +66,448 @@ You will have some output that looks like this::
       [kameleon]: Log file : /home/cristian/Repositories/exptools/setup_complex_exp/tests/new_version/kameleon.log
 
 With this simple example, we have already introduced most of the Kameleon concepts and syntax.
-First, how recipes are structured. Which could be done using: sections, steps, microsteps.
+First, how recipes are structured using a hierarchy composed of: sections, steps, microsteps.
 
-* Sections: The sections correspond to the minimal actions that have to be performed in order to have a software
+* Sections: correspond to the minimal actions that have to be performed in order to have a software
   stack that can be run almost anywhere. This brings to Kameleon a high degree of customizability, reuse of
-  code and users have total control in when and where the
+  code and users have total control over when and where the
   sections have to take place. This minimal actions are: bootstrap, setup and export.
 
-* Steps: It refers to a specific action to be done inside a section.
+* Steps: It refers to a specific action to be done inside a section
+  (e.g., software installation, network configuration, configure kernel).
   Steps can be declared in independent files that improves the degree of reusability.
 
 * Microsteps: procedures composed of shell commands. The goal of dividing steps into microsteps is the
-  possibility of activating certain actions within a step.
-
-The Kameleon hierarchy encourages the reuse (shareability) of code and modularity of procedures.
+  possibility of activating certain actions within a step and performing a better checkpoint.
 
-The minimal building block are the commands exec_ which wraps shell commands adding
+Kameleon hierarchy encourages the reuse (shareability) of code and modularity of procedures.
+The minimal building block are the commands *exec_* which wraps shell commands adding
 a simple error handling and interactivenes in case of a problem.
 These commands are executed in a given context. Which could be: local, in, out.
-That are going to be defined later. They can be used as follows::
+They can be used as follows::
 
      setup:
-     - first_step:
-       - hello_microstep:
-         - exec_local: echo "Hello world"
-	 - exec_in: echo "Hello world"
-	 - exec_out: echo "Hello world"
+       - first_step:
+         - hello_microstep:
+           - exec_local: echo "Hello world"
+	   - exec_in: echo "Hello world"
+	   - exec_out: echo "Hello world"
      # The end
 
 
 * Local context: It represents the Kameleon execution environment. Normally is the user’s machine.
 
 * OUT context: It is where the appliance will be bootstraped. Some procedures have to be carried out in
-  order create the place where the software appliance is built (In context).
-  This can be: the same user’s machine using chroot.
+  order to create the place where the software appliance is built (In context).
+  One example is: the same user’s machine using chroot.
   Thus, in this context is where the setup of the chroot takes place.
-  Establishing the proper environmental variables in order to have a clean environment.
-  Other examples are: setting up a virtual machine, access to an infrastructure in order to get an instance and be able to deploy, setting
+  Other examples are: setting up a virtual machine, accessing an infrastructure in order to get a reservation and be able to deploy, setting
   a Docker container, etc.
 
-* IN context: It makes reference to inside the newly
+* IN context: It refers to inside the newly
   created appliance. It can be mapped to a chroot,
   virtual machine, physical machine, Linux container, etc.
+
+In the last example all the contexts are executed on the user's machine.
+Which is the default behavior that can be customized (it will be shown later on this tutorial).
+Most of the time, users take advantage of the *In context* in order to customize a given a appliance.
+
+We can add variables as well::
+
+     setup:
+       - first_step:
+         - message: "Hello world"
+         - hello_microstep:
+           - exec_local: echo "Variable value $$message"
+
+
+Let's apply the syntax to a real example in the next section.
+
+----------------------------------------
+Building a simple Debian based appliance
+----------------------------------------
+
+Kameleon already provides tested recipes for building different software appliances based
+on different Linux flavors. We can take a look at the provided templates by typing::
+
+     $ kameleon templates
+
+Which will output::
+
+    The following templates are available in /home/cristian/Repositories/kameleon_v2/templates:
+    NAME                 | DESCRIPTION
+    ---------------------|-------------------------------------------------------------
+    archlinux            | Build an Archlinux base system system.
+    archlinux-desktop    | Archlinux GNOME Desktop edition.
+    debian-testing       | Debian Testing base system
+    debian7              | Debian 7 (Wheezy) base system
+    debian7-desktop      | Debian 7 (Wheezy) GNOME Desktop edition.
+    debian7-oar-dev      | Debian 7 dev appliance with OAR-2.5 (node/server/frontend).
+    fedora-rawhide       | Fedora Rawhide base system
+    fedora20             | Fedora 20 base system
+    fedora20-desktop     | Fedora 20 GNOME Desktop edition
+    old-debian7          | [deprecated] Build a debian wheezy appliance using chroot...
+    ubuntu-12.04         | Ubuntu 12.04 LTS (Precise Pangolin) base system.
+    ubuntu-12.04-desktop | Ubuntu 12.04 LTS (Precise Pangolin) Desktop edition.
+    ubuntu-14.04         | Ubuntu 14.04 LTS (Trusty Tahr) base system.
+    ubuntu-14.04-desktop | Ubuntu 14.04 LTS (Trusty Tahr) Desktop edition.
+    vagrant-debian7      | A standard Debian 7 vagrant base box
+
+
+Let's import the template debian7::
+
+    $ kameleon import debian7
+
+This will generate the following files in the current directory::
+
+    ├── debian7.yaml
+    ├── kameleon.log
+    └── steps
+        ├── aliases
+        |   └── defaults.yaml
+	├── bootstrap
+	│   ├── debian
+	│   │   └── debootstrap.yaml
+	│   ├── initialize_disk_qemu.yaml
+	│   ├── install_bootloader.yaml
+	│   ├── prepare_qemu.yaml
+	│   └── start_qemu.yaml
+	├── checkpoints
+	│   └── qemu.yaml
+	├── export
+	│   └── save_appliance.yaml
+	└── setup
+	    ├── create_group.yaml
+	    ├── create_user.yaml
+	    └── debian
+	        ├── configure_apt.yaml
+		├── configure_kernel.yaml
+		├── configure_keyboard.yaml
+		├── configure_network.yaml
+		├── configure_system.yaml
+		├── install_software.yaml
+		└── upgrade_system.yaml
+
+     8 directories, 19 files
+
+Here we can observe that a directory has been generated.
+This directory contains all the steps needed to build the final software appliance.
+These steps are organized by sections. There is a directory checkpoints that is going
+to be explained later on.
+
+Here we can notice that all the process of building is based on steps files written with Kameleon syntax.
+Separating the steps in different files gives a high degree of reusability.
+
+The recipe looks like this:
+
+.. literalinclude:: debian7.yaml
+   :lines: 69-125
+
+The previous recipe build a debian wheezy using qemu.
+It looks verbose but normally you as user you wont see it.
+You will use it as a template in a way that will be explained later.
+The recipe specify all the steps, configurations values that are going to be used
+to build the appliance. Kameleon recipes gives many details to you, few things are hidden.
+Which is good for reproducibility purposes and when reporting bugs.
+
+If we have all the dependencies required as qemu, qemu-tools and debootstrap we can start to build the appliance
+doing the following::
+
+     $ kamelon build debian7.yaml
+
+The process will start and in about few minutes
+a directory called builds will be generated in the current directory,
+you will have a qemu virtual disk with a base debian wheezy installed in it.
+That you can try out by executing::
+
+     $ sudo qemu-system-x86_64 -enable-kvm builds/debian7/debian7.qcow2
+
+
+
+Customizing a software appliance
+================================
+
+Now, lets customize a given template in order to create a software appliance that have OpenMPI, Taktuk and tools necessary to compile source code.
+Kameleon allows us to extend a given template. We will use this for adding the necessary software. Type the following::
+
+     $ kameleon new debian_customized debian7
+
+This will create the file debian_customized.yaml which contents are::
+
+     ---
+     extend: debian7
+
+     global:
+     # You can see the base template `debian7.yaml` to know the
+     # variables that you can override
+
+     bootstrap:
+       - @base
+
+     setup:
+       - @base
+
+     export:
+       - @base
+
+If we try to build this recipe, it will generate the exact same image as before.
+But the idea here is to change it in order to install the desired software.
+Therefore, we will modify the setup section like this::
+
+     extend: debian7
+
+     global:
+     # You can see the base template `debian7.yaml` to know the
+     # variables that you can override
+
+     bootstrap:
+       - @base
+
+     setup:
+       - @base
+       - install_software:
+         - packages: >
+            g++ make taktuk openssh-server openmpi-bin openmpi-common openmpi-dev
+
+     export:
+       - @base
+
+
+For building execute::
+
+     $ kameleon build debian_customized.yaml
+
+Then, you can follow the same steps as before to try it out and verify that the software was installed.
+Now, let's make things a little more complicated. We will now compile and install TAU in our system.
+So, for that let's create a step file that will look like this::
+
+     - get_tau:
+       - exec_in: cd /tmp/
+       - exec_in: wget  -q http://www.cs.uoregon.edu/research/tau/tau_releases/tau-2.22.2.tar.gz
+       - exec_in: wget -q http://www.cs.uoregon.edu/research/tau/pdt_releases/pdt-3.19.tar.gz
+
+     - pdt_install:
+       - exec_in: cd /tmp/
+       - exec_in: tar -xzf pdt-3.19.tar.gz
+       - exec_in: cd /tmp/pdtoolkit-3.19
+       - exec_in: ./configure -prefix=/usr/local/pdt-install
+       - exec_in: make clean install
+
+     - tau_install:
+       - exec_in: cd /tmp/
+       - exec_in: tar -xzf tau-2.22.2.tar.gz
+       - exec_in: cd /tmp/tau-2.22.2
+       - exec_in: ./configure -prefix=/usr/local/tau-install -pdt=/usr/local/pdt-install/ -mpiinc=/usr/local/openmpi-install/include -mpilib=/usr/local/openmpi-install/lib
+       - exec_in: make install
+
+
+You have to put it under the directory *steps/setup/* and you can call it tau_install.
+In order to use it in your recipe, modify it as follows::
+
+     extend: debian7
+
+     global:
+     # You can see the base template `debian7.yaml` to know the
+     # variables that you can override
+
+     bootstrap:
+       - @base
+
+     setup:
+       - @base
+       - install_software:
+         - packages: >
+            g++ make taktuk openssh-server openmpi-bin openmpi-common openmpi-dev
+       - tau_install
+     export:
+       - @base
+
+
+And rebuild the image again, you will see that it wont start from the beginning.
+It will take advantage of the checkpoint system and it will start from the last
+successfull executed step.
+
+When building there is the following error::
+
+
+     [kameleon]: Step 46 : setup/tau_install/tau_install
+     [kameleon]:  ---> Running step
+        [in_ctx]: Unset ParaProf's cubeclasspath...
+	[in_ctx]: Unset Perfdmf cubeclasspath...
+	[in_ctx]: Error: Cannot access MPI include directory /usr/local/openmpi-install/include
+     [kameleon]: Error occured when executing the following command :
+     [kameleon]:
+     [kameleon]: > exec_in: ./configure -prefix=/usr/local/tau-install -pdt=/usr/local/pdt-install/ -mpiinc=/usr/local/openmpi-install/include -mpilib=/usr/local/openmpi-install/lib
+     [kameleon]: Press [r] to retry
+     [kameleon]: Press [c] to continue with execution
+     [kameleon]: Press [a] to abort execution
+     [kameleon]: Press [l] to switch to local_context shell
+     [kameleon]: Press [o] to switch to out_context shell
+     [kameleon]: Press [i] to switch to in_context shell
+     [kameleon]: answer ? [c/a/r/l/o/i]:
+
+We can observe that the problem is related with the configure script that cannot access the MPI path.
+It can be debugged by using the interactive shell provided by Kameleon.
+The interactive shell allows us to log into a given context.
+For this case we see that the error happened in the in context, so let's type i in order to enter to this context::
+
+  [kameleon]: User choice : [i] launch in_context
+     [in_ctx]: Starting interactive shell
+  [kameleon]: Starting process: "LC_ALL=POSIX ssh -F /tmp/kameleon/debian_customized/ssh_config debian_customized -t /bin/bash"
+  (in_context) root@cristiancomputer: / #
+
+The commands executed by Kameleon remain in the bash history.
+Therefore, It can be rexecuted manually.
+For this case, we only need to change the path for the OpenMPI libraries.
+As we have installed it using the packages they are avaiable under the directories:
+*/usr/include/openmpi/*, */usr/lib/openmpi/* respectively.
+If we try with the following parameters::
+
+    # ./configure -prefix=/usr/local/tau-install -pdt=/usr/local/pdt-install/ -mpiinc=/usr/include/openmpi/ -mpilib=/usr/lib/openmpi/
+
+It will finish without any problem. We have found the bug, therefore we can just logout by typing *exit* and
+then *abort* for stopping the execution and update the step file with the previous line.
+If you carry out the building again you will see that now everything goes smoothly.
+Again Kameleon will use the checkpoint system to avoid starting from scratch.
+
+---------------------------------
+Creating a Grid'5000 environment.
+---------------------------------
+
+Now, let's use the extend and export functionalities for creating a Grid'5000 environment.
+With this step we will see how code can be re-used with Kameleon.
+Therefore, we can extend the recipe created before::
+
+     ---
+     extend: debian_customized
+
+     global:
+         # You can see the base template `debian7.yaml` to know the
+         # variables that you can override
+
+     bootstrap:
+       - @base
+
+     setup:
+       - @base
+
+     export:
+       - save_appliance:
+         - input: $$image_disk
+         - output: $$kameleon_cwd/$$kameleon_recipe_name
+         - save_as_tgz
+
+       - g5k_custom:
+         - kadeploy_file:
+           - write_local:
+             - $$kameleon_cwd/$$kameleon_recipe_name.yaml
+             - |
+               #
+               # Kameleon generated based on kadeploy description file
+               #
+               ---
+               name: $$kameleon_recipe_name
+
+               version: 1
+
+               os: linux
+
+               image:
+                 file: $$kameleon_recipe_name.tar.gz
+                 kind: tar
+                 compression: gzip
+
+               postinstalls:
+                 - archive: server:///grid5000/postinstalls/debian-x64-base-2.5-post.tgz
+                   compression: gzip
+                   script: traitement.ash /rambin
+
+               boot:
+                 kernel: /vmlinuz
+                 initrd: /initrd.img
+
+               filesystem: $$filesystem_type
+
+This recipe will generate in the build directory a tar.gz image and a configuration file for Kadeploy.
+For example::
+
+     $ ls builds
+     total 8831536
+     -rw-r--r-- 1 root root 18767806464 juin  15 23:04 base_debian_g5k.qcow2
+     -rw-r--r-- 1 root root   206403737 juin  15 23:04 debian_g5k.tar.gz
+     -rw-r--r-- 1 root root         379 juin  15 23:04 debian_g5k.yaml
+     -rw-r--r-- 1 root root         426 juin  15 23:03 fstab.orig
+     -rw------- 1 root root         672 juin  15 23:01 insecure_ssh_key
+
+Therefore if we log in a Grid'5000 site for instance (Grenoble) we can submit a deploy job and
+deploy the image using kadeploy::
+
+
+  user@fgrenoble:~$ oarsub -I t deploy
+  [ADMISSION RULE] Set default walltime to 3600.
+  [ADMISSION RULE] Modify resource description with type constraints
+  Generate a job key...
+  OAR_JOB_ID=1663465
+  Interactive mode : waiting...
+  Starting...
+
+  Connect to OAR job 1663465 via the node fgrenoble.grenoble.grid5000.fr
+
+  user@fgrenoble:~$ kadeploy -a debian_g5k.yaml -f $OAR_NODEFILE
+
+
+With luck the image will be deployed on baremetal after some few minutes.
+
+
+Playing with Kameleon contexts
+------------------------------
+
+The environment that has just been deployed is a basic debian.
+It doesn't have the modules required for infiniband and
+other configuration that site administrators do for a specific hardware
+or politics of the site.
+In this case would be good to be able to use the environments already
+provided by Grid'5000. This can be done by using Kameleon contexts.
+The idea is to re-utilize the same recipe we have written before.
+
+Kameleon already provides a recipe for interacting with Grid'5000 where
+the configuration of the context is as follows:
+
+* Local context: it is the user's machine.
+
+* Context out: it is the site frontend.
+  It is used for submitting a job and deploying
+  a given Grid'5000 environment.
+
+* Context in: will be inside the deployed node.
+
+
+First, we import the G5k recipe::
+
+  $ kameleon import debian7-g5k
+
+And we can just make a copy of our previous recipe (debian customized) and
+we call it for instance debian_customized_g5k.yaml.
+This recipe will look like this:
+
+.. literalinclude:: debian_customized_g5k.yaml
+
+
+But there will be a problem with the installation of TAU. Because
+we download the tarball directly from its web site which is an
+operation not allowed in Grid'5000. Just certain sites are accessible
+using a web proxy.
+To solve this we have to modify the step *tau_install* like this:
+
+.. literalinclude:: tau_install_g5k.yaml
+
+Here, we change the context for performing the operation of download.
+For now on, it will be the local context that is going to download the
+tarballs. Then we have to put them into the *in contex* for
+this operation we use a pipe. Pipes are a means for communicating
+contexts. We use a pipe between our local context and the in contex.
+
+With those changes we will be able to build a G5k environment with
+our already tested configuration. The recipe saves
+the environment on the Kameleon workdir on the frontend.
+Thus the environment is accessible to be deployed the number of times needed.