kameleon-builder 2.0.0 → 2.1.0

data/docs/source/checkpoint.rst

@@ -1,3 +1,5 @@
+.. _`checkpoint`:
+
 ----------
 Checkpoint
 ----------

data/docs/source/commands.rst

@@ -1,3 +1,5 @@
+.. _`commands`:
+
 --------
 Commands
 --------
@@ -12,8 +14,7 @@ The exec command is a simple command execute, in the given context, the user
 command in argument. The context is specified by the name suffix local, out or
 in like this ``exec_[in/out/local]``.
 
-It is currently used most to execute bash script, but you can use any tools
-callable with bash.
+It is mostly used to execute bash script or other comand line application.
 
 For example this command save the message "Hello world:" in the hello.txt file
 within the workdir of the *in* context:
@@ -40,7 +41,7 @@ to the new_file within the in context workdir:
         - exec_out: cat my_file
         - exec_in: cat > new_file
 
-This command are usually not used directly but with Aliases_.
+This command are usually not used directly but with :ref:`Aliases`.
 
 Hooks
 ~~~~~

data/docs/source/conf.py

@@ -27,8 +27,8 @@ import datetime
 # Add any Sphinx extension module names here, as strings. They can be
 # extensions coming with Sphinx (named 'sphinx.ext.*') or your custom
 # ones.
-extensions = []
-
+extensions = ['sphinx.ext.todo']
+AUTHORS = "Michael Mercier, Salem Harrache"
 # Add any paths that contain templates here, relative to this directory.
 templates_path = ['_templates']
 
@@ -124,7 +124,7 @@ if not on_rtd:  # only import and set the theme if we're building docs locally
 
 # The name of an image file (relative to this directory) to place at the top
 # of the sidebar.
-#html_logo = None
+# html_logo = '_static/kameleon-long.png'
 
 # The name of an image file (within the static path) to use as favicon of the
 # docs.  This file should be a Windows icon file (.ico) being 16x16 or 32x32
@@ -188,14 +188,22 @@ htmlhelp_basename = 'Kameleondoc'
 
 # -- Options for LaTeX output ---------------------------------------------
 
-latex_elements = {}
+latex_logo = '_static/kameleon-long.png'
+
+latex_elements = {
+    'tableofcontents': "\\tableofcontents",
+    'preamble': '''%
+    \pagestyle{plain}
+    \pagenumbering{arabic}
+    ''',
+}
 
 # Grouping the document tree into LaTeX files. List of tuples
 # (source start file, target name, title,
 #  author, documentclass [howto, manual, or own class]).
 latex_documents = [
     ('index', 'Kameleon.tex', u'Kameleon Documentation',
-     u'Salem Harrache, Michael Mercier', 'manual'),
+     AUTHORS, 'howto'),
 ]
 
 # The name of an image file (relative to this directory) to place at the top of
@@ -225,7 +233,7 @@ latex_documents = [
 # (source start file, name, description, authors, manual section).
 man_pages = [
     ('index', 'kameleon', u'Kameleon Documentation',
-     [u'Salem Harrache', u'Michael Mercier'], 1)
+     AUTHORS.split(', '), 1)
 ]
 
 # If true, show URL addresses after external links.
@@ -239,7 +247,7 @@ man_pages = [
 #  dir menu entry, description, category)
 texinfo_documents = [
     ('index', 'Kameleon', u'Kameleon Documentation',
-     u'Salem Harrache, Michael Mercier', 'Kameleon',
+     AUTHORS, 'Kameleon',
      'One line description of project.', 'Miscellaneous'),
 ]
 

data/docs/source/context.rst

@@ -1,11 +1,14 @@
+.. _`context`:
+
 -------
 Context
 -------
 
-To understand how Kameleon work you have to get the *context* notion. A context
-is an execution environnement with his variables (like $PATH, $TERM,...), his
-tools (debootstrap, yum, ...) and all his specifics (filesystem, local/remote,
-...). When you build an appliance you deal with 3 contexts:
+To understand how Kameleon works you have to get the *context* notion. A context
+is an execution environnement with its variables (like $PATH, $TERM,...), its
+tools (debootstrap, yum, ...) and all its specifics (filesystem, local/remote,
+...). It also manage the connections to your context and make it easy and reliable. 
+When you build an appliance you deal with 3 contexts:
 
 - The *local* context which is the Kameleon execution environnement
 - The *out* context where you will bootstrap the appliance

data/docs/source/faq.rst

@@ -2,4 +2,42 @@
 FAQ
 ---
 
-Work in progress...
+.. note::
+  Work in progress...
+
+Why my step file is not valid?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+When you try to build your recipe, you gat this message::
+
+  [global]: The macrostep /path/to/my/step/step.yaml is not valid
+
+It means your step is not a valid YAML file. Be sure that you did not use some tabulations
+instead of spaces or mix dictionnary and list in the same level. It is recommended to (re)read
+the :doc:`recipe` documentation.
+
+I have some troubles with qemu-nbd, what should I do?
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+qemu-nbd is heavily used is some recipes and when the recipe crash it can be left in a 
+bad state. So, when you build you got this::
+
+  qemu-nbd: Failed to bdrv_open ...
+  nbd device /dev/nbd2 is unavailable
+  
+It means the that the ``/dev/ndb2`` device is not available anymore so go to the recipe and
+change the value of ``nbd_device`` to something else like ``/dev/nbd3``.
+
+.. note::
+  This is a cleaning problem and this is just a workaround. If you find an way to be clean
+  the nbd device correctly lest us know!
+
+If the problem persist you can try to reboot your computer or remove the entire build directory.
+
+What is a *mapping value*?
+~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+When you write a command and you got this error::
+  
+  Error: ... mapping values are not allowed in this context at line 6 column 27
+  
+In YAML the mapping value is a ``:`` character. If you put this in a command the YAML parser 
+will not understand and raise an error. Just put some quotes around your command and that's it! 

data/docs/source/getting_started.rst

@@ -2,5 +2,231 @@
 Getting Started
 ---------------
 
+.. note::
+    This page is a work in progress...
 
-Work in progress...
+Installation
+~~~~~~~~~~~~
+
+To install Kameleon have a look at the :doc:`installation` section.
+
+Create a new recipe from template
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+
+First thing to know is that Kameleon is an automation tool for bash. It brings
+the ease of error handling, retry, checkpointing, easy debugging and cleaning
+to your scripts to help you build your software appliance.
+
+The template system built-in allows you quickly build a system and to understand the
+Kameleon basics. Let see the template list::
+
+    kameleon templates
+
+It shows the templates names and descriptions directly from the templates files shipped
+with  Kameleon::
+
+    The following templates are available in /var/lib/gems/1.9.1/gems/kameleon-builder-2.0.0/templates:
+    NAME                 | DESCRIPTION
+    ---------------------|-----------------------------------------------------------
+    fedora-docker        | Fedora base image [Work in progress].
+    debian-wheezy-chroot | Build a debian wheezy appliance using chroot and qemu-nbd.
+    debian-wheezy-docker | Build a debian wheezy appliance using Docker.
+
+Let's pick one of these. The ``debian-wheezy-chroot`` is a good example. Now you
+will create a new recipe from this template.  Let's name it ``debian_test``::
+
+    kameleon new debian_test -t debian-wheezy-chroot
+
+Kameleon make a direct copy of the YAML template recipe file and all
+the other required files like steps or aliases ones. You can see that in the
+``new`` command output::
+
+   [cli]: Cloning template 'debian-wheezy-chroot'
+   [recipe]: Loading /var/lib/gems/1.9.1/gems/kameleon-builder-2.0.0/templates/debian-wheezy-chroot.yaml
+   [recipe]: Loading aliases /var/lib/gems/1.9.1/gems/kameleon-builder-2.0.0/templates/aliases/defaults.yaml
+   [recipe]: Loading checkpoint configuration /var/lib/gems/1.9.1/gems/kameleon-builder-2.0.0/templates/checkpoints/qcow2.yaml
+   [recipe]: Loading macrostep /var/lib/gems/1.9.1/gems/kameleon-builder-2.0.0/templates/steps/bootstrap/debian/debootstrap.yaml
+   [recipe]: Loading macrostep /var/lib/gems/1.9.1/gems/kameleon-builder-2.0.0/templates/steps/bootstrap/prepare_appliance_with_nbd.yaml
+   [recipe]: Loading macrostep /var/lib/gems/1.9.1/gems/kameleon-builder-2.0.0/templates/steps/bootstrap/start_chroot.yaml
+   [recipe]: Loading macrostep /var/lib/gems/1.9.1/gems/kameleon-builder-2.0.0/templates/steps/setup/debian/software_install.yaml
+   [recipe]: Loading macrostep /var/lib/gems/1.9.1/gems/kameleon-builder-2.0.0/templates/steps/setup/debian/kernel_install.yaml
+   [recipe]: Loading macrostep /var/lib/gems/1.9.1/gems/kameleon-builder-2.0.0/templates/steps/setup/debian/system_config.yaml
+   [recipe]: Loading macrostep /var/lib/gems/1.9.1/gems/kameleon-builder-2.0.0/templates/steps/setup/debian/keyboard_config.yaml
+   [recipe]: Loading macrostep /var/lib/gems/1.9.1/gems/kameleon-builder-2.0.0/templates/steps/setup/debian/network_config.yaml
+   [recipe]: Loading macrostep /var/lib/gems/1.9.1/gems/kameleon-builder-2.0.0/templates/steps/setup/create_user.yaml
+   [recipe]: Loading macrostep /var/lib/gems/1.9.1/gems/kameleon-builder-2.0.0/templates/steps/export/save_appliance_from_nbd.yaml
+   [recipe]: Loading recipe metadata
+      [cli]: New recipe "debian_test" as been created in /home/mercierm/kameleon_ws/recipes/debian_test.yaml
+
+You can check that you got all the files in your workspace for example with the
+UNIX ``tree`` command::
+
+    tree
+    .
+    `-- recipes
+        |-- aliases
+        |   `-- defaults.yaml
+        |-- checkpoints
+        |   `-- qcow2.yaml
+        |-- debian_test.yaml
+        `-- steps
+            |-- bootstrap
+            |   |-- debian
+            |   |   `-- debootstrap.yaml
+            |   |-- prepare_appliance_with_nbd.yaml
+            |   `-- start_chroot.yaml
+            |-- export
+            |   `-- save_appliance_from_nbd.yaml
+            `-- setup
+                |-- create_user.yaml
+                `-- debian
+                    |-- kernel_install.yaml
+                    |-- keyboard_config.yaml
+                    |-- network_config.yaml
+                    |-- software_install.yaml
+                    `-- system_config.yaml
+    
+    9 directories, 13 files
+
+To understand this hierarchy please refer to the :doc:`recipe` documentation.
+
+Build my new recipe
+~~~~~~~~~~~~~~~~~~~
+
+.. note::
+    Be sure to be `root` to run the following steps. It is needed for loading
+    modules, chrooting,...
+
+There is no magic in Kameleon, everything is written in YAML:
+from your system bootstrap to its export. It empowers you to customize anything
+you want at anytime during the appliance build. But before changing anything
+just build the template to see how it works::
+
+     kameleon build debian_test
+
+Oups! Maybe you get an error like this::
+
+    ...
+    [engine]: qemu-img is missing
+    [engine]: Press [c] to continue with execution
+    [engine]: Press [a] to abort execution
+    [engine]: Press [l] to switch to local_context shell
+    [engine]: Press [o] to switch to out_context shell
+    [engine]: answer ? [c/a/l/o]:
+
+This is the interactive prompt of Kameleon. 
+It is a powerful tool that offers you the possibility to fix a problem
+if something goes wrong during the build process.
+For this example, the problem is due to the missing ``qemu-img`` binary. 
+So you have to install it on your ``out`` context (to read more about context see the
+:doc:`context` page). Just type the ``o`` key and ``Enter``. Now you are logged
+in your out context. If you are on a Debian based system install the missing
+package::
+
+    apt-get install qemu-utils
+
+Press ``Ctrl-d`` or type ``exit`` to go back to the Kameleon prompt then press
+``c`` and ``Enter`` to continue the build.
+
+The first time it will take a while to finish the building process. But, Thanks
+to the :doc:`checkpoint` mechanism you can abort with ``Ctrl-c`` anytime during
+the build without problem. Every step is backed up and if you start the
+build again, it will skip all the steps already done 
+to restart at the point you have just stopped. Moreover, if you change anything in the recipe
+Kameleon will know it (using recipe and steps hashes), so your next build will
+automatically restart at the right steps. Here is a good example: The first
+time you built your recipe you should have something like this::
+
+    ...
+    [cli]: Build recipe 'debian_test' is completed !
+    [cli]: Build total duration : 424 secs
+    ...
+
+Now you can run your appliance using qemu::
+
+    qemu-system-x86_64 -enable-kvm builds/debian_test/debian_test.qcow2
+
+.. note::
+    If you do not have access to a graphical server use the ``-curses`` option
+
+How to use the checkpoint
+~~~~~~~~~~~~~~~~~~~~~~~~~
+
+You just have to run the build again and you will notice that it is much faster::
+
+    kameleon build debian_test
+    ...
+    [engine]: Step 1 : bootstrap/_init_bootstrap/_init_0_debootstrap
+    [engine]:  ---> fac7c7045b6f
+    [engine]:  ---> Using cache
+    ...
+    [cli]: Build recipe 'debian_test' is completed !
+    [cli]: Build total duration : 4 secs
+    ...
+
+As you can see Kameleon has used the checkpoint cache for each step and in
+doing so it takes just 4 seconds to build the recipe again. Well the recipe did
+not change so there is no real challenge to build it so fast. Let's change
+the user name for example. Open the ``debian_test.yaml`` recipe file and in the
+global section change the user name like this::
+
+    user_name: my_user
+
+Save the file and re-build the recipe again. This is a part of the outputs you
+should see::
+
+    kameleon build debian_test
+    ...
+    [engine]: Step 29 : setup/create_user/create_group
+    [engine]:  ---> ad19db198510
+    [engine]:  ---> Using cache
+    [engine]: Step 30 : setup/create_user/add_user
+    [engine]:  ---> 6bde599e7ed1
+    [engine]:  ---> Running step
+    [engine]:  ---> Creating checkpoint : 6bde599e7ed1
+    [engine]: Step 31 : setup/create_user/add_group_to_sudoers
+    [engine]:  ---> 28b7a1fae5e2
+    [engine]:  ---> Running step
+    [engine]:  ---> Creating checkpoint : 28b7a1fae5e2 
+    ...
+    [cli]: Build recipe 'debian_test' is completed !
+    [cli]: Build total duration : 29 secs
+    ...
+
+This need a little explanation: You have change the ``user_name`` value in the
+recipe. This variable is firstly used in the ``add_user`` :ref:`microstep`, in the
+create_user :ref:`step` within the setup section. That is why all microsteps
+before this one (the 30 in our case) are using the cache but all the microsteps after
+are build again, to prevent side effects of this change, even if they are not
+using the ``add_user`` value.
+
+Add a new step
+~~~~~~~~~~~~~~
+
+Let's assume that you want to add a step to put a timestamp in your image to
+know when it was built. First, you have to create a step file in the
+``steps/setup`` folder because you want your timestamp to be added inside the
+newly created appliance before exporting it. Let's call it ``add_timestamp.yaml``:
+
+.. literalinclude:: ../../contrib/steps/setup/add_timestamp.yaml
+    :language: yaml
+
+Then you should have this step to the recipe at the end of the setup section::
+
+    ...
+    setup:
+        ...
+        - add_timestamp
+
+Then build again your recipe and run it like before to see that your timestamp
+has been truly added.
+To get more information about steps definition and usage like default
+variable and microstep selection see :ref:`step`.
+
+Advanced Features
+~~~~~~~~~~~~~~~~~
+
+Kameleon gives you a lot of extension and customization possibilities. You can
+define you own :doc:`aliases` and even your own :doc:`checkpoint` mechanism. You
+are invited to go through the rest of the documentation to fully understand
+Kameleon and it's great possibilities.

data/docs/source/grid5000_tutorial.rst

@@ -0,0 +1,110 @@
+==================
+Grid'5000 Tutorial
+==================
+
+This tutorial will introduce Kameleon a tool to build software appliances that can be
+deployed on different infrastructures such as: virtualization, cloud computing, baremetal, etc.
+
+---------------
+Kameleon basics
+---------------
+
+First of all, let's see all the syntax flavors that Kameleon have to offer.
+From this point, we assume that kameleon have been installed and it's already working
+in your system, otherwise will refer to[1].
+Kameleon can be seen as a shell sequencier which will boost your shell scripts.
+It is based on the execution of shell scripts but it provides some syntax sugar that makes
+the work with shell scripts less painfull.
+
+We will start with the basics
+
+Kameleon Hello world
+~~~~~~~~~~~~~~~~~~~~
+
+Everything we want to build have to be specified by a recipe. Kameleon will read this recipe
+and it will execute the appropiate actions. Let's create a hello world recipe for kameleon.
+Open a text editor and write the following::
+
+     setup:
+     - first_step:
+       - hello_microstep:
+         - exec_local: echo "Hello world"
+     # The end
+
+save the privious file as a YAML file. For instance hello_world.yaml.
+
+.. note::
+    Be sure of respecting the YAML syntax `yaml`_.
+
+.. _yaml: http://www.yaml.org/
+
+
+Then, you run it like this::
+
+     kameleon build hello_world.yaml
+
+You will have some output that looks like this::
+
+      [kameleon]: Starting recipe consistency check
+      [kameleon]: Resolving variables
+      [kameleon]: Calculating microstep identifiers
+      [kameleon]: Creating kameleon working directory : /home/cristian/Repositories/exptools/setup_complex_exp/tests/new_version/build/hello_world
+      [kameleon]: Building local context [local]
+      [kameleon]: Building external context [out]
+      [kameleon]: Building internal context [in]
+      [kameleon]: Starting build recipe 'hello_world.yaml'
+      [kameleon]: Step 1 : setup/first_step/hello_microstep
+      [kameleon]:  ---> Running step
+      [kameleon]: Starting process: "bash"
+      [local_ctx]: The local_context has been initialized
+      [local_ctx]: Hello world
+      [kameleon]:
+      [kameleon]: Build recipe 'hello_world.yaml' is completed !
+      [kameleon]: Build total duration : 0 secs
+      [kameleon]: Build directory : /home/cristian/Repositories/exptools/setup_complex_exp/tests/new_version/build/hello_world
+      [kameleon]: Build recipe file : /home/cristian/Repositories/exptools/setup_complex_exp/tests/new_version/build/hello_world/kameleon_build_recipe.yaml
+      [kameleon]: Log file : /home/cristian/Repositories/exptools/setup_complex_exp/tests/new_version/kameleon.log
+
+With this simple example, we have already introduced most of the Kameleon concepts and syntax.
+First, how recipes are structured. Which could be done using: sections, steps, microsteps.
+
+* Sections: The sections correspond to the minimal actions that have to be performed in order to have a software
+  stack that can be run almost anywhere. This brings to Kameleon a high degree of customizability, reuse of
+  code and users have total control in when and where the
+  sections have to take place. This minimal actions are: bootstrap, setup and export.
+
+* Steps: It refers to a specific action to be done inside a section.
+  Steps can be declared in independent files that improves the degree of reusability.
+
+* Microsteps: procedures composed of shell commands. The goal of dividing steps into microsteps is the
+  possibility of activating certain actions within a step.
+
+The Kameleon hierarchy encourages the reuse (shareability) of code and modularity of procedures.
+
+The minimal building block are the commands exec_ which wraps shell commands adding
+a simple error handling and interactivenes in case of a problem.
+These commands are executed in a given context. Which could be: local, in, out.
+That are going to be defined later. They can be used as follows::
+
+     setup:
+     - first_step:
+       - hello_microstep:
+         - exec_local: echo "Hello world"
+	 - exec_in: echo "Hello world"
+	 - exec_out: echo "Hello world"
+     # The end
+
+
+* Local context: It represents the Kameleon execution environment. Normally is the user’s machine.
+
+* OUT context: It is where the appliance will be bootstraped. Some procedures have to be carried out in
+  order create the place where the software appliance is built (In context).
+  This can be: the same user’s machine using chroot.
+  Thus, in this context is where the setup of the chroot takes place.
+  Establishing the proper environmental variables in order to have a clean environment.
+  Other examples are: setting up a virtual machine, access to an infrastructure in order to get an instance and be able to deploy, setting
+  a Docker container, etc.
+
+* IN context: It makes reference to inside the newly
+  created appliance. It can be mapped to a chroot,
+  virtual machine, physical machine, Linux container, etc.