kameleon-builder 2.0.0 → 2.1.0

data/docs/source/index.rst

@@ -10,6 +10,9 @@ Welcome to Kameleon's documentation!
 .. note::
     This documentation is currently being written...
 
+.. image:: _static/kameleon-long.png
+     :align: right
+
 -----
 About
 -----
@@ -17,7 +20,7 @@ About
 Kameleon should be seen as a simple but powerful tool to generate customized
 appliances. With Kameleon, you make your recipe that describes how to create
 step by step your own distribution. At start Kameleon is used to create custom
-kvm, LXC, VirtualBox, iso images, ..., but as it is designed to be very generic
+kvm, docker, VirtualBox, ..., but as it is designed to be very generic
 you can probably do a lot more than that.
 
 
@@ -31,13 +34,16 @@ User Documentation
 
     installation.rst
     getting_started.rst
+    use_cases.rst
     recipe.rst
     context.rst
     commands.rst
     workspace.rst
     checkpoint.rst
+    persistent_cache.rst
     aliases.rst
     faq.rst
+    grid5000_tutorial.rst
 
 Indices and tables
 ==================
@@ -45,4 +51,3 @@ Indices and tables
 * :ref:`genindex`
 * :ref:`modindex`
 * :ref:`search`
-

data/docs/source/installation.rst

@@ -1,7 +1,8 @@
 ------------
 Installation
 ------------
-
+.. note::
+    On debian based distribution be sure to install the ``ruby-dev`` package first
 
 To install the latest release from `RubyGems`_:
 
@@ -11,11 +12,18 @@ To install the latest release from `RubyGems`_:
 
     gem install kameleon-builder --pre
 
-Or from source:
-
-::
+Or from source::
 
     git clone https://github.com/oar-team/kameleon.git
     cd kameleon
     gem build kameleon-builder.gemspec
     gem install kameleon-builder-<version>.gem
+
+
+Any troubles?
+~~~~~~~~~~~~~
+If you got an error message like this one::
+
+    ``/usr/lib/ruby/1.9.1/rubygems/custom_require.rb:36:in `require': cannot load such file -- mkmf (LoadError)``
+
+It's because you need the ``ruby-dev`` package to fit the dependancies.

data/docs/source/persistent_cache.rst

@@ -0,0 +1,34 @@
+.. _`persistent_cache`:
+
+----------------
+Persistent Cache
+----------------
+
+.. versionadded:: 2.0.1
+
+In order to exactly reconstruct a software appliance with the same exact
+version of packages. Kameleon offer the option of creating a persistent cache
+that will catch all the software packages during the building of the software
+appliance.  Enabling other to reconstruct the same exact software appliance
+with the right package versions. Kameleon uses Polipo [1]_ which is a tiny and
+lightweight web proxy to cache most of the packages that comes form the
+network.  First of all, you have to install Polipo on your system.  If you are
+under a debian distribution you can install it using the package manager::
+
+   sudo apt-get install polipo
+
+You can as well build it from sources and then specify the path of the
+generated binary using the option ``--proxy_path``. To use, you just have to
+add the option ``--cache`` as an argument of the build command.
+For example::
+
+  kameleon build debian_test -b /tmp/kameleon/ --cache
+
+This will create a tar file in the build directory ``/tmp/kameleon`` called
+``debian_test-cache.tar``.  In order to use this generated cache file in
+another build, we have just to use the options ``--from_cache`` as follows::
+
+   kameleon build debian_test -b /tmp/kameleon/ --from_cache /tmp/debian_test-cache.tar
+
+
+.. [1] http://www.pps.univ-paris-diderot.fr/~jch/software/polipo/

data/docs/source/recipe.rst

@@ -3,20 +3,24 @@ Recipe
 ------
 
 Kameleon compute YAML files, named  *recipes*, that describes how you will
-build your appliance. These files are stored in the root of your `Workspace`_.
-A recipe is a hierarchical structure of `Sections`_, `Steps`_, `Microsteps`_ and
-`Commands`_. Here is an overview of this structure:
+build your appliance. These files are stored in the root of your :ref:`workspace`.
+A recipe is a hierarchical structure of `Section`_, `Step`_, `Microstep`_ and
+:ref:`commands`. Here is an overview of this structure:
 
 .. code-block:: yaml
 
     recipe
-    └── section
-        └── step
-            └── microstep
-                └──command
+    |
+    `-- section
+        |
+        `-- step
+            |
+            `-- microstep
+                |
+                `-- command
 
 The recipe also contains set of `Global variables`_ declaration and some
-imports like `Aliases`_ and `Checkpoint`_.
+imports like :ref:`aliases` and :ref:`checkpoint`.
 
 This is an example of a recipe:
 
@@ -89,7 +93,7 @@ This is an example of a recipe:
       - system_config:
         - locales: fr_FR en_US
         - lang: fr_FR.UTF-8
-        - timezone: Europe/Paris
+        - timezone: UTC
       - keyboard_config:
         - layout: "fr,us"
       - network_config:
@@ -112,11 +116,11 @@ This is an example of a recipe:
 Section
 -------
 
-Each section is a list of `Steps`_. Currently, there is 3 sections:
+Each section is a list of steps. Currently, there is 3 sections:
 
 bootstrap
     This section contains the bootstrap of the new system and create the *in*
-    context (see `Contexts`_).
+    context (see :ref:`context`).
 
 setup
     This one is dedicated to the install and configuration steps.
@@ -125,10 +129,13 @@ export
     Export the generated appliance in the format of your choice.
 
 
-Step
-----
+.. _`step`:
+.. _`microstep`:
 
-Each *step* contains a list of microsteps that contains a list of Commands_
+Step and microstep
+-------------------
+
+Each *step* contains a list of *microsteps* that contains a list of :ref:`commands`
 written in one YAML file.  To be found by Kameleon this file must be named by
 with the step name plus the YAML extension ``.yaml``. For example the
 ``software_install.yaml`` step file looks like this:
@@ -171,7 +178,7 @@ Steps path
 ~~~~~~~~~~
 
 The steps are YAML formated files stored in the ``recipe/steps`` directory of
-the Workspace_. To enable a better recipe reuse and ease of write the steps
+the :ref:`workspace`. To enable a better recipe reuse and ease of write the steps
 are stored by default in specific folders depending on the sections.
 
 Kameleon is looking for the steps files using the ``include_steps`` list value,
@@ -227,7 +234,7 @@ Global variables
 ~~~~~~~~~~~~~~~~~
 
 Global variables are defined in the ``global`` dictionary of the recipe.
-Kameleon use some global variable to enable the appliance build. See Context_
+Kameleon use some global variable to enable the appliance build. See :ref:`context`
 and `Steps path`_ for more details
 
 

data/docs/source/use_cases.rst

@@ -0,0 +1,93 @@
+.. _`use cases`:
+
+---------
+Use Cases
+---------
+
+Here it is described different use cases for Kameleon. It should give you a better
+idea, through examples, of how Kameleon is useful for.
+
+Distribute an environnement to your co-workers/students/friends/...
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+Whit Kameleon you can easily export your environnement in any format. The
+export section of the recipe is made for this. For example, if you would like
+to export your image in vdi format to use VirtualBox you just have to uncomment
+the right line. Like in the debian7 template::
+
+    #== Export the generated appliance in the format of your choice
+    export:
+      - save_appliance_from_nbd:
+        - mountdir: $$rootfs
+        - filename: "$${kameleon_recipe_name}"
+        - save_as_qcow2
+        # - save_as_qed
+        # - save_as_tgz
+        # - save_as_raw
+        # - save_as_vmdk
+        - save_as_vdi
+
+Make a Linux virtual machine with graphical support
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+You can directly use the templates that provide a desktop. However, here is an
+example of adding desktop capability to the ``debian8`` template. First create
+a new recipe from this template::
+
+     kameleon new debian8-dektop -t debian8
+
+Then edit the recipe file ``debian8-dektop.yaml`` and add ``gnome-core`` and ``xorg``
+packages to the install list::
+
+    setup:
+        # Install
+        - software_install:
+          - packages: >
+              debian-keyring ntp zip unzip rsync sudo less vim bash-completion
+              gnome-core xorg
+
+These packages take some extra space, so add some space on the disk. 4G should
+be enough::
+
+    bootstrap:
+        ...
+        - create_disk_nbd:
+            - image_size: 4G
+
+Build your recipe::
+
+    sudo kameleon build debian8-desktop
+
+When the build has finished, you can try you image with Qemu::
+
+    sudo qemu-system-x86_64 -m 1024 --enable-kvm -vga std \
+        builds/debian8-desktop/debian8-desktop.qcow2
+
+Alternatively, you could use ``virt-manager`` that provide a good GUI to manage
+your virtual machines.
+
+.. note::
+    If you want a better integration between the host and the guest like
+    copy/paste use ``spice`` (http://www.linux-kvm.org/page/SPICE)
+
+
+Create a fully reproducible experimental environement
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+You should use the :ref:`persistent_cache` feature.
+
+.. todo::
+    Give a complete use case.
+
+Create a persistent live USB key
+~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+A dirty but reliable method to do this is to cat the entire raw disk on the USB
+key. First be sure that the disk size is equal or smaller then your
+Then, export you image in raw format (this is the disk content bit by bit) and
+dump it to your USB key. Once your image is built, if your USB key is the
+``/dev/sdb`` device, be sure that it is not mounted and just do this::
+
+    cat my_image.raw > /dev/my_key
+
+.. warning::
+    This is a dangerous operation, you usb key will erase without
+    warning! Be sure that you pick the right device (use lsblk): it should be
+    ``/dev/sdX`` where X is a letter. Do NOT use the ``dev/sdXY``. unmount it
+    and use the root device instead.

data/docs/source/workspace.rst

@@ -1,3 +1,5 @@
+.. _`workspace`:
+
 ---------
 Workspace
 ---------

data/kameleon-builder.gemspec

@@ -1,4 +1,6 @@
 # coding: utf-8
+RUBYONEX = RUBY_VERSION < "2.0"
+
 lib = File.expand_path('../lib', __FILE__)
 $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
 require 'kameleon/version'
@@ -24,10 +26,14 @@ Gem::Specification.new do |s|
   s.test_files    = s.files.grep(%r{^(tests|s|features)/})
   s.require_paths = ["lib"]
 
-  s.add_dependency 'childprocess', '~> 0.3'
+  s.required_ruby_version     = RUBYONEX ? '>= 1.9.3' : '>= 2.0.0'
+
+  s.add_dependency 'childprocess', '~> 0.5'
   s.add_dependency 'thor', '~> 0.15'
   s.add_dependency 'table_print', '~> 1.5'
   s.add_dependency 'log4r-color', '~> 1.2'
+  s.add_dependency 'syck', '~> 1.0.0' unless RUBYONEX
+  s.add_dependency 'diffy', '~> 3.0.4'
 
   s.add_development_dependency 'pry', '~> 0.9'
   s.add_development_dependency 'pry-debugger', '~> 0.2'

data/lib/kameleon.rb

@@ -1,5 +1,3 @@
-require 'securerandom'
-require 'yaml'
 require 'fileutils'
 require 'optparse'
 require 'erb'
@@ -12,11 +10,9 @@ require 'log4r-color'
 require 'log4r-color/configurator'
 require 'pathname'
 require 'table_print'
+require 'diffy'
 
 module Kameleon
-  # to force yaml to dump ASCII-8Bit strings as strings
-  YAML::ENGINE.yamler='syck'
-
   # add a PROGRESS and NOTICE level
   Log4r::Configurator.custom_levels(:DEBUG, :INFO, :NOTICE,
                                     :PROGRESS, :WARN, :ERROR,
@@ -56,7 +52,7 @@ module Kameleon
     end
 
     def logger
-      @logger ||= Log4r::Logger.new("kameleon::[global]")
+      @logger ||= Log4r::Logger.new("kameleon::[kameleon]")
     end
 
     def env
@@ -67,6 +63,7 @@ module Kameleon
 end
 
 # Load the things which must be loaded before anything else
+require 'kameleon/compat'
 require 'kameleon/utils'
 require 'kameleon/error'
 require 'kameleon/cli'

data/lib/kameleon/cli.rb

@@ -5,40 +5,62 @@ require 'kameleon/utils'
 module Kameleon
   class CLI < Thor
 
-    class_option :no_color, :type => :boolean, :default => false,
-                 :desc => "Disable colorization in output"
+
+    class_option :color, :type => :boolean, :default => true,
+                 :desc => "Enable colorization in output"
     class_option :debug, :type => :boolean, :default => false,
                  :desc => "Enable debug output"
-    class_option :workspace, :aliases => '-w', :type => :string,
-                 :desc => 'Change the kameleon current work directory. ' \
-                          '(The folder containing your recipes folder).' \
-                          ' Default : ./'
+    map %w(-h --help) => :help
+
     no_commands do
       def logger
-        @logger ||= Log4r::Logger.new("kameleon::[cli]")
+        @logger ||= Log4r::Logger.new("kameleon::[kameleon]")
+      end
+    end
+
+    method_option :force,:type => :boolean,
+                  :default => false, :aliases => "-f",
+                  :desc => "Overwrite all existing files"
+    desc "import [TEMPLATE_NAME]", "Imports the given template"
+    def import(template_name)
+      templates_path = Kameleon.env.templates_path
+      template_path = File.join(templates_path, template_name) + '.yaml'
+      begin
+        template_recipe = RecipeTemplate.new(template_path)
+      rescue
+        raise TemplateNotFound, "Template '#{template_name}' not found. " \
+                                "To see all templates, run the command "\
+                                "`kameleon templates`"
+      else
+        logger.notice("Importing template '#{template_name}'...")
+        template_recipe.copy_template(options[:force])
+        logger.notice("done")
       end
     end
 
-    method_option :template, :aliases => "-t",
-                  :desc => "Starting from a template", :required => true,
-                  :enum => Kameleon.templates_names
     method_option :force,:type => :boolean,
                   :default => false, :aliases => "-f",
-                  :desc => "overwrite the recipe"
-    desc "new [RECIPE_NAME]", "Creates a new recipe"
-    def new(recipe_name)
-      logger.notice("Cloning template '#{options[:template]}'")
+                  :desc => "Overwrite all existing files"
+    desc "new [RECIPE_NAME] [TEMPLATE_NAME]", "Creates a new recipe"
+    def new(recipe_name, template_name)
+      if recipe_name == template_name
+        fail RecipeError, "Recipe name should be different from template name"
+      end
       templates_path = Kameleon.env.templates_path
-      recipes_path = Kameleon.env.recipes_path
-
-      template_path = File.join(templates_path, options[:template]) + '.yaml'
-      template_recipe = RecipeTemplate.new(template_path)
-      template_recipe.copy_template(recipes_path,
-                                    recipe_name,
-                                    options[:force])
-      recipe_path = File.join(recipes_path, recipe_name + ".yaml")
-      logger.notice("New recipe \"#{recipe_name}\" "\
-                    "as been created in #{recipe_path}")
+      template_path = File.join(templates_path, template_name) + '.yaml'
+      begin
+        template_recipe = RecipeTemplate.new(template_path)
+      rescue
+        raise TemplateNotFound, "Template '#{template_name}' not found. " \
+                                "To see all templates, run the command "\
+                                "`kameleon templates`"
+      else
+        logger.notice("Cloning template '#{template_name}'...")
+        template_recipe.copy_template(options[:force])
+        logger.notice("Creating extended recipe from template '#{template_name}'...")
+        template_recipe.copy_extended_recipe(recipe_name, options[:force])
+        logger.notice("done")
+      end
     end
 
     desc "templates", "Lists all defined templates"
@@ -58,6 +80,7 @@ module Kameleon
           raise e if Kameleon.env.debug
         end
       end
+      templates_hash = templates_hash.sort_by{ |k| k["name"] }
       tp templates_hash, {"name" => {:width => 30}}, { "description" => {:width => 60}}
     end
 
@@ -68,61 +91,80 @@ module Kameleon
     end
     map %w(-v --version) => :version
 
-
-    desc "build [RECIPE_NAME]", "Builds the appliance from the recipe"
+    desc "build [RECIPE_PATH]", "Builds the appliance from the given recipe"
     method_option :build_path, :type => :string ,
                   :default => nil, :aliases => "-b",
                   :desc => "Set the build directory path"
+    method_option :clean, :type => :boolean ,
+                  :default => false,
+                  :desc => "Run the command `kameleon clean` first"
     method_option :from_checkpoint, :type => :string ,
                   :default => nil,
                   :desc => "Using specific checkpoint to build the image. " \
                            "Default value is the last checkpoint."
-    method_option :no_checkpoint, :type => :boolean ,
-                  :default => false,
+    method_option :checkpoint, :type => :boolean ,
+                  :default => true,
                   :desc => "Do not use checkpoints"
-    def build(recipe_name)
-      logger.notice("Starting build recipe '#{recipe_name}'")
-      start_time = Time.now.to_i
-      recipe_path = File.join(Kameleon.env.recipes_path, recipe_name) + '.yaml'
+    method_option :cache, :type => :boolean,
+                  :default => false,
+                  :desc => "Generate a persistent cache for the appliance."
+    method_option :from_cache, :type => :string ,
+                  :default => nil,
+                  :desc => "Using a persistent cache tar file to build the image."
+    method_option :proxy_path, :type => :string ,
+                  :default => nil,
+                  :desc => "Full path of the proxy binary to use for the persistent cache."
+
+    def build(recipe_path)
+      clean(recipe_path) if options[:clean]
       engine = Kameleon::Engine.new(Recipe.new(recipe_path), options)
+      logger.notice("Starting build recipe '#{recipe_path}'")
+      start_time = Time.now.to_i
       engine.build
       total_time = Time.now.to_i - start_time
       logger.notice("")
-      logger.notice("Build recipe '#{recipe_name}' is completed !")
+      logger.notice("Build recipe '#{recipe_path}' is completed !")
       logger.notice("Build total duration : #{total_time} secs")
       logger.notice("Build directory : #{engine.cwd}")
       logger.notice("Build recipe file : #{engine.build_recipe_path}")
       logger.notice("Log file : #{Kameleon.env.log_file}")
     end
 
-    desc "checkpoints [RECIPE_NAME]", "Lists all availables checkpoints"
+    desc "checkpoints [RECIPE_PATH]", "Lists all availables checkpoints"
     method_option :build_path, :type => :string ,
                   :default => nil, :aliases => "-b",
                   :desc => "Set the build directory path"
-    def checkpoints(recipe_name)
+    def checkpoints(recipe_path)
       Log4r::Outputter['console'].level = Log4r::ERROR unless Kameleon.env.debug
-      recipe_path = File.join(Kameleon.env.recipes_path, recipe_name) + '.yaml'
       engine = Kameleon::Engine.new(Recipe.new(recipe_path), options)
       engine.pretty_checkpoints_list
     end
 
-    desc "clear [RECIPE_NAME]", "Cleaning out context and removing all checkpoints"
+    desc "clean [RECIPE_PATH]", "Cleaning 'out' and 'local' contexts and removing all checkpoints"
     method_option :build_path, :type => :string ,
                   :default => nil, :aliases => "-b",
                   :desc => "Set the build directory path"
-    def clear(recipe_name)
+    def clean(recipe_path)
       Log4r::Outputter['console'].level = Log4r::INFO
-      recipe_path = File.join(Kameleon.env.recipes_path, recipe_name) + '.yaml'
       engine = Kameleon::Engine.new(Recipe.new(recipe_path), options)
       engine.clear
     end
+    map %w(clear) => :clean
+
+    desc "commands", "Lists all available commands", :hide => true
+    def commands
+      puts CLI.all_commands.keys - ["commands", "completions"]
+    end
+
+    desc "source_root", "Prints the kameleon directory path", :hide => true
+    def source_root
+      puts Kameleon.source_root
+    end
 
     # Hack Thor to init Kameleon env soon
     def self.init(base_config)
-      options = base_config[:shell].base.options
-      workspace ||= options[:workspace] || ENV['KAMELEON_WORKSPACE'] || Dir.pwd
-      env_options = options.merge({:workspace => workspace})
-      FileUtils.mkdir_p workspace
+      env_options = Hash.new
+      env_options.merge! base_config[:shell].base.options.clone
       # configure logger
       env_options["debug"] = true if ENV["KAMELEON_LOG"] == "debug"
       ENV["KAMELEON_LOG"] = "debug" if env_options["debug"]
@@ -138,34 +180,46 @@ module Kameleon
       begin
         level = Log4r.const_get(level_name.upcase)
       rescue NameError
-        fail KameleonError, "Invalid KAMELEON_LOG level is set: #{level_name}.\n" \
-                            "Please use one of the standard log levels: debug," \
-                            " info, warn, or error"
+        level = Log4r.const_get("INFO")
+        $stderr << "Invalid KAMELEON_LOG level is set: #{level_name}.\n" \
+                   "Please use one of the standard log levels: debug," \
+                   " info, warn, or error\n"
+        raise KameleonError
       end
       format = ConsoleFormatter.new
       # format = Log4r::PatternFormatter.new(:pattern => '%11c: %M')
-      if !$stdout.tty? or env_options["no_color"]
+      if !$stdout.tty? or !env_options["color"]
         console_output = Log4r::StdoutOutputter.new('console',
                                                     :formatter => format)
+        Diffy::Diff.default_format = :text
       else
         console_output = Log4r::ColorOutputter.new 'console', {
           :colors => { :debug  => :light_black,
                        :info   => :green,
+                       :progress_info => :green,
                        :notice => :light_blue,
-                       :progress => :light_blue,
+                       :progress_notice => :light_blue,
                        :warn   => :yellow,
                        :error  => :red,
+                       :progress_error => :red,
                        :fatal  => :red,
                      },
           :formatter => format,
         }
+        Diffy::Diff.default_format = :color
       end
       logger = Log4r::Logger.new('kameleon')
-      logger.outputters << console_output
       logger.level = level
-      logger = nil
+      logger.outputters << console_output
+      format_file = FileFormatter.new
       Kameleon.logger.debug("`kameleon` invoked: #{ARGV.inspect}")
       Kameleon.env = Kameleon::Environment.new(env_options)
+      filelog = Log4r::FileOutputter.new('logfile',
+                                         :trunc=>false,
+                                         :filename => Kameleon.env.log_file.to_s,
+                                         :formatter => format_file)
+      logger.outputters << filelog
+      logger = nil
     end
 
     def self.start(given_args=ARGV, config={})