kameleon-builder 2.1.0 → 2.1.1
Sign up to get free protection for your applications and to get access to all the features.
- data/.editorconfig +0 -0
- data/CHANGELOG.rst +87 -0
- data/docs/Makefile +1 -1
- data/docs/source/context.rst +3 -1
- data/docs/source/debian7.yaml +128 -0
- data/docs/source/debian_customized.yaml +28 -0
- data/docs/source/debian_customized_g5k.yaml +21 -0
- data/docs/source/grid5000_tutorial.rst +435 -32
- data/docs/source/installation.rst +8 -0
- data/docs/source/tau_install_g5k.yaml +24 -0
- data/ext/mkrf_conf.rb +58 -0
- data/kameleon-builder.gemspec +7 -6
- data/lib/kameleon.rb +6 -2
- data/lib/kameleon/compat.rb +23 -1
- data/lib/kameleon/context.rb +8 -2
- data/lib/kameleon/engine.rb +19 -17
- data/lib/kameleon/persistent_cache.rb +11 -6
- data/lib/kameleon/recipe.rb +15 -4
- data/lib/kameleon/shell.rb +12 -13
- data/lib/kameleon/step.rb +19 -6
- data/lib/kameleon/utils.rb +5 -2
- data/templates/debian7-g5k.yaml +2 -2
- data/templates/debian7-kameleon.yaml +43 -0
- data/templates/debian7.yaml +1 -1
- data/templates/docker-debian7.yaml +107 -0
- data/templates/steps/bootstrap/initialize_disk_chroot.yaml +1 -1
- data/templates/steps/bootstrap/initialize_disk_qemu.yaml +1 -1
- data/templates/steps/bootstrap/prepare_docker.yaml +37 -35
- data/templates/steps/bootstrap/prepare_qemu.yaml +1 -1
- data/templates/steps/bootstrap/start_docker.yaml +6 -2
- data/templates/steps/bootstrap/start_qemu.yaml +2 -2
- data/templates/steps/checkpoints/qemu.yaml +1 -1
- data/templates/steps/export/save_appliance_from_g5k.yaml +1 -1
- data/version.txt +1 -1
- metadata +15 -7
- data/templates/debian7-docker.yaml +0 -111
data/.editorconfig
CHANGED
File without changes
|
data/CHANGELOG.rst
ADDED
@@ -0,0 +1,87 @@
|
|
1
|
+
Kameleon CHANGELOG
|
2
|
+
==================
|
3
|
+
|
4
|
+
version 2.1.0
|
5
|
+
-------------
|
6
|
+
|
7
|
+
Released on May 12th 2014
|
8
|
+
|
9
|
+
- [core] Fixed psych yaml parsing (#1)
|
10
|
+
- [core] Changed option ``--no-no-color`` to ``--color``
|
11
|
+
- [core] Saved the contexts state files in their WORKDIR (#3)
|
12
|
+
- [core] Set context in/out/local cmd to /bin/bash by default (#5)
|
13
|
+
- [core] Made global section non mandatory
|
14
|
+
- [core] Made writing embedded step in recipe possible (#12)
|
15
|
+
- [core] Improved the readability of logs and the progress bar
|
16
|
+
- [core] Moved aliases and checkpoints folders to steps
|
17
|
+
- [core] Removed the ``recipes`` folder and the ``workspace`` (#2)
|
18
|
+
- [core] Make a safe copy with ``kameleon new`` command
|
19
|
+
- [core] Added a simple extend recipe feature (#11)
|
20
|
+
- [core] Introduced the keyword "@base" in the extended recipes (#11)
|
21
|
+
- [core] Don't log identifier of microstep during build process
|
22
|
+
- [core] Added ``kameleon import`` command (#11)
|
23
|
+
- [core] Added ``--clean`` option to ``kameleon build`` command
|
24
|
+
- [core] Added the lazy context initialization (#10)
|
25
|
+
- [core] Set the variable ``KAMELEON_WORKDIR`` for all contexts
|
26
|
+
- [core] Used ``KAMELEON_WORKDIR`` when working with PIPE
|
27
|
+
- [core] Added persistent cache feature to Kameleon, So far it is caching just packages comming from the network using Polipo
|
28
|
+
- [template] Added new templates :
|
29
|
+
|
30
|
+
- archlinux
|
31
|
+
- archlinux-desktop
|
32
|
+
- debian-testing
|
33
|
+
- debian7
|
34
|
+
- debian7-desktop
|
35
|
+
- debian7-oar-dev
|
36
|
+
- fedora-rawhide
|
37
|
+
- fedora20
|
38
|
+
- fedora20-desktop
|
39
|
+
- ubuntu-12.04
|
40
|
+
- ubuntu-12.04-desktop
|
41
|
+
- ubuntu-14.04
|
42
|
+
- ubuntu-14.04-desktop
|
43
|
+
- vagrant-debian7
|
44
|
+
- [template] Installed the extlinux bootloader depending on distributions
|
45
|
+
- [template] New way to bootstrap fedora using Liveos image
|
46
|
+
- [template] Installed linux kernel and extlinux bootloader from bootstrap section
|
47
|
+
- [template] Used parted instead of sfdisk
|
48
|
+
- [template] Added save_as_qed step
|
49
|
+
- [template] Removed insecure ssh key before any export
|
50
|
+
- [template] Added shell auto-completion for bash, zsh and fish shell
|
51
|
+
- [template] Default user group is sudo
|
52
|
+
- [template] Added a new qemu/kvm template with full-snapshot support
|
53
|
+
- [template] Ability to add user in multiple groups (with usermod -G)
|
54
|
+
- [template] Improved I/O performance with qemu/kvm
|
55
|
+
- [template] Removed force-unsafe-io for dpkg to avoid corrupted filesystem
|
56
|
+
- [template] Used qemu by default instead of chroot
|
57
|
+
- [template] Added option to disable debootstrap cache
|
58
|
+
- [template] Refactor qcow2 backing file checkpoints
|
59
|
+
- [template] Make QEMU checkpoint more robust and avoid disk corruption
|
60
|
+
- [template] Major revision of steps to make it easier to use in different templates
|
61
|
+
- [template] Rename steps for more semantic consistency
|
62
|
+
- [template] Making the 'save_appliance' step not dependent on the state of the machine
|
63
|
+
- [template] Enabled cache for arch_bootstrap
|
64
|
+
- [template] Added openssh in arch-bootstrap and enabled sshd.service/dhcp.service
|
65
|
+
- [template] Added user 'nobody' to allow sshd to run in the archlinux virtual machine
|
66
|
+
- [template] Enabled checkpoints (backing-file) only in the "setup" stage
|
67
|
+
- [template] Fixed .ssh and authorized_keys permissions
|
68
|
+
- [template] Avoid crash of in_context when we send a shutdown command to the virtual machine
|
69
|
+
- [template] Exclude special files with rsync (proc/dev...) when copying rootfs to the disk
|
70
|
+
- [template] Force stop qemu if still running
|
71
|
+
- [template] Make debian-chroot depreciated
|
72
|
+
- [template] Refactor archlinux template to use it with qemu/kvm
|
73
|
+
- [template] Improved the LiveOS fedora bootstrap step to get the system running with qemu/kvm
|
74
|
+
- [template] Refactor fedora20/debian8 templates to use them with qemu/kvm
|
75
|
+
- [template] Set timezone to UTC by default
|
76
|
+
- [template] Used ProxyCommand to improve the debian7-g5k recipe
|
77
|
+
- [aliases] Updated write_file and append_file aliases to support double quotes
|
78
|
+
- [aliases] Defined new aliases for unmounting devices
|
79
|
+
- [docs] More documentation
|
80
|
+
|
81
|
+
|
82
|
+
version 2.0.0
|
83
|
+
=============
|
84
|
+
|
85
|
+
Released on February 17th 2014
|
86
|
+
|
87
|
+
Initial public release of kameleon 2
|
data/docs/Makefile
CHANGED
@@ -65,7 +65,7 @@ singlehtml:
|
|
65
65
|
@echo "Build finished. The HTML page is in $(BUILDDIR)/singlehtml."
|
66
66
|
|
67
67
|
publish: html
|
68
|
-
rsync -avh -e "ssh" $(BUILDDIR)/html/ kameleon.website:~/www
|
68
|
+
rsync -avh --exclude 'pkg' -e "ssh" $(BUILDDIR)/html/ kameleon.website:~/www
|
69
69
|
|
70
70
|
|
71
71
|
pickle:
|
data/docs/source/context.rst
CHANGED
@@ -7,7 +7,9 @@ Context
|
|
7
7
|
To understand how Kameleon works you have to get the *context* notion. A context
|
8
8
|
is an execution environnement with its variables (like $PATH, $TERM,...), its
|
9
9
|
tools (debootstrap, yum, ...) and all its specifics (filesystem, local/remote,
|
10
|
-
...).
|
10
|
+
...).
|
11
|
+
|
12
|
+
It also manage the connections to your context and make it easy and reliable.
|
11
13
|
When you build an appliance you deal with 3 contexts:
|
12
14
|
|
13
15
|
- The *local* context which is the Kameleon execution environnement
|
@@ -0,0 +1,128 @@
|
|
1
|
+
#==============================================================================
|
2
|
+
# vim: softtabstop=2 shiftwidth=2 expandtab fenc=utf-8 cc=81 tw=80
|
3
|
+
#==============================================================================
|
4
|
+
#
|
5
|
+
# DESCRIPTION: Debian 7 (Wheezy) base system
|
6
|
+
#
|
7
|
+
#==============================================================================
|
8
|
+
|
9
|
+
---
|
10
|
+
# Loads some helpful aliases
|
11
|
+
aliases: defaults.yaml
|
12
|
+
# Enables qemu checkpoint
|
13
|
+
checkpoint: qemu.yaml
|
14
|
+
#== Global variables use by Kameleon engine and the steps
|
15
|
+
global:
|
16
|
+
## User varibales : used by the recipe
|
17
|
+
user_name: kameleon
|
18
|
+
user_password: $$user_name
|
19
|
+
|
20
|
+
# Distribution
|
21
|
+
distrib: debian
|
22
|
+
release: wheezy
|
23
|
+
arch: amd64
|
24
|
+
|
25
|
+
## QEMU options
|
26
|
+
qemu_enable_kvm: true
|
27
|
+
qemu_cpu: 2
|
28
|
+
qemu_memory_size: 512
|
29
|
+
qemu_monitor_port: 10023
|
30
|
+
qemu_ssh_port: 55423
|
31
|
+
qemu_arch: x86_64
|
32
|
+
|
33
|
+
## Disk options
|
34
|
+
nbd_device: /dev/nbd1
|
35
|
+
image_disk: $$kameleon_cwd/base_$$kameleon_recipe_name.qcow2
|
36
|
+
image_size: 10G
|
37
|
+
filesystem_type: ext4
|
38
|
+
|
39
|
+
# rootfs options
|
40
|
+
rootfs: $$kameleon_cwd/rootfs
|
41
|
+
rootfs_download_path: /var/cache/kameleon/$$distrib/$$release/$$arch/rootfs
|
42
|
+
|
43
|
+
## System variables. Required by kameleon engine
|
44
|
+
# Include specific steps
|
45
|
+
include_steps:
|
46
|
+
- $$distrib/$$release
|
47
|
+
- $$distrib
|
48
|
+
|
49
|
+
# Apt options
|
50
|
+
apt_repository: http://ftp.debian.org/debian/
|
51
|
+
apt_enable_contrib: true
|
52
|
+
apt_enable_nonfree: true
|
53
|
+
apt_install_recommends: false
|
54
|
+
|
55
|
+
# Shell session from where we launch exec_out commands. There is often a
|
56
|
+
# local bash session, but it can be a remote shell on other machines or on
|
57
|
+
# any shell. (eg. bash, chroot, fakechroot, ssh, tmux, lxc...)
|
58
|
+
out_context:
|
59
|
+
cmd: bash
|
60
|
+
workdir: $$kameleon_cwd
|
61
|
+
|
62
|
+
# Shell session that allows us to connect to the building machine in order to
|
63
|
+
# configure it and setup additional programs
|
64
|
+
ssh_config_file: $$kameleon_cwd/ssh_config
|
65
|
+
in_context:
|
66
|
+
cmd: LC_ALL=POSIX ssh -F $$ssh_config_file $$kameleon_recipe_name -t /bin/bash
|
67
|
+
workdir: /
|
68
|
+
|
69
|
+
#== Bootstrap the new system and create the 'in_context'
|
70
|
+
bootstrap:
|
71
|
+
- debootstrap:
|
72
|
+
- include_pkg: >
|
73
|
+
ifupdown locales libui-dialog-perl dialog isc-dhcp-client netbase
|
74
|
+
net-tools iproute acpid openssh-server pciutils extlinux
|
75
|
+
linux-image-$$arch
|
76
|
+
- release: $$release
|
77
|
+
- arch: $$arch
|
78
|
+
- repository: $$apt_repository
|
79
|
+
- enable_cache: true
|
80
|
+
- initialize_disk_qemu
|
81
|
+
- prepare_qemu
|
82
|
+
- install_bootloader
|
83
|
+
- start_qemu
|
84
|
+
|
85
|
+
#== Install and configuration steps
|
86
|
+
# WARNING: this part should be independante from the out context (whenever
|
87
|
+
# possible...)
|
88
|
+
setup:
|
89
|
+
# Install
|
90
|
+
- configure_apt:
|
91
|
+
- repository: $$apt_repository
|
92
|
+
- enable_contrib_repo: $$apt_enable_contrib
|
93
|
+
- enable_nonfree_repo: $$apt_enable_nonfree
|
94
|
+
- install_recommends: $$apt_install_recommends
|
95
|
+
- upgrade_system:
|
96
|
+
- dist_upgrade: true
|
97
|
+
- install_software:
|
98
|
+
- packages: >
|
99
|
+
debian-keyring ntp zip unzip rsync sudo less vim bash-completion
|
100
|
+
- configure_kernel:
|
101
|
+
- arch: $$arch
|
102
|
+
# Configuration
|
103
|
+
- configure_system:
|
104
|
+
- locales: POSIX C en_US fr_FR de_DE
|
105
|
+
- lang: en_US.UTF-8
|
106
|
+
- timezone: UTC
|
107
|
+
- configure_keyboard:
|
108
|
+
- layout: "us,fr,de"
|
109
|
+
- configure_network:
|
110
|
+
- hostname: kameleon-$$distrib
|
111
|
+
- create_group:
|
112
|
+
- name: admin
|
113
|
+
- create_user:
|
114
|
+
- name: $$user_name
|
115
|
+
- groups: sudo admin
|
116
|
+
- password: $$user_password
|
117
|
+
|
118
|
+
#== Export the generated appliance in the format of your choice
|
119
|
+
export:
|
120
|
+
- save_appliance:
|
121
|
+
- input: $$image_disk
|
122
|
+
- output: $$kameleon_cwd/$$kameleon_recipe_name
|
123
|
+
- save_as_qcow2
|
124
|
+
# - save_as_qed
|
125
|
+
# - save_as_tgz
|
126
|
+
# - save_as_raw
|
127
|
+
# - save_as_vmdk
|
128
|
+
# - save_as_vdi
|
@@ -0,0 +1,28 @@
|
|
1
|
+
#==============================================================================
|
2
|
+
# vim: softtabstop=2 shiftwidth=2 expandtab fenc=utf-8 cc=81 tw=80
|
3
|
+
#==============================================================================
|
4
|
+
#
|
5
|
+
# DESCRIPTION: <MY RECIPE DESCRIPTION>
|
6
|
+
#
|
7
|
+
#==============================================================================
|
8
|
+
|
9
|
+
---
|
10
|
+
extend: debian7
|
11
|
+
|
12
|
+
global:
|
13
|
+
# You can see the base template `debian7.yaml` to know the
|
14
|
+
# variables that you can override
|
15
|
+
# g5k_user: cruizsanabria # CHANGE ME
|
16
|
+
# g5k_site: grenoble # CHANGE ME
|
17
|
+
|
18
|
+
bootstrap:
|
19
|
+
- @base
|
20
|
+
|
21
|
+
setup:
|
22
|
+
- @base
|
23
|
+
- install_software:
|
24
|
+
- packages: >
|
25
|
+
g++ make taktuk openssh-server openmpi-bin openmpi-common openmpi-dev
|
26
|
+
# - tau_install
|
27
|
+
export:
|
28
|
+
- @base
|
@@ -0,0 +1,21 @@
|
|
1
|
+
|
2
|
+
---
|
3
|
+
extend: debian7-g5k
|
4
|
+
|
5
|
+
global:
|
6
|
+
# You can see the base template `debian7.yaml` to know the
|
7
|
+
# variables that you can override
|
8
|
+
g5k_user: cruizsanabria # CHANGE ME
|
9
|
+
g5k_site: grenoble # CHANGE ME
|
10
|
+
|
11
|
+
bootstrap:
|
12
|
+
- @base
|
13
|
+
|
14
|
+
setup:
|
15
|
+
- @base
|
16
|
+
- install_software:
|
17
|
+
- packages: >
|
18
|
+
g++ make taktuk openssh-server openmpi-bin openmpi-common openmpi-dev
|
19
|
+
- tau_install
|
20
|
+
export:
|
21
|
+
- @base
|
@@ -2,27 +2,27 @@
|
|
2
2
|
Grid'5000 Tutorial
|
3
3
|
==================
|
4
4
|
|
5
|
-
This tutorial will introduce Kameleon a tool to build software appliances
|
6
|
-
deployed on different
|
5
|
+
This tutorial will introduce Kameleon, a tool to build software appliances.
|
6
|
+
With Kameleon it is possible to generate appliances that can be deployed on different virtualization hypervisors or on baremetal.
|
7
|
+
It targets an important activity in Grid'5000 which is the customization of the experimental environments.
|
7
8
|
|
8
9
|
---------------
|
9
10
|
Kameleon basics
|
10
11
|
---------------
|
11
12
|
|
12
|
-
First of all, let's see all the syntax flavors that Kameleon
|
13
|
-
From this point, we assume that
|
14
|
-
in your system, otherwise
|
15
|
-
Kameleon can be seen as a shell
|
13
|
+
First of all, let's see all the syntax flavors that *Kameleon* has to offer.
|
14
|
+
From this point, we assume that *Kameleon* have been installed and it's already working
|
15
|
+
in your system, otherwise go to :ref:`installation` section.
|
16
|
+
Kameleon can be seen as a shell sequencer which will boost your shell scripts.
|
16
17
|
It is based on the execution of shell scripts but it provides some syntax sugar that makes
|
17
|
-
the work with shell scripts less
|
18
|
-
|
19
|
-
We will start with the basics
|
18
|
+
the work with shell scripts less painful.
|
19
|
+
Let's start with the basics
|
20
20
|
|
21
21
|
Kameleon Hello world
|
22
22
|
~~~~~~~~~~~~~~~~~~~~
|
23
23
|
|
24
|
-
Everything we want to build have to be specified by a recipe. Kameleon
|
25
|
-
and
|
24
|
+
Everything we want to build have to be specified by a recipe. Kameleon reads this recipe
|
25
|
+
and executes the appropriate actions. Let's create a hello world recipe using Kameleon.
|
26
26
|
Open a text editor and write the following::
|
27
27
|
|
28
28
|
setup:
|
@@ -31,10 +31,10 @@ Open a text editor and write the following::
|
|
31
31
|
- exec_local: echo "Hello world"
|
32
32
|
# The end
|
33
33
|
|
34
|
-
|
34
|
+
Save the previous file as a YAML file. For instance, hello_world.yaml.
|
35
35
|
|
36
36
|
.. note::
|
37
|
-
Be sure of respecting the YAML syntax `yaml`_.
|
37
|
+
Be sure of respecting the YAML syntax and indentation `yaml`_.
|
38
38
|
|
39
39
|
.. _yaml: http://www.yaml.org/
|
40
40
|
|
@@ -66,45 +66,448 @@ You will have some output that looks like this::
|
|
66
66
|
[kameleon]: Log file : /home/cristian/Repositories/exptools/setup_complex_exp/tests/new_version/kameleon.log
|
67
67
|
|
68
68
|
With this simple example, we have already introduced most of the Kameleon concepts and syntax.
|
69
|
-
First, how recipes are structured
|
69
|
+
First, how recipes are structured using a hierarchy composed of: sections, steps, microsteps.
|
70
70
|
|
71
|
-
* Sections:
|
71
|
+
* Sections: correspond to the minimal actions that have to be performed in order to have a software
|
72
72
|
stack that can be run almost anywhere. This brings to Kameleon a high degree of customizability, reuse of
|
73
|
-
code and users have total control
|
73
|
+
code and users have total control over when and where the
|
74
74
|
sections have to take place. This minimal actions are: bootstrap, setup and export.
|
75
75
|
|
76
|
-
* Steps: It refers to a specific action to be done inside a section
|
76
|
+
* Steps: It refers to a specific action to be done inside a section
|
77
|
+
(e.g., software installation, network configuration, configure kernel).
|
77
78
|
Steps can be declared in independent files that improves the degree of reusability.
|
78
79
|
|
79
80
|
* Microsteps: procedures composed of shell commands. The goal of dividing steps into microsteps is the
|
80
|
-
possibility of activating certain actions within a step.
|
81
|
-
|
82
|
-
The Kameleon hierarchy encourages the reuse (shareability) of code and modularity of procedures.
|
81
|
+
possibility of activating certain actions within a step and performing a better checkpoint.
|
83
82
|
|
84
|
-
|
83
|
+
Kameleon hierarchy encourages the reuse (shareability) of code and modularity of procedures.
|
84
|
+
The minimal building block are the commands *exec_* which wraps shell commands adding
|
85
85
|
a simple error handling and interactivenes in case of a problem.
|
86
86
|
These commands are executed in a given context. Which could be: local, in, out.
|
87
|
-
|
87
|
+
They can be used as follows::
|
88
88
|
|
89
89
|
setup:
|
90
|
-
|
91
|
-
|
92
|
-
|
93
|
-
|
94
|
-
|
90
|
+
- first_step:
|
91
|
+
- hello_microstep:
|
92
|
+
- exec_local: echo "Hello world"
|
93
|
+
- exec_in: echo "Hello world"
|
94
|
+
- exec_out: echo "Hello world"
|
95
95
|
# The end
|
96
96
|
|
97
97
|
|
98
98
|
* Local context: It represents the Kameleon execution environment. Normally is the user’s machine.
|
99
99
|
|
100
100
|
* OUT context: It is where the appliance will be bootstraped. Some procedures have to be carried out in
|
101
|
-
order create the place where the software appliance is built (In context).
|
102
|
-
|
101
|
+
order to create the place where the software appliance is built (In context).
|
102
|
+
One example is: the same user’s machine using chroot.
|
103
103
|
Thus, in this context is where the setup of the chroot takes place.
|
104
|
-
|
105
|
-
Other examples are: setting up a virtual machine, access to an infrastructure in order to get an instance and be able to deploy, setting
|
104
|
+
Other examples are: setting up a virtual machine, accessing an infrastructure in order to get a reservation and be able to deploy, setting
|
106
105
|
a Docker container, etc.
|
107
106
|
|
108
|
-
* IN context: It
|
107
|
+
* IN context: It refers to inside the newly
|
109
108
|
created appliance. It can be mapped to a chroot,
|
110
109
|
virtual machine, physical machine, Linux container, etc.
|
110
|
+
|
111
|
+
In the last example all the contexts are executed on the user's machine.
|
112
|
+
Which is the default behavior that can be customized (it will be shown later on this tutorial).
|
113
|
+
Most of the time, users take advantage of the *In context* in order to customize a given a appliance.
|
114
|
+
|
115
|
+
We can add variables as well::
|
116
|
+
|
117
|
+
setup:
|
118
|
+
- first_step:
|
119
|
+
- message: "Hello world"
|
120
|
+
- hello_microstep:
|
121
|
+
- exec_local: echo "Variable value $$message"
|
122
|
+
|
123
|
+
|
124
|
+
Let's apply the syntax to a real example in the next section.
|
125
|
+
|
126
|
+
----------------------------------------
|
127
|
+
Building a simple Debian based appliance
|
128
|
+
----------------------------------------
|
129
|
+
|
130
|
+
Kameleon already provides tested recipes for building different software appliances based
|
131
|
+
on different Linux flavors. We can take a look at the provided templates by typing::
|
132
|
+
|
133
|
+
$ kameleon templates
|
134
|
+
|
135
|
+
Which will output::
|
136
|
+
|
137
|
+
The following templates are available in /home/cristian/Repositories/kameleon_v2/templates:
|
138
|
+
NAME | DESCRIPTION
|
139
|
+
---------------------|-------------------------------------------------------------
|
140
|
+
archlinux | Build an Archlinux base system system.
|
141
|
+
archlinux-desktop | Archlinux GNOME Desktop edition.
|
142
|
+
debian-testing | Debian Testing base system
|
143
|
+
debian7 | Debian 7 (Wheezy) base system
|
144
|
+
debian7-desktop | Debian 7 (Wheezy) GNOME Desktop edition.
|
145
|
+
debian7-oar-dev | Debian 7 dev appliance with OAR-2.5 (node/server/frontend).
|
146
|
+
fedora-rawhide | Fedora Rawhide base system
|
147
|
+
fedora20 | Fedora 20 base system
|
148
|
+
fedora20-desktop | Fedora 20 GNOME Desktop edition
|
149
|
+
old-debian7 | [deprecated] Build a debian wheezy appliance using chroot...
|
150
|
+
ubuntu-12.04 | Ubuntu 12.04 LTS (Precise Pangolin) base system.
|
151
|
+
ubuntu-12.04-desktop | Ubuntu 12.04 LTS (Precise Pangolin) Desktop edition.
|
152
|
+
ubuntu-14.04 | Ubuntu 14.04 LTS (Trusty Tahr) base system.
|
153
|
+
ubuntu-14.04-desktop | Ubuntu 14.04 LTS (Trusty Tahr) Desktop edition.
|
154
|
+
vagrant-debian7 | A standard Debian 7 vagrant base box
|
155
|
+
|
156
|
+
|
157
|
+
Let's import the template debian7::
|
158
|
+
|
159
|
+
$ kameleon import debian7
|
160
|
+
|
161
|
+
This will generate the following files in the current directory::
|
162
|
+
|
163
|
+
├── debian7.yaml
|
164
|
+
├── kameleon.log
|
165
|
+
└── steps
|
166
|
+
├── aliases
|
167
|
+
| └── defaults.yaml
|
168
|
+
├── bootstrap
|
169
|
+
│ ├── debian
|
170
|
+
│ │ └── debootstrap.yaml
|
171
|
+
│ ├── initialize_disk_qemu.yaml
|
172
|
+
│ ├── install_bootloader.yaml
|
173
|
+
│ ├── prepare_qemu.yaml
|
174
|
+
│ └── start_qemu.yaml
|
175
|
+
├── checkpoints
|
176
|
+
│ └── qemu.yaml
|
177
|
+
├── export
|
178
|
+
│ └── save_appliance.yaml
|
179
|
+
└── setup
|
180
|
+
├── create_group.yaml
|
181
|
+
├── create_user.yaml
|
182
|
+
└── debian
|
183
|
+
├── configure_apt.yaml
|
184
|
+
├── configure_kernel.yaml
|
185
|
+
├── configure_keyboard.yaml
|
186
|
+
├── configure_network.yaml
|
187
|
+
├── configure_system.yaml
|
188
|
+
├── install_software.yaml
|
189
|
+
└── upgrade_system.yaml
|
190
|
+
|
191
|
+
8 directories, 19 files
|
192
|
+
|
193
|
+
Here we can observe that a directory has been generated.
|
194
|
+
This directory contains all the steps needed to build the final software appliance.
|
195
|
+
These steps are organized by sections. There is a directory checkpoints that is going
|
196
|
+
to be explained later on.
|
197
|
+
|
198
|
+
Here we can notice that all the process of building is based on steps files written with Kameleon syntax.
|
199
|
+
Separating the steps in different files gives a high degree of reusability.
|
200
|
+
|
201
|
+
The recipe looks like this:
|
202
|
+
|
203
|
+
.. literalinclude:: debian7.yaml
|
204
|
+
:lines: 69-125
|
205
|
+
|
206
|
+
The previous recipe build a debian wheezy using qemu.
|
207
|
+
It looks verbose but normally you as user you wont see it.
|
208
|
+
You will use it as a template in a way that will be explained later.
|
209
|
+
The recipe specify all the steps, configurations values that are going to be used
|
210
|
+
to build the appliance. Kameleon recipes gives many details to you, few things are hidden.
|
211
|
+
Which is good for reproducibility purposes and when reporting bugs.
|
212
|
+
|
213
|
+
If we have all the dependencies required as qemu, qemu-tools and debootstrap we can start to build the appliance
|
214
|
+
doing the following::
|
215
|
+
|
216
|
+
$ kamelon build debian7.yaml
|
217
|
+
|
218
|
+
The process will start and in about few minutes
|
219
|
+
a directory called builds will be generated in the current directory,
|
220
|
+
you will have a qemu virtual disk with a base debian wheezy installed in it.
|
221
|
+
That you can try out by executing::
|
222
|
+
|
223
|
+
$ sudo qemu-system-x86_64 -enable-kvm builds/debian7/debian7.qcow2
|
224
|
+
|
225
|
+
|
226
|
+
|
227
|
+
Customizing a software appliance
|
228
|
+
================================
|
229
|
+
|
230
|
+
Now, lets customize a given template in order to create a software appliance that have OpenMPI, Taktuk and tools necessary to compile source code.
|
231
|
+
Kameleon allows us to extend a given template. We will use this for adding the necessary software. Type the following::
|
232
|
+
|
233
|
+
$ kameleon new debian_customized debian7
|
234
|
+
|
235
|
+
This will create the file debian_customized.yaml which contents are::
|
236
|
+
|
237
|
+
---
|
238
|
+
extend: debian7
|
239
|
+
|
240
|
+
global:
|
241
|
+
# You can see the base template `debian7.yaml` to know the
|
242
|
+
# variables that you can override
|
243
|
+
|
244
|
+
bootstrap:
|
245
|
+
- @base
|
246
|
+
|
247
|
+
setup:
|
248
|
+
- @base
|
249
|
+
|
250
|
+
export:
|
251
|
+
- @base
|
252
|
+
|
253
|
+
If we try to build this recipe, it will generate the exact same image as before.
|
254
|
+
But the idea here is to change it in order to install the desired software.
|
255
|
+
Therefore, we will modify the setup section like this::
|
256
|
+
|
257
|
+
extend: debian7
|
258
|
+
|
259
|
+
global:
|
260
|
+
# You can see the base template `debian7.yaml` to know the
|
261
|
+
# variables that you can override
|
262
|
+
|
263
|
+
bootstrap:
|
264
|
+
- @base
|
265
|
+
|
266
|
+
setup:
|
267
|
+
- @base
|
268
|
+
- install_software:
|
269
|
+
- packages: >
|
270
|
+
g++ make taktuk openssh-server openmpi-bin openmpi-common openmpi-dev
|
271
|
+
|
272
|
+
export:
|
273
|
+
- @base
|
274
|
+
|
275
|
+
|
276
|
+
For building execute::
|
277
|
+
|
278
|
+
$ kameleon build debian_customized.yaml
|
279
|
+
|
280
|
+
Then, you can follow the same steps as before to try it out and verify that the software was installed.
|
281
|
+
Now, let's make things a little more complicated. We will now compile and install TAU in our system.
|
282
|
+
So, for that let's create a step file that will look like this::
|
283
|
+
|
284
|
+
- get_tau:
|
285
|
+
- exec_in: cd /tmp/
|
286
|
+
- exec_in: wget -q http://www.cs.uoregon.edu/research/tau/tau_releases/tau-2.22.2.tar.gz
|
287
|
+
- exec_in: wget -q http://www.cs.uoregon.edu/research/tau/pdt_releases/pdt-3.19.tar.gz
|
288
|
+
|
289
|
+
- pdt_install:
|
290
|
+
- exec_in: cd /tmp/
|
291
|
+
- exec_in: tar -xzf pdt-3.19.tar.gz
|
292
|
+
- exec_in: cd /tmp/pdtoolkit-3.19
|
293
|
+
- exec_in: ./configure -prefix=/usr/local/pdt-install
|
294
|
+
- exec_in: make clean install
|
295
|
+
|
296
|
+
- tau_install:
|
297
|
+
- exec_in: cd /tmp/
|
298
|
+
- exec_in: tar -xzf tau-2.22.2.tar.gz
|
299
|
+
- exec_in: cd /tmp/tau-2.22.2
|
300
|
+
- exec_in: ./configure -prefix=/usr/local/tau-install -pdt=/usr/local/pdt-install/ -mpiinc=/usr/local/openmpi-install/include -mpilib=/usr/local/openmpi-install/lib
|
301
|
+
- exec_in: make install
|
302
|
+
|
303
|
+
|
304
|
+
You have to put it under the directory *steps/setup/* and you can call it tau_install.
|
305
|
+
In order to use it in your recipe, modify it as follows::
|
306
|
+
|
307
|
+
extend: debian7
|
308
|
+
|
309
|
+
global:
|
310
|
+
# You can see the base template `debian7.yaml` to know the
|
311
|
+
# variables that you can override
|
312
|
+
|
313
|
+
bootstrap:
|
314
|
+
- @base
|
315
|
+
|
316
|
+
setup:
|
317
|
+
- @base
|
318
|
+
- install_software:
|
319
|
+
- packages: >
|
320
|
+
g++ make taktuk openssh-server openmpi-bin openmpi-common openmpi-dev
|
321
|
+
- tau_install
|
322
|
+
export:
|
323
|
+
- @base
|
324
|
+
|
325
|
+
|
326
|
+
And rebuild the image again, you will see that it wont start from the beginning.
|
327
|
+
It will take advantage of the checkpoint system and it will start from the last
|
328
|
+
successfull executed step.
|
329
|
+
|
330
|
+
When building there is the following error::
|
331
|
+
|
332
|
+
|
333
|
+
[kameleon]: Step 46 : setup/tau_install/tau_install
|
334
|
+
[kameleon]: ---> Running step
|
335
|
+
[in_ctx]: Unset ParaProf's cubeclasspath...
|
336
|
+
[in_ctx]: Unset Perfdmf cubeclasspath...
|
337
|
+
[in_ctx]: Error: Cannot access MPI include directory /usr/local/openmpi-install/include
|
338
|
+
[kameleon]: Error occured when executing the following command :
|
339
|
+
[kameleon]:
|
340
|
+
[kameleon]: > exec_in: ./configure -prefix=/usr/local/tau-install -pdt=/usr/local/pdt-install/ -mpiinc=/usr/local/openmpi-install/include -mpilib=/usr/local/openmpi-install/lib
|
341
|
+
[kameleon]: Press [r] to retry
|
342
|
+
[kameleon]: Press [c] to continue with execution
|
343
|
+
[kameleon]: Press [a] to abort execution
|
344
|
+
[kameleon]: Press [l] to switch to local_context shell
|
345
|
+
[kameleon]: Press [o] to switch to out_context shell
|
346
|
+
[kameleon]: Press [i] to switch to in_context shell
|
347
|
+
[kameleon]: answer ? [c/a/r/l/o/i]:
|
348
|
+
|
349
|
+
We can observe that the problem is related with the configure script that cannot access the MPI path.
|
350
|
+
It can be debugged by using the interactive shell provided by Kameleon.
|
351
|
+
The interactive shell allows us to log into a given context.
|
352
|
+
For this case we see that the error happened in the in context, so let's type i in order to enter to this context::
|
353
|
+
|
354
|
+
[kameleon]: User choice : [i] launch in_context
|
355
|
+
[in_ctx]: Starting interactive shell
|
356
|
+
[kameleon]: Starting process: "LC_ALL=POSIX ssh -F /tmp/kameleon/debian_customized/ssh_config debian_customized -t /bin/bash"
|
357
|
+
(in_context) root@cristiancomputer: / #
|
358
|
+
|
359
|
+
The commands executed by Kameleon remain in the bash history.
|
360
|
+
Therefore, It can be rexecuted manually.
|
361
|
+
For this case, we only need to change the path for the OpenMPI libraries.
|
362
|
+
As we have installed it using the packages they are avaiable under the directories:
|
363
|
+
*/usr/include/openmpi/*, */usr/lib/openmpi/* respectively.
|
364
|
+
If we try with the following parameters::
|
365
|
+
|
366
|
+
# ./configure -prefix=/usr/local/tau-install -pdt=/usr/local/pdt-install/ -mpiinc=/usr/include/openmpi/ -mpilib=/usr/lib/openmpi/
|
367
|
+
|
368
|
+
It will finish without any problem. We have found the bug, therefore we can just logout by typing *exit* and
|
369
|
+
then *abort* for stopping the execution and update the step file with the previous line.
|
370
|
+
If you carry out the building again you will see that now everything goes smoothly.
|
371
|
+
Again Kameleon will use the checkpoint system to avoid starting from scratch.
|
372
|
+
|
373
|
+
---------------------------------
|
374
|
+
Creating a Grid'5000 environment.
|
375
|
+
---------------------------------
|
376
|
+
|
377
|
+
Now, let's use the extend and export functionalities for creating a Grid'5000 environment.
|
378
|
+
With this step we will see how code can be re-used with Kameleon.
|
379
|
+
Therefore, we can extend the recipe created before::
|
380
|
+
|
381
|
+
---
|
382
|
+
extend: debian_customized
|
383
|
+
|
384
|
+
global:
|
385
|
+
# You can see the base template `debian7.yaml` to know the
|
386
|
+
# variables that you can override
|
387
|
+
|
388
|
+
bootstrap:
|
389
|
+
- @base
|
390
|
+
|
391
|
+
setup:
|
392
|
+
- @base
|
393
|
+
|
394
|
+
export:
|
395
|
+
- save_appliance:
|
396
|
+
- input: $$image_disk
|
397
|
+
- output: $$kameleon_cwd/$$kameleon_recipe_name
|
398
|
+
- save_as_tgz
|
399
|
+
|
400
|
+
- g5k_custom:
|
401
|
+
- kadeploy_file:
|
402
|
+
- write_local:
|
403
|
+
- $$kameleon_cwd/$$kameleon_recipe_name.yaml
|
404
|
+
- |
|
405
|
+
#
|
406
|
+
# Kameleon generated based on kadeploy description file
|
407
|
+
#
|
408
|
+
---
|
409
|
+
name: $$kameleon_recipe_name
|
410
|
+
|
411
|
+
version: 1
|
412
|
+
|
413
|
+
os: linux
|
414
|
+
|
415
|
+
image:
|
416
|
+
file: $$kameleon_recipe_name.tar.gz
|
417
|
+
kind: tar
|
418
|
+
compression: gzip
|
419
|
+
|
420
|
+
postinstalls:
|
421
|
+
- archive: server:///grid5000/postinstalls/debian-x64-base-2.5-post.tgz
|
422
|
+
compression: gzip
|
423
|
+
script: traitement.ash /rambin
|
424
|
+
|
425
|
+
boot:
|
426
|
+
kernel: /vmlinuz
|
427
|
+
initrd: /initrd.img
|
428
|
+
|
429
|
+
filesystem: $$filesystem_type
|
430
|
+
|
431
|
+
This recipe will generate in the build directory a tar.gz image and a configuration file for Kadeploy.
|
432
|
+
For example::
|
433
|
+
|
434
|
+
$ ls builds
|
435
|
+
total 8831536
|
436
|
+
-rw-r--r-- 1 root root 18767806464 juin 15 23:04 base_debian_g5k.qcow2
|
437
|
+
-rw-r--r-- 1 root root 206403737 juin 15 23:04 debian_g5k.tar.gz
|
438
|
+
-rw-r--r-- 1 root root 379 juin 15 23:04 debian_g5k.yaml
|
439
|
+
-rw-r--r-- 1 root root 426 juin 15 23:03 fstab.orig
|
440
|
+
-rw------- 1 root root 672 juin 15 23:01 insecure_ssh_key
|
441
|
+
|
442
|
+
Therefore if we log in a Grid'5000 site for instance (Grenoble) we can submit a deploy job and
|
443
|
+
deploy the image using kadeploy::
|
444
|
+
|
445
|
+
|
446
|
+
user@fgrenoble:~$ oarsub -I t deploy
|
447
|
+
[ADMISSION RULE] Set default walltime to 3600.
|
448
|
+
[ADMISSION RULE] Modify resource description with type constraints
|
449
|
+
Generate a job key...
|
450
|
+
OAR_JOB_ID=1663465
|
451
|
+
Interactive mode : waiting...
|
452
|
+
Starting...
|
453
|
+
|
454
|
+
Connect to OAR job 1663465 via the node fgrenoble.grenoble.grid5000.fr
|
455
|
+
|
456
|
+
user@fgrenoble:~$ kadeploy -a debian_g5k.yaml -f $OAR_NODEFILE
|
457
|
+
|
458
|
+
|
459
|
+
With luck the image will be deployed on baremetal after some few minutes.
|
460
|
+
|
461
|
+
|
462
|
+
Playing with Kameleon contexts
|
463
|
+
------------------------------
|
464
|
+
|
465
|
+
The environment that has just been deployed is a basic debian.
|
466
|
+
It doesn't have the modules required for infiniband and
|
467
|
+
other configuration that site administrators do for a specific hardware
|
468
|
+
or politics of the site.
|
469
|
+
In this case would be good to be able to use the environments already
|
470
|
+
provided by Grid'5000. This can be done by using Kameleon contexts.
|
471
|
+
The idea is to re-utilize the same recipe we have written before.
|
472
|
+
|
473
|
+
Kameleon already provides a recipe for interacting with Grid'5000 where
|
474
|
+
the configuration of the context is as follows:
|
475
|
+
|
476
|
+
* Local context: it is the user's machine.
|
477
|
+
|
478
|
+
* Context out: it is the site frontend.
|
479
|
+
It is used for submitting a job and deploying
|
480
|
+
a given Grid'5000 environment.
|
481
|
+
|
482
|
+
* Context in: will be inside the deployed node.
|
483
|
+
|
484
|
+
|
485
|
+
First, we import the G5k recipe::
|
486
|
+
|
487
|
+
$ kameleon import debian7-g5k
|
488
|
+
|
489
|
+
And we can just make a copy of our previous recipe (debian customized) and
|
490
|
+
we call it for instance debian_customized_g5k.yaml.
|
491
|
+
This recipe will look like this:
|
492
|
+
|
493
|
+
.. literalinclude:: debian_customized_g5k.yaml
|
494
|
+
|
495
|
+
|
496
|
+
But there will be a problem with the installation of TAU. Because
|
497
|
+
we download the tarball directly from its web site which is an
|
498
|
+
operation not allowed in Grid'5000. Just certain sites are accessible
|
499
|
+
using a web proxy.
|
500
|
+
To solve this we have to modify the step *tau_install* like this:
|
501
|
+
|
502
|
+
.. literalinclude:: tau_install_g5k.yaml
|
503
|
+
|
504
|
+
Here, we change the context for performing the operation of download.
|
505
|
+
For now on, it will be the local context that is going to download the
|
506
|
+
tarballs. Then we have to put them into the *in contex* for
|
507
|
+
this operation we use a pipe. Pipes are a means for communicating
|
508
|
+
contexts. We use a pipe between our local context and the in contex.
|
509
|
+
|
510
|
+
With those changes we will be able to build a G5k environment with
|
511
|
+
our already tested configuration. The recipe saves
|
512
|
+
the environment on the Kameleon workdir on the frontend.
|
513
|
+
Thus the environment is accessible to be deployed the number of times needed.
|