dev-lxc 2.0.3 → 2.1.0

data/docs/base_containers.md

@@ -0,0 +1,44 @@
+### Base Containers
+
+The container that is used as the base container for a cluster's containers must exist before
+the cluster can be built. The cluster's containers are cloned from the base container using
+the btrfs filesystem to very quickly provide a lightweight duplicate of the container.
+
+This container provides the chosen OS platform and version (e.g. b-ubuntu-1404).
+
+A typical LXC container has minimal packages installed so `dev-lxc` makes sure that the
+same packages used in Chef's [bento boxes](https://github.com/opscode/bento) are
+installed to provide a more typical server environment.
+A few additional packages are also installed.
+
+Base containers have openssh-server installed and running with unique SSH Host Keys.
+
+Base containers have a "dev-lxc" user with "dev-lxc" password and passwordless sudo.
+
+*Once this base container is created there is rarely a need to delete it.*
+
+### Create a dev-lxc Base Container
+
+dev-lxc is able to create base containers that have openssh-server installed and running with unique SSH Host Keys.
+
+dev-lxc base containers have a "dev-lxc" user with "dev-lxc" password and passwordless sudo.
+
+You can see a menu of base containers that `dev-lxc` can create by using the following command.
+
+```
+dev-lxc create-base-container
+```
+
+The initial creation of base containers can take awhile so let's go ahead and start creating
+an Ubuntu 14.04 container now.
+
+```
+dev-lxc create-base-container b-ubuntu-1404
+```
+
+Note: It is possible to pass additional arguments to the underlying LXC create command.
+For example:
+
+```
+dev-lxc create-base-container b-ubuntu-1404 -o -- '--no-validate --keyserver http://my.key.server.com'
+```

data/docs/configuration.md

@@ -0,0 +1,182 @@
+### dev-lxc.yml Config Files
+
+dev-lxc uses a YAML configuration file named `dev-lxc.yml` to define a cluster.
+
+The `init` command generates sample config files for various server types.
+
+Let's generate a config for a cluster with a standalone Chef Server, Chef Automate server,
+private Supermarket server, build node and an infrastructure node.
+
+```
+dev-lxc init --chef --automate --supermarket --build-nodes --nodes > dev-lxc.yml
+```
+
+The contents of `dev-lxc.yml` should look like this.
+
+```
+# base_container must be the name of an existing container
+base_container: b-ubuntu-1404
+
+# list any host directories you want mounted into the servers
+#mounts:
+#  - /root/dev root/dev
+
+# list any SSH public keys you want added to /home/dev-lxc/.ssh/authorized_keys
+#ssh-keys:
+#  - /root/dev/clusters/id_rsa.pub
+
+# DHCP reserved (static) IPs must be selected from the IP range 10.0.3.150 - 254
+
+chef-server:
+  users:          # a user's password will be the same as its username
+    - mary-admin
+    - joe-user
+  orgs:
+    demo:
+      admins:
+        - mary-admin
+      non-admins:
+        - joe-user
+  servers:
+    chef.lxc:
+      ipaddress: 10.0.3.203
+      products:
+        chef-server:
+        manage:
+        push-jobs-server:
+        reporting:
+
+supermarket:
+  servers:
+    supermarket.lxc:
+      ipaddress: 10.0.3.206
+      products:
+        supermarket:
+
+automate:
+  servers:
+    automate.lxc:
+      ipaddress: 10.0.3.200
+      products:
+        delivery:
+        chefdk:     # downloaded only for build nodes
+      license_path: /CHANGE/ME
+      chef_org: delivery
+      enterprise_name: demo-ent
+
+build-nodes:
+  servers:
+    build-node-1.lxc:
+
+nodes:
+  chef_server_url: https://chef.lxc/organizations/demo
+  validation_client_name: demo-validator
+  # comment out or remove the validation_key path to use chef-server keys generated by dev-lxc
+  validation_key: # /path/to/org/validation/key
+  servers:
+    node-1.lxc:
+      products:
+        chef:
+```
+
+The dev-lxc.yml config file is very customizable. You can add or remove mounts, products or servers,
+change ip addresses, server names, the base_container and more.
+
+As you can see there are four server types represented by five servers.
+
+1. chef-server - chef.lxc
+2. analytics - analytics.lxc
+3. supermarket - supermarket.lxc
+4. nodes - node-1.lxc
+
+#### Global Settings
+
+The global settings used by each of the server types are the `base_container`, a list of `mounts` and
+a list of `ssh-keys`. These settings are described in the config comments.
+
+Be sure to set `base_container` in the `dev-lxc.yml` to an existing container's name.  
+This container will be cloned to create each container in the cluster.  
+If you don't already have a container to use as a `base_container` then you can follow the instructions in the  
+[Create a dev-lxc Base Container section](https://github.com/jeremiahsnapp/dev-lxc#create-a-dev-lxc-base-container) to create one.
+
+#### Server Specific Settings
+
+It is possible to define different values for `base_container`, `mounts` or `ssh-keys` for a particular server type or even for a particular server as you can see in the following snippet.
+
+```
+nodes:
+  base_container: b-ubuntu-1404
+  servers:
+    node-1.lxc:
+      base_container: b-centos-6
+    node-2.lxc:
+```
+
+IP addresses from the range 10.0.3.150 - 254 can be assigned to the servers. If an IP address
+is not specified then a dynamic IP address is assigned when the server starts.
+
+#### mixlib-install Library Automatically Manages a Cache of Product Packages
+
+dev-lxc uses the [mixlib-install](https://github.com/chef/mixlib-install) library to download Chef products
+to a cache in `/var/dev-lxc` in the host VM. This cache is automatically mounted into each server when it starts.
+
+A list of Chef products to be installed can be defined for each server
+using [product names that mixlib-install understands](https://github.com/chef/mixlib-install/blob/master/PRODUCT_MATRIX.md).
+
+The channel and version of the product can be defined also.
+
+Channel can be `current`, `stable` or `unstable` with `stable` as the default.
+Version can be `latest` or a version number with `latest` as the default.
+
+For example, the following specifies the `current` channel and version `0.16.1` of the `chefdk` product.
+
+```
+nodes:
+  servers:
+    node-1.lxc:
+      products:
+        chefdk:
+          channel: current
+          version: 0.16.1
+```
+
+The `package_source` setting can be used to specify a package file on disk.
+
+```
+nodes:
+  servers:
+    node-1.lxc:
+      products:
+        chefdk:
+          package_source: /root/chefdk_0.16.1-1_amd64.deb
+```
+
+#### Chef Server Orgs/Users
+
+When defining a Chef Server you can include organizations and users that will be automatically created and associated accordingly.
+
+#### Automatic Integration Between Servers
+
+dev-lxc knows how to automatically configure Chef Server standalone, Chef Server tier topology,
+Chef Server HA 2.0 as well as Chef Automate, Chef Client, Analytics, Compliance and Supermarket.
+
+If a Chef Automate, Analytics server or Supermarket server is defined in the same config file as
+a Chef Server then each server will automatically be integrated with that Chef Server.
+
+If a node server with Chef Client or Chef DK installed is defined in the same config file as
+a Chef Server then the Chef Client will automatically be configured to use that Chef Server.
+
+Alternatively, values for `chef_server_url`, `validation_client_name` and `validation_key` can
+be set in the config file either for all nodes or for each individual node.
+
+```
+nodes:
+  chef_server_url: https://chef.lxc/organizations/demo
+  validation_client_name: demo-validator
+  # comment out or remove the validation_key path to use chef-server keys generated by dev-lxc
+  validation_key: # /path/to/org/validation/key
+  servers:
+    node-1.lxc:
+      products:
+        chef:
+```

data/docs/dev-lxc_version_2.md

@@ -0,0 +1,10 @@
+# dev-lxc 2.0 is Available
+
+Here are some of the new features which provide a significantly simplified and streamlined usage.
+
+* mixlib-install library is used to automatically manage a cache of product packages
+* Genuine container snapshot management (make as many snapshots as you want)
+* New "nodes" server type which auto configures nodes for a Chef Server in the same cluster
+  * Removed all xc-... bash functions because the new "nodes" server type replaces this functionality
+* Able to build Chef Server HA 2.0 cluster using chef-backend
+* Updated and simplified READMEs

data/docs/manage_multiple_clusters.md

@@ -0,0 +1,30 @@
+### Maintain Uniqueness Across Multiple Clusters
+
+The default cluster configs are already designed to be unique from each other but as you build
+more clusters you have to maintain uniqueness across the YAML config files for the following items.
+
+* Server names, `api_fqdn` and `analytics_fqdn`
+
+    Server names should really be unique across all clusters.
+
+    Even when cluster A is shutdown, if cluster B uses the same server names when it is created it
+	will use the already existing servers from cluster A.
+
+    `api_fqdn` and `analytics_fqdn` uniqueness only matters when clusters with the same `api_fqdn`
+	and `analytics_fqdn` are running.
+
+    If cluster B is started with the same `api_fqdn` or `analytics_fqdn` as an already running cluster A,
+	then cluster B will overwrite cluster A's DNS resolution of `api_fqdn` or `analytics_fqdn`.
+
+* IP Addresses
+
+    IP addresses uniqueness only matters when clusters with the same IP's are running.
+
+    If cluster B is started with the same IP's as an already running cluster A, then cluster B
+	will overwrite cluster A's DHCP reservation of the IP's but dnsmasq will still refuse to
+	assign the IP's to cluster B because they already in use by cluster A. dnsmasq then assigns
+	random IP's from the DHCP pool to cluster B leaving it in an unexpected state.
+
+    The `dev-lxc-platform` creates the IP range 10.0.3.150 - 254 for DHCP reserved IP's.
+
+    Use unique IP's from that range when configuring clusters.

data/docs/prerequisites.md

@@ -0,0 +1,75 @@
+
+#### Download and Install Prerequisites
+
+Download and install [VirtualBox](https://www.virtualbox.org/wiki/Downloads).
+
+Download and install [Vagrant](https://www.vagrantup.com/downloads.html).
+
+Install the vagrant-persistent-storage plugin.
+
+```
+vagrant plugin install vagrant-persistent-storage
+```
+
+Download and install [ChefDK](http://downloads.chef.io/chef-dk/).
+
+#### Download dev-lxc-platform
+
+```
+git clone https://github.com/jeremiahsnapp/dev-lxc-platform.git
+```
+
+#### Configure .kitchen.yml
+
+The cpus and memory .kitchen.yml values are set high to give enough resources to comfortably run multiple containers.
+
+Configure .kitchen.yml settings such as cpus, memory, synced_folders as desired.
+
+#### Build the dev-lxc-platform VM
+
+This should take less than 15 minutes.
+
+While the VM is being created you are free to open a separate terminal and follow the remaining setup instructions.
+
+```
+cd dev-lxc-platform
+kitchen converge
+```
+
+#### Enable Vagrant Control for VM
+
+Typically we want to be able to shutdown and startup the dev-lxc-platform VM rather than use the usual kitchen model of converge and destroy so we need to enable Vagrant control over the VM for easier management.
+
+Install [direnv](http://direnv.net/) to use the `.envrc` file included in the
+dev-lxc-platform repo to automatically set `VAGRANT_CWD` upon entering the top level directory
+of the dev-lxc-platform repo.
+
+Vagrant commands run from this directory such as `vagrant up`, `vagrant ssh` and `vagrant halt` will manage the dev-lxc-platform VM.
+
+```
+brew install direnv
+```
+
+Be sure to follow the [direnv install instructions](http://direnv.net/) to add the appropriate line to your user's shell rc file.
+
+#### Setup Networking
+
+Your workstation needs to know how to resolve the .lxc domain that dev-lxc containers use.
+
+For OS X you can run the following command.
+
+```
+sudo mkdir -p /etc/resolver
+echo nameserver 10.0.3.1 | sudo tee /etc/resolver/lxc
+```
+
+Adding a route entry to the workstation enables direct communication between
+the workstation and any dev-lxc container.
+
+For OS X run the following command.
+
+The route entry won't survive a workstation reboot. You will have to recreate it as needed.
+
+```
+sudo route -n add 10.0.3.0/24 33.33.34.13
+```

data/docs/usage.md

@@ -0,0 +1,187 @@
+## Usage
+
+### Show Calculated Configuration
+
+Mostly for debugging purposes you have the ability to print the calculated cluster configuration.
+
+```
+dev-lxc show-config
+```
+
+### Cluster status
+
+Run the following command to see the status of the cluster.
+
+```
+dev-lxc status
+```
+
+This is an example of the output.
+
+```
+chef.lxc            NOT_CREATED
+
+analytics.lxc       NOT_CREATED
+
+supermarket.lxc     NOT_CREATED
+
+node-1.lxc          NOT_CREATED
+```
+
+### Specifying a Subset of Servers
+
+Many dev-lxc subcommands can act on a subset of the cluster's servers by specifying a regular expression that matches the desired server names.
+
+For example, the following command will show the status of the Chef Server.
+
+```
+dev-lxc status chef
+```
+
+### cluster-view, tks, tls commands
+
+The dev-lxc-platform comes with some commands that create and manage helpful
+tmux/byobu sessions to more easily see the state of a cluster.
+
+Running the `cluster-view` command in the same directory as a `dev-lxc.yml` file
+creates a tmux/byobu session with the same name as the cluster's directory.
+
+`cluster-view` can also be run with the parent directory of a `dev-lxc.yml` file
+as the first argument and `cluster-view` will change to that directory before
+creating the tmux/byobu session.
+
+The session's first window is named "cluster".
+
+The left side is for running dev-lxc commands.
+
+The right side updates every 0.5 seconds with the cluster's status provided by `dev-lxc status`.
+
+The session's second window is named "shell". It opens in the same directory as the
+cluster's `dev-lxc.yml` file.
+
+The `tls` and `tks` commands are really aliases.
+
+`tls` is an alias for `tmux list-sessions` and is used to see what tmux/byobu sessions
+are running.
+
+`tks` is an alias for `tmux kill-session -t` and is used to kill tmux/byobu sessions.
+When specifying the session to be killed you only need as many characters of the session
+name that are required to make the name unique among the list of running sessions.
+
+I recommend switching to a different running tmux/byobu session before killing the current
+tmux/byobu session. Otherwise you will need to reattach to the remaining tmux/byobu session.
+Use the keyboard shortcuts Alt-Up/Down to easily switch between tmux/byobu sessions.
+
+### Start cluster
+
+Starting the cluster the first time takes awhile since it has a lot to download and build.
+
+```
+dev-lxc up
+```
+
+A test org, users, knife.rb and keys are automatically created in
+the bootstrap backend server in `/root/chef-repo/.chef` for testing purposes.
+
+The `knife-opc` plugin is installed in the embedded ruby environment of the
+Private Chef and Enterprise Chef server to facilitate the creation of the test
+org and user.
+
+Note: You also have the option of running the `prepare-product-cache` subcommand which downloads required product packages to the cache.  
+This can be helpful when you don't want to start building the cluster yet but you want the package cache ready when you build the cluster later.
+
+```
+dev-lxc prepare-product-cache
+```
+
+### Print Chef Automate Credentials
+
+If the cluster has a Chef Automate server you can use the `print-automate-credentials` subcommand to see what the login credentials.
+
+```
+dev-lxc print-automate-credentials
+```
+
+### Create chef-repo
+
+Create a local chef-repo with appropriate knife.rb and pem files.
+
+Use the `-p` option to also get pivotal.pem and pivotal.rb files.
+
+Use the `-f` option to overwrite existing knife.rb and pivotal.rb files.
+
+```
+dev-lxc chef-repo
+```
+
+Now you can easily use knife to access the cluster.
+
+```
+cd chef-repo
+knife client list
+```
+
+### Stop and start the cluster
+
+```
+dev-lxc halt
+dev-lxc up
+```
+
+### Run arbitrary commands in each server
+
+```
+dev-lxc run-command chef 'uptime'
+```
+
+### Attach the terminal to a server
+
+Attach the terminal to a server in the cluster that matches the REGEX pattern given.
+
+```
+dev-lxc attach chef
+```
+
+### Create a snapshot of the servers
+
+Save the changes in the servers to snapshots with a comment.
+
+```
+dev-lxc halt
+dev-lxc snapshot -c 'this is a snapshot comment'
+```
+
+### List snapshots
+
+```
+dev-lxc snapshot -l
+```
+
+### Restore snapshots
+
+Restore snapshots by name.
+
+Leave out the snapshot name or specify `LAST` to restore the most recent snapshot.
+
+```
+dev-lxc snapshot -r
+dev-lxc up
+```
+
+### Destroy snapshots
+
+Destroy snapshots by name or destroy all snapshots by specifying `ALL`.
+
+Leave out the snapshot name or specify `LAST` to destroy the most recent snapshots.
+
+```
+dev-lxc snapshot -d
+```
+
+### Destroy cluster
+
+Use the following command to destroy the cluster's servers.
+
+```
+dev-lxc destroy
+```

data/lib/dev-lxc/cli.rb

@@ -22,6 +22,12 @@ module DevLXC::CLI
       end
     }
 
+    desc "show-config", "Show calculated configuration"
+    option :config, :desc => "Specify a cluster's YAML config file. `./dev-lxc.yml` will be used by default"
+    def show_config
+      get_cluster(options[:config]).show_config
+    end
+
     desc "create-base-container [BASE_CONTAINER_NAME]", "Create a base container"
     option :options, :aliases => "-o", :desc => "Specify additional options for the lxc create"
     def create_base_container(base_container_name=nil)
@@ -46,6 +52,8 @@ module DevLXC::CLI
     option :analytics, :type => :boolean, :desc => "Analytics Server"
     option :compliance, :type => :boolean, :desc => "Compliance Server"
     option :supermarket, :type => :boolean, :desc => "Supermarket Server"
+    option :automate, :type => :boolean, :desc => "Automate Server"
+    option :build_nodes, :type => :boolean, :desc => "Build Nodes"
     option :adhoc, :type => :boolean, :desc => "Adhoc Servers"
     option :append, :aliases => "-a", :type => :boolean, :desc => "Do not generate the global config header"
     option :filename, :aliases => "-f", :desc => "Write generated content to FILE rather than standard output."
@@ -67,6 +75,15 @@ base_container: b-ubuntu-1404
 chef-server:
   topology: tier
   api_fqdn: chef.lxc
+  users:          # a user's password will be the same as its username
+    - mary-admin
+    - joe-user
+  orgs:
+    demo:
+      admins:
+        - mary-admin
+      non-admins:
+        - joe-user
   servers:
     chef-be.lxc:
       ipaddress: 10.0.3.201
@@ -87,6 +104,15 @@ chef-server:
 )
       chef_config = %Q(
 chef-server:
+  users:          # a user's password will be the same as its username
+    - mary-admin
+    - joe-user
+  orgs:
+    demo:
+      admins:
+        - mary-admin
+      non-admins:
+        - joe-user
   servers:
     chef.lxc:
       ipaddress: 10.0.3.203
@@ -95,6 +121,23 @@ chef-server:
         manage:
         push-jobs-server:
         reporting:
+)
+      automate_config = %Q(
+automate:
+  servers:
+    automate.lxc:
+      ipaddress: 10.0.3.200
+      products:
+        delivery:
+        chefdk:     # downloaded only for build nodes
+      license_path: /CHANGE/ME
+      chef_org: delivery
+      enterprise_name: demo-ent
+)
+      build_nodes_config = %Q(
+build-nodes:
+  servers:
+    build-node-1.lxc:
 )
       analytics_config = %Q(
 analytics:
@@ -129,6 +172,15 @@ adhoc:
       chef_backend_config = %Q(
 chef-backend:
   api_fqdn: chef.lxc
+  users:          # a user's password will be the same as its username
+    - mary-admin
+    - joe-user
+  orgs:
+    demo:
+      admins:
+        - mary-admin
+      non-admins:
+        - joe-user
   servers:
     chef-backend1.lxc:
       ipaddress: 10.0.3.208
@@ -156,6 +208,10 @@ chef-backend:
 )
       nodes_config = %Q(
 nodes:
+  chef_server_url: https://chef.lxc/organizations/demo
+  validation_client_name: demo-validator
+  # comment out or remove the validation_key path to use chef-server keys generated by dev-lxc
+  validation_key: # /path/to/org/validation/key
   servers:
     node-1.lxc:
       products:
@@ -168,6 +224,8 @@ nodes:
       config += analytics_config if options[:analytics]
       config += compliance_config if options[:compliance]
       config += supermarket_config if options[:supermarket]
+      config += automate_config if options[:automate]
+      config += build_nodes_config if options[:build_nodes]
       config += adhoc_config if options[:adhoc]
       config += chef_backend_config if options[:chef_backend]
       config += nodes_config if options[:nodes]
@@ -241,6 +299,12 @@ nodes:
       get_cluster(options[:config]).chef_repo(options[:force], options[:pivotal])
     end
 
+    desc "print-automate-credentials", "Print Automate credentials"
+    option :config, :desc => "Specify a cluster's YAML config file. `./dev-lxc.yml` will be used by default"
+    def print_automate_credentials
+      get_cluster(options[:config]).print_automate_credentials
+    end
+
     desc "run-command [SERVER_NAME_REGEX] [COMMAND]", "Runs a command in each server"
     option :config, :desc => "Specify a cluster's YAML config file. `./dev-lxc.yml` will be used by default"
     def run_command(server_name_regex=nil, command)
@@ -249,6 +313,16 @@ nodes:
       print_elapsed_time(Time.now - start_time)
     end
 
+    desc "prepare-product-cache [SERVER_NAME_REGEX]", "Download required product packages to cache"
+    option :config, :desc => "Specify a cluster's YAML config file. `./dev-lxc.yml` will be used by default"
+    def prepare_product_cache(server_name_regex=nil)
+      start_time = Time.now
+      cluster = get_cluster(options[:config])
+      servers = cluster.get_sorted_servers(server_name_regex)
+      cluster.prep_product_cache(servers, true)
+      print_elapsed_time(Time.now - start_time)
+    end
+
     desc "up [SERVER_NAME_REGEX]", "Start servers - This is the default if no subcommand is given"
     option :config, :desc => "Specify a cluster's YAML config file. `./dev-lxc.yml` will be used by default"
     def up(server_name_regex=nil)