bosh_openstack_cpi 0.0.7 → 1.5.0.pre.1113

data/README.md

@@ -1,7 +1,5 @@
 # OpenStack BOSH Cloud Provider Interface
 
-[![Build Status](https://secure.travis-ci.org/piston/openstack-bosh-cpi.png)](http://travis-ci.org/piston/openstack-bosh-cpi)
-
 ## Bringing the world’s most popular open source platform-as-a-service to the world’s most popular open source infrastructure-as-a-service platform
 
 This repo contains software designed to manage the deployment of Cloud Foundry on top of OpenStack, using Cloud Foundry BOSH. Say what?
@@ -18,9 +16,6 @@ Cloud Foundry is the leading open source platform-as-a-service (PaaS) offering w
 
 Cloud Foundry BOSH is an open source tool chain for release engineering, deployment and lifecycle management of large scale distributed services. In this manual we describe the architecture, topology, configuration, and use of BOSH, as well as the structure and conventions used in packaging and deployment.
 
- * BOSH Source Code: https://github.com/cloudfoundry/bosh
- * BOSH Documentation: https://github.com/cloudfoundry/oss-docs/blob/master/bosh/documentation.md
-
 ## OpenStack and Cloud Foundry, Together using BOSH
 
 Cloud Foundry BOSH defines a Cloud Provider Interface API that enables platform-as-a-service deployment across multiple cloud providers - initially VMWare's vSphere and AWS. Piston Cloud has partnered with VMWare to provide a CPI for OpenStack, opening up Cloud Foundry deployment to an entire ecosystem of public and private OpenStack deployments.

data/USAGE.md

@@ -1,4 +1,5 @@
 # BOSH OpenStack Cloud Provider Interface
+# Copyright (c) 2009-2013 VMware, Inc.
 # Copyright (c) 2012 Piston Cloud Computing, Inc.
 
 For online documentation see: http://rubydoc.info/gems/bosh_openstack_cpi/
@@ -9,6 +10,8 @@ These options are passed to the OpenStack CPI when it is instantiated.
 
 ### OpenStack options
 
+The registry options are passed to the Openstack CPI by the BOSH director based on the settings in `director.yml`:
+
 * `auth_url` (required)
   URL of the OpenStack Identity endpoint to connect to
 * `username` (required)
@@ -20,45 +23,55 @@ These options are passed to the OpenStack CPI when it is instantiated.
 * `region` (optional)
   OpenStack region
 * `endpoint_type` (optional)
-  OpenStack endpoint type for Glance
+  OpenStack endpoint type (publicURL (default), adminURL, internalURL)
+* `state_timeout` (optional)
+  Timeout (in seconds) for OpenStack resources desired state (by default 300)
+* `stemcell_public_visibility` (optional)
+  Set public visibility for stemcells (true or false (default))
 * `default_key_name` (required)
   default OpenStack ssh key name to assign to created virtual machines
 * `default_security_group` (required)
   default OpenStack security group to assign to created virtual machines
-* `private_key` (required)
-  local path to the ssh private key, must match `default_key_name`
 
 ### Registry options
 
-The registry options are passed to the Openstack CPI by the BOSH director based on the settings in `director.yml`, but can be overridden if needed.
+The registry options are passed to the Openstack CPI by the BOSH director based on the settings in `director.yml`:
 
 * `endpoint` (required)
   OpenStack registry URL
 * `user` (required)
   OpenStack registry user
 * `password` (required)
-  rOpenStack egistry password
+  OpenStack registry password
 
 ### Agent options
 
-Agent options are passed to the OpenStack  CPI by the BOSH director based on the settings in `director.yml`, but can be overridden if needed.
+The agent options are passed to the OpenStack CPI by the BOSH director based on the settings in `director.yml`:
+
+## Network options
+
+The OpenStack CPI supports these networks types:
+
+* `type` (required)
+  can be `dynamic` for a DHCP assigned IP by OpenStack, `manual` for a static IP assigned manually at the BOSH deployment manifest or `vip` to use a Floating IP (which needs to be already allocated)
 
-### Resource pool options
+These options are specified under `cloud_properties` in the `networks` section of a BOSH deployment manifest:
 
-These options are specified under `cloud_options` in the `resource_pools` section of a BOSH deployment manifest.
+* `security_groups` (optional)
+  the OpenStack security groups to assign to VMs. If not specified, it'll use the default security groups set at the OpenStack options
+
+* `net_id` (required for `manual` networks)
+  the OpenStack Quantum network UUID to attach as a NIC to VMs.
+
+## Resource pool options
+
+These options are specified under `cloud_properties` in the `resource_pools` section of a BOSH deployment manifest:
 
 * `instance_type` (required)
   which type of instance (OpenStack flavor) the VMs should belong to
 * `availability_zone` (optional)
   the OpenStack availability zone the VMs should be created in
 
-### Network options
-
-These options are specified under `cloud_options` in the `networks` section of a BOSH deployment manifest.
-
-* `type` (required)
-  can be either `dynamic` for a DHCP assigned IP by OpenStack, or `vip` to use a Floating IP (which needs to be already allocated)
-
 ## Example
 
 This is a sample of how OpenStack specific properties are used in a BOSH deployment manifest:
@@ -70,36 +83,51 @@ This is a sample of how OpenStack specific properties are used in a BOSH deploym
     ...
 
     networks:
-      - name: nginx_network
-        type: vip
-        cloud_properties: {}
       - name: default
         type: dynamic
         cloud_properties:
           security_groups:
-          - default
-
+            - default
+          net_id: 2438bca2-24fa-450f-ae7b-ec2e53b51984
+      - name: static
+        type: manual
+        subnets:
+          - name: private
+            range: 10.0.1.0/24
+            gateway: 10.0.1.1
+            reserved:
+              - 10.0.1.2 - 10.0.1.9
+            static:
+              - 10.0.1.10 - 10.0.1.20
+            cloud_properties:
+              security_groups:
+                - default
+              net_id: 8d8b84b4-faa6-4605-9fbf-c179bdae4282
+      - name: floating
+        type: vip
+        cloud_properties: {}
     ...
 
     resource_pools:
       - name: common
         network: default
-        size: 3
+        size: 1
         stemcell:
-          name: bosh-stemcell
-          version: 0.6.7
+          name: bosh-openstack-kvm-ubuntu
+          version: latest
         cloud_properties:
           instance_type: m1.small
+          availability_zone:
 
     ...
 
     properties:
       openstack:
-        auth_url: http://pistoncloud.com/:5000/v2.0/tokens
+        auth_url: http://pistoncloud.com/:5000/v2.0
         username: christopher
         api_key: QRoqsenPsNGX6
         tenant: Bosh
-        region: us-west
+        region: RegionOne
+        endpoint_type: publicURL
         default_key_name: bosh
-        default_security_groups: ["bosh"]
-        private_key: /home/bosh/.ssh/bosh.pem
+        default_security_groups: ["bosh"]

data/bin/bosh_openstack_console

@@ -1,5 +1,6 @@
 #!/usr/bin/env ruby
 
+# Copyright (c) 2009-2013 VMware, Inc.
 # Copyright (c) 2012 Piston Cloud Computing, Inc.
 
 # Usage example:
@@ -10,15 +11,6 @@
 #                                [],
 #                                {"foo" =>"bar"})
 
-gemfile = File.expand_path("../../Gemfile", __FILE__)
-
-if File.exists?(gemfile)
-  ENV["BUNDLE_GEMFILE"] = gemfile
-  require "rubygems"
-  require "bundler/setup"
-end
-
-$:.unshift(File.expand_path("../../lib", __FILE__))
 require "bosh_openstack_cpi"
 require "irb"
 require "irb/completion"
@@ -38,7 +30,7 @@ unless config_file
   exit(1)
 end
 
-@config = YAML.load_file(config_file)
+@config = Psych.load_file(config_file)
 
 module ConsoleHelpers
   def cpi

data/lib/bosh_openstack_cpi.rb

@@ -1,3 +1,4 @@
+# Copyright (c) 2009-2013 VMware, Inc.
 # Copyright (c) 2012 Piston Cloud Computing, Inc.
 
 require "cloud/openstack"

data/lib/cloud/openstack.rb

@@ -1,3 +1,4 @@
+# Copyright (c) 2009-2013 VMware, Inc.
 # Copyright (c) 2012 Piston Cloud Computing, Inc.
 
 module Bosh
@@ -6,24 +7,28 @@ end
 
 require "fog"
 require "httpclient"
+require "json"
 require "pp"
 require "set"
 require "tmpdir"
-require "uuidtools"
+require "securerandom"
 require "yajl"
 
+require "common/exec"
 require "common/thread_pool"
 require "common/thread_formatter"
 
+require 'bosh/registry/client'
 require "cloud"
 require "cloud/openstack/helpers"
 require "cloud/openstack/cloud"
-require "cloud/openstack/registry_client"
+require "cloud/openstack/tag_manager"
 require "cloud/openstack/version"
 
 require "cloud/openstack/network_configurator"
 require "cloud/openstack/network"
 require "cloud/openstack/dynamic_network"
+require "cloud/openstack/manual_network"
 require "cloud/openstack/vip_network"
 
 module Bosh

data/lib/cloud/openstack/cloud.rb

@@ -1,3 +1,4 @@
+# Copyright (c) 2009-2013 VMware, Inc.
 # Copyright (c) 2012 Piston Cloud Computing, Inc.
 
 module Bosh::OpenStackCloud
@@ -6,6 +7,9 @@ module Bosh::OpenStackCloud
   class Cloud < Bosh::Cloud
     include Helpers
 
+    BOSH_APP_DIR = "/var/vcap/bosh"
+    FIRST_DEVICE_NAME_LETTER = "b"
+
     attr_reader :openstack
     attr_reader :registry
     attr_reader :glance
@@ -22,15 +26,21 @@ module Bosh::OpenStackCloud
       @options = options.dup
 
       validate_options
+      initialize_registry
 
       @logger = Bosh::Clouds::Config.logger
 
       @agent_properties = @options["agent"] || {}
       @openstack_properties = @options["openstack"]
-      @registry_properties = @options["registry"]
 
       @default_key_name = @openstack_properties["default_key_name"]
       @default_security_groups = @openstack_properties["default_security_groups"]
+      @state_timeout = @openstack_properties["state_timeout"]
+      @stemcell_public_visibility = @openstack_properties["stemcell_public_visibility"]
+
+      unless @openstack_properties["auth_url"].match(/\/tokens$/)
+        @openstack_properties["auth_url"] = @openstack_properties["auth_url"] + "/tokens"
+      end
 
       openstack_params = {
         :provider => "OpenStack",
@@ -38,9 +48,15 @@ module Bosh::OpenStackCloud
         :openstack_username => @openstack_properties["username"],
         :openstack_api_key => @openstack_properties["api_key"],
         :openstack_tenant => @openstack_properties["tenant"],
-        :openstack_region => @openstack_properties["region"]
+        :openstack_region => @openstack_properties["region"],
+        :openstack_endpoint_type => @openstack_properties["endpoint_type"]
       }
-      @openstack = Fog::Compute.new(openstack_params)
+      begin
+        @openstack = Fog::Compute.new(openstack_params)
+      rescue Exception => e
+        @logger.error(e)
+        cloud_error("Unable to connect to the OpenStack Compute API. Check task debug log for details.")  
+      end
 
       glance_params = {
         :provider => "OpenStack",
@@ -49,17 +65,14 @@ module Bosh::OpenStackCloud
         :openstack_api_key => @openstack_properties["api_key"],
         :openstack_tenant => @openstack_properties["tenant"],
         :openstack_region => @openstack_properties["region"],
-        :openstack_endpoint_type => @openstack_properties["endpoint_type"] ||
-                                    "publicURL"
+        :openstack_endpoint_type => @openstack_properties["endpoint_type"]
       }
-      @glance = Fog::Image.new(glance_params)
-
-      registry_endpoint = @registry_properties["endpoint"]
-      registry_user = @registry_properties["user"]
-      registry_password = @registry_properties["password"]
-      @registry = RegistryClient.new(registry_endpoint,
-                                     registry_user,
-                                     registry_password)
+      begin
+        @glance = Fog::Image.new(glance_params)
+      rescue Exception => e
+        @logger.error(e)
+        cloud_error("Unable to connect to the OpenStack Image Service API. Check task debug log for details.")
+      end
 
       @metadata_lock = Mutex.new
     end
@@ -75,12 +88,8 @@ module Bosh::OpenStackCloud
     # @option cloud_properties [String] infrastructure Stemcell infraestructure
     # @option cloud_properties [String] disk_format Image disk format
     # @option cloud_properties [String] container_format Image container format
-    # @option cloud_properties [optional, String] kernel_id UUID of the kernel
-    #   image stored at OpenStack
     # @option cloud_properties [optional, String] kernel_file Name of the
     #   kernel image file provided at the stemcell archive
-    # @option cloud_properties [optional, String] ramdisk_id UUID of the
-    #   ramdisk image stored at OpenStack
     # @option cloud_properties [optional, String] ramdisk_file Name of the
     #   ramdisk image file provided at the stemcell archive
     # @return [String] OpenStack image UUID of the stemcell
@@ -88,83 +97,41 @@ module Bosh::OpenStackCloud
       with_thread_name("create_stemcell(#{image_path}...)") do
         begin
           Dir.mktmpdir do |tmp_dir|
-            @logger.info("Extracting stemcell to `#{tmp_dir}'...")
-            image_name = "BOSH-#{generate_unique_name}"
-
-            # 1. Unpack image to temp directory
-            unpack_image(tmp_dir, image_path)
-            root_image = File.join(tmp_dir, "root.img")
-        
-            # 2. If image contains a kernel file, upload it to glance service
-            kernel_id = nil
-            if cloud_properties["kernel_id"]
-              kernel_id = cloud_properties["kernel_id"]
-            elsif cloud_properties["kernel_file"]
-              kernel_image = File.join(tmp_dir, cloud_properties["kernel_file"])
-              unless File.exists?(kernel_image)
-                cloud_error("Kernel image " \
-                            "#{cloud_properties['kernel_file']} " \
-                            "is missing from stemcell archive")
-              end
-              kernel_params = {
-                :name => "#{image_name}-AKI",
-                :disk_format => "aki",
-                :container_format => "aki",
-                :location => kernel_image,
-                :properties => {
-                  :stemcell => image_name
-                }
-              }
-              @logger.info("Uploading kernel image...")
-              kernel_id = upload_image(kernel_params)
-            end
-
-            # 3. If image contains a ramdisk file, upload it to glance service
-            ramdisk_id = nil
-            if cloud_properties["ramdisk_id"]
-              ramdisk_id = cloud_properties["ramdisk_id"]
-            elsif cloud_properties["ramdisk_file"]
-              ramdisk_image = File.join(tmp_dir, cloud_properties["ramdisk_file"])
-              unless File.exists?(ramdisk_image)
-                cloud_error("Ramdisk image " \
-                            "#{cloud_properties['ramdisk_file']} " \
-                            "is missing from stemcell archive")
-              end
-              ramdisk_params = {
-                :name => "#{image_name}-ARI",
-                :disk_format => "ari",
-                :container_format => "ari",
-                :location => ramdisk_image,
-                :properties => {
-                  :stemcell => image_name
-                }
-              }
-              @logger.info("Uploading ramdisk image...")
-              ramdisk_id = upload_image(ramdisk_params)
-            end
-
-            # 4. Upload image using Glance service
+            @logger.info("Creating new image...")
             image_params = {
-              :name => image_name,
+              :name => "BOSH-#{generate_unique_name}",
               :disk_format => cloud_properties["disk_format"],
               :container_format => cloud_properties["container_format"],
-              :location => root_image,
-              :is_public => true
+              :is_public => @stemcell_public_visibility.nil? ? false : @stemcell_public_visibility,
             }
+            
             image_properties = {}
-            image_properties[:kernel_id] = kernel_id if kernel_id
-            image_properties[:ramdisk_id] = ramdisk_id if ramdisk_id
-            if cloud_properties["name"]
-              image_properties[:stemcell_name] = cloud_properties["name"]
-            end
-            if cloud_properties["version"]
-              image_properties[:stemcell_version] = cloud_properties["version"]
-            end
-            unless image_properties.empty?
-              image_params[:properties] = image_properties
+            vanilla_options = ["name", "version", "os_type", "os_distro", "architecture", "auto_disk_config"]
+            vanilla_options.reject{ |o| cloud_properties[o].nil? }.each do |key|
+              image_properties[key.to_sym] = cloud_properties[key]
+            end            
+            image_params[:properties] = image_properties unless image_properties.empty?
+            
+            # If image_location is set in cloud properties, then pass the copy-from parm. Then Glance will fetch it 
+            # from the remote location on a background job and store it in its repository.
+            # Otherwise, unpack image to temp directory and upload to Glance the root image.
+            if cloud_properties["image_location"]
+              @logger.info("Using remote image from `#{cloud_properties["image_location"]}'...")
+              image_params[:copy_from] = cloud_properties["image_location"]
+            else
+              @logger.info("Extracting stemcell file to `#{tmp_dir}'...")
+              unpack_image(tmp_dir, image_path)
+              image_params[:location] = File.join(tmp_dir, "root.img")
             end
-            @logger.info("Uploading image...")
-            upload_image(image_params)
+
+            # Upload image using Glance service            
+            @logger.debug("Using image parms: `#{image_params.inspect}'")
+            image = with_openstack { @glance.images.create(image_params) }
+            
+            @logger.info("Creating new image `#{image.id}'...")
+            wait_resource(image, :active)
+            
+            image.id.to_s
           end
         rescue => e
           @logger.error(e)
@@ -182,33 +149,9 @@ module Bosh::OpenStackCloud
     def delete_stemcell(stemcell_id)
       with_thread_name("delete_stemcell(#{stemcell_id})") do
         @logger.info("Deleting stemcell `#{stemcell_id}'...")
-        image = @glance.images.find_by_id(stemcell_id)
+        image = with_openstack { @glance.images.find_by_id(stemcell_id) }
         if image
-          kernel_id = image.properties["kernel_id"]
-          if kernel_id
-            kernel = @glance.images.find_by_id(kernel_id)
-            if kernel && kernel.properties["stemcell"]
-              if kernel.properties["stemcell"] == image.name
-                @logger.info("Deleting kernel `#{kernel_id}'...")
-                kernel.destroy
-                @logger.info("Kernel `#{kernel_id}' is now deleted")
-              end
-            end
-          end
-
-          ramdisk_id = image.properties["ramdisk_id"]
-          if ramdisk_id
-            ramdisk = @glance.images.find_by_id(ramdisk_id)
-            if ramdisk && ramdisk.properties["stemcell"]
-              if ramdisk.properties["stemcell"] == image.name
-                @logger.info("Deleting ramdisk `#{ramdisk_id}'...")
-                ramdisk.destroy
-                @logger.info("Ramdisk `#{ramdisk_id}' is now deleted")
-              end
-            end
-          end
-
-          image.destroy
+          with_openstack { image.destroy }
           @logger.info("Stemcell `#{stemcell_id}' is now deleted")
         else
           @logger.info("Stemcell `#{stemcell_id}' not found. Skipping.")
@@ -240,50 +183,76 @@ module Bosh::OpenStackCloud
         server_name = "vm-#{generate_unique_name}"
 
         network_configurator = NetworkConfigurator.new(network_spec)
-        security_groups =
-          network_configurator.security_groups(@default_security_groups)
-        @logger.debug("Using security groups: `#{security_groups.join(', ')}'")
 
-        image = @openstack.images.find { |i| i.id == stemcell_id }
-        if image.nil?
-          cloud_error("Image `#{stemcell_id}' not found")
+        openstack_security_groups = with_openstack { @openstack.security_groups }.collect { |sg| sg.name }
+        security_groups = network_configurator.security_groups(@default_security_groups)
+        security_groups.each do |sg|
+          cloud_error("Security group `#{sg}' not found") unless openstack_security_groups.include?(sg)
         end
+        @logger.debug("Using security groups: `#{security_groups.join(', ')}'")
+
+        nics = network_configurator.nics
+        @logger.debug("Using NICs: `#{nics.join(', ')}'")
+
+        image = with_openstack { @openstack.images.find { |i| i.id == stemcell_id } }
+        cloud_error("Image `#{stemcell_id}' not found") if image.nil?
         @logger.debug("Using image: `#{stemcell_id}'")
 
-        flavor = @openstack.flavors.find { |f|
-          f.name == resource_pool["instance_type"] }
-        if flavor.nil?
-          cloud_error("Flavor `#{resource_pool["instance_type"]}' not found")
+        flavor = with_openstack { @openstack.flavors.find { |f| f.name == resource_pool["instance_type"] } }
+        cloud_error("Flavor `#{resource_pool["instance_type"]}' not found") if flavor.nil?
+        if flavor_has_ephemeral_disk?(flavor)
+          if flavor.ram
+            # Ephemeral disk size should be at least the double of the vm total memory size, as agent will need:
+            # - vm total memory size for swapon,
+            # - the rest for /vcar/vcap/data
+            min_ephemeral_size = (flavor.ram / 1024) * 2
+            if flavor.ephemeral < min_ephemeral_size
+              cloud_error("Flavor `#{resource_pool["instance_type"]}' should have at least #{min_ephemeral_size}Gb " +
+                          "of ephemeral disk")
+            end
+          end
         end
         @logger.debug("Using flavor: `#{resource_pool["instance_type"]}'")
 
+        keyname = resource_pool["key_name"] || @default_key_name
+        keypair = with_openstack { @openstack.key_pairs.find { |k| k.name == keyname } }
+        cloud_error("Key-pair `#{keyname}' not found") if keypair.nil?
+        @logger.debug("Using key-pair: `#{keypair.name}' (#{keypair.fingerprint})")
+
         server_params = {
           :name => server_name,
           :image_ref => image.id,
           :flavor_ref => flavor.id,
-          :key_name => resource_pool["key_name"] || @default_key_name,
+          :key_name => keypair.name,
           :security_groups => security_groups,
-          :user_data => Yajl::Encoder.encode(user_data(server_name,
-                                                       network_spec))
+          :nics => nics,
+          :user_data => Yajl::Encoder.encode(user_data(server_name, network_spec)),
+          :personality => [{
+                            "path" => "#{BOSH_APP_DIR}/user_data.json",
+                            "contents" => Yajl::Encoder.encode(user_data(server_name, network_spec, keypair.public_key))
+                          }]
         }
 
-        availability_zone = select_availability_zone(disk_locality,
-                              resource_pool["availability_zone"])
-        if availability_zone
-          server_params[:availability_zone] = availability_zone
-        end
+        availability_zone = select_availability_zone(disk_locality, resource_pool["availability_zone"])
+        server_params[:availability_zone] = availability_zone if availability_zone
 
-        server = @openstack.servers.create(server_params)
+        @logger.debug("Using boot parms: `#{server_params.inspect}'")
+        server = with_openstack { @openstack.servers.create(server_params) }
 
         @logger.info("Creating new server `#{server.id}'...")
-        wait_resource(server, :active, :state)
+        begin
+          wait_resource(server, :active, :state)
+        rescue Bosh::Clouds::CloudError => e
+          @logger.warn("Failed to create server: #{e.message}")
+          raise Bosh::Clouds::VMCreationFailed.new(true)
+        end
 
         @logger.info("Configuring network for server `#{server.id}'...")
         network_configurator.configure(@openstack, server)
 
         @logger.info("Updating settings for server `#{server.id}'...")
-        settings = initial_agent_settings(server_name, agent_id, network_spec,
-                                          environment)
+        settings = initial_agent_settings(server_name, agent_id, network_spec, environment,
+                                          flavor_has_ephemeral_disk?(flavor))
         @registry.update_settings(server.name, settings)
 
         server.id.to_s
@@ -298,10 +267,10 @@ module Bosh::OpenStackCloud
     def delete_vm(server_id)
       with_thread_name("delete_vm(#{server_id})") do
         @logger.info("Deleting server `#{server_id}'...")
-        server = @openstack.servers.get(server_id)
+        server = with_openstack { @openstack.servers.get(server_id) }
         if server
-          server.destroy
-          wait_resource(server, :terminated, :state, true)
+          with_openstack { server.destroy }
+          wait_resource(server, [:terminated, :deleted], :state, true)
 
           @logger.info("Deleting settings for server `#{server.id}'...")
           @registry.delete_settings(server.name)
@@ -311,6 +280,18 @@ module Bosh::OpenStackCloud
       end
     end
 
+    ##
+    # Checks if an OpenStack server exists
+    #
+    # @param [String] server_id OpenStack server UUID
+    # @return [Boolean] True if the vm exists
+    def has_vm?(server_id)
+      with_thread_name("has_vm?(#{server_id})") do
+        server = with_openstack { @openstack.servers.get(server_id) }
+        !server.nil? && ![:terminated, :deleted].include?(server.state.downcase.to_sym)
+      end
+    end
+
     ##
     # Reboots an OpenStack Server
     #
@@ -318,10 +299,9 @@ module Bosh::OpenStackCloud
     # @return [void]
     def reboot_vm(server_id)
       with_thread_name("reboot_vm(#{server_id})") do
-        server = @openstack.servers.get(server_id)
-        unless server
-          cloud_error("Server `#{server_id}' not found")
-        end
+        server = with_openstack { @openstack.servers.get(server_id) }
+        cloud_error("Server `#{server_id}' not found") unless server
+
         soft_reboot(server)
       end
     end
@@ -332,28 +312,20 @@ module Bosh::OpenStackCloud
     # @param [String] server_id OpenStack server UUID
     # @param [Hash] network_spec Raw network spec passed by director
     # @return [void]
-    # @raise [Bosh::Clouds:NotSupported] if the security groups change
+    # @raise [Bosh::Clouds:NotSupported] If there's a network change that requires the recreation of the VM
     def configure_networks(server_id, network_spec)
       with_thread_name("configure_networks(#{server_id}, ...)") do
         @logger.info("Configuring `#{server_id}' to use the following " \
                      "network settings: #{network_spec.pretty_inspect}")
-
         network_configurator = NetworkConfigurator.new(network_spec)
-        server = @openstack.servers.get(server_id)
-
-        sg = @openstack.list_security_groups(server_id).body["security_groups"]
-        actual = sg.collect { |s| s["name"] }.sort
-        new = network_configurator.security_groups(@default_security_groups)
-
-        # If the security groups change, we need to recreate the VM
-        # as you can't change the security group of a running server,
-        # we need to send the InstanceUpdater a request to do it for us
-        unless actual == new
-          raise Bosh::Clouds::NotSupported,
-                "security groups change requires VM recreation: %s to %s" %
-                [actual.join(", "), new.join(", ")]
-        end
 
+        server = with_openstack { @openstack.servers.get(server_id) }
+        cloud_error("Server `#{server_id}' not found") unless server
+
+        compare_security_groups(server, network_configurator.security_groups(@default_security_groups))
+
+        compare_private_ip_addresses(server, network_configurator.private_ip)
+        
         network_configurator.configure(@openstack, server)
 
         update_agent_settings(server) do |settings|
@@ -371,17 +343,9 @@ module Bosh::OpenStackCloud
     # @return [String] OpenStack volume UUID
     def create_disk(size, server_id = nil)
       with_thread_name("create_disk(#{size}, #{server_id})") do
-        unless size.kind_of?(Integer)
-          raise ArgumentError, "Disk size needs to be an integer"
-        end
-
-        if (size < 1024)
-          cloud_error("Minimum disk size is 1 GiB")
-        end
-
-        if (size > 1024 * 1000)
-          cloud_error("Maximum disk size is 1 TiB")
-        end
+        raise ArgumentError, "Disk size needs to be an integer" unless size.kind_of?(Integer)
+        cloud_error("Minimum disk size is 1 GiB") if (size < 1024)
+        cloud_error("Maximum disk size is 1 TiB") if (size > 1024 * 1000)
 
         volume_params = {
           :name => "volume-#{generate_unique_name}",
@@ -390,14 +354,14 @@ module Bosh::OpenStackCloud
         }
 
         if server_id
-          server = @openstack.servers.get(server_id)
+          server = with_openstack { @openstack.servers.get(server_id) }
           if server && server.availability_zone
             volume_params[:availability_zone] = server.availability_zone
           end
         end
 
         @logger.info("Creating new volume...")
-        volume = @openstack.volumes.create(volume_params)
+        volume = with_openstack { @openstack.volumes.create(volume_params) }
 
         @logger.info("Creating new volume `#{volume.id}'...")
         wait_resource(volume, :available)
@@ -415,14 +379,14 @@ module Bosh::OpenStackCloud
     def delete_disk(disk_id)
       with_thread_name("delete_disk(#{disk_id})") do
         @logger.info("Deleting volume `#{disk_id}'...")
-        volume = @openstack.volumes.get(disk_id)
+        volume = with_openstack { @openstack.volumes.get(disk_id) }
         if volume
           state = volume.status
           if state.to_sym != :available
             cloud_error("Cannot delete volume `#{disk_id}', state is #{state}")
           end
 
-          volume.destroy
+          with_openstack { volume.destroy }
           wait_resource(volume, :deleted, :status, true)
         else
           @logger.info("Volume `#{disk_id}' not found. Skipping.")
@@ -438,14 +402,11 @@ module Bosh::OpenStackCloud
     # @return [void]
     def attach_disk(server_id, disk_id)
       with_thread_name("attach_disk(#{server_id}, #{disk_id})") do
-        server = @openstack.servers.get(server_id)
-        unless server
-          cloud_error("Server `#{server_id}' not found")
-        end
-        volume = @openstack.volumes.get(disk_id)
-        unless server
-          cloud_error("Volume `#{disk_id}' not found")
-        end
+        server = with_openstack { @openstack.servers.get(server_id) }
+        cloud_error("Server `#{server_id}' not found") unless server
+
+        volume = with_openstack { @openstack.volumes.get(disk_id) }
+        cloud_error("Volume `#{disk_id}' not found") unless volume
 
         device_name = attach_volume(server, volume)
 
@@ -465,14 +426,11 @@ module Bosh::OpenStackCloud
     # @return [void]
     def detach_disk(server_id, disk_id)
       with_thread_name("detach_disk(#{server_id}, #{disk_id})") do
-        server = @openstack.servers.get(server_id)
-        unless server
-          cloud_error("Server `#{server_id}' not found")
-        end
-        volume = @openstack.volumes.get(disk_id)
-        unless server
-          cloud_error("Volume `#{disk_id}' not found")
-        end
+        server = with_openstack { @openstack.servers.get(server_id) }
+        cloud_error("Server `#{server_id}' not found") unless server
+
+        volume = with_openstack { @openstack.volumes.get(disk_id) }
+        cloud_error("Volume `#{disk_id}' not found") unless volume
 
         detach_volume(server, volume)
 
@@ -484,6 +442,64 @@ module Bosh::OpenStackCloud
       end
     end
 
+    ##
+    # Takes a snapshot of an OpenStack volume
+    #
+    # @param [String] disk_id OpenStack volume UUID
+    # @param [Hash] metadata Metadata key/value pairs to add to snapshot
+    # @return [String] OpenStack snapshot UUID
+    # @raise [Bosh::Clouds::CloudError] if volume is not found
+    def snapshot_disk(disk_id, metadata)
+      with_thread_name("snapshot_disk(#{disk_id})") do
+        volume = with_openstack { @openstack.volumes.get(disk_id) }
+        cloud_error("Volume `#{disk_id}' not found") unless volume
+
+        devices = []
+        volume.attachments.each { |attachment| devices << attachment["device"] unless attachment.empty? }
+       
+        description = [:deployment, :job, :index].collect { |key| metadata[key] }
+        description << devices.first.split('/').last unless devices.empty?
+        snapshot_params = {
+          :name => "snapshot-#{generate_unique_name}",
+          :description => description.join('/'),
+          :volume_id => volume.id
+        }
+
+        @logger.info("Creating new snapshot for volume `#{disk_id}'...")
+        snapshot = @openstack.snapshots.new(snapshot_params)
+        with_openstack { snapshot.save(true) }
+
+        @logger.info("Creating new snapshot `#{snapshot.id}' for volume `#{disk_id}'...")
+        wait_resource(snapshot, :available)
+
+        snapshot.id.to_s
+      end
+    end
+
+    ##
+    # Deletes an OpenStack volume snapshot
+    #
+    # @param [String] snapshot_id OpenStack snapshot UUID
+    # @return [void]
+    # @raise [Bosh::Clouds::CloudError] if snapshot is not in available state
+    def delete_snapshot(snapshot_id)
+      with_thread_name("delete_snapshot(#{snapshot_id})") do
+        @logger.info("Deleting snapshot `#{snapshot_id}'...")
+        snapshot = with_openstack { @openstack.snapshots.get(snapshot_id) }
+        if snapshot
+          state = snapshot.status
+          if state.to_sym != :available
+            cloud_error("Cannot delete snapshot `#{snapshot_id}', state is #{state}")
+          end
+
+          with_openstack { snapshot.destroy }
+          wait_resource(snapshot, :deleted, :status, true)
+        else
+          @logger.info("Snapshot `#{snapshot_id}' not found. Skipping.")
+        end
+      end
+    end
+
     ##
     # Set metadata for an OpenStack server
     #
@@ -492,14 +508,13 @@ module Bosh::OpenStackCloud
     # @return [void]
     def set_vm_metadata(server_id, metadata)
       with_thread_name("set_vm_metadata(#{server_id}, ...)") do
-        server = @openstack.servers.get(server_id)
-        unless server
-          cloud_error("Server `#{server_id}' not found")
-        end
+        with_openstack do
+          server = @openstack.servers.get(server_id)
+          cloud_error("Server `#{server_id}' not found") unless server
 
-        metadata.each do |name, value|
-          value = "" if value.nil? # value is required
-          server.metadata.update(name => value)
+          metadata.each do |name, value|
+            TagManager.tag(server, name, value)
+          end
         end
       end
     end
@@ -524,7 +539,7 @@ module Bosh::OpenStackCloud
     # @note this is a private method that is public to make it easier to test
     def select_availability_zone(volumes, resource_pool_az)
       if volumes && !volumes.empty?
-        disks = volumes.map { |vid| @openstack.volumes.get(vid) }
+        disks = volumes.map { |vid| with_openstack { @openstack.volumes.get(vid) } }
         ensure_same_availability_zone(disks, resource_pool_az)
         disks.first.availability_zone
       else
@@ -555,7 +570,7 @@ module Bosh::OpenStackCloud
     #
     # @return [String] Unique name
     def generate_unique_name
-      UUIDTools::UUID.random_create.to_s
+      SecureRandom.uuid
     end
 
     ##
@@ -564,11 +579,12 @@ module Bosh::OpenStackCloud
     # @param [String] server_name server name
     # @param [Hash] network_spec network specification
     # @return [Hash] server user data
-    def user_data(server_name, network_spec)
+    def user_data(server_name, network_spec, public_key = nil)
       data = {}
 
       data["registry"] = { "endpoint" => @registry.endpoint }
       data["server"] = { "name" => server_name }
+      data["openssh"] = { "public_key" => public_key } if public_key
 
       with_dns(network_spec) do |servers|
         data["dns"] = { "nameserver" => servers }
@@ -592,13 +608,12 @@ module Bosh::OpenStackCloud
     end
 
     ##
-    # Generates initial agent settings. These settings will be read by agent
-    # from OpenStack registry (also a BOSH component) on a target server. Disk
-    # conventions for OpenStack are:
-    # system disk: /dev/vda
-    # ephemeral disk: /dev/ vdb
-    # OpenStack volumes can be configured to map to other device names later
-    # (vdc through vdz, also some kernels will remap vd* to xvd*).
+    # Generates initial agent settings. These settings will be read by Bosh Agent from Bosh Registry on a target 
+    # server. Disk conventions in Bosh Agent for OpenStack are:
+    # - system disk: /dev/sda
+    # - ephemeral disk: /dev/sdb
+    # - persistent disks: /dev/sdc through /dev/sdz
+    # As some kernels remap device names (from sd* to vd* or xvd*), Bosh Agent will lookup for the proper device name 
     #
     # @param [String] server_name Name of the OpenStack server (will be picked
     #   up by agent to fetch registry settings)
@@ -606,8 +621,9 @@ module Bosh::OpenStackCloud
     #   assume its identity
     # @param [Hash] network_spec Agent network spec
     # @param [Hash] environment Environment settings
+    # @param [Boolean] has_ephemeral Has Ephemeral disk?
     # @return [Hash] Agent settings
-    def initial_agent_settings(server_name, agent_id, network_spec, environment)
+    def initial_agent_settings(server_name, agent_id, network_spec, environment, has_ephemeral)
       settings = {
         "vm" => {
           "name" => server_name
@@ -615,12 +631,12 @@ module Bosh::OpenStackCloud
         "agent_id" => agent_id,
         "networks" => network_spec,
         "disks" => {
-          "system" => "/dev/vda",
-          "ephemeral" => "/dev/vdb",
+          "system" => "/dev/sda",
           "persistent" => {}
         }
       }
 
+      settings["disks"]["ephemeral"] = has_ephemeral ? "/dev/sdb" : nil
       settings["env"] = environment if environment
       settings.merge(@agent_properties)
     end
@@ -630,9 +646,7 @@ module Bosh::OpenStackCloud
     #
     # @param [Fog::Compute::OpenStack::Server] server OpenStack server
     def update_agent_settings(server)
-      unless block_given?
-        raise ArgumentError, "Block is not provided"
-      end
+      raise ArgumentError, "Block is not provided" unless block_given?
 
       @logger.info("Updating settings for server `#{server.id}'...")
       settings = @registry.read_settings(server.name)
@@ -647,7 +661,7 @@ module Bosh::OpenStackCloud
     # @return [void]
     def soft_reboot(server)
       @logger.info("Soft rebooting server `#{server.id}'...")
-      server.reboot
+      with_openstack { server.reboot }
       wait_resource(server, :active, :state)
     end
 
@@ -658,7 +672,7 @@ module Bosh::OpenStackCloud
     # @return [void]
     def hard_reboot(server)
       @logger.info("Hard rebooting server `#{server.id}'...")
-      server.reboot(type = 'HARD')
+      with_openstack { server.reboot(type = 'HARD') }
       wait_resource(server, :active, :state)
     end
 
@@ -669,31 +683,58 @@ module Bosh::OpenStackCloud
     # @param [Fog::Compute::OpenStack::Volume] volume OpenStack volume
     # @return [String] Device name
     def attach_volume(server, volume)
-      volume_attachments = @openstack.get_server_volumes(server.id).
-                           body['volumeAttachments']
-      device_names = Set.new(volume_attachments.collect! { |v| v["device"] })
-
-      new_attachment = nil
-      ("c".."z").each do |char|
-        dev_name = "/dev/vd#{char}"
-        if device_names.include?(dev_name)
-          @logger.warn("`#{dev_name}' on `#{server.id}' is taken")
-          next
-        end
-        @logger.info("Attaching volume `#{volume.id}' to `#{server.id}', " \
-                     "device name is `#{dev_name}'")
-        if volume.attach(server.id, dev_name)
-          wait_resource(volume, :"in-use")
-          new_attachment = dev_name
-        end
-        break
+      @logger.info("Attaching volume `#{volume.id}' to server `#{server.id}'...")
+      volume_attachments = with_openstack { server.volume_attachments }
+      device = volume_attachments.find { |a| a["volumeId"] == volume.id }
+
+      if device.nil?                
+        device_name = select_device_name(volume_attachments, first_device_name_letter(server))
+        cloud_error("Server has too many disks attached") if device_name.nil?
+
+        @logger.info("Attaching volume `#{volume.id}' to server `#{server.id}', device name is `#{device_name}'")
+        with_openstack { volume.attach(server.id, device_name) }
+        wait_resource(volume, :"in-use")        
+      else
+        device_name = device["device"]
+        @logger.info("Volume `#{volume.id}' is already attached to server `#{server.id}' in `#{device_name}'. Skipping.")
       end
 
-      if new_attachment.nil?
-        cloud_error("Server has too many disks attached")
+      device_name
+    end
+
+    ##
+    # Select the first available device name
+    #
+    # @param [Array] volume_attachments Volume attachments
+    # @param [String] first_device_name_letter First available letter for device names
+    # @return [String] First available device name or nil is none is available
+    def select_device_name(volume_attachments, first_device_name_letter)
+      (first_device_name_letter.."z").each do |char|
+        # Some kernels remap device names (from sd* to vd* or xvd*). 
+        device_names = ["/dev/sd#{char}", "/dev/vd#{char}", "/dev/xvd#{char}"]
+        # Bosh Agent will lookup for the proper device name if we set it initially to sd*.
+        return "/dev/sd#{char}" if volume_attachments.select { |v| device_names.include?( v["device"]) }.empty?
+        @logger.warn("`/dev/sd#{char}' is already taken")
       end
 
-      new_attachment
+      nil
+    end
+
+    ##
+    # Returns the first letter to be used on device names
+    #
+    # @param [Fog::Compute::OpenStack::Server] server OpenStack server
+    # @return [String] First available letter
+    def first_device_name_letter(server)
+      letter = "#{FIRST_DEVICE_NAME_LETTER}"
+      return letter if server.flavor.nil?
+      return letter unless server.flavor.has_key?('id')
+      flavor = with_openstack { @openstack.flavors.find { |f| f.id == server.flavor['id'] } }
+      return letter if flavor.nil?
+
+      letter.succ! if flavor_has_ephemeral_disk?(flavor)
+      letter.succ! if flavor_has_swap_disk?(flavor)
+      letter
     end
 
     ##
@@ -703,33 +744,68 @@ module Bosh::OpenStackCloud
     # @param [Fog::Compute::OpenStack::Volume] volume OpenStack volume
     # @return [void]
     def detach_volume(server, volume)
-      volume_attachments = @openstack.get_server_volumes(server.id).
-                           body['volumeAttachments']
-      device_map = volume_attachments.collect! { |v| v["volumeId"] }
+      @logger.info("Detaching volume `#{volume.id}' from `#{server.id}'...")
+      volume_attachments = with_openstack { server.volume_attachments }
+      if volume_attachments.find { |a| a["volumeId"] == volume.id }
+        with_openstack { volume.detach(server.id, volume.id) }
+        wait_resource(volume, :available)
+      else
+        @logger.info("Disk `#{volume.id}' is not attached to server `#{server.id}'. Skipping.")
+      end
+    end
 
-      unless device_map.include?(volume.id)
-        cloud_error("Disk `#{volume.id}' is not attached to " \
-                    "server `#{server.id}'")
+    ##
+    # Compares actual server security groups with those specified at the network spec
+    #
+    # @param [Fog::Compute::OpenStack::Server] server OpenStack server
+    # @param [Array] specified_sg_names Security groups specified at the network spec
+    # @return [void]
+    # @raise [Bosh::Clouds:NotSupported] If the security groups change, we need to recreate the VM as you can't 
+    # change the security group of a running server, so we need to send the InstanceUpdater a request to do it for us
+    def compare_security_groups(server, specified_sg_names)
+      actual_sg_names = with_openstack { server.security_groups }.collect { |sg| sg.name }
+
+      unless actual_sg_names.sort == specified_sg_names.sort
+        raise Bosh::Clouds::NotSupported,
+              "security groups change requires VM recreation: %s to %s" %
+              [actual_sg_names.join(", "), specified_sg_names.join(", ")]
       end
+    end
 
-      @logger.info("Detaching volume `#{volume.id}' from `#{server.id}'...")
-      volume.detach(server.id, volume.id)
-      wait_resource(volume, :available)
+    ##
+    # Compares actual server private IP addresses with the IP address specified at the network spec
+    #
+    # @param [Fog::Compute::OpenStack::Server] server OpenStack server
+    # @param [String] specified_ip_address IP address specified at the network spec (if Manual network)
+    # @return [void]
+    # @raise [Bosh::Clouds:NotSupported] If the IP address change, we need to recreate the VM as you can't 
+    # change the IP address of a running server, so we need to send the InstanceUpdater a request to do it for us
+    def compare_private_ip_addresses(server, specified_ip_address)
+      actual_ip_addresses = with_openstack { server.private_ip_addresses }
+
+      unless specified_ip_address.nil? || actual_ip_addresses.include?(specified_ip_address)
+        raise Bosh::Clouds::NotSupported,
+              "IP address change requires VM recreation: %s to %s" %
+              [actual_ip_addresses.join(", "), specified_ip_address]
+      end
     end
 
     ##
-    # Uploads a new image to OpenStack via Glance
+    # Checks if the OpenStack flavor has ephemeral disk
     #
-    # @param [Hash] image_params Image params
-    # @return [String] OpenStack image UUID
-    def upload_image(image_params)
-      @logger.info("Creating new image...")
-      started_at = Time.now
-      image = @glance.images.create(image_params)
-      total = Time.now - started_at
-      @logger.info("Created new image `#{image.id}', took #{total}s")
+    # @param [Fog::Compute::OpenStack::Flavor] OpenStack flavor
+    # @return [Boolean] true if flavor has ephemeral disk, false otherwise
+    def flavor_has_ephemeral_disk?(flavor)
+      flavor.ephemeral.nil? || flavor.ephemeral.to_i <= 0 ? false : true
+    end
 
-      image.id.to_s
+    ##
+    # Checks if the OpenStack flavor has swap disk
+    #
+    # @param [Fog::Compute::OpenStack::Flavor] OpenStack flavor
+    # @return [Boolean] true if flavor has swap disk, false otherwise
+    def flavor_has_swap_disk?(flavor)
+      flavor.swap.nil? || flavor.swap.to_i <= 0 ? false : true
     end
 
     ##
@@ -739,10 +815,11 @@ module Bosh::OpenStackCloud
     # @param [String] image_path Local filesystem path to a stemcell image
     # @return [void]
     def unpack_image(tmp_dir, image_path)
-      output = `tar -C #{tmp_dir} -xzf #{image_path} 2>&1`
-      if $?.exitstatus != 0
-        cloud_error("Failed to unpack stemcell root image" \
-                    "tar exit status #{$?.exitstatus}: #{output}")
+      result = Bosh::Exec.sh("tar -C #{tmp_dir} -xzf #{image_path} 2>&1", :on_error => :return)
+      if result.failed?
+        @logger.error("Extracting stemcell root image failed in dir #{tmp_dir}, " +
+                      "tar returned #{result.exit_status}, output: #{result.output}")
+        cloud_error("Extracting stemcell root image failed. Check task debug log for details.")
       end
       root_image = File.join(tmp_dir, "root.img")
       unless File.exists?(root_image)
@@ -775,8 +852,15 @@ module Bosh::OpenStackCloud
       end
     end
 
-    def task_checkpoint
-      Bosh::Clouds::Config.task_checkpoint
+    def initialize_registry
+      registry_properties = @options.fetch('registry')
+      registry_endpoint   = registry_properties.fetch('endpoint')
+      registry_user       = registry_properties.fetch('user')
+      registry_password   = registry_properties.fetch('password')
+
+      @registry = Bosh::Registry::Client.new(registry_endpoint,
+                                             registry_user,
+                                             registry_password)
     end
 
   end