knife-azure 1.0.2 → 1.1.0

data/README.md

@@ -0,0 +1,275 @@
+# Knife Azure
+
+## Description
+A [knife] (http://docs.opscode.com/knife.html) plugin to create,
+delete, and enumerate
+[Windows Azure] (https://www.windowsazure.com)
+resources to be managed by Chef.
+
+## Installation
+Be sure you are running the latest version of Chef, which can be installed
+via:
+
+    gem install chef
+
+This plugin is distributed as a Ruby Gem. To install it, run:
+
+    gem install knife-azure
+    
+Depending on your system's configuration, you may need to run this command
+with root/administrator privileges.
+
+## Configuration
+For this plugin to interact with Azure's REST API, you will need to give Knife
+information about your Azure account and credentials. The easiest way to do
+this is to sign in to the Azure portal and download a publishsettings file
+from https://manage.windowsazure.com/publishsettings/index?client=xplat to a
+local file system location, and
+then refer to the local file via an entry in your knife.rb:
+
+    knife[:azure_publish_settings_file] = "~/myazure.publishsettings"
+    
+Alternatively, all subcommands for this plugin will accept an
+--azure-publish-settings-file option to allow you to specify the path to that
+file with each command invocation.
+
+The plug-in also accepts authentication information specified using an
+alternative set of options -- see the section on "Alternative Management
+Certificate Specification" for details.
+
+## Basic Examples
+The following examples assume that you've configured the publishsettings file
+location in your knife.rb:
+
+      # List images for use in creating new VM's:
+      $ knife azure image list
+        
+      # List all VM's (including those not be managed by Chef)
+      $ knife azure server list
+  
+      # Create and bootstrap an Ubuntu VM over ssh
+      $ knife azure server create -N MyNewNode --azure-vm-size Medium --I 8fcc3d_Ubuntu-12_04-amd64-30GB -m 'West US' --ssh-user myuser --identity-file ~/.ssh/myprivatekey_rsa
+  
+      # Create and bootstrap a Windows VM over winrm
+      $ knife azure server create --azure-dns-name MyNewServerName --azure-vm-size Medium --I 8fcc3d_Win2012-amd64-30GB -m 'West US' --winrm-user myuser --winrm-password 'mypassword' --bootstrap-protocol winrm --distro 'windows-chef-client-msi'
+  
+      # Delete a server and purge it from the Chef server
+      $ knife azure server delete MyNewNode --purge -y
+
+Use the --help option to read more about each subcommand. Eg:
+
+    knife azure server create --help
+
+## Detailed Usage
+
+### Common Configuration
+Most configuration options can be specified either in your knife.rb file or as command line parameters. The CLI parameters override the knife.rb parameters.
+
+The following options are required for all subcommands:
+
+    option :azure_publish_settings_file      Path to your .publishsettings file
+
+OR
+
+    option :azure_subscription_id            Your Azure subscription ID
+    option :azure_mgmt_cert                  Management certificate in PEM format
+    option :azure_api_host_name              Your Azure API host name
+
+### Azure Image List Subcommand
+Outputs a list of all linux images that are available to use for provisioning. You should choose one of these to use for the :azure_source_image parameter to the server create command. You can use the filter option to see a detailed image list.
+
+    knife azure image list
+
+### Azure Server Create Subcommand
+This subcommand provisions a new server in Azure and then performs a Chef bootstrap. The goal of the bootstrap is to get Chef installed on the target system so it can run Chef Client with a Chef Server.
+
+#### Windows Bootstrapping Requirements
+knife-azure depends on knife-windows: https://github.com/opscode/knife-windows
+to bootstrap Windows machines via winrm (Basic, NTLM and Kerberos authentication) or ssh.
+
+The distro/template to be used for bootstrapping is: https://github.com/opscode/knife-windows/blob/master/lib/chef/knife/bootstrap/windows-chef-client-msi.erb
+
+Windows source images should have the WinRM service enabled and the
+authentication should be set accordingly (Basic, NTLM and Kerberos). Firewall rules should be added accordingly to the source images. Refer to the link to configure this:
+https://github.com/opscode/knife-windows#nodes
+
+#### Azure-specific Options
+      :azure_dns_name                   Required. The DNS prefix name that can be used to access the cloud
+                                        service which is unique within Windows Azure. If you want to add
+                                        new VM to an existing service/deployment, specify an exiting
+                                        dns-name, along with --azure-connect-to-existing-dns option. Otherwise
+                                        a new deployment is created.
+      :azure_service_location           Required. Specifies the geographic location of the resource as the
+                                        name of a datacenter location that is valid for your subscription. Eg:
+                                        West US, East US, East Asia, Southeast Asia, North Europe, West Europe.
+      :azure_source_image               Required. Specifies the name of the disk image to use to create
+                                        the virtual machine. Do a "knife azure image list" to see a
+                                        list of available images.
+      :azure_storage_account            A name for the storage account that is unique within Windows Azure.
+                                        Storage account names must be between 3 and 24 characters in
+                                        length and use numbers and lower-case letters only. This name is
+                                        the DNS prefix name and can be used to access blobs, queues, and
+                                        tables in the storage account.
+      :azure_vm_name                    Specifies the name for the virtual machine. The name must
+                                        be unique within the deployment.
+      :azure_os_disk_name               Optional. Specifies the friendly name of the disk containing
+                                        the guest OS image in the image repository.
+      :azure_vm_size                    Size of virtual machine. Default is Small.
+                                        (ExtraSmall, Small, Medium, Large, ExtraLarge)
+      :azure_connect_to_existing_dns    Set this flag to add the new VM to an existing
+                                        deployment/service. Must give the name of the existing
+                                        DNS correctly in the --azure-dns-name option
+
+#### Azure VM Quick Create
+You can create a server with minimal configuration. On the Azure Management Portal, this corresponds to a "Quick Create - VM". Sample command for quick create (for an Ubuntu instance):
+
+    knife azure server create
+                --azure-publish-settings-file '/path/to/your/cert.publishsettingsfile'
+                --azure-dns-name 'myservice'
+                --azure-service-location 'West US'
+                --azure-source-image 'source-image-name'
+                --ssh-user 'jetstream'
+                --identity-file '~/.ssh/myazure_rsa'
+
+Note that the --identity-file option, which enables specification of a private
+key authorized to communicate securely with the created server during the
+bootstrap process, will also configure the user specified by --ssh-user with
+the public key that corresponds to the private key specified by
+--identity-file. This configuration persists even after the create subcommand
+has completed successfully, so that the key specified with --identity-file can
+be used with ssh clients for subsequent access to the server as the user
+specified by --ssh-user.
+
+You can set these options from knife.rb. A typical knife.rb is
+shown below:
+
+    knife[:azure_publish_settings_file] = '/path/to/your/cert.publishsettingsfile'
+    knife[:azure_dns_name] = 'myservice'
+    knife[:azure_service_location] = 'West US'
+    knife[:azure_source_image] = 'source-image-name'
+    knife[:ssh_user] = 'jetstream'
+    knife[:identity_file] = '~/.ssh/myazure_rsa'
+
+#### Azure VM Advanced Create
+You can set various other options in the advanced create.
+  Eg: If you want to set the Azure VM Name different from that of the Azure DNS Name, set the option :azure_vm_name.
+  Eg: If you want to specify a Storage Account Name, set the option :azure_storage_account
+
+To connect to an existing DNS/service, you can use a command as below:
+
+    knife azure server create
+                --azure-subscription-id 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
+                --azure-mgmt-cert '/path/to/your/mgmtCert.pem'
+                --azure-api-host-name 'management.core.windows.net'
+                --azure-connect-to-existing-dns
+                --azure-dns-name 'myservice'
+                --azure-vm-name 'myvm02'
+                --azure-service-location 'West US'
+                --azure-source-image 'source-image-name'
+                --ssh-user 'jetstream'
+                --ssh-password 'jetstream@123'
+
+These options may also be configured from knife.rb, as in this example:
+
+    knife[:azure_subscription_id] = 'xxxxxxxx-xxxx-xxxx-xxxx-xxxxxxxxxxxx'
+    knife[:azure_mgmt_cert] = '/path/to/your/mgmtCert.pem'
+    knife[:azure_api_host_name] = 'management.core.windows.net'
+    knife[:azure_service_location] = 'West US'
+    knife[:azure_dns_name]='myservice'
+    knife[:azure_vm_name]='myvm02'
+    knife[:ssh_user]='jetstream'
+    knife[:identity_file]='/path/to/RSA/private/key'
+    knife[:azure_storage_account]='auxpreview104'
+    knife[:azure_os_disk_name]='disk107'
+    knife[:tcp_endpoints]='66'
+    knife[:udp_endpoints]='77,88,99'
+
+#### Options for Bootstrapping a Windows Node in Azure
+
+    :bootstrap_protocol           Default is winrm for a windows image
+    :winrm_password               The WinRM password
+    :winrm_port                   The WinRM port, by default this is 5985
+    :winrm_transport              The WinRM transport type.  valid choices are [ssl, plaintext]
+    :kerberos_keytab_file         The Kerberos keytab file used for authentication
+    :kerberos_realm               The Kerberos realm used for authentication
+    :kerberos_service             The Kerberos service used for authentication
+    :ca_trust_file                The Certificate Authority (CA) trust file used for SSL transport
+
+#### Azure Windows Node Create
+The quick create option requires the following options for a windows instance:
+    
+    knife azure server create
+                --azure-publish-settings-file '/path/to/your/cert.publishsettingsfile'
+                --azure-dns-name 'myserverdnsname'
+                --azure-service-location 'West US'
+                --azure-source-image 'windows-2012-image-id'
+                --winrm-user 'jetstream'
+                --winrm-password 'jetstream@123'                
+                --distro 'windows-chef-client-msi'
+
+Sample knife.rb for bootstrapping Windows Node with basic authentication
+
+    knife[:bootstrap_protocol] = 'winrm'
+    knife[:winrm_password] = 'mgcvTuvV2Rh'
+    knife[:winrm_user] = 'myuser'
+    knife[:winrm_port] = '5985'
+    knife[:distro] = 'windows-chef-client-msi'
+    knife[:azure_source_image]='windows-2012-image-id'
+
+### Azure Server Delete Subcommand
+Deletes an existing server in the currently configured Azure account. By
+default, this does not delete the associated node and client objects from the
+Chef server. To do so, add the --purge flag. Also by default, the DNS name, also called "cloud service", is deleted if you are deleting the last VM from that service. By default, the OS disk is also deleted. If you want to retain them add the --preserve flag as shown below. To delete the storage account, add the --delete-azure-storage-account flag since by default the storage account is not deleted.
+
+    knife azure server delete "myvm01"
+    knife azure server delete "myvm01" --purge  #purge chef node
+    knife azure server delete "myvm01" --preserve-azure-os-disk
+    knife azure server delete "myvm01" --preserve-azure-dns-name
+    knife azure server delete "myvm01" --delete-azure-storage-account
+
+Since the VM name can be the same across DNS name, you must specify the DNS
+name also to delete the VM. Sample command to delete a VM for a given DNS name:
+
+    knife azure server delete "myvm01" --azure-dns-name "mydnsname"
+    knife azure server delete "myvm01" "myvm02" --azure-dns-name "mydnsname"
+
+### Azure Server List Subcommand
+Outputs a list of all servers in the currently configured Azure account. PLEASE NOTE - this shows all instances associated with the account, some of which may not be currently managed by the Chef server.
+
+    knife azure server list
+
+## Alternative Management Certificate Specification
+In addition to specifying the management certificate using the publishsettings
+file, you can also specify it in PEM format. Follow these steps to generate the certificate in the PEM format:
+
+1. Download the settings file from https://manage.windowsazure.com/publishsettings/index?client=xplat
+2. Extract the data from the ManagementCertificate field into a separate file named - cert.pfx
+3. Decode the certificate file:
+
+#### On Linux/Mac(Homebrew)
+
+    base64 -d cert.pfx > cert_decoded.pfx
+
+#### On Windows
+You can decode and extract the PFX file using powershell or a free windows base 64 decoder such as http://www.fourmilab.ch/webtools/base64/base64.zip,
+
+    base64.exe -d cert.pfx -> cert_decoded.pfx
+
+4. Convert the decoded PFX file to a PEM file
+
+#### On Linux/Mac(Homebrew)
+
+    openssl pkcs12 -in cert_decoded.pfx -out managementCertificate.pem -nodes
+
+#### On Windows
+ Use powershell & run following command. If openssl.exe is not already installed it can be downloaded from http://www.openssl.org/related/binaries.html (Note: openssl depends on Microsoft Visual C++ Redistributable package (x86) which must be installed for openssl to function properly).
+
+    openssl base64 -d -A -in cert_decoded.pfx -out cert_decode.der
+
+    openssl pkcs12 -in cert_decoded.der -out managementCertificate.pem -nodes
+
+You might be asked to enter a password which is usually blank.
+You might be also asked to enter a passphrase. Please enter the phrase of your choice.
+
+It is possible to generate your own certificates and upload them. More Detailed Documentation about the Management Certificates is available : https://www.windowsazure.com/en-us/manage/linux/common-tasks/manage-certificates/
+

data/lib/azure/certificate.rb

@@ -0,0 +1,87 @@
+#
+# Author:: Mukta Aphale (mukta.aphale@clogeny.com)
+# Copyright:: Copyright (c) 2010-2011 Opscode, Inc.
+# License:: Apache License, Version 2.0
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#     http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+#
+
+class Azure
+  class Certificates
+    def initialize(connection)
+      @connection=connection
+    end
+    def create(params)
+      certificate = Certificate.new(@connection)
+      certificate.create(params)
+    end
+  end
+end
+
+class Azure
+  class Certificate
+    attr_accessor :connection
+    attr_accessor :cert_data, :fingerprint, :certificate_version
+    def initialize(connection)
+      @connection = connection
+      @certificate_version = 2 # cf. RFC 5280 - to make it a "v3" certificate
+    end
+    def create(params)
+      # If RSA private key has been specified, then generate an x 509 certificate from the
+      # public part of the key
+      @cert_data = generate_public_key_certificate_data({:ssh_key => params[:identity_file],
+                                             :ssh_key_passphrase => params[:identity_file_passphrase]})
+      # Generate XML to call the API
+      # Add certificate to the hosted service
+      builder = Nokogiri::XML::Builder.new do |xml|
+        xml.CertificateFile('xmlns'=>'http://schemas.microsoft.com/windowsazure') {
+          xml.Data @cert_data
+          xml.CertificateFormat 'pfx'
+          xml.Password 'knifeazure'
+        }
+      end
+      # Windows Azure API call
+      @connection.query_azure("hostedservices/#{params[:azure_dns_name]}/certificates", "post", builder.to_xml)
+      # Return the fingerprint to be used while adding role
+      @fingerprint
+    end
+
+    def generate_public_key_certificate_data (params)
+      # Generate OpenSSL RSA key from the mentioned ssh key path (and passphrase)
+      key = OpenSSL::PKey::RSA.new(File.read(params[:ssh_key]), params[:ssh_key_passphrase])
+      # Generate X 509 certificate
+      ca = OpenSSL::X509::Certificate.new
+      ca.version = @certificate_version
+      ca.serial = Random.rand(65534) + 1 # 2 digit byte range random number for better security aspect
+      ca.subject = OpenSSL::X509::Name.parse "/DC=org/DC=knife-plugin/CN=Opscode CA"
+      ca.issuer = ca.subject # root CA's are "self-signed"
+      ca.public_key = key.public_key # Assign the ssh-key's public part to the certificate
+      ca.not_before = Time.now
+      ca.not_after =  ca.not_before + 2 * 365 * 24 * 60 * 60 # 2 years validity
+      ef = OpenSSL::X509::ExtensionFactory.new
+      ef.subject_certificate = ca
+      ef.issuer_certificate = ca
+      ca.add_extension(ef.create_extension("basicConstraints","CA:TRUE",true))
+      ca.add_extension(ef.create_extension("keyUsage","keyCertSign, cRLSign", true))
+      ca.add_extension(ef.create_extension("subjectKeyIdentifier","hash",false))
+      ca.add_extension(ef.create_extension("authorityKeyIdentifier","keyid:always",false))
+      ca.sign(key, OpenSSL::Digest::SHA256.new)
+      # Generate the SHA1 fingerprint of the der format of the X 509 certificate
+      @fingerprint =  OpenSSL::Digest::SHA1.new(ca.to_der)
+      # Create the pfx format of the certificate
+      pfx = OpenSSL::PKCS12.create('knifeazure', 'knife-azure-pfx',  key,  ca)
+      # Encode the pfx format - upload this certificate
+      Base64.strict_encode64(pfx.to_der)
+    end
+  end
+end

data/lib/azure/connection.rb

@@ -23,11 +23,12 @@ require File.expand_path('../deploy', __FILE__)
 require File.expand_path('../role', __FILE__)
 require File.expand_path('../disk', __FILE__)
 require File.expand_path('../image', __FILE__)
+require File.expand_path('../certificate', __FILE__)
 
 class Azure
   class Connection
     include AzureAPI
-    attr_accessor :hosts, :rest, :images, :deploys, :roles, :disks, :storageaccounts
+    attr_accessor :hosts, :rest, :images, :deploys, :roles, :disks, :storageaccounts, :certificates
     def initialize(params={})
       @rest = Rest.new(params)
       @hosts = Hosts.new(self)
@@ -36,6 +37,7 @@ class Azure
       @deploys = Deploys.new(self)
       @roles = Roles.new(self)
       @disks = Disks.new(self)
+      @certificates = Certificates.new(self)
     end
     def query_azure(service_name, verb = 'get', body = '')
       Chef::Log.info 'calling ' + verb + ' ' + service_name

data/lib/azure/deploy.rb

@@ -21,40 +21,69 @@ class Azure
     def initialize(connection)
       @connection=connection
     end
-    def all
-      deploys = Array.new
-      hosts = @connection.hosts.all
-      hosts.each do |host|
-        deploy = Deploy.new(@connection)
-        deploy.retrieve(host.name)
-        unless deploy.name == nil
-          deploys << deploy
+    # force_load should be true when there is something in local cache and we want to reload
+    # first call is always load.
+    def load(force_load = false)
+      if not @deploys || force_load
+        @deploys = begin
+          deploys = Array.new
+          hosts = @connection.hosts.all
+          hosts.each do |host|
+            deploy = Deploy.new(@connection)
+            deploy.retrieve(host.name)
+            if deploy.name
+              host.add_deploy(deploy)
+              deploys << deploy
+            end
+          end
+          deploys
         end
       end
-      deploys
+      @deploys
+    end
+
+    def all
+      self.load
     end
-    def find(hostedservicename)
-      deployName = nil
-      self.all.each do |deploy|
-        next unless deploy.hostedservicename == hostedservicename
-        deployName = deploy.name
+
+    # TODO - Current knife-azure plug-in seems to have assumption that single hostedservice
+    # will always have one deployment (production). see Deploy#retrieve below
+    def get_deploy_name_for_hostedservice(hostedservicename)
+      host = @connection.hosts.find(hostedservicename)
+      if host && host.deploys.length > 0
+        host.deploys[0].name
+      else
+        nil
       end
-      deployName
     end
+
     def create(params)
-      unless @connection.hosts.exists(params[:hosted_service_name])
-        @connection.hosts.create(params)
-      end 
-      unless @connection.storageaccounts.exists(params[:storage_account])
+      if params[:azure_connect_to_existing_dns]
+        unless @connection.hosts.exists?(params[:azure_dns_name])
+          Chef::Log.fatal 'The specified Azure DNS Name does not exist.'
+          exit 1
+        end
+      else
+        ret_val = @connection.hosts.create(params)
+        if ret_val.css('Error Code').length > 0
+          Chef::Log.fatal 'Unable to create DNS:' + ret_val.at_css('Error Code').content + ' : ' + ret_val.at_css('Error Message').content
+          exit 1
+        end
+      end
+      unless @connection.storageaccounts.exists?(params[:azure_storage_account])
         @connection.storageaccounts.create(params)
       end
-      params['deploy_name'] = find(params[:hosted_service_name])
+      if params[:identity_file]
+        params[:fingerprint] = @connection.certificates.create(params)
+      end
+      params['deploy_name'] = get_deploy_name_for_hostedservice(params[:azure_dns_name])
+
       if params['deploy_name'] != nil
         role = Role.new(@connection)
         roleXML = role.setup(params)
         ret_val = role.create(params, roleXML)
       else
-        params['deploy_name'] = params[:hosted_service_name]
+        params['deploy_name'] = params[:azure_dns_name]
         deploy = Deploy.new(@connection)
         deployXML = deploy.setup(params)
         ret_val = deploy.create(params, deployXML)
@@ -63,7 +92,7 @@ class Azure
           Chef::Log.fatal 'Unable to create role:' + ret_val.at_css('Error Code').content + ' : ' + ret_val.at_css('Error Message').content
           exit 1
       end
-      @connection.roles.find(params[:role_name])
+      @connection.roles.find_in_hosted_service(params[:azure_vm_name], params[:azure_dns_name])
     end
     def delete(rolename)
     end
@@ -71,7 +100,8 @@ class Azure
 
   class Deploy
     include AzureUtility
-    attr_accessor :connection, :name, :status, :url, :roles, :hostedservicename
+    attr_accessor :connection, :name, :status, :url, :hostedservicename
+
     def initialize(connection)
       @connection = connection
     end
@@ -82,12 +112,12 @@ class Azure
         @name = xml_content(deployXML, 'Deployment Name')
         @status = xml_content(deployXML,'Deployment Status')
         @url = xml_content(deployXML, 'Deployment Url')
-        @roles = Array.new
+        @roles = Hash.new
         rolesXML = deployXML.css('Deployment RoleInstanceList RoleInstance')
         rolesXML.each do |roleXML|
           role = Role.new(@connection)
           role.parse(roleXML, hostedservicename, @name)
-          @roles << role
+          @roles[role.name] = role
         end
       end
     end
@@ -98,7 +128,7 @@ class Azure
       builder = Nokogiri::XML::Builder.new do |xml|
         xml.Deployment(
           'xmlns'=>'http://schemas.microsoft.com/windowsazure',
-          'xmlns:i'=>'http://www.w3.org/2001/XMLSchema-instance' 
+          'xmlns:i'=>'http://www.w3.org/2001/XMLSchema-instance'
         ) {
           xml.Name params['deploy_name']
           xml.DeploymentSlot 'Production'
@@ -110,8 +140,22 @@ class Azure
       builder.doc
     end
     def create(params, deployXML)
-      servicecall = "hostedservices/#{params[:hosted_service_name]}/deployments"
-      @connection.query_azure(servicecall, "post", deployXML.to_xml) 
+      servicecall = "hostedservices/#{params[:azure_dns_name]}/deployments"
+      @connection.query_azure(servicecall, "post", deployXML.to_xml)
+    end
+
+    def roles
+      @roles.values if @roles
+    end
+
+    # just delete from local cache
+    def delete_role_if_present(role)
+      @roles.delete(role.name) if @roles
     end
+
+    def find_role(name)
+      @roles[name] if @roles
+    end
+
   end
 end