bosh-cloudfoundry 0.7.0.alpha.7 → 0.7.0.alpha.8

data/ChangeLog.md

@@ -17,6 +17,7 @@ The rewrite introduces some new implementation/feature concepts:
 * using `bosh diff` (aka biff) to generate the deployment file
 * bundles all final releases into the project & distributed rubygem/plugin (no runtime dependency on cf-release git repository; only the public blobstore)
 * templates are versioned for each final release (unless new templates not required for new release)
-* different sizes of deployments (orders of magnitude), such as small, medium & large
+* different sizes of deployments (orders of magnitude), such as small, medium & large: `bosh create cf --deployment-size large`
+* mutable/changable properties (and immutable properties) for each template version: `bosh change cf attributes persistent_disk=8192`
 
 The latter means that new versions of this rubygem can be published that are backwards compatible with aging deployments of Cloud Foundry. There should not be any forced coupling of old `bosh-cloudfoundry` to old `cf-release` final releases.

data/README.md

@@ -2,12 +2,13 @@
 
 This is a simple `bosh` CLI plugin to boot up Cloud Foundry and then grow and upgrade and maintain it. Initially runs on AWS or OpenStack via bosh.
 
-Example create/delete scenario:
+Example create/scale/delete scenario:
 
 ```
 $ bosh prepare cf
 $ bosh create cf --dns mycloud.com --public-ip 1.2.3.4
-
+...
+$ bosh change cf attributes persistent_disk=8192
 ...
 $ bosh delete cf
 ```
@@ -93,6 +94,8 @@ The `bosh_cli` gem is currently only available from S3, rather than RubyGem itse
 
 ## Usage
 
+### Create initial Cloud Foundry
+
 Each time you install the latest `bosh-cloudfoundry` gem you may want to re-upload the latest available Cloud Foundry bosh release to your bosh. If no newer release is available then nothing good nor bad will occur.
 
 ```
@@ -123,14 +126,16 @@ $ bosh create cf --security-group cf-core
 * TODO - how to update Cloud Foundry servers to a different instance size/flavor
 * TODO - how to update the persistent disks of the deployment
 
-## Initializing Cloud Foundry
+### Initializing Cloud Foundry
 
 Once Cloud Foundry is up and running, follow these steps to login (and discover your password) and create an initial organization and space:
 
 ```
 $ cf target api.mycloud.com
-$ bosh show cf passwords
-Common password: 6d7fe84f828b
+$ bosh show cf attributes
+...
+common_password: 6d7fe84f828b
+...
 $ cf login admin
 Password> 6d7fe84f828b
 
@@ -139,6 +144,16 @@ $ cf create-space production
 $ cf switch-space production
 ```
 
+### Scaling Cloud Foundry
+
+If your persistent disks start filling up (monitor via `bosh vms --vitals`) then you can scale them up by running:
+
+```
+$ bosh change cf attributes persistent_disk=8192
+```
+
+The initial size of persistent disks is `4096` (4Gb).
+
 ## Releasing new plugin gem versions
 
 There are two reasons to release new versions of this plugin.

data/bosh-cloudfoundry.gemspec

@@ -2,7 +2,7 @@
 
 Gem::Specification.new do |spec|
   spec.name          = "bosh-cloudfoundry"
-  spec.version       = "0.7.0.alpha.7"
+  spec.version       = "0.7.0.alpha.8"
   spec.authors       = ["Dr Nic Williams"]
   spec.email         = ["drnicwilliams@gmail.com"]
   spec.description   = %q{Create & manage Cloud Foundry deployments}

data/lib/bosh/cli/commands/cf.rb

@@ -73,11 +73,7 @@ module Bosh::Cli::Command
       say "Security group: #{attrs.validated_color(:security_group)}"
       nl
 
-      step("Validating deployment size", "Available deployment sizes are #{attrs.available_deployment_sizes.join(', ')}", :fatal) do
-        attrs.validate(:deployment_size)
-      end
-
-      validate_dns_mapping
+      validate_deployment_attributes
 
       unless confirmed?("Security group #{attrs.validated_color(:security_group)} exists with ports #{attrs.required_ports.join(", ")}")
         cancel_deployment
@@ -88,34 +84,64 @@ module Bosh::Cli::Command
 
       raise Bosh::Cli::ValidationHalted unless errors.empty?
 
-      deployment_file = DeploymentFile.new(@release_version_cpi_size, attrs, bosh_status)
-      deployment_file.prepare_environment
-      deployment_file.create_deployment_file
-      deployment_file.deploy(options)
-
+      @deployment_file = DeploymentFile.new(@release_version_cpi_size, attrs, bosh_status)
+      perform_deploy(options)
     rescue Bosh::Cli::ValidationHalted
       errors.each do |error|
         say error.make_red
       end
+      exit 1
     end
 
-    usage "show cf properties"
-    desc "display the deployment properties, indicate which are changable"
-    def show_cf_properties
+    usage "show cf attributes"
+    desc "display the deployment attributes, indicate which are changable"
+    def show_cf_attributes
       setup_deployment_attributes
       reconstruct_deployment_file
       nl
-      say "Immutable properties:"
+      say "Immutable attributes:"
       attrs.immutable_attributes.each do |attr_name|
         say "#{attr_name}: #{attrs.validated_color(attr_name.to_sym)}"
       end
       nl
-      say "Mutable (changable) properties:"
+      say "Mutable (changable) attributes:"
       attrs.mutable_attributes.each do |attr_name|
         say "#{attr_name}: #{attrs.validated_color(attr_name.to_sym)}"
       end
     end
 
+    usage "change cf attributes"
+    desc "change deployment attributes and perform bosh deploy"
+    def change_cf_attributes(*attribute_values)
+      setup_deployment_attributes
+      reconstruct_deployment_file
+
+      # TODO fail if setting immutable attributes
+      attribute_values.each do |attr_value|
+        # FIXME check that correct format of input: xyz=123 and give feedback
+        attr_name, value = attr_value.split(/=/)
+        previous_value = attrs.validated_color(attr_name)
+        step("Checking '#{attr_name}' is a valid mutable attribute",
+             "Attribute '#{attr_name}' is not a valid mutable attribute (see 'bosh show cf attributes')", :non_fatal) do
+          attrs.mutable_attribute?(attr_name)
+        end
+        attrs.set_mutable(attr_name, value)
+      end
+
+      validate_deployment_attributes
+      # TODO show validated attributes like "create cf"
+
+      raise Bosh::Cli::ValidationHalted unless errors.empty?
+
+      perform_deploy(options)
+
+    rescue Bosh::Cli::ValidationHalted
+      errors.each do |error|
+        say error.make_red
+      end
+      exit 1
+    end
+
     protected
     def setup_deployment_attributes
       @release_version_cpi = ReleaseVersionCpi.latest_for_cpi(bosh_cpi)
@@ -134,6 +160,10 @@ module Bosh::Cli::Command
       @release_version_cpi_size = @deployment_file.release_version_cpi_size
     end
 
+    def perform_deploy(deploy_options={})
+      @deployment_file.perform(deploy_options)
+    end
+
     def bosh_release_dir
       File.expand_path("../../../../../bosh_release", __FILE__)
     end
@@ -193,8 +223,10 @@ module Bosh::Cli::Command
       cmd
     end
 
-    def validate_dns_mapping
+    def validate_deployment_attributes
+      attrs.validate_deployment_size
       attrs.validate_dns_mapping
     end
+
   end
 end

data/lib/bosh/cloudfoundry/deployment_attributes.rb

@@ -54,12 +54,16 @@ module Bosh::Cloudfoundry
 
     # Attributes & their values that can be changed via setters & deployment re-deployed successfully
     def mutable_attributes
-      release_version_cpi.mutable_attributes
+      release_version_cpi.mutable_attributes.map(&:to_sym)
     end
 
     # Attributes & their values that are not to be changed over time
     def immutable_attributes
-      release_version_cpi.immutable_attributes
+      release_version_cpi.immutable_attributes.map(&:to_sym)
+    end
+
+    def mutable_attribute?(attribute)
+      mutable_attributes.include?(attribute.to_sym)
     end
 
     def set_unless_nil(attribute, value)
@@ -67,7 +71,15 @@ module Bosh::Cloudfoundry
     end
 
     def set(attribute, value)
-      attributes[attribute.to_sym] = value if value
+      attributes[attribute.to_sym] = value
+    end
+
+    def set_mutable(attribute, value)
+      if mutable_attribute?(attribute)
+        attributes[attribute.to_sym] = value
+      else
+        false
+      end
     end
 
     def validate(attribute)
@@ -81,6 +93,12 @@ module Bosh::Cloudfoundry
       end
     end
 
+    def validate_deployment_size
+      step("Validating deployment size", "Available deployment sizes are #{available_deployment_sizes.join(', ')}", :fatal) do
+        validate(:deployment_size)
+      end
+    end
+
     # FIXME only supports a single ip_address
     def validate_dns_mapping
       validate_dns_a_record("api.#{dns}", ip_addresses.first)

data/lib/bosh/cloudfoundry/deployment_file.rb

@@ -19,6 +19,12 @@ module Bosh::Cloudfoundry
       @bosh_status = bosh_status
     end
 
+    def perform(deploy_options={})
+      prepare_environment
+      create_deployment_file
+      deploy(deploy_options)
+    end
+
     def prepare_environment
       step("Checking/creating #{deployment_file_dir} for deployment files",
            "Failed to create #{deployment_file_dir} for deployment files", :fatal) do
@@ -31,9 +37,6 @@ module Bosh::Cloudfoundry
     # ---
     # name: NAME
     # director_uuid: 4ae3a0f0-70a5-4c0d-95f2-7fafaefe8b9e
-    # releases:
-    #  - name: cf-release
-    #    version: 132
     # networks: {}
     # properties:
     #   cf:
@@ -51,10 +54,6 @@ module Bosh::Cloudfoundry
           file << {
             "name" => deployment_attributes.name,
             "director_uuid" => bosh_uuid,
-            "releases" => {
-              "name" => release_name,
-              "version" => release_version_number
-            },
             "networks" => {},
             "properties" => {
               self.class.properties_key => deployment_attributes.attributes_with_string_keys
@@ -105,8 +104,6 @@ module Bosh::Cloudfoundry
       end
       release_version = release["version"]
       release_version_cpi = ReleaseVersionCpi.new(release_version, bosh_cpi)
-      deployment_size = deployment_file["properties"]["deployment_size"]
-      release_version_cpi_size = ReleaseVersionCpiSize.new(release_version_cpi, deployment_size)
 
       attributes = deployment_file["properties"][properties_key]
       # convert string keys to symbol keys
@@ -115,6 +112,9 @@ module Bosh::Cloudfoundry
       end
       deployment_attributes = DeploymentAttributes.new(director_client, bosh_status, release_version_cpi, attributes)
 
+      deployment_size = deployment_attributes.deployment_size
+      release_version_cpi_size = ReleaseVersionCpiSize.new(release_version_cpi, deployment_size)
+
       self.new(release_version_cpi_size, deployment_attributes, bosh_status)
     end
 

data/lib/bosh/cloudfoundry/validations/dns_ip_mapping_validation.rb

@@ -45,9 +45,19 @@ module Bosh::Cloudfoundry::Validations
 
     protected
     def resolve_dns(domain)
+      domain += "." if domain[-1] != "."
       packet = Net::DNS::Resolver.start(domain, Net::DNS::A)
-      resolved_a_records = packet.answer.map(&:value)
-      [(packet.answer.size > 0), resolved_a_records]
+      resolved_a_records = packet.answer.select { |record|
+        record.name == domain
+      }.map { |record|
+        case record.type
+        when "CNAME"
+          resolve_dns(record.value)[1]
+        when "A"
+          record.value
+        end
+      }.flatten
+      [(resolved_a_records.size > 0), resolved_a_records]
     end
   end
-end
+end

data/spec/deployment_attributes_spec.rb

@@ -38,14 +38,28 @@ describe Bosh::Cloudfoundry::DeploymentAttributes do
       subject.immutable_attributes.should == immutable_attributes
     end
 
-    # mutable_attributes ultimately determined by ReleaseVersion (from templates/vXYZ/spec)
-    it "mutable_attributes derived from release_version[_cpi]" do
-      mutable_attributes = %w[ip_addresses persistent_disk security_group].map(&:to_sym)
-      release_version_cpi.should_receive(:mutable_attributes).and_return(mutable_attributes)
-      subject = Bosh::Cloudfoundry::DeploymentAttributes.new(director, bosh_status, release_version_cpi, {
-        dns: "mycloud.com", ip_addresses: ["1.2.3.4"]
-      })
-      subject.mutable_attributes.should == mutable_attributes
+    context "mutable attributes" do
+      let(:mutable_attributes) { %w[ip_addresses persistent_disk security_group].map(&:to_sym) }
+      before { release_version_cpi.should_receive(:mutable_attributes).and_return(mutable_attributes) }
+
+      subject do
+        Bosh::Cloudfoundry::DeploymentAttributes.new(director, bosh_status, release_version_cpi, {
+          dns: "mycloud.com", ip_addresses: ["1.2.3.4"]
+        })
+      end
+
+      # mutable_attributes ultimately determined by ReleaseVersion (from templates/vXYZ/spec)
+      it "mutable_attributes derived from release_version[_cpi]" do
+        subject.mutable_attributes.should == mutable_attributes
+      end
+
+      it "mutable_attribute? is true for mutable attributes" do
+        subject.should be_mutable_attribute("persistent_disk")
+      end
+
+      it "mutable_attribute? is false for immutable attributes" do
+        subject.should_not be_mutable_attribute("name")
+      end
     end
   end
 end

data/spec/deployment_file_spec.rb

@@ -76,18 +76,18 @@ describe Bosh::Cloudfoundry::DeploymentFile do
         subject.deploy(non_interactive: true)
       end
     end
-  # 
-  # it "generates a large deployment" do
-  #   in_home_dir do
-  #     command.add_option(:deployment_size, "large")
-  # 
-  #     command.create_cf
-  #     files_match(spec_asset("v132/aws/large.yml"), command.deployment_file)
-  # 
-  #     manifest = YAML.load_file(command.deployment_file)
-  #     Bosh::Cli::DeploymentManifest.new(manifest).normalize
-  #   end
-  # end
+
+    # it "large size" do
+    #   in_home_dir do
+    #     command.add_option(:deployment_size, "large")
+    # 
+    #     command.create_cf
+    #     files_match(spec_asset("v133/aws/large.yml"), command.deployment_file)
+    # 
+    #     manifest = YAML.load_file(command.deployment_file)
+    #     Bosh::Cli::DeploymentManifest.new(manifest).normalize
+    #   end
+    # end
   # 
   # it "specifies core size" do
   #   in_home_dir do

data/spec/plugin_spec.rb

@@ -4,13 +4,44 @@ describe Bosh::Cli::Command::CloudFoundry do
   include FileUtils
 
   let(:command) { Bosh::Cli::Command::CloudFoundry.new }
+  let(:director) { instance_double("Bosh::Cli::Director") }
+
+  def setup_deployment
+    deployment_file = home_file("deployment.yml")
+    command.stub(:deployment).and_return(deployment_file)
+    File.open(deployment_file, "w") do |f|
+      f << {
+        "releases" => [
+          {"name" => "cf-release", "version" => 132}
+        ],
+        "properties" => {
+          "cf" => {
+            # immutable attributes (determined via ReleaseVersion via templates/vXYZ/spec)
+            "name" => "demo",
+            "deployment_size" => "medium",
+            "dns" => "mycloud.com",
+            "common_password" => "qwerty",
+            # mutable attributes (determined via ReleaseVersion via templates/vXYZ/spec)
+            "ip_addresses" => ["1.2.3.4"],
+            "persistent_disk" => 4096,
+            "security_group" => "cf"
+          }
+        }
+      }.to_yaml
+    end
+    deployment_file
+  end
 
   before(:all) do
     # Let us have pretty access to all protected methods which are protected from the bosh_cli plugin system.
     Bosh::Cli::Command::CloudFoundry.send(:public, *Bosh::Cli::Command::CloudFoundry.protected_instance_methods)
   end
 
-  before { setup_home_dir }
+  before do
+    setup_home_dir
+    command.add_option(:config, home_file(".bosh_config"))
+    command.add_option(:non_interactive, true)
+  end
 
   it "shows help" do
     subject.cf_help
@@ -18,11 +49,8 @@ describe Bosh::Cli::Command::CloudFoundry do
 
   context "prepare cf" do
     before do
-      command.add_option(:config, home_file(".bosh_config"))
-      command.add_option(:non_interactive, true)
       command.should_receive(:auth_required)
 
-      director = instance_double("Bosh::Cli::Director")
       director.should_receive(:get_status).and_return({"uuid" => "UUID", "cpi" => "aws"})
       command.stub(:director_client).and_return(director)
     end
@@ -51,7 +79,6 @@ describe Bosh::Cli::Command::CloudFoundry do
   context "create cf" do
     context "validation failures" do
       before do
-        director = instance_double("Bosh::Cli::Director")
         director.stub(:get_status).and_return({"uuid" => "UUID", "cpi" => "aws"})
         command.stub(:director_client).and_return(director)
       end
@@ -70,65 +97,75 @@ describe Bosh::Cli::Command::CloudFoundry do
 
     context "with requirements" do
       it "creates cf deployment" do
-        command.add_option(:config, home_file(".bosh_config"))
-        command.add_option(:non_interactive, true)
         command.add_option(:name, "demo")
         command.add_option(:ip, ["1.2.3.4"])
         command.add_option(:dns, "mycloud.com")
         command.add_option(:common_password, "qwertyasdfgh")
 
         command.should_receive(:auth_required)
-        command.should_receive(:validate_dns_mapping)
+        command.should_receive(:validate_deployment_attributes)
 
-        director = instance_double("Bosh::Cli::Director")
         director.should_receive(:get_status).and_return({"uuid" => "UUID", "cpi" => "aws"})
         command.stub(:director_client).and_return(director)
 
         command.stub(:deployment).and_return(home_file("deployments/cf/demo.yml"))
 
-        deployment_file = mock("deployment_file")
+        deployment_file = instance_double("Bosh::Cloudfoundry::DeploymentFile")
         Bosh::Cloudfoundry::DeploymentFile.should_receive(:new).
           and_return(deployment_file)
-        deployment_file.should_receive(:prepare_environment)
-        deployment_file.should_receive(:create_deployment_file)
-        deployment_file.should_receive(:deploy)
+        deployment_file.should_receive(:perform)
 
         command.create_cf
       end
 
     end
+  end
+  
+  context "existing deployment" do
+    before do
+      @deployment_file_path = setup_deployment
 
-    it "displays the list of attributes/properties" do
-      command.add_option(:config, home_file(".bosh_config"))
-      command.add_option(:non_interactive, true)
-
-      director = instance_double("Bosh::Cli::Director")
       director.should_receive(:get_status).and_return({"uuid" => "UUID", "cpi" => "aws"})
       command.stub(:director_client).and_return(director)
+    end
 
-      command.stub(:deployment).and_return(home_file("deployment.yml"))
-      File.open(home_file("deployment.yml"), "w") do |f|
-        f << {
-          "releases" => [
-            {"name" => "cf-release", "version" => 132}
-          ],
-          "properties" => {
-            "cf" => {
-              # immutable attributes (determined via ReleaseVersion via templates/vXYZ/spec)
-              "name" => "demo",
-              "deployment_size" => "medium",
-              "dns" => "mycloud.com",
-              "common_password" => "qwerty",
-              # mutable attributes (determined via ReleaseVersion via templates/vXYZ/spec)
-              "ip_addresses" => ["1.2.3.4"],
-              "persistent_disk" => 4096,
-              "security_group" => "cf"
-            }
-          }
-        }.to_yaml
+    it "displays the list of attributes/properties" do
+      command.show_cf_attributes
+    end
+
+    context "modifies attributes/properties and redeploys" do
+      let(:deployment_attributes) { instance_double("Bosh::Cloudfoundry::DeploymentAttributes")}
+      before do
+        deployment_file = instance_double("Bosh::Cloudfoundry::DeploymentFile")
+        Bosh::Cloudfoundry::DeploymentFile.should_receive(:new).
+          and_return(deployment_file)
+        deployment_file.should_receive(:deployment_attributes).and_return(deployment_attributes)
+        deployment_file.should_receive(:release_version_cpi_size)
+        deployment_file.should_receive(:perform)
+
+        command.should_receive(:validate_deployment_attributes)
+
+        deployment_attributes.stub(:mutable_attributes).and_return([:persistent_disk, :security_group])
+      end
+
+      it "for single property" do
+        deployment_attributes.should_receive(:validated_color).with("persistent_disk")
+        deployment_attributes.should_receive(:mutable_attribute?).with("persistent_disk").and_return(true)
+        deployment_attributes.should_receive(:set_mutable).with("persistent_disk", "8192")
+
+        command.change_cf_attributes("persistent_disk=8192")
+      end
+
+      it "for multiple properties" do
+        deployment_attributes.should_receive(:validated_color).with("persistent_disk")
+        deployment_attributes.should_receive(:validated_color).with("security_group")
+        deployment_attributes.should_receive(:mutable_attribute?).with("persistent_disk").and_return(true)
+        deployment_attributes.should_receive(:mutable_attribute?).with("security_group").and_return(true)
+        deployment_attributes.should_receive(:set_mutable).with("persistent_disk", "8192")
+        deployment_attributes.should_receive(:set_mutable).with("security_group", "cf-core")
+
+        command.change_cf_attributes("persistent_disk=8192", "security_group=cf-core")
       end
-      command.show_cf_properties
     end
   end
-
 end

data/templates/v132/aws/large/deployment_file.yml.erb

@@ -39,7 +39,7 @@ director_uuid: <%= find("director_uuid") %>
 
 releases:
  - name: cf-release
-   version: <%= find("releases.version") %>
+   version: 132
 
 networks:
 - name: floating

data/templates/v132/aws/medium/deployment_file.yml.erb

@@ -39,7 +39,7 @@ director_uuid: <%= find("director_uuid") %>
 
 releases:
  - name: cf-release
-   version: <%= find("releases.version") %>
+   version: 132
 
 networks:
 - name: floating

data/templates/v132/openstack/large/deployment_file.yml.erb

@@ -39,7 +39,7 @@ director_uuid: <%= find("director_uuid") %>
 
 releases:
  - name: cf-release
-   version: <%= find("releases.version") %>
+   version: 132
 
 networks:
 - name: floating

data/templates/v132/openstack/medium/deployment_file.yml.erb

@@ -39,7 +39,7 @@ director_uuid: <%= find("director_uuid") %>
 
 releases:
  - name: cf-release
-   version: <%= find("releases.version") %>
+   version: 132
 
 networks:
 - name: floating

data/templates/v133/aws/large/deployment_file.yml.erb

@@ -39,7 +39,7 @@ director_uuid: <%= find("director_uuid") %>
 
 releases:
  - name: cf-release
-   version: <%= find("releases.version") %>
+   version: 133
 
 networks:
 - name: floating

data/templates/v133/aws/medium/deployment_file.yml.erb

@@ -39,7 +39,7 @@ director_uuid: <%= find("director_uuid") %>
 
 releases:
  - name: cf-release
-   version: <%= find("releases.version") %>
+   version: 133
 
 networks:
 - name: floating

data/templates/v133/openstack/large/deployment_file.yml.erb

@@ -39,7 +39,7 @@ director_uuid: <%= find("director_uuid") %>
 
 releases:
  - name: cf-release
-   version: <%= find("releases.version") %>
+   version: 133
 
 networks:
 - name: floating

data/templates/v133/openstack/medium/deployment_file.yml.erb

@@ -39,7 +39,7 @@ director_uuid: <%= find("director_uuid") %>
 
 releases:
  - name: cf-release
-   version: <%= find("releases.version") %>
+   version: 133
 
 networks:
 - name: floating

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: bosh-cloudfoundry
 version: !ruby/object:Gem::Version
-  version: 0.7.0.alpha.7
+  version: 0.7.0.alpha.8
   prerelease: 6
 platform: ruby
 authors:
@@ -494,7 +494,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
       version: '0'
       segments:
       - 0
-      hash: -1355577384199561262
+      hash: -3134788032320103940
 required_rubygems_version: !ruby/object:Gem::Requirement
   none: false
   requirements: