stack_master 0.3.1 → 0.4.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: b8d7c06c818e3f34290f0cf825ff2edbf0345e7c
-  data.tar.gz: 570f49fdf0bb8c83729e3eb3e381a9a5ef1de805
+  metadata.gz: 134af119f4a1e7ac352e26c21921f3b1c31cdd4f
+  data.tar.gz: b2945a13aa2168a9f4b2c8c4ae60883fa10d0f49
 SHA512:
-  metadata.gz: 11cc834e25489e928352ee4aa92937b1e63ba12889644decce84b57f57c08dd6c22f777c2f69a4eeda78271bb0c7a655b12354e8d16ed0d2b696664dae36a440
-  data.tar.gz: ead584da13331f9714bac80c224aa2f9f59222cb87950a8181b227e334ce41adfbf04a865e568316bbeb06dc30e32df4ba90c37a61f1f6236bff60cac5bc75f9
+  metadata.gz: d2fb9a998fc5ac28093551d67b3fe5832ba3c0ce411812590070cff1a87552f1ff30d8d9ea30b82d2c94e40961598a0dc15ac911d1672484a2bc85d209a71a94
+  data.tar.gz: b6f3477bdaaa9f42d8a75367a55c1b40938e7550eb9ce37f2c1bea11fa093e57064732c2b5919b2a0d215981fd9a08706fd574418ac5a079aa1b844865b2f968

data/README.md

@@ -1,17 +1,29 @@
 ![StackMaster](/logo.png?raw=true)
 
-StackMaster is a sure-footed way of creating, updating and keeping track of
-Amazon (AWS) CloudFormation stacks.
-
-- See the changes you are making to a stack before you apply them
-- Connect stacks
-- Keep secrets secret
-- Customise stacks in different environments
-- Apply descriptive labels to regions
-
-StackMaster provides an easy command line interface to managing CloudFormation
-stacks defined with templates specified in either the
-[SparkleFormation](http://www.sparkleformation.io) DSL or standard JSON format.
+StackMaster is a CLI tool to manage CloudFormation stacks, with the following features:
+
+- Synchronous visibility into stack updates. See exactly what is changing and
+  what will happen before agreeing to apply a change.
+- Dynamic parameter resolvers.
+- Template compiler support for YAML and [SparkleFormation](http://www.sparkleformation.io).
+
+Stack updates can cause a lot of damage if applied blindly. StackMaster helps
+with this by providing the operator with as much information about the proposed
+change as possible before asking for confirmation to continue. That information
+includes:
+
+- Template body and parameter diffs.
+- [Change
+  sets](https://aws.amazon.com/blogs/aws/new-change-sets-for-aws-cloudformation/)
+are displayed for review.
+- Once the diffs & change set have been reviewed, the change can be applied and
+  stack events monitored.
+- Stack events will be displayed until an end state is reached.
+
+Stack parameters can be dynamically resolved at runtime using one of the
+built in parameter resolvers. Parameters can be sourced from GPG encrypted YAML
+files, other stacks outputs, by querying various AWS API's to get resource ARNs
+etc.
 
 ## Installation
 
@@ -71,18 +83,34 @@ stacks:
 
 ## Directories
 
-- `templates` - CloudFormation or SparkleFormation templates.
+- `templates` - CloudFormation, SparkleFormation or CfnDsl templates.
 - `polices` - Stack policies.
 - `parameters` - Parameters as YAML files.
 - `secrets` - GPG encrypted secret files.
+- `policies` - Stack policy JSON files.
+
+## Templates
+
+StackMaster supports CloudFormation templates in plain JSON or YAML. Any `.yml` or `.yaml` file will be processed as
+YAML. While any `.json` file will be processed as JSON.
+
+### Ruby DSLs
+By default, any template ending with `.rb` will be processed as a [SparkleFormation](https://github.com/sparkleformation/sparkle_formation)
+template. However, if you want to use [CfnDsl](https://github.com/stevenjack/cfndsl) templates you can add
+the following lines to your `stack_master.yml`.
+
+```
+template_compilers:
+  rb: cfndsl
+```
 
 ## Parameters
 
 Parameters are loaded from multiple YAML files, merged from the following lookup paths:
 
 - parameters/[stack_name].yml
-- parameters/[region]/[stack_name].yml
-- parameters/[region_alias]/[stack_name].yml
+- parameters/[region]/[underscored_stack_name].yml
+- parameters/[region_alias]/[underscored_stack_name].yml
 
 A simple parameter file could look like this:
 
@@ -94,13 +122,11 @@ Keys in parameter files are automatically converted to camel case.
 
 ## Parameter Resolvers
 
-Parameter resolvers enable dynamic resolution of parameter values. A parameter
-using a resolver will be a hash with one key where the key is the name of the
-resolver.
+Parameter values can be sourced dynamically using parameter resolvers.
 
-One benefit of using resolvers instead of hard coding values like VPC ID's and
-resource ARNs is that the same configuration works cross region, even though
-the resolved values will be different.
+One benefit of using parameter resolvers instead of hard coding values like VPC
+ID's and resource ARNs is that the same configuration works cross
+region/account, even though the resolved values will be different.
 
 ### Stack Output
 
@@ -112,6 +138,10 @@ vpc_id:
   stack_output: my-vpc-stack/VpcId
 ```
 
+This is the most used parameter resolver because it enables stacks to be split
+up into their separated concerns (VPC, web, database etc) with outputs feeding
+into parameters of dependent stacks.
+
 ### Secret
 
 The secret parameters resolver expects a `secret_file` to be defined in the
@@ -269,11 +299,13 @@ stack_master status # Displays the status of each stack
 
 The apply command does the following:
 
-- Builds the proposed stack json and resolves parameters.
+- Compiles the proposed stack template and resolves parameters.
 - Fetches the current state of the stack from CloudFormation.
 - Displays a diff of the current stack and the proposed stack.
+- Creates a change set and displays the actions that CloudFormation will take
+  to perform the update (if the stack already exists).
 - Asks if the update should continue.
-- If yes, the API call is made to update or create the stack.
+- If yes, the API calls are made to update or create the stack.
 - Stack events are displayed until CloudFormation has finished applying the changes.
 
 Demo:

data/lib/stack_master.rb

@@ -40,6 +40,8 @@ require "stack_master/stack_definition"
 require "stack_master/template_compiler"
 require "stack_master/template_compilers/sparkle_formation"
 require "stack_master/template_compilers/json"
+require "stack_master/template_compilers/yaml"
+require "stack_master/template_compilers/cfndsl"
 require "stack_master/commands/terminal_helper"
 require "stack_master/commands/apply"
 require "stack_master/change_set"

data/lib/stack_master/command.rb

@@ -22,8 +22,8 @@ module StackMaster
         catch(:halt) do
           super
         end
-      rescue Aws::CloudFormation::Errors::ServiceError => e
-        failed "#{e.class} #{e.message}"
+      rescue Aws::CloudFormation::Errors::ServiceError, TemplateCompiler::TemplateCompilationFailed => e
+        failed error_message(e)
       end
     end
 
@@ -33,6 +33,12 @@ module StackMaster
 
     private
 
+    def error_message(e)
+      msg = "#{e.class} #{e.message}"
+      msg << "\n Caused by: #{e.cause.class} #{e.cause.message}" if e.cause
+      msg
+    end
+
     def failed(message = nil)
       StackMaster.stderr.puts(message) if message
       @failed = true

data/lib/stack_master/config.rb

@@ -42,7 +42,7 @@ module StackMaster
       end
       @region_defaults = normalise_region_defaults(config.fetch('region_defaults', {}))
       @stacks = []
-      @template_compilers = default_template_compilers
+      load_template_compilers(config)
       load_config
     end
 
@@ -62,11 +62,28 @@ module StackMaster
     end
 
     private
+    def load_template_compilers(config)
+      @template_compilers = {}
+      populate_template_compilers(config.fetch('template_compilers', {}))
+      merge_defaults_to_user_defined_compilers
+    end
+
+    def merge_defaults_to_user_defined_compilers
+      @template_compilers = default_template_compilers.merge(@template_compilers)
+    end
+
+    def populate_template_compilers user_defined_compilers
+      user_defined_compilers.each do |key, val|
+        @template_compilers[key.to_sym] = val.to_sym
+      end
+    end
 
     def default_template_compilers
       {
         rb: :sparkle_formation,
         json: :json,
+        yml:  :yaml,
+        yaml: :yaml,
       }
     end
 

data/lib/stack_master/parameter_resolvers/secret.rb

@@ -26,6 +26,7 @@ module StackMaster
       end
 
       def decrypt_with_dotgpg
+        Dotgpg.interactive = true
         dir = Dotgpg::Dir.closest(secret_file_path)
         stream = StringIO.new
         dir.decrypt(secret_path_relative_to_base, stream)

data/lib/stack_master/template_compiler.rb

@@ -1,8 +1,11 @@
 module StackMaster
   class TemplateCompiler
+    TemplateCompilationFailed = Class.new(RuntimeError)
 
     def self.compile(config, template_file_path)
       template_compiler_for_file(template_file_path, config).compile(template_file_path)
+    rescue
+      raise TemplateCompilationFailed.new("Failed to compile #{template_file_path}.")
     end
 
     def self.register(name, klass)

data/lib/stack_master/template_compilers/cfndsl.rb

@@ -0,0 +1,11 @@
+module StackMaster::TemplateCompilers
+  class Cfndsl
+
+    def self.compile(template_file_path)
+      require 'cfndsl'
+      CfnDsl.eval_file_with_extras(template_file_path).to_json
+    end
+
+    StackMaster::TemplateCompiler.register(:cfndsl, self)
+  end
+end

data/lib/stack_master/template_compilers/yaml.rb

@@ -0,0 +1,11 @@
+module StackMaster::TemplateCompilers
+  class Yaml
+
+    def self.compile(template_file_path)
+      template_body = File.read(template_file_path)
+      JSON.dump(YAML.load(template_body))
+    end
+
+    StackMaster::TemplateCompiler.register(:yaml, self)
+  end
+end

data/lib/stack_master/version.rb

@@ -1,3 +1,3 @@
 module StackMaster
-  VERSION = "0.3.1"
+  VERSION = "0.4.1"
 end

data/spec/fixtures/stack_master.yml

@@ -4,6 +4,8 @@ region_aliases:
 stack_defaults:
   tags:
     application: my-awesome-blog
+template_compilers:
+  rb: ruby_dsl
 region_defaults:
   us_east_1:
     tags:

data/spec/fixtures/templates/json/valid_myapp_vpc.json

@@ -0,0 +1,53 @@
+{
+  "Description": "A test VPC template",
+  "Resources": {
+    "Vpc": {
+      "Type": "AWS::EC2::VPC",
+      "Properties": {
+        "CidrBlock": "10.200.0.0/16"
+      }
+    },
+    "PublicSubnet": {
+      "Type": "AWS::EC2::Subnet",
+      "Properties": {
+        "VpcId": {
+          "Ref": "Vpc"
+        },
+        "CidrBlock": "10.200.1.0/24",
+        "AvailabilityZone": {
+          "Ref": "VpcAz1"
+        },
+        "Tags": [
+          {
+            "Key": "Name",
+            "Value": "PublicSubnet"
+          },
+          {
+            "Key": "network",
+            "Value": "public"
+          }
+        ]
+      }
+    }
+  },
+  "Parameters": {
+    "VpcAz1": {
+      "Description": "VPC AZ 1",
+      "Type": "AWS::EC2::AvailabilityZone::Name"
+    }
+  },
+  "Outputs": {
+    "VpcId": {
+      "Description": "VPC ID",
+      "Value": {
+        "Ref": "Vpc"
+      }
+    },
+    "PublicSubnet": {
+      "Description": "Public subnet",
+      "Value": {
+        "Ref": "PublicSubnet"
+      }
+    }
+  }
+}

data/spec/fixtures/templates/rb/cfndsl/sample.json

@@ -0,0 +1,28 @@
+{
+  "Parameters" : {
+    "One" : {
+      "Type" : "String",
+      "Default" : "Test",
+      "MaxLength" : 15
+    }
+  },
+  "Resources" : {
+    "MyInstance" : {
+      "Type" : "AWS::EC2::Instance",
+      "Properties" : {
+        "ImageId" : "ami-12345678"
+      }
+    }
+  },
+  "AWSTemplateFormatVersion" : "2010-09-09",
+  "Outputs" : {
+    "One" : {
+      "Value" : {
+        "Fn::Base64" : {
+          "Ref" : "One"
+        }
+      }
+    }
+  },
+  "Description" : "Test"
+}

data/spec/fixtures/templates/rb/cfndsl/sample.rb

@@ -0,0 +1,16 @@
+CloudFormation {
+  Description "Test"
+
+  Parameter("One") {
+    String
+    Default "Test"
+    MaxLength 15
+  }
+
+  Output(:One,FnBase64( Ref("One")))
+
+  EC2_Instance(:MyInstance) {
+    ImageId "ami-12345678"
+  }
+
+}

data/spec/fixtures/templates/yml/valid_myapp_vpc.yml

@@ -0,0 +1,35 @@
+---
+  Description: "A test VPC template"
+  Resources:
+    Vpc:
+      Type: "AWS::EC2::VPC"
+      Properties:
+        CidrBlock: "10.200.0.0/16"
+    PublicSubnet:
+      Type: "AWS::EC2::Subnet"
+      Properties:
+        VpcId:
+          Ref: "Vpc"
+        CidrBlock: "10.200.1.0/24"
+        AvailabilityZone:
+          Ref: "VpcAz1"
+        Tags:
+          -
+            Key: "Name"
+            Value: "PublicSubnet"
+          -
+            Key: "network"
+            Value: "public"
+  Parameters:
+    VpcAz1:
+      Description: "VPC AZ 1"
+      Type: "AWS::EC2::AvailabilityZone::Name"
+  Outputs:
+    VpcId:
+      Description: "VPC ID"
+      Value:
+        Ref: "Vpc"
+    PublicSubnet:
+      Description: "Public subnet"
+      Value:
+        Ref: "PublicSubnet"

data/spec/stack_master/command_spec.rb

@@ -45,4 +45,22 @@ RSpec.describe StackMaster::Command do
       expect { command_class.perform(error_proc) }.to output(/the message/).to_stderr
     end
   end
+
+  context 'when a template compilation error occurs' do
+    it 'outputs the message' do
+      error_proc = proc {
+        raise StackMaster::TemplateCompiler::TemplateCompilationFailed.new('the message')
+      }
+      expect { command_class.perform(error_proc) }.to output(/the message/).to_stderr
+    end
+
+    it 'outputs the exception\'s cause' do
+      exception_with_cause = StackMaster::TemplateCompiler::TemplateCompilationFailed.new('the message')
+      allow(exception_with_cause).to receive(:cause).and_return(RuntimeError.new('the cause message'))
+      error_proc = proc {
+        raise exception_with_cause
+      }
+      expect { command_class.perform(error_proc) }.to output(/Caused by: RuntimeError the cause message/).to_stderr
+    end
+  end
 end

data/spec/stack_master/config_spec.rb

@@ -15,7 +15,7 @@ RSpec.describe StackMaster::Config do
     )
   }
 
-  context ".load!" do
+  describe ".load!" do
     it "fails to load the config if no stack_master.yml in parent directories" do
       expect { StackMaster::Config.load!('stack_master.yml') }.to raise_error Errno::ENOENT
     end
@@ -70,6 +70,16 @@ RSpec.describe StackMaster::Config do
     })
   end
 
+  it 'loads template compiler mappings' do
+    expect(loaded_config.template_compilers).to eq({
+                                                     rb: :ruby_dsl,
+                                                     json: :json,
+                                                     yml: :yaml,
+                                                     yaml: :yaml,
+
+                                                   })
+  end
+
   it 'loads region defaults' do
     expect(loaded_config.region_defaults).to eq({
       'us-east-1' => {

data/spec/stack_master/template_compiler_spec.rb

@@ -16,6 +16,16 @@ RSpec.describe StackMaster::TemplateCompiler do
         expect(TestTemplateCompiler).to receive(:compile).with(template_file_path)
         StackMaster::TemplateCompiler.compile(config, template_file_path)
       end
+
+      context 'when template compilation fails' do
+        before { allow(TestTemplateCompiler).to receive(:compile).and_raise(RuntimeError) }
+
+        it 'raise TemplateCompilationFailed exception' do
+          expect{ StackMaster::TemplateCompiler.compile(config, template_file_path)
+          }.to raise_error(
+                 StackMaster::TemplateCompiler::TemplateCompilationFailed,"Failed to compile #{template_file_path}.")
+        end
+      end
     end
   end
 end

data/spec/stack_master/template_compilers/cfndsl_spec.rb

@@ -0,0 +1,17 @@
+RSpec.describe StackMaster::TemplateCompilers::Cfndsl do
+  describe '.compile' do
+    def compile
+      described_class.compile(template_file_path)
+    end
+
+    context 'valid cfndsl template' do
+      let(:template_file_path) { 'spec/fixtures/templates/rb/cfndsl/sample.rb' }
+      let(:valid_compiled_json_path) { 'spec/fixtures/templates/rb/cfndsl/sample.json' }
+
+      it 'produces valid JSON' do
+        valid_compiled_json = File.read(valid_compiled_json_path)
+        expect(JSON.parse(compile)).to eq(JSON.parse(valid_compiled_json))
+      end
+    end
+  end
+end

data/spec/stack_master/template_compilers/yaml_spec.rb

@@ -0,0 +1,18 @@
+RSpec.describe StackMaster::TemplateCompilers::Yaml do
+  describe '.compile' do
+    def compile
+      described_class.compile(template_file_path)
+    end
+
+    context 'valid YAML template' do
+      let(:template_file_path) { 'spec/fixtures/templates/yml/valid_myapp_vpc.yml' }
+
+      it 'produces valid JSON' do
+        valid_myapp_vpc_as_json = File.read('spec/fixtures/templates/json/valid_myapp_vpc.json')
+        valid_myapp_vpc_as_hash = JSON.parse(valid_myapp_vpc_as_json)
+
+        expect(JSON.parse(compile)).to eq(valid_myapp_vpc_as_hash)
+      end
+    end
+  end
+end

data/stack_master.gemspec

@@ -36,4 +36,5 @@ Gem::Specification.new do |spec|
   spec.add_dependency "table_print"
   spec.add_dependency "dotgpg"
   spec.add_dependency "deep_merge"
+  spec.add_dependency "cfndsl"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: stack_master
 version: !ruby/object:Gem::Version
-  version: 0.3.1
+  version: 0.4.1
 platform: ruby
 authors:
 - Steve Hodgkiss
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2016-04-01 00:00:00.000000000 Z
+date: 2016-04-07 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
@@ -263,6 +263,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: cfndsl
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 description: ''
 email:
 - steve@hodgkiss.me
@@ -339,8 +353,10 @@ files:
 - lib/stack_master/stack_states.rb
 - lib/stack_master/stack_status.rb
 - lib/stack_master/template_compiler.rb
+- lib/stack_master/template_compilers/cfndsl.rb
 - lib/stack_master/template_compilers/json.rb
 - lib/stack_master/template_compilers/sparkle_formation.rb
+- lib/stack_master/template_compilers/yaml.rb
 - lib/stack_master/test_driver/cloud_formation.rb
 - lib/stack_master/testing.rb
 - lib/stack_master/utils.rb
@@ -352,7 +368,11 @@ files:
 - script/buildkite_rspec.sh
 - spec/fixtures/parameters/myapp_vpc.yml
 - spec/fixtures/stack_master.yml
+- spec/fixtures/templates/json/valid_myapp_vpc.json
 - spec/fixtures/templates/myapp_vpc.json
+- spec/fixtures/templates/rb/cfndsl/sample.json
+- spec/fixtures/templates/rb/cfndsl/sample.rb
+- spec/fixtures/templates/yml/valid_myapp_vpc.yml
 - spec/spec_helper.rb
 - spec/stack_master/change_set_spec.rb
 - spec/stack_master/command_spec.rb
@@ -382,8 +402,10 @@ files:
 - spec/stack_master/stack_events/streamer_spec.rb
 - spec/stack_master/stack_spec.rb
 - spec/stack_master/template_compiler_spec.rb
+- spec/stack_master/template_compilers/cfndsl_spec.rb
 - spec/stack_master/template_compilers/json_spec.rb
 - spec/stack_master/template_compilers/sparkle_formation_spec.rb
+- spec/stack_master/template_compilers/yaml_spec.rb
 - spec/stack_master/test_driver/cloud_formation_spec.rb
 - spec/stack_master/utils_spec.rb
 - spec/stack_master/validator_spec.rb
@@ -413,7 +435,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 2.5.1
+rubygems_version: 2.4.5.1
 signing_key: 
 specification_version: 4
 summary: StackMaster is a sure-footed way of creating, updating and keeping track
@@ -434,7 +456,11 @@ test_files:
 - features/validate.feature
 - spec/fixtures/parameters/myapp_vpc.yml
 - spec/fixtures/stack_master.yml
+- spec/fixtures/templates/json/valid_myapp_vpc.json
 - spec/fixtures/templates/myapp_vpc.json
+- spec/fixtures/templates/rb/cfndsl/sample.json
+- spec/fixtures/templates/rb/cfndsl/sample.rb
+- spec/fixtures/templates/yml/valid_myapp_vpc.yml
 - spec/spec_helper.rb
 - spec/stack_master/change_set_spec.rb
 - spec/stack_master/command_spec.rb
@@ -464,8 +490,10 @@ test_files:
 - spec/stack_master/stack_events/streamer_spec.rb
 - spec/stack_master/stack_spec.rb
 - spec/stack_master/template_compiler_spec.rb
+- spec/stack_master/template_compilers/cfndsl_spec.rb
 - spec/stack_master/template_compilers/json_spec.rb
 - spec/stack_master/template_compilers/sparkle_formation_spec.rb
+- spec/stack_master/template_compilers/yaml_spec.rb
 - spec/stack_master/test_driver/cloud_formation_spec.rb
 - spec/stack_master/utils_spec.rb
 - spec/stack_master/validator_spec.rb