stack_master 0.0.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: be7dbbbfb5ac18598037c53174655253fef33f73
+  data.tar.gz: e1b2cb9dd3846993f497dc51a606053fc7670907
+SHA512:
+  metadata.gz: 5fea5f38e52302091b2ab880cb59ca68ad9e60547a8f09b7eceb52611aa23475b4c3131fa62b9074580264315f7a75c0796c9fe9bebb1fd535c63c11ebe7d130
+  data.tar.gz: 4442a6772b1a8efd992660b9ba720f73208935a1bb0f5760a5e4b2b8ea5d4a1c3675964f4f4c3288d968bf45d3c752c4b05fba49f8487894f995e8cfc4537dbb

data/.gitignore

@@ -0,0 +1,18 @@
+*.gem
+*.rbc
+.bundle
+.config
+.yardoc
+Gemfile.lock
+InstalledFiles
+_yardoc
+coverage
+doc/
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp
+/spec/examples.txt

data/.rspec

@@ -0,0 +1,2 @@
+--color
+--require spec_helper

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in stack_master.gemspec
+gemspec

data/LICENSE.txt

@@ -0,0 +1,22 @@
+Copyright (c) 2015 Steve Hodgkiss
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,208 @@
+![StackMaster](/logo.png?raw=true)
+
+StackMaster is a sure-footed way of creating, updating and keeping track of
+Amazon (AWS) CloudFormation stacks.
+
+- See the changes you are making to a stack before you apply them
+- Connect stacks
+- Keep secrets secret
+- Customise stacks in different environments
+- Apply descriptive labels to regions
+
+StackMaster provides an easy command line interface to managing CloudFormation
+stacks defined with templates specified in either the
+[SparkleFormation](http://www.sparkleformation.io) DSL or standard JSON format.
+
+## Installation
+
+System-wide: `gem install stack_master`
+
+With bundler:
+
+- Add `gem 'stack_master'` to your Gemfile.
+- Run `bundle install`
+- Run `bundle exec stack_master init` to generate a directory structure and stack_master.yml file
+
+## Configuration
+
+Stacks are defined inside a `stack_master.yml` YAML file. When running
+`stack_master`, it is assumed that this file will exist in the current working
+directory, or that the file is passed in with `--config
+/path/to/stack_master.yml`.  Here's an example configuration file:
+
+```
+region_aliases:
+  production: us-east-1
+  staging: ap-southeast-2
+stack_defaults:
+  tags:
+    application: my-awesome-app
+region_defaults:
+  us-east-1:
+    secret_file: production.yml.gpg
+    tags:
+      environment: production
+    notification_arns:
+      - test_arn
+  ap-southeast-2:
+    secret_file: staging.yml.gpg
+    tags:
+      environment: staging
+stacks:
+  production:
+    myapp-vpc:
+      template: myapp_vpc.rb
+    myapp-db:
+      template: myapp_db.rb
+      stack_policy_file: db_stack_policy.json
+    myapp-web:
+      template: myapp_web.rb
+  staging:
+    myapp-vpc:
+      template: myapp_vpc.rb
+    myapp-db:
+      template: myapp_db.rb
+    myapp-web:
+      template: myapp_web.rb
+  eu-central-1:
+    myapp-vpc:
+      template: myapp_vpc.rb
+```
+
+## Directories
+
+- `templates` - CloudFormation or SparkleFormation templates.
+- `polices` - Stack policies.
+- `parameters` - Parameters as YAML files.
+- `secrets` - GPG encrypted secret files.
+
+## Parameters
+
+Parameters are loaded from multiple YAML files, merged from the following lookup paths:
+
+- parameters/[stack_name].yml
+- parameters/[region]/[stack_name].yml
+- parameters/[region_alias]/[stack_name].yml
+
+A simple parameter file could look like this:
+
+```
+key_name: myapp-us-east-1
+```
+
+Keys in parameter files are automatically converted to camel case.
+
+## Parameter Resolvers
+
+Parameter resolvers enable dynamic resolution of parameter values. A parameter
+using a resolver will be a hash with one key where the key is the name of the
+resolver.
+
+One benefit of using resolvers instead of hard coding values like VPC ID's and
+resource ARNs is that the same configuration works cross region, even though
+the resolved values will be different.
+
+### Stack Output
+
+The stack output parameter resolver looks up outputs from other stacks in the
+same region. The expected format is `[stack-name]/[OutputName]`.
+
+```yaml
+vpc_id:
+  stack_output: my-vpc-stack/VpcId
+```
+
+### Secret
+
+The secret parameters resolver expects a `secret_file` to be defined in the
+stack definition which is a GPG encrypted YAML file. Once decrypted and parsed,
+the value provided to the secret resolver is used to lookup the associated key
+in the secret file. A common use case for this is to store database passwords.
+
+stack_master.yml:
+
+```yaml
+stacks:
+  us-east-1:
+    my_app:
+      template: my_app.json
+      secret_file: production.yml.gpg
+```
+
+secrets/production.yml.gpg, when decrypted:
+
+```yaml
+db_password: my-password
+```
+
+parameters/my_app.yml:
+
+```yaml
+db_password:
+  secret: db_password
+```
+
+### Security Group
+
+Looks up a security group by name and returns the ARN.
+
+```yaml
+ssh_sg:
+  security_group: SSHSecurityGroup
+```
+
+### SNS Topic
+
+Looks up an SNS topic by name and returns the ARN.
+
+```yaml
+notification_topic:
+  sns_topic: PagerDuty
+```
+
+## Commands
+
+```bash
+stack_master help # Display up to date docs on the commands available
+stack_master init # Initialises a directory structure and stack_master.yml file
+stack_master list # Lists stack definitions
+stack_master apply [region-or-alias] [stack-name] # Create or update a stack
+stack_master diff [region-or-alias] [stack-name] # Display a stack tempalte and parameter diff
+stack_master delete [region-or-alias] [stack-name] # Delete a stack
+stack_master events [region-or-alias] [stack-name] # Display events for a stack
+stack_master outputs [region-or-alias] [stack-name] # Display outputs for a stack
+stack_master resources [region-or-alias] [stack-name] # Display outputs for a stack
+stack_master status # Displays the status of each stacks
+```
+
+## Applying updates
+
+The apply command does the following:
+
+- Builds the proposed stack json and resolves parameters.
+- Fetches the current state of the stack from CloudFormation.
+- Displays a diff of the current stack and the proposed stack.
+- Asks if the update should continue.
+- If yes, the API call is made to update or create the stack.
+- Stack events are displayed until CloudFormation has finished applying the changes.
+
+Demo:
+
+![Apply Demo](/apply_demo.gif?raw=true)
+
+## Maintainers
+
+- [Steve Hodgkiss](https://github.com/stevehodgkiss)
+- [Glen Stampoultzis](https://github.com/gstamp)
+
+## License
+
+StackMaster uses the MIT license. See [LICENSE.txt](https://github.com/envato/stack_master/blob/master/LICENSE.txt) for details.
+
+## Contributing
+
+1. Fork it ( http://github.com/envato/stack_master/fork )
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create new Pull Request

data/Rakefile

@@ -0,0 +1,11 @@
+require "bundler/gem_tasks"
+require 'bundler/setup'
+
+task :environment do
+  require 'stack_master'
+end
+
+task :console => :environment do
+  require 'pry'
+  binding.pry
+end

data/bin/stack_master

@@ -0,0 +1,16 @@
+#!/usr/bin/env ruby
+
+require 'rubygems'
+require 'stack_master'
+
+if ENV['STUB_AWS'] == 'true'
+  require 'stack_master/testing'
+end
+
+trap("SIGINT") { raise StackMaster::CtrlC }
+
+begin
+  StackMaster::CLI.new(ARGV.dup).execute!
+rescue StackMaster::CtrlC
+  StackMaster.stdout.puts "Exiting..."
+end

data/example/simple/Gemfile

@@ -0,0 +1,3 @@
+source 'https://rubygems.org'
+
+gem 'stack_master', path: '../../'

data/example/simple/parameters/myapp_vpc.yml

@@ -0,0 +1 @@
+vpc_az_1: ap-southeast-2a

data/example/simple/parameters/myapp_web.yml

@@ -0,0 +1,2 @@
+VpcId:
+  stack_output: myapp-vpc/VpcId

data/example/simple/stack_master.yml

@@ -0,0 +1,13 @@
+stack_defaults:
+  tags:
+    application: my-awesome-app
+region_defaults:
+  ap_southeast_2:
+    tags:
+      environment: production
+stacks:
+  ap_southeast_2:
+    myapp_vpc:
+      template: myapp_vpc.rb
+    myapp_web:
+      template: myapp_web.rb

data/example/simple/templates/myapp_vpc.rb

@@ -0,0 +1,39 @@
+SparkleFormation.new(:myapp_vpc) do
+  description "A test VPC template"
+
+  resources.vpc do
+    type 'AWS::EC2::VPC'
+    properties do
+      cidr_block '10.200.0.0/16'
+    end
+  end
+
+  parameters.vpc_az_1 do
+    description 'VPC AZ 1'
+    type 'AWS::EC2::AvailabilityZone::Name'
+  end
+
+  resources.public_subnet do
+    type 'AWS::EC2::Subnet'
+    properties do
+      vpc_id ref!(:vpc)
+      cidr_block '10.200.1.0/24'
+      availability_zone ref!(:vpc_az_1)
+      tags _array(
+        { Key: 'Name', Value: 'PublicSubnet' },
+        { Key: 'network', Value: 'public' }
+      )
+    end
+  end
+
+  outputs do
+    vpc_id do
+      description 'VPC ID'
+      value ref!(:vpc)
+    end
+    public_subnet do
+      description 'Public subnet'
+      value ref!(:public_subnet)
+    end
+  end
+end

data/example/simple/templates/myapp_web.rb

@@ -0,0 +1,16 @@
+SparkleFormation.new(:myapp_web) do
+  description "Test web template"
+
+  parameters.vpc_id do
+    description 'VPC ID'
+    type 'String'
+  end
+
+  resources.web_sg do
+    type 'AWS::EC2::SecurityGroup'
+    properties do
+      group_description 'Security group for web instances'
+      vpc_id ref!(:vpc_id)
+    end
+  end
+end

data/features/apply.feature

@@ -0,0 +1,241 @@
+Feature: Apply command
+
+  Background:
+    Given a file named "stack_master.yml" with:
+      """
+      stacks:
+        us_east_1:
+          myapp_vpc:
+            template: myapp_vpc.rb
+          myapp_web:
+            template: myapp_web.rb
+      """
+    And a directory named "parameters"
+    And a file named "parameters/myapp_vpc.yml" with:
+      """
+      KeyName: my-key
+      """
+    And a directory named "templates"
+    And a file named "templates/myapp_vpc.rb" with:
+      """
+      SparkleFormation.new(:myapp_vpc) do
+        description "Test template"
+        set!('AWSTemplateFormatVersion', '2010-09-09')
+
+        parameters.key_name do
+          description 'Key name'
+          type 'String'
+        end
+
+        resources.vpc do
+          type 'AWS::EC2::VPC'
+          properties do
+            cidr_block '10.200.0.0/16'
+          end
+        end
+
+        outputs do
+          vpc_id do
+            description 'A VPC ID'
+            value ref!(:vpc)
+          end
+        end
+      end
+      """
+    And a file named "templates/myapp_web.rb" with:
+      """
+      SparkleFormation.new(:myapp_web) do
+        description "Test template"
+        set!('AWSTemplateFormatVersion', '2010-09-09')
+
+        parameters.vpc_id do
+          description 'VPC ID'
+          type 'AWS::EC2::VPC::Id'
+        end
+
+        resources.test_sg do
+          type 'AWS::EC2::SecurityGroup'
+          properties do
+            group_description 'Test SG'
+            vpc_id ref!(:vpc_id)
+          end
+        end
+      end
+      """
+    And I set the environment variables to:
+      | variable | value |
+      | STUB_AWS | true  |
+
+  Scenario: Run apply and create a new stack
+    Given I set the environment variables to:
+      | variable | value |
+      | ANSWER   | y     |
+    And I stub the following stack events:
+      | stack_id | event_id | stack_name | logical_resource_id | resource_status | resource_type              | timestamp           |
+      | 1        | 1        | myapp-vpc  | TestSg              | CREATE_COMPLETE | AWS::EC2::SecurityGroup    | 2020-10-29 00:00:00 |
+      | 1        | 1        | myapp-vpc  | myapp-vpc           | CREATE_COMPLETE | AWS::CloudFormation::Stack | 2020-10-29 00:00:00 |
+    When I run `stack_master apply us-east-1 myapp-vpc --trace` interactively
+    And the output should contain all of these lines:
+      | Stack diff:                                                                    |
+      | +    "Vpc": {                                                                  |
+      | Parameters diff:                                                               |
+      | KeyName: my-key                                                                |
+      | 2020-10-29 00:00:00 +1100 myapp-vpc AWS::CloudFormation::Stack CREATE_COMPLETE |
+    Then the exit status should be 0
+
+  Scenario: Run apply and don't create the stack
+    Given I set the environment variables to:
+      | variable | value |
+      | ANSWER   | n     |
+    When I run `stack_master apply us-east-1 myapp-vpc --trace` interactively
+    And the output should contain all of these lines:
+      | Stack diff:      |
+      | +    "Vpc": {    |
+      | Parameters diff: |
+      | KeyName: my-key  |
+      | aborted          |
+    And the output should not contain all of these lines:
+      | 2020-10-29 00:00:00 +1100 myapp-vpc AWS::CloudFormation::Stack CREATE_COMPLETE |
+    Then the exit status should be 0
+
+  Scenario: Run apply on an existing stack
+    Given I set the environment variables to:
+      | variable | value |
+      | ANSWER   | y     |
+    And I stub the following stack events:
+      | stack_id | event_id | stack_name | logical_resource_id | resource_status | resource_type              | timestamp           |
+      | 1        | 1        | myapp-vpc  | TestSg              | CREATE_COMPLETE | AWS::EC2::SecurityGroup    | 2020-10-29 00:00:00 |
+      | 1        | 1        | myapp-vpc  | myapp-vpc           | CREATE_COMPLETE | AWS::CloudFormation::Stack | 2020-10-29 00:00:00 |
+    And I stub the following stacks:
+      | stack_id | stack_name | parameters       | region    |
+      | 1        | myapp-vpc  | KeyName=my-key   | us-east-1 |
+    And I stub a template for the stack "myapp-vpc":
+      """
+      {
+        "Description": "Test template",
+        "AWSTemplateFormatVersion": "2010-09-09",
+        "Parameters": {
+          "KeyName": {
+            "Description": "Key Name",
+            "Type": "String"
+          }
+        },
+        "Resources": {
+          "TestSg": {
+            "Type": "AWS::EC2::SecurityGroup",
+            "Properties": {
+              "GroupDescription": "Test SG",
+              "VpcId": {
+                "Ref": "VpcId"
+              }
+            }
+          },
+          "TestSg2": {
+            "Type": "AWS::EC2::SecurityGroup",
+            "Properties": {
+              "GroupDescription": "Test SG 2",
+              "VpcId": {
+                "Ref": "VpcId"
+              }
+            }
+          }
+        }
+      }
+      """
+    When I run `stack_master apply us-east-1 myapp-vpc --trace` interactively
+    And the output should contain all of these lines:
+      | Stack diff:                                                                    |
+      | -    "TestSg2": {                                                              |
+      | Parameters diff: No changes                                                    |
+    Then the exit status should be 0
+
+  Scenario: Create a stack using a stack output resolver
+    Given I set the environment variables to:
+      | variable | value |
+      | ANSWER   | y     |
+    And a file named "parameters/myapp_web.yml" with:
+      """
+      VpcId:
+        stack_output: myapp-vpc/VpcId
+      """
+    And I stub the following stack events:
+      | stack_id | event_id | stack_name | logical_resource_id | resource_status | resource_type              | timestamp           |
+      | 1        | 1        | myapp-web  | TestSg              | CREATE_COMPLETE | AWS::EC2::SecurityGroup    | 2020-10-29 00:00:00 |
+      | 1        | 1        | myapp-web  | myapp-web           | CREATE_COMPLETE | AWS::CloudFormation::Stack | 2020-10-29 00:00:00 |
+    And I stub the following stacks:
+      | stack_id | stack_name | region    | outputs          |
+      | 1        | myapp-vpc  | us-east-1 | VpcId=vpc-xxxxxx |
+    When I run `stack_master apply us-east-1 myapp-web --trace` interactively
+    And the output should contain all of these lines:
+      | Stack diff:                                                                    |
+      | +    "TestSg": {                                                               |
+      | Parameters diff:                                                               |
+      | VpcId: vpc-xxxxxx                                                              |
+      | 2020-10-29 00:00:00 +1100 myapp-web AWS::CloudFormation::Stack CREATE_COMPLETE |
+    Then the exit status should be 0
+
+  Scenario: Create a stack with a notification ARN and a stack update policy
+    Given a file named "stack_master.yml" with:
+      """
+      stacks:
+        us_east_1:
+          myapp_vpc:
+            template: myapp_vpc.rb
+            notification_arns:
+              - test_arn
+            stack_policy_file: no_rds_replacement.json
+      """
+    And a file named "policies/no_rds_replacement.json" with:
+      """
+      {}
+      """
+    And I set the environment variables to:
+      | variable | value |
+      | ANSWER   | y     |
+    And I stub the following stack events:
+      | stack_id | event_id | stack_name | logical_resource_id | resource_status | resource_type              | timestamp           |
+      | 1        | 1        | myapp-vpc  | TestSg              | CREATE_COMPLETE | AWS::EC2::SecurityGroup    | 2020-10-29 00:00:00 |
+      | 1        | 1        | myapp-vpc  | myapp-vpc           | CREATE_COMPLETE | AWS::CloudFormation::Stack | 2020-10-29 00:00:00 |
+    And I stub the following stacks:
+      | stack_id | stack_name | parameters       | region    |
+      | 1        | myapp-vpc  | KeyName=my-key   | us-east-1 |
+    And I stub a template for the stack "myapp-vpc":
+      """
+      {
+        "Description": "Test template",
+        "AWSTemplateFormatVersion": "2010-09-09",
+        "Parameters": {
+          "KeyName": {
+            "Description": "Key Name",
+            "Type": "String"
+          }
+        },
+        "Resources": {
+          "TestSg": {
+            "Type": "AWS::EC2::SecurityGroup",
+            "Properties": {
+              "GroupDescription": "Test SG",
+              "VpcId": {
+                "Ref": "VpcId"
+              }
+            }
+          },
+          "TestSg2": {
+            "Type": "AWS::EC2::SecurityGroup",
+            "Properties": {
+              "GroupDescription": "Test SG 2",
+              "VpcId": {
+                "Ref": "VpcId"
+              }
+            }
+          }
+        }
+      }
+      """
+    When I run `stack_master apply us-east-1 myapp-vpc --trace` interactively
+    Then the stack "myapp-vpc" should have a policy with the following:
+      """
+      {}
+      """
+    And the stack "myapp-vpc" should contain this notification ARN "test_arn"
+    Then the exit status should be 0