cfn-flow 0.0.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 9b11eb9cc9a03afce395768b4e29f5b59764646f
+  data.tar.gz: 18fa02cf2b9834e8ab4c870d23ecb548dfd7a064
+SHA512:
+  metadata.gz: 81a7ce12e778b4b5beb6fd566143900ec6a9803b19f56a9b4b4700dd65d4128a792ec9f48be6a3741c329f24ad5a7c3023a2288c4a80014c836a86348910266a
+  data.tar.gz: 01491f61b53acc8ece495146a7e8b9864f96dd897ec7fc4465ce97910d773ba88fd04da9952abd9db7a1314eeb3630ae533276b465b0b67866354163166b9365

data/README.md

@@ -0,0 +1,133 @@
+# cfn-flow
+An opinionated command-line workflow for developing AWS CloudFormation templates. Track template changes in git and upload versioned releases to AWS S3.
+
+#### Opinions
+
+1. *Optimize for onboarding.* The workflow should be simple to learn & understand.
+2. *Optimize for happiness.* The workflow should be easy and enjoyable to use.
+3. *Auditable history.* Know who changed what when. Leverage git for auditing.
+4. *Immutable releases.* The code in a release never changes.
+
+## Installation
+
+Via [rubygems](https://rubygems.org/gems/cfn-flow):
+```
+gem install cfn-flow
+```
+
+The `git` command is also needed.
+
+## Usage
+
+Poke around:
+```
+cfn-flow help
+```
+
+#### Dev mode (default)
+
+Dev mode allows you to quickly test template changes.
+`cfn-flow` validates all templates and uploads them to your personal prefix, overwriting existing templates.
+
+Dev mode does not verify that your local changes are
+committed to git (as opposed to release mode).
+
+You should use dev mode for testing & verifying changes in non-production stacks.
+
+```
+# Set a personal name to prefix your templates.
+export CFN_FLOW_DEV_NAME=aaron
+
+# Validate and upload all CloudFormation templates in your working directory to
+s3://my-bucket/dev/aaron/*
+# NB that this overwrites existing templates in your CFN_FLOW_DEV_NAME
+namespace.
+
+cfn-init
+```
+
+You can launch or update test stacks using your dev template path to quickly test your
+template changes.
+
+#### Release mode
+
+Release mode publishes your templates to a versioned S3 path, and pushes a git
+tag of the version.
+
+```
+# uploads templates to `s3://my-bucket/release/1.0.0/*` and pushes a 1.0.0 git
+tag
+cfn-flow --release 1.0.0
+```
+
+Release mode ensures there are no uncommitted changes in your git working
+directory, and pushes a `1.0.0` git tag.
+
+Inspecting the differences between releases is possible using `git log` and `git
+diff`.
+
+## Configuration
+
+You can configure cfn-flow defaults by creating a `cfn-flow.yml` file in same
+directory you run `cfn-flow` (presumably the root of your project).
+
+Settings in the configuration file are overridden by environment variables. And
+environment variables are overridden by command line arguments.
+
+```
+# cfn-flow.yml in the root of your project
+# All options in this config can be overridden with command line arguments
+---
+# S3 bucket where templates are uploaded. No default.
+# Override with CFN_FLOW_BUCKET
+bucket: 'my-s3-bucket'
+
+# S3 path prefix. Default: none
+# Override with CFN_FLOW_TO
+to: my/project/prefix
+
+# Local path in which to recursively search for templates. Default: .
+# Override with CFN_FLOW_FROM
+from: my/local/prefix
+
+# AWS Region
+# Override with AWS_REGION
+region: us-east-1 # AWS region
+```
+
+#### AWS credentials
+
+AWS credentials can only be set using the
+`AWS_ACCESS_KEY_ID` and `AWS_SECRET_ACCESS_KEY` environment variables; or by
+using an EC2 instance's IAM role.
+
+
+## Sweet Features
+
+#### YAML > JSON
+
+`cfn-flow` lets you write templates in either JSON or
+[YAML](http://www.yaml.org). YAML is a superset of JSON that allows a terser,
+less cluttered syntax, inline comments, and code re-use with variables. YAML
+templates are transparently converted to JSON when uploaded to S3 for use in
+CloudFormation stacks.
+
+#### Use versions in nested stack template URLs
+
+`cfn-flow` works great with [nested stack
+resources](http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-stack.html). Use [Fn::Join](http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/intrinsic-function-reference-join.html) to construct the `TemplateURL` from a parameter:
+
+```
+{
+   "Type" : "AWS::CloudFormation::Stack",
+   "Properties" : {
+      "TemplateURL" : {
+        "Fn::Join" : [ ":",r
+          [ "https://s3.amazonaws.com/my-bucket", {"Ref": "prefix"}, "my-template.json" ]
+          ]
+      }
+   }
+}
+```
+
+While testing, set the `prefix` parameter to dev prefix like `dev/aaron`. When you're confident your changes work, release them with cfn-flow and change the `prefix` parameter to `release/1.0.0` for production.

data/Rakefile

@@ -0,0 +1,18 @@
+require "rubygems"
+require "bundler/setup"
+require 'rake/testtask'
+
+namespace :test do
+  Rake::TestTask.new(:units) do |t|
+    t.pattern = "spec/*_spec.rb"
+  end
+
+  Rake::TestTask.new(:integration) do |t|
+    t.pattern = "spec/integration/*_spec.rb"
+  end
+end
+
+desc 'Run tests'
+task :test => %w[test:units test:integration]
+
+task :default => :test

data/bin/cfn-flow

@@ -0,0 +1,3 @@
+#!/usr/bin/env ruby
+require 'cfn-flow'
+CfnFlow.start

data/lib/cfn-flow/template.rb

@@ -0,0 +1,59 @@
+class CfnFlow::Template
+  attr_reader :from, :prefix, :bucket
+  def initialize(from:, prefix:, bucket:)
+    @from, @prefix, @bucket = from, prefix, bucket
+  end
+
+  def yaml?
+    from.end_with?('.yml')
+  end
+
+  def json?
+    ! yaml?
+  end
+
+  # Determine if this file is a CFN template
+  def is_cfn_template?
+    from_data.is_a?(Hash) && from_data.key?('Resources')
+  end
+
+  def validate!
+    cfn.validate_template(template_body: to_json)
+  end
+
+  def key
+    # Replace leading './' in from, rename *.yml to *.json
+    File.join(prefix, from.sub(/\A\.\//, '').sub(/\.yml\Z/, '.json'))
+  end
+
+  def upload!
+    s3_object.put(body: to_json)
+  end
+
+  def url
+    s3_object.public_url
+  end
+
+  def from_data
+    # We *could* load JSON as YAML, but that would generate confusing errors
+    # in the case of a JSON syntax error.
+    @from_data ||= yaml? ? YAML.load_file(from) : MultiJson.load(File.read(from))
+  rescue
+    puts "Error loading #{from}"
+    raise $!
+  end
+
+  def to_json
+    @to_json ||= MultiJson.dump(from_data, pretty: true)
+  end
+
+  private
+  def cfn
+    @cfn ||= Aws::CloudFormation::Client.new
+  end
+
+  def s3_object
+    @s3_object ||= Aws::S3::Object.new(bucket, key)
+  end
+
+end

data/lib/cfn-flow/version.rb

@@ -0,0 +1,3 @@
+module CfnFlow
+  VERSION = '0.0.1'
+end

data/lib/cfn-flow.rb

@@ -0,0 +1,143 @@
+require 'thor'
+require 'aws-sdk'
+require 'multi_json'
+require 'yaml'
+
+class CfnFlow < Thor
+  class GitError < StandardError; end
+
+  require 'cfn-flow/template'
+  def self.shared_options
+
+    method_option :bucket,     type: :string, desc: 'S3 bucket for templates'
+    method_option :to,         type: :string, desc: 'S3 path prefix for templates'
+    method_option :from,       type: :string, desc: 'Local source directory for templates'
+    method_option 'dev-name',  type: :string, desc: 'Personal development prefix'
+    method_option :region,     type: :string, desc: 'AWS Region'
+
+    method_option :verbose, type: :boolean, desc: 'Verbose output', default: false
+  end
+
+  no_commands do
+    def load_config
+      defaults = { 'from' => '.' }
+      file_config = begin
+        YAML.load_file('./cfn-flow.yml')
+      rescue Errno::ENOENT
+        {}
+      end
+      env_config = {
+        'bucket'      => ENV['CFN_FLOW_BUCKET'],
+        'to'          => ENV['CFN_FLOW_TO'],
+        'from'        => ENV['CFN_FLOW_FROM'],
+        'dev-name'    => ENV['CFN_FLOW_DEV_NAME'],
+        'region'      => ENV['AWS_REGION']
+      }.delete_if {|_,v| v.nil?}
+
+      # Env vars override config file. Command args override env vars.
+      self.options = defaults.merge(file_config).merge(env_config).merge(options)
+
+      # Ensure region env var is set for AWS client
+      ENV['AWS_REGION'] = options['region']
+
+      # TODO: validate required options are present
+    end
+
+    shared_options
+    def load_templates
+      load_config
+      glob = File.join(options['from'], '**/*.{yml,json,template}')
+
+      @templates = Dir.glob(glob).map { |path|
+        CfnFlow::Template.new(from: path, bucket: options['bucket'], prefix: prefix)
+      }.select! {|t|
+        verbose "Checking file #{t.from}... "
+        if t.is_cfn_template?
+          verbose "loaded"
+          true
+        else
+          verbose "skipped."
+          false
+        end
+      }
+    end
+  end
+
+  desc :validate, 'Validates templates'
+  shared_options
+  def validate
+    load_templates
+    @templates.each do |t|
+      begin
+        verbose "Validating #{t.from}... "
+        t.validate!
+        verbose "valid."
+      rescue Aws::CloudFormation::Errors::ValidationError
+        say "Error validating #{t.from}. Message:"
+        say $!.message
+      end
+    end
+  end
+
+  desc :upload, 'Validate & upload templates to the CFN_DEV_FLOW_NAME prefix'
+  shared_options
+  method_option :release, type: :string, desc: 'Upload & tag release'
+  def upload
+    tag_release if options['release']
+
+    validate
+    @templates.each do |t|
+      verbose "Uploading #{t.from} to #{t.url}"
+      t.upload!
+    end
+
+    push_release if options['release']
+
+  end
+  default_task :upload
+
+  private
+  def verbose(msg)
+    say msg if options['verbose']
+  end
+
+  def prefix
+    # Add the release or dev name to the prefix
+    parts = []
+    parts << options['prefix'] unless options['prefix'].empty?
+    if options['release']
+      parts += [ 'release',  options['release'] ]
+    else
+      parts += [ 'dev', options['dev-name'] ]
+    end
+    File.join(*parts)
+  end
+
+  def tag_release
+    # Check git status
+    unless `git status -s`.empty?
+      git_error "Git working directory is not clean. Please commit or reset changes in order to release."
+    end
+    unless $?.success?
+      git_error "Error running `git status`"
+    end
+
+    say "Tagging release #{options['release']}"
+    `git tag -a -m #{options['release']}, #{options['release']}`
+    unless $?.success?
+      git_error "Error tagging release."
+    end
+  end
+
+  def push_tag
+    `git push origin #{options['release']}`
+    unless $?.success?
+      git_error "Error pushing tag to origin."
+    end
+  end
+
+  def git_error(message)
+    say message, :red
+    raise GitError.new(message)
+  end
+end

data/spec/sqs.template

@@ -0,0 +1,66 @@
+{
+  "AWSTemplateFormatVersion" : "2010-09-09",
+ 
+  "Description" : "AWS CloudFormation Sample Template SQS_With_CloudWatch_Alarms: Sample template showing how to create an SQS queue with AWS CloudWatch alarms on queue depth. **WARNING** This template creates an Amazon SQS Queue and one or more Amazon CloudWatch alarms. You will be billed for the AWS resources used if you create a stack from this template.",
+ 
+  "Parameters" : {
+    "AlarmEMail": {
+      "Description": "EMail address to notify if there are any operational issues",
+      "Type": "String",
+      "AllowedPattern": "([a-zA-Z0-9_\\-\\.]+)@((\\[[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.)|(([a-zA-Z0-9\\-]+\\.)+))([a-zA-Z]{2,4}|[0-9]{1,3})(\\]?)",
+      "ConstraintDescription": "must be a valid email address."
+    }
+  },
+ 
+  "Resources" : {
+    "MyQueue" : {
+      "Type" : "AWS::SQS::Queue",
+      "Properties" : {
+      }
+    },
+
+    "AlarmTopic": {
+      "Type": "AWS::SNS::Topic",
+      "Properties": {
+        "Subscription": [{
+          "Endpoint": { "Ref": "AlarmEMail" },
+          "Protocol": "email"
+        }]
+      }
+    },
+
+    "QueueDepthAlarm": {
+      "Type": "AWS::CloudWatch::Alarm",
+      "Properties": {
+        "AlarmDescription": "Alarm if queue depth grows beyond 10 messages",
+        "Namespace": "AWS/SQS",
+        "MetricName": "ApproximateNumberOfMessagesVisible",
+        "Dimensions": [{
+          "Name": "QueueName",
+          "Value" : { "Fn::GetAtt" : ["MyQueue", "QueueName"] }
+        }],
+        "Statistic": "Sum",
+        "Period": "300",
+        "EvaluationPeriods": "1",
+        "Threshold": "10",
+        "ComparisonOperator": "GreaterThanThreshold",
+        "AlarmActions": [{ "Ref": "AlarmTopic" }],
+        "InsufficientDataActions": [{ "Ref": "AlarmTopic" }]
+      }
+    }
+  },
+  "Outputs" : {
+    "QueueURL" : {
+      "Description" : "URL of newly created SQS Queue",
+      "Value" : { "Ref" : "MyQueue" }
+    },
+    "QueueARN" : {
+      "Description" : "ARN of newly created SQS Queue",
+      "Value" : { "Fn::GetAtt" : ["MyQueue", "Arn"]}
+    },
+    "QueueName" : {
+      "Description" : "Name newly created SQS Queue",
+      "Value" : { "Fn::GetAtt" : ["MyQueue", "QueueName"]}
+    }
+  }
+}

data/spec/sqs.yml

@@ -0,0 +1,62 @@
+---
+AWSTemplateFormatVersion: '2010-09-09'
+Description: 'AWS CloudFormation Sample Template SQS_With_CloudWatch_Alarms: Sample
+  template showing how to create an SQS queue with AWS CloudWatch alarms on queue
+  depth. **WARNING** This template creates an Amazon SQS Queue and one or more Amazon
+  CloudWatch alarms. You will be billed for the AWS resources used if you create a
+  stack from this template.'
+Parameters:
+  AlarmEMail:
+    Description: EMail address to notify if there are any operational issues
+    Type: String
+    AllowedPattern: "([a-zA-Z0-9_\\-\\.]+)@((\\[[0-9]{1,3}\\.[0-9]{1,3}\\.[0-9]{1,3}\\.)|(([a-zA-Z0-9\\-]+\\.)+))([a-zA-Z]{2,4}|[0-9]{1,3})(\\]?)"
+    ConstraintDescription: must be a valid email address.
+Resources:
+  MyQueue:
+    Type: AWS::SQS::Queue
+    Properties: {}
+  AlarmTopic:
+    Type: AWS::SNS::Topic
+    Properties:
+      Subscription:
+      - Endpoint:
+          Ref: AlarmEMail
+        Protocol: email
+  QueueDepthAlarm:
+    Type: AWS::CloudWatch::Alarm
+    Properties:
+      AlarmDescription: Alarm if queue depth grows beyond 10 messages
+      Namespace: AWS/SQS
+      MetricName: ApproximateNumberOfMessagesVisible
+      Dimensions:
+      - Name: QueueName
+        Value:
+          Fn::GetAtt:
+          - MyQueue
+          - QueueName
+      Statistic: Sum
+      Period: '300'
+      EvaluationPeriods: '1'
+      Threshold: '10'
+      ComparisonOperator: GreaterThanThreshold
+      AlarmActions:
+      - Ref: AlarmTopic
+      InsufficientDataActions:
+      - Ref: AlarmTopic
+Outputs:
+  QueueURL:
+    Description: URL of newly created SQS Queue
+    Value:
+      Ref: MyQueue
+  QueueARN:
+    Description: ARN of newly created SQS Queue
+    Value:
+      Fn::GetAtt:
+      - MyQueue
+      - Arn
+  QueueName:
+    Description: Name newly created SQS Queue
+    Value:
+      Fn::GetAtt:
+      - MyQueue
+      - QueueName

metadata

@@ -0,0 +1,139 @@
+--- !ruby/object:Gem::Specification
+name: cfn-flow
+version: !ruby/object:Gem::Version
+  version: 0.0.1
+platform: ruby
+authors:
+- Aaron Suggs
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2015-01-27 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: aws-sdk
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 2.0.20.pre
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: 2.0.20.pre
+- !ruby/object:Gem::Dependency
+  name: thor
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.18'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.18'
+- !ruby/object:Gem::Dependency
+  name: multi_json
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.0'
+- !ruby/object:Gem::Dependency
+  name: minitest
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: appraisal
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+description: An opinionate worflow for AWS CloudFormation
+email: aaron@ktheory.com
+executables:
+- cfn-flow
+extensions: []
+extra_rdoc_files: []
+files:
+- README.md
+- Rakefile
+- bin/cfn-flow
+- lib/cfn-flow.rb
+- lib/cfn-flow/template.rb
+- lib/cfn-flow/version.rb
+- spec/sqs.template
+- spec/sqs.yml
+homepage: http://github.com/kickstarter/cfn-flow
+licenses:
+- MIT
+metadata: {}
+post_install_message: 
+rdoc_options:
+- "--charset=UTF-8"
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 2.0.0
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.4.5
+signing_key: 
+specification_version: 4
+summary: A CLI for CloudFormation templates
+test_files:
+- spec/sqs.template
+- spec/sqs.yml