lono 1.1.3 → 2.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 9f79d721191571eaa17df7a5e037281b4631ca0e
-  data.tar.gz: f38b1dbae7db19838efae137471d0685710b43d8
+  metadata.gz: 380e874615eea57c1cc0788f77aef520d453e8c1
+  data.tar.gz: fb70edb0ff4a31521f6df6d3085fdde5fd47014d
 SHA512:
-  metadata.gz: 6d3eeb1096333c819438a73d03c81de71fcc367e5ab1ba34393fae138b39b99e8a5f7bd9cec1d4fcdcbd38af62eca13d91d6fbce93ab41b592f95da78f93e2f7
-  data.tar.gz: 250cabca938f4f5eb8b48f15843ac6cd70387ed7e322e5ff40cd00918f22da36b5de247b73ba55138bff77ca8eb98aed6ebccc1b167a298ebe471d6c18ce9568
+  metadata.gz: c529da1117901155fe7ddc3033d779611370d3654ad01e76e5c5364a917f956911501d4014228166e22386457f3d2a2dbfa335ba9272eb1ef63ef0b85934dc4a
+  data.tar.gz: c31a6c825e2c42a1828d333fc22034c9605566c2cd6433104df21df25be0ea17e7990d555e55bb97ce07ecace12c119e8262153c12bf636b6e1c85df7bd74cd4

data/.gitmodules

@@ -0,0 +1,3 @@
+[submodule "vendor/plissken"]
+	path = vendor/plissken
+	url = git@github.com:tongueroo/plissken.git

data/CHANGELOG.md

@@ -3,6 +3,14 @@
 All notable changes to this project will be documented in this file.
 This project *tries* to adhere to [Semantic Versioning](http://semver.org/), even before v1.0.
 
+## [2.0.0]
+- added lono cfn subcommand to launch CloudFormation stacks
+- added lono params subcommand to generate CloudFormation parameter files
+- moved lono template methods into subcommand.  lono generate still available at top level.
+
+## [1.1.4]
+- allow --help or -h at the end of the command
+
 ## [1.1.3]
 - make display output path prettier
 

data/README.md

@@ -2,30 +2,27 @@
 
 [![ReadmeCI](http://www.readmeci.com/images/readmeci-badge.svg)](http://www.readmeci.com/tongueroo/lono)
 [![Gem Version](https://badge.fury.io/rb/lono.png)](http://badge.fury.io/rb/lono)
-[![Build History][2]][1]
+[![CircleCI](https://circleci.com/gh/tongueroo/lono.svg?style=svg)](https://circleci.com/gh/tongueroo/lono)
 [![Code Climate][3]][4]
 [![Dependency Status](https://gemnasium.com/tongueroo/lono.png)](https://gemnasium.com/tongueroo/lono)
 [![Coverage Status](https://coveralls.io/repos/tongueroo/lono/badge.png)](https://coveralls.io/r/tongueroo/lono)
 
-[1]: http://travis-ci.org/tongueroo/lono
-[2]: https://secure.travis-ci.org/tongueroo/lono.png?branch=master
 [3]: https://codeclimate.com/repos/51d7f1407e00a4042c010ab4/badges/5273fe6cdb5a13e58554/gpa.png
 [4]: https://codeclimate.com/repos/51d7f1407e00a4042c010ab4/feed
 
-Lono is a CloudFormation Template generator.  Lono generates CloudFormation templates based on ERB ruby templates in either `yaml` or `json` format.
+Lono is a tool for managing CloudFormation templates.
 
-Lono can be used with [lono-cfn](https://github.com/tongueroo/lono-cfn). Lono-cfn automatically calls `lono generate` as part of launching the stack so you never forget to run it.  These blog posts cover both lono and lono-cfn:
+* Lono generates CloudFormation templates based on ERB ruby templates in either `yaml` or `json` format.
+* Lono takes simple env-like files to and generates the CloudFormation parameter files.
+* Lono wraps the CloudFormation api calls in a simple interface using the generated files to launch the CloudFormation stacks.
+
+These blog posts lono:
 
 * [Why Generate CloudFormation Templates with Lono](https://medium.com/boltops/why-generate-cloudformation-templates-with-lono-65b8ea5eb87d)
 * [Generating CloudFormation Templates with Lono](https://medium.com/boltops/generating-cloudformation-templates-with-lono-4709afa1299b)
 * [AutoScaling CloudFormation Template with Lono](https://medium.com/boltops/autoscaling-cloudformation-template-with-lono-3dc520480c5f)
-* [CloudFormation Tools: lono, lono-params and lono-cfn Together
-](https://medium.com/boltops/cloudformation-tools-lono-lono-params-and-lono-cfn-play-together-620af51e616)
-* [AWS CloudFormation dry-run with lono-cfn plan](https://medium.com/boltops/aws-cloudformation-dry-run-with-lono-cfn-plan-2a1e0f80d13c)
-
-**UPDATE**: lono now supports yaml format for CloudFormation! The old json format is still supported. For current projects, you do not have to change anything as the format is autodetected as part of `lono generate`.
-
-Newly generated projects with `lono new infra` will use the yaml format by default.  If you prefer the json format use `lono new --format json infra`. You cannot mix and match json and yaml format in one project, it's either all json or yaml. If any one has questions feel free to ping me: tongueroo@gmail.com.
+* [CloudFormation Tools: lono, lono-params and lono cfn Together](https://medium.com/boltops/cloudformation-tools-lono-lono-params-and-lono-cfn-play-together-620af51e616)
+* [AWS CloudFormation dry-run with lono cfn preview](https://medium.com/boltops/aws-cloudformation-dry-run-with-lono-cfn-plan-2a1e0f80d13c)
 
 ## Usage
 
@@ -33,14 +30,18 @@ Newly generated projects with `lono new infra` will use the yaml format by defau
 $ lono new infra
 </pre>
 
-This sets up a starter lono project called infra with example templates.  Next you cd into the folder and generate the templates.
+This sets up a starter lono project called infra with example templates.  Next you cd into the folder and generate the template and parameter files.
 
 <pre>
 $ cd infra
 $ lono generate
 </pre>
 
-This generates the templates in the `config` and `templates` folders to the `output` folder.
+This CloudFormation template and parameter files are now available under `output` and `output/params`.
+
+### Generate Template Files
+
+The CloudFormation templates files that were generated from the `config` and `templates` folders and written to the `output` folder.
 
 The starter lono template project config files looks like [this](lib/starter_project_yaml/config/lono.rb) and [this](lib/starter_project_yaml/config/lono/api.rb).  Here's a snippet from one of the config files: [config/lono.rb](lib/starter_project_yaml/config/lono.rb).
 
@@ -171,6 +172,31 @@ You can use the generated CloudFormation templates in the `output` folder just a
 
 ![Lono flowchart](http://tongueroo.com/images/github-readmes/lono-flowchart.png "Lono flowchart")
 
+### Generate Parameter Files
+
+With lono you write simpler params file that is formatted with a simple `key=value` env-like file.  The `params/mystack.txt` is shorter format versus the verbose standard CloudFormation parameters json file. Here is an example of a simple `params/mystack.txt` param file.
+
+```bash
+Param1=1
+Param2=2
+```
+
+The `lono generate` command generates the CloudFormation parameter file in `output/params/mystack.json` which looks like this:
+
+
+```json
+[
+  {
+    "ParameterKey": "Param1",
+    "ParameterValue": "1"
+  },
+  {
+    "ParameterKey": "Param2",
+    "ParameterValue": "2"
+  }
+]
+```
+
 ## Template helper methods
 
 There are helper methods that are available in templates.
@@ -189,7 +215,7 @@ If you have a lot of templates, the config/lono.rb file can get unwieldy long.
 
 ## Generate
 
-You can generate the CF templates by running:
+You can generate the CloudFormation templates by running:
 
 <pre>
 $ lono generate
@@ -202,39 +228,124 @@ The lono init command also sets up guard-lono.  Guard-lono continuously generate
 $ guard
 </pre>
 
-## lono-cfn and lono-params
+# lono cfn
 
-Running `lono generate` and building up the `aws cloudformation create-stack` command repeatedly gets old. The `lono-cfn` tool will automatically run `lono generate` and then launch the CloudFormation stack all in one command.  Example usage:
+## Summary
+Lono also provides a `lono cfn` wrapper command that allows you to launch stacks from the lono templates.  The `lono cfn` tool automatically runs `lono generate` internally and then launches the CloudFormation stack all in one command.  Provided that you are in a lono project and have a `my-stack` lono template definition.  To create a stack you can simply run:
 
 ```
-$ lono-cfn create mystack-$(date +%Y%m%d%H%M%S) --template mystack --params mystack
-$ lono-cfn create mystack-$(date +%Y%m%d%H%M%S) # shorthand if template and params file matches.
-$ lono-cfn update mystack-1493859659
-$ lono-cfn delete mystack-1493859659
+$ lono cfn create my-stack
 ```
 
-More info about lono-cfn here: [lono-cfn](https://github.com/tongueroo/lono-cfn) - Wrapper cfn tool to quickly create CloudFormation stacks from lono templates and params files.
+The above command will generate files to `output/my-stack.json` and `output/params/my-stack.txt` and use them to create a CloudFormation stack.  Here are some more examples of cfn commands:
 
-The params file is formatted with a simple `key=value`, env-like file.  It is cleaner to have a `params/mystack.txt` file like so:
+```
+$ lono cfn create mystack-$(date +%Y%m%d%H%M%S) --template mystack --params mystack
+$ lono cfn create mystack-$(date +%Y%m%d%H%M%S) # shorthand if template and params file matches.
+$ lono cfn diff mystack-1493859659
+$ lono cfn preview mystack-1493859659
+$ lono cfn update mystack-1493859659
+$ lono cfn delete mystack-1493859659
+$ lono cfn create -h # getting help
+```
 
-```bash
-Param1=1
-Param2=2
+### Conventions
+
+The template by convention defaults to the name of the stack.  In turn, the params by convention defaults to the name of the template.
+
+* stack - This is a required parameter and is the CLI first parameter.
+* template - By convention matches the stack name but can be overriden with `--template`.
+* params - By convention matches the template name but can be overriden with `--params`.
+
+The conventions allows the command to be a very nice short command that can be easily remembered.  For example, these 2 commands are the same:
+
+### lono cfn create
+
+Long form:
+
+```
+$ lono cfn create my-stack --template my-stack --params --my-stack
 ```
 
-Verus the rather verbose standard CloudFormation parameters json file:
+Short form:
 
-```json
-[
-  {
-    "ParameterKey": "Param1",
-    "ParameterValue": "1"
-  },
-  {
-    "ParameterKey": "Param2",
-    "ParameterValue": "2"
-  }
-]
 ```
+$ lono cfn create my-stack
+```
+
+
+Both template and params conventions can be overridden.  Here are examples of overriding the template and params name conventions.
+
+```
+$ lono cfn create my-stack --template different-name1
+```
+
+The template that will be use is output/different-name1.json and the parameters will use params/different-name1.json.
 
-More info about lono-params here: [lono-params](https://github.com/tongueroo/lono-params) - Tool to generate a CloudFormation parameters json formatted file from a simplier env like file.
+```
+$ lono cfn create my-stack --params different-name2
+```
+
+The template that will be use is output/different-name2.json and the parameters will use params/different-name2.json.
+
+```
+$ lono cfn create my-stack --template different-name3 --params different-name4
+```
+
+The template that will be use is output/different-name3.json and the parameters will use params/different-name4.json.
+
+### lono cfn update
+
+To update stacks you can use `lono cfn update`:
+
+```
+$ lono cfn update my-stack --template template-name --params params-name
+```
+
+By default the update command will display a preview of the stack changes before applying the update and prompt to check if you are sure.  If you want to bypass the are you sure prompt, use the `--sure` option.
+
+```
+$ lono cfn update my-stack --template template-name --params params-name --sure
+```
+
+### lono cfn delete
+
+```
+$ lono cfn delete my-stack --sure
+```
+
+### lono cfn preview
+
+If you want to see the CloudFormation preview without updating the stack you can also use the `lono cfn preview` command.  The preview command is also covered in this blog post: [AWS CloudFormation dry-run with lono cfn preview](https://medium.com/boltops/aws-cloudformation-dry-run-with-lono cfn-plan-2a1e0f80d13c)
+
+```
+$ lono cfn preview example --template single_instance --params single_instance
+Using template: output/single_instance.yml
+Using parameters: params/single_instance.txt
+Generating CloudFormation templates:
+  ./output/single_instance.yml
+Params file generated for example at ./output/params/example.json
+Generating CloudFormation Change Set for preview.....
+CloudFormation preview for 'example' stack update. Changes:
+Remove AWS::Route53::RecordSet: DnsRecord testsubdomain.sub.tongueroo.com
+$
+```
+
+
+## Contributing
+
+1. Fork it
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create new Pull Request
+
+## Developing
+
+There is a submodule in this project, so when you check out remember to sync the submodule.
+
+```bash
+$ git clone git@github.com:yourfork/lono.git
+$ git submodule sync
+$ git submodule update --init
+```

data/bin/lono

@@ -1,4 +1,4 @@
 #!/usr/bin/env ruby
 
-require File.expand_path("../../lib/lono", __FILE__)
-Lono::CLI.start
+require_relative "../lib/lono"
+Lono::CLI.start

data/circle.yml

@@ -0,0 +1,4 @@
+checkout:
+  post:
+    - git submodule sync --recursive
+    - git submodule update --init --recursive

data/lib/lono.rb

@@ -4,10 +4,19 @@ require 'pp'
 require 'colorize'
 require 'fileutils'
 
-$:.unshift File.dirname(__FILE__)
-require 'lono/version'
-require 'lono/cli'
-require 'lono/new'
-require 'lono/template'
-require 'lono/dsl'
-require 'lono/bashify'
+# vendor because need https://github.com/futurechimp/plissken/pull/6 to be merged
+$:.unshift(File.expand_path("../../vendor/plissken/lib", __FILE__))
+require "plissken"
+
+$:.unshift(File.expand_path('../', __FILE__))
+module Lono
+  autoload :VERSION, 'lono/version'
+  autoload :Help, 'lono/help'
+  autoload :ProjectChecker, 'lono/project_checker'
+  autoload :Command, 'lono/command'
+  autoload :CLI, 'lono/cli'
+  autoload :New, 'lono/new'
+  autoload :Template, 'lono/template'
+  autoload :Cfn, 'lono/cfn'
+  autoload :Param, 'lono/param'
+end

data/lib/lono/cfn.rb

@@ -0,0 +1,64 @@
+require "thor"
+
+module Lono
+  class Cfn < Command
+    autoload :Help, 'lono/cfn/help'
+    autoload :AwsServices, 'lono/cfn/aws_services'
+    autoload :Util, 'lono/cfn/util'
+    autoload :CLI, 'lono/cfn/cli'
+    autoload :Base, 'lono/cfn/base'
+    autoload :Create, 'lono/cfn/create'
+    autoload :Update, 'lono/cfn/update'
+    autoload :Delete, 'lono/cfn/delete'
+    autoload :Preview, 'lono/cfn/preview'
+    autoload :Diff, 'lono/cfn/diff'
+
+    class_option :verbose, type: :boolean
+    class_option :noop, type: :boolean
+    class_option :project_root, desc: "Project folder.  Defaults to current directory", default: "."
+    class_option :region, desc: "AWS region"
+
+    # common to create and update
+    class_option :template, desc: "override convention and specify the template file to use"
+    class_option :param, desc: "override convention and specify the param file to use"
+    class_option :lono, type: :boolean, desc: "invoke lono to generate CloudFormation templates", default: true
+
+    desc "create STACK", "create a CloudFormation stack"
+    long_desc Help.create
+    def create(name)
+      Create.new(name, options).run
+    end
+
+    desc "update STACK", "update a CloudFormation stack"
+    long_desc Help.update
+    option :change_set, type: :boolean, default: true, desc: "Uses generated change set to update the stack.  If false, will perform normal update-stack."
+    option :diff, type: :boolean, default: true, desc: "Show diff of the source code template changes before continuing."
+    option :preview, type: :boolean, default: true, desc: "Show preview of the stack changes before continuing."
+    option :sure, type: :boolean, desc: "Skips are you sure prompt"
+    def update(name)
+      Update.new(name, options).run
+    end
+
+    desc "delete STACK", "delete a CloudFormation stack"
+    long_desc Help.delete
+    option :sure, type: :boolean, desc: "Skips are you sure prompt"
+    def delete(name)
+      Delete.new(name, options).run
+    end
+
+    desc "preview STACK", "preview a CloudFormation stack update"
+    long_desc Help.preview
+    option :keep, type: :boolean, desc: "keep the changeset instead of deleting it afterwards"
+    option :diff, type: :boolean, default: true, desc: "Show diff of the source code template changes also."
+    def preview(name)
+      Diff.new(name, options).run if options[:diff]
+      Preview.new(name, options).run
+    end
+
+    desc "diff STACK", "diff of newly generated template vs existing template in AWS"
+    long_desc Help.diff
+    def diff(name)
+      Diff.new(name, options).run
+    end
+  end
+end

data/lib/lono/cfn/aws_services.rb

@@ -0,0 +1,37 @@
+require "aws-sdk"
+
+module Lono::Cfn::AwsServices
+  def cfn
+    @cfn ||= Aws::CloudFormation::Client.new
+  end
+
+  def stack_exists?(stack_name)
+    return true if testing_update?
+    return false if @options[:noop]
+
+    exist = nil
+    begin
+      # When the stack does not exist an exception is raised. Example:
+      # Aws::CloudFormation::Errors::ValidationError: Stack with id blah does not exist
+      resp = cfn.describe_stacks(stack_name: stack_name)
+      exist = true
+    rescue Aws::CloudFormation::Errors::ValidationError => e
+      if e.message =~ /does not exist/
+        exist = false
+      elsif e.message.include?("'stackName' failed to satisfy constraint")
+        # Example of e.message when describe_stack with invalid stack name
+        # "1 validation error detected: Value 'instance_and_route53' at 'stackName' failed to satisfy constraint: Member must satisfy regular expression pattern: [a-zA-Z][-a-zA-Z0-9]*|arn:[-a-zA-Z0-9:/._+]*"
+        puts "Invalid stack name: #{stack_name}"
+        puts "Full error message: #{e.message}"
+        exit 1
+      else
+        raise # re-raise exception  because unsure what other errors can happen
+      end
+    end
+    exist
+  end
+
+  def testing_update?
+    ENV['TEST'] && self.class.name == "LonoCfn::Update"
+  end
+end

data/lib/lono/cfn/base.rb

@@ -0,0 +1,144 @@
+require "lono"
+
+class Lono::Cfn::Base
+  include Lono::Cfn::AwsServices
+  include Lono::Cfn::Util
+
+  def initialize(stack_name, options={})
+    @stack_name = stack_name
+    @options = options
+    @project_root = options[:project_root] || '.'
+    ProjectChecker.check(@project_root) unless options[:lono] # already ran checker in lono generate
+
+    template_name = options[:template] || @stack_name
+    param_name = options[:param] || template_name
+    @template_path = get_source_path(template_name, :template)
+    @param_path = get_source_path(param_name, :param)
+    puts "Using template: #{@template_path}" unless @options[:mute_using]
+    puts "Using parameters: #{@param_path}" unless @options[:mute_using]
+  end
+
+  def run
+    params = generate_all
+    save_stack(params) # defined in the sub class
+  end
+
+  def generate_all
+    generate_templates if @options[:lono]
+    check_for_errors
+    generate_params(mute: @options[:mute_params])
+  end
+
+  def generate_templates
+    Lono::Template::DSL.new(
+      project_root: @project_root,
+      pretty: true
+    ).run
+  end
+
+  def generate_params(options={})
+    generator_options = {
+      project_root: @project_root,
+      path: @param_path,
+      allow_no_file: true
+    }.merge(options)
+    generator = Lono::Param::Generator.new(@stack_name, generator_options)
+    generator.generate  # Writes the json file in CamelCase keys format
+    generator.params    # Returns Array in underscore keys format
+  end
+
+  def check_for_errors
+    errors, warns = check_files
+    unless errors.empty?
+      puts "Please double check the command you ran.  There were some errors."
+      puts "ERROR: #{errors.join("\n")}".colorize(:red)
+      exit
+    end
+    unless warns.empty?
+      puts "Please double check the command you ran.  There were some warnings."
+      puts "WARN: #{warns.join("\n")}".colorize(:yellow)
+    end
+  end
+
+  def check_files
+    errors, warns = [], []
+    unless File.exist?(@template_path)
+      warns << "Template file missing: could not find #{@template_path}"
+    end
+    if @options[:param] && !File.exist?(@param_path)
+      warns << "Parameters file missing: could not find #{@param_path}"
+    end
+    [errors, warns]
+  end
+
+  # if existing in params path then use that
+  # if it doesnt assume it is a full path and check that
+  # else fall back to convention, which also eventually gets checked in check_for_errors
+  #
+  # Type - :param or :template
+  def get_source_path(path, type)
+    if path.nil?
+      default_convention_path = convention_path(@stack_name, type)
+    else
+      # convention path based on the input from the user
+      convention_path(path, type)
+    end
+  end
+
+  def convention_path(name, type)
+    path = case type
+    when :template
+      format = detect_format
+      "#{@project_root}/output/#{name}.#{format}"
+    when :param
+      "#{@project_root}/params/#{name}.txt"
+    else
+      raise "hell: dont come here"
+    end
+    path.sub(/^\.\//, '')
+  end
+
+  # Returns String with value of "yml" or "json".
+  def detect_format
+    formats = Dir.glob("#{@project_root}/output/**/*").map { |path| path }.
+                reject { |s| s =~ %r{/params/} }. # reject output/params folder
+                map { |path| File.extname(path) }.
+                reject { |s| s.empty? }. # reject ""
+                uniq
+    if formats.size > 1
+      puts "ERROR: Detected multiple formats: #{formats.join(", ")}".colorize(:red)
+      puts "All the output files must use the same format.  Either all json or all yml."
+      exit 1
+    elsif formats.size == 1
+      formats.first.sub(/^\./,'')
+    else
+      puts "WARN: Did not detect any template formats. Defaulting to yml.  Please check the config and templates folders."
+      "yml" # default format
+    end
+  end
+
+  # All CloudFormation states listed here:
+  # http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/using-cfn-describing-stacks.html
+  def stack_status(stack_name)
+    return true if testing_update?
+    return false if @options[:noop]
+
+    resp = cfn.describe_stacks(stack_name: stack_name)
+    status = resp.stacks[0].stack_status
+  end
+
+  def exist_unless_updatable(status)
+    return true if testing_update?
+    return false if @options[:noop]
+
+    unless status =~ /_COMPLETE$/
+      puts "Cannot create a change set for the stack because the #{@stack_name} is not in an updatable state.  Stack status: #{status}"
+      quit(1)
+    end
+  end
+
+  # To allow mocking in specs
+  def quit(signal)
+    exit signal
+  end
+end