lono 2.1.0 → 3.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 01b405f56d85200ea0063749316264a0587ea359
-  data.tar.gz: 8963150180a2ff89f2cf125bea08099e27b73917
+  metadata.gz: 2d0a86c90a4434f8330f76b0dd3c804449d42c68
+  data.tar.gz: d8a703e5267a622bf1280d60283e80aeb4cf41ad
 SHA512:
-  metadata.gz: c9c78aa0037081874d77499a5ae30e127558fcf36c99098a318bb87dd3b493c6c59b5092c316ce7c77be55d71634ec70143e1520463f0657d1312732376c24b8
-  data.tar.gz: 9cf8a0df5380cf41469efcb82ec35466d5e9a7737d60275c727f4897f47ed0f55c3ee4b88aec7cb02c308a63bbb8a701fd4fe1a64275cc2bf906419cbb1a5625
+  metadata.gz: 5051c3a781e0eb8b9e00c2eb38072c7f764e2d75516622a61a823d13810753e24718829783a50ffd4fdc24ebfe37b90c9df70b25cb3b587ce505d7aa15d8a9c7
+  data.tar.gz: 2e1180d405393ed872c504727f2d612dea0e301b7197a6aefa82de3f3396c83cb464276e4639be60d5c6824a567eab6f8066c7a58c2a6bb15881115b749f5df1

data/CHANGELOG.md

@@ -3,6 +3,17 @@
 All notable changes to this project will be documented in this file.
 This project *tries* to adhere to [Semantic Versioning](http://semver.org/), even before v1.0.
 
+## [3.0.0]
+- Major feature changes
+- Lono structural changes
+- Layering Support
+- Shared Variable Support
+- Nested Stacks Support
+- Format and Extension Detection
+- Custom Helper Support
+- Source Name Convention Support
+- Settings File Support
+
 ## [2.1.0]
 - improve instance_eval error when lono.rb errors, print out line of code and context
 

data/README.md

@@ -1,22 +1,24 @@
 # Lono
 
-[![ReadmeCI](http://www.readmeci.com/images/readmeci-badge.svg)](http://www.readmeci.com/tongueroo/lono)
 [![Gem Version](https://badge.fury.io/rb/lono.png)](http://badge.fury.io/rb/lono)
 [![CircleCI](https://circleci.com/gh/tongueroo/lono.svg?style=svg)](https://circleci.com/gh/tongueroo/lono)
 [![Code Climate][3]][4]
 [![Dependency Status](https://gemnasium.com/tongueroo/lono.png)](https://gemnasium.com/tongueroo/lono)
 [![Coverage Status](https://coveralls.io/repos/tongueroo/lono/badge.png)](https://coveralls.io/r/tongueroo/lono)
+[![ReadmeCI](http://www.readmeci.com/images/readmeci-badge.svg)](http://www.readmeci.com/tongueroo/lono)
+[![Join the chat at https://gitter.im/tongueroo/lono](https://badges.gitter.im/tongueroo/lono.svg)](https://gitter.im/tongueroo/lono?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
+[![Support](https://img.shields.io/badge/get-support-blue.svg)](https://boltops.com?utm_source=badge&utm_medium=badge&utm_campaign=lono)
 
 [3]: https://codeclimate.com/repos/51d7f1407e00a4042c010ab4/badges/5273fe6cdb5a13e58554/gpa.png
 [4]: https://codeclimate.com/repos/51d7f1407e00a4042c010ab4/feed
 
-Lono is a tool for managing CloudFormation templates.
+Lono is a tool to help you easily manage your CloudFormation templates. Lono handles the entire CloudFormation lifecyle. It starts with helping you craft of the templates and helps you all the way to end when you provision of the infrastructure.
 
 * Lono generates CloudFormation templates based on ERB ruby templates in either `yaml` or `json` format.
 * Lono takes simple env-like files to and generates the CloudFormation parameter files.
 * Lono wraps the CloudFormation api calls in a simple interface using the generated files to launch the CloudFormation stacks.
 
-Lono documentation is at [lono.cloud](http://lono.cloud).
+See [lono.cloud](http://lono.cloud) for full lono documentation.
 
 These blog posts also cover lono:
 
@@ -26,213 +28,24 @@ These blog posts also cover lono:
 * [CloudFormation Tools: lono, lono-params and lono cfn Together](https://medium.com/boltops/cloudformation-tools-lono-lono-params-and-lono-cfn-play-together-620af51e616)
 * [AWS CloudFormation dry-run with lono cfn preview](https://medium.com/boltops/aws-cloudformation-dry-run-with-lono-cfn-plan-2a1e0f80d13c)
 
-## Usage
-
-<pre>
-$ lono new infra
-</pre>
-
-This sets up a starter lono project called infra with example templates.  Next you cd into the folder and generate the template and parameter files.
-
-<pre>
-$ cd infra
-$ lono generate
-</pre>
-
-This CloudFormation template and parameter files are now available under `output` and `output/params`.
+## Quick Usage
 
-### Generate Template Files
+It only takes a couple of commands to start using lono.
 
-The CloudFormation templates files that were generated from the `config` and `templates` folders and written to the `output` folder.
-
-The starter lono template project config files looks like [this](lib/starter_project_yaml/config/lono.rb) and [this](lib/starter_project_yaml/config/lono/api.rb).  Here's a snippet from one of the config files: [config/lono.rb](lib/starter_project_yaml/config/lono.rb).
-
-```ruby
-template "api-web-prod.yml" do
-  app,role,env = name.sub('.yml','').split('-')
-  source "web.yml.erb"
-  variables(
-    env: env,
-    app: app,
-    role: role,
-    ami: "ami-123",
-    instance_type: "m1.small",
-    port: "80",
-    high_threshold: "15",
-    high_periods: "4",
-    low_threshold: "5",
-    low_periods: "10",
-    max_size: "24",
-    min_size: "6",
-    down_adjustment: "-3",
-    up_adjustment: "3",
-    ssl_cert: "arn:aws:iam::12345:server-certificate/wildcard"
-  )
-end
+```sh
+brew cask install boltopslabs/software/bolts
+lono new infra
+cd infra
+lono generate # not needed but showing for explanation
+lono cfn create example
 ```
 
-Here is the corresponding ERB template [templates/web.yml.erb](lib/starter_project_yaml/templates/web.yml.erb).  Note that some of the source code has been shorten for brevity.
-
-```yaml
-<% @app,@role,@env = name.sub('.yml','').split('-') -%>
----
-AWSTemplateFormatVersion: '2010-09-09'
-Description: <%= @app.capitalize %> Stack
-Mappings:
-...
-Outputs:
-...
-Parameters:
-  Application:
-    Default: <%= @app %>
-    Description: Application name
-    Type: String
-...
-Resources:
-  CPUAlarmHigh:
-    Properties:
-      AlarmActions:
-      - Ref: WebServerScaleUpPolicy
-      AlarmDescription: Scale-up if CPU > <%= @high_threshold %>% for <%= @high_mins %>
-...
-<%= partial("host_record.yml.erb", domain: "mydomain.com") %>
-  LaunchConfig:
-    Properties:
-      BlockDeviceMappings:
-      - DeviceName: "/dev/sdb"
-        VirtualName: ephemeral0
-      ImageId:
-...
-      UserData:
-        Fn::Base64: !Sub | # No more Fn::Join needed
-          #!/bin/bash -lexv
-          <% stack_name = "#{@env}-#{@app}-#{@role}" %>
-          exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
-          echo <%= stack_name %> > /tmp/stack_name
-          cat /proc/uptime | cut -f1 -d'.' > /tmp/time-to-boot
-    Type: AWS::AutoScaling::LaunchConfiguration
-
-```
-
-The generated `output/blog-web-prod.yml` CloudFormation template looks like this:
-
-```yaml
----
-AWSTemplateFormatVersion: '2010-09-09'
-Description: Api Stack
-Mappings:
-...
-Outputs:
-...
-Parameters:
-  Application:
-    Default: api
-    Description: Application name
-    Type: String
-...
-Resources:
-  CPUAlarmHigh:
-    Properties:
-      AlarmActions:
-      - Ref: WebServerScaleUpPolicy
-      AlarmDescription: Scale-up if CPU > 15% for
-...
-  HostRecord:
-    Properties:
-      Comment: DNS name for mydomain.com
-      HostedZoneName: ".mydomain.net."
-      Name:
-        Fn::Join:
-        - ''
-        - - Ref: AWS::StackName
-          - mydomain.com
-      ResourceRecords:
-      - Fn::GetAtt:
-        - elb
-        - DNSName
-      TTL: '60'
-      Type: CNAME
-    Type: AWS::Route53::RecordSet
-  LaunchConfig:
-    Properties:
-      BlockDeviceMappings:
-      - DeviceName: "/dev/sdb"
-        VirtualName: ephemeral0
-      ImageId:
-...
-      UserData:
-        Fn::Base64: |
-          #!/bin/bash -lexv
-
-          exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
-          echo api-web-prod > /tmp/stack_name
-          cat /proc/uptime | cut -f1 -d'.' > /tmp/time-to-boot
-    Type: AWS::AutoScaling::LaunchConfiguration
-```
-
-You can use the generated CloudFormation templates in the `output` folder just as you would a normal CloudFormation template.  Here's a flow chart of the overall process.
+This sets up a starter lono project called infra with example templates.  You cd into the folder and call `lono cfn create` which automatically generates the CloudFormation template and parameter files to `output` and `output/params` and launches the stack.
 
 ![Lono flowchart](http://tongueroo.com/images/github-readmes/lono-flowchart.png "Lono flowchart")
 
-### Generate Parameter Files
-
-With lono you write simpler params file that is formatted with a simple `key=value` env-like file.  The `params/mystack.txt` is shorter format versus the verbose standard CloudFormation parameters json file. Here is an example of a simple `params/mystack.txt` param file.
-
-```bash
-Param1=1
-Param2=2
-```
-
-The `lono generate` command generates the CloudFormation parameter file in `output/params/mystack.json` which looks like this:
+### lono cfn summary
 
-
-```json
-[
-  {
-    "ParameterKey": "Param1",
-    "ParameterValue": "1"
-  },
-  {
-    "ParameterKey": "Param2",
-    "ParameterValue": "2"
-  }
-]
-```
-
-## Template helper methods
-
-There are helper methods that are available in templates.
-
-* partial(relative_path, variables, options) - Can be use to embed other files in a template.  The partial view should be placed in the `templates/partial` folder of the project.  So:
-  * partial('launch\_config.json.erb') -> `templates/partial/launch_config.json.erb`
-  * partial('launch\_config.json.erb', foo: "bar", hello: "world") - variables can be passed to the partial helper method are available to the partial as instance variables.  So, in this case `@foo` and `@hello` will be available in the `launch_config.json.erb` partial.
-  * partial('user_data/bootstrap.sh.erb', {}, indent: 10) - Indent the result partial by 10 spaces.  Useful for yaml format.
-
-* user\_data - Helpful if you are using CloudFormation json format. More info on the [wiki page](https://github.com/tongueroo/lono/wiki/user_data-helper-for-json-format).
-
-## Breaking up config/lono.rb
-
-If you have a lot of templates, the config/lono.rb file can get unwieldy long.  You can break up the lono.rb file and put template defintions in the config/lono directory.  Any file in this directory will be automatically loaded. An [example](lib/starter_project_yaml/config/lono/api.rb) is in the starter project.
-
-
-## lono generate
-
-You can generate the CloudFormation templates by running:
-
-<pre>
-$ lono generate
-$ lono g -c # shortcut
-</pre>
-
-The lono init command also sets up guard-lono.  Guard-lono continuously generates the cloudformation templates.  Just run guard.
-
-<pre>
-$ guard
-</pre>
-
-# lono cfn
-
-## Summary
 Lono also provides a `lono cfn` management command that allows you to launch stacks from the lono templates.  The `lono cfn` tool automatically runs `lono generate` internally and then launches the CloudFormation stack all in one command.  Provided that you are in a lono project and have a `my-stack` lono template definition.  To create a stack you can simply run:
 
 ```
@@ -251,88 +64,7 @@ $ lono cfn delete mystack-1493859659
 $ lono cfn create -h # getting help
 ```
 
-### Conventions
-
-The template by convention defaults to the name of the stack.  In turn, the params by convention defaults to the name of the template.
-
-* stack - This is a required parameter and is the CLI first parameter.
-* template - By convention matches the stack name but can be overriden with `--template`.
-* params - By convention matches the template name but can be overriden with `--params`.
-
-The conventions allows the command to be a very nice short command that can be easily remembered.  For example, these 2 commands are the same:
-
-### lono cfn create
-
-Long form:
-
-```
-$ lono cfn create my-stack --template my-stack --params --my-stack
-```
-
-Short form:
-
-```
-$ lono cfn create my-stack
-```
-
-
-Both template and params conventions can be overridden.  Here are examples of overriding the template and params name conventions.
-
-```
-$ lono cfn create my-stack --template different-name1
-```
-
-The template that will be use is output/different-name1.json and the parameters will use params/different-name1.json.
-
-```
-$ lono cfn create my-stack --params different-name2
-```
-
-The template that will be use is output/different-name2.json and the parameters will use params/different-name2.json.
-
-```
-$ lono cfn create my-stack --template different-name3 --params different-name4
-```
-
-The template that will be use is output/different-name3.json and the parameters will use params/different-name4.json.
-
-### lono cfn update
-
-To update stacks you can use `lono cfn update`:
-
-```
-$ lono cfn update my-stack --template template-name --params params-name
-```
-
-By default the update command will display a preview of the stack changes before applying the update and prompt to check if you are sure.  If you want to bypass the are you sure prompt, use the `--sure` option.
-
-```
-$ lono cfn update my-stack --template template-name --params params-name --sure
-```
-
-### lono cfn delete
-
-```
-$ lono cfn delete my-stack --sure
-```
-
-### lono cfn preview
-
-If you want to see the CloudFormation preview without updating the stack you can also use the `lono cfn preview` command.  The preview command is also covered in this blog post: [AWS CloudFormation dry-run with lono cfn preview](https://medium.com/boltops/aws-cloudformation-dry-run-with-lono-cfn-plan-2a1e0f80d13c)
-
-```
-$ lono cfn preview example --template single_instance --params single_instance
-Using template: output/single_instance.yml
-Using parameters: params/single_instance.txt
-Generating CloudFormation templates:
-  ./output/single_instance.yml
-Params file generated for example at ./output/params/example.json
-Generating CloudFormation Change Set for preview.....
-CloudFormation preview for 'example' stack update. Changes:
-Remove AWS::Route53::RecordSet: DnsRecord testsubdomain.sub.tongueroo.com
-$
-```
-
+See [lono.cloud](http://lono.cloud) for full lono documentation.
 
 ## Contributing
 

data/lib/lono.rb

@@ -3,6 +3,8 @@ require 'yaml'
 require 'pp'
 require 'colorize'
 require 'fileutils'
+# http://guides.rubyonrails.org/active_support_core_extensions.html#inflections
+require 'active_support/core_ext/string'
 
 # vendor because need https://github.com/futurechimp/plissken/pull/6 to be merged
 $:.unshift(File.expand_path("../../vendor/plissken/lib", __FILE__))
@@ -11,6 +13,7 @@ require "plissken"
 $:.unshift(File.expand_path('../', __FILE__))
 module Lono
   autoload :VERSION, 'lono/version'
+  autoload :Env, 'lono/env'
   autoload :Help, 'lono/help'
   autoload :ProjectChecker, 'lono/project_checker'
   autoload :Command, 'lono/command'
@@ -20,4 +23,7 @@ module Lono
   autoload :Cfn, 'lono/cfn'
   autoload :Param, 'lono/param'
   autoload :Clean, 'lono/clean'
+  autoload :Settings, 'lono/settings'
 end
+
+Lono::Env.setup!

data/lib/lono/cfn.rb

@@ -21,8 +21,13 @@ class Lono::Cfn < Lono::Command
   class_option :template, desc: "override convention and specify the template file to use"
   class_option :param, desc: "override convention and specify the param file to use"
   class_option :lono, type: :boolean, desc: "invoke lono to generate CloudFormation templates", default: true
+  class_option :s3_upload, type: :boolean, desc: "uploads templates to s3 if s3.path detected", default: true
+  class_option :capabilities, type: :array, desc: "iam capabilities. Ex: CAPABILITY_IAM, CAPABILITY_NAMED_IAM"
+  class_option :iam, type: :boolean, desc: "Shortcut for common IAM capabilities: CAPABILITY_IAM, CAPABILITY_NAMED_IAM"
+  class_option :rollback, type: :boolean, desc: "rollback", default: true
 
   desc "create STACK", "create a CloudFormation stack"
+  option :randomize_stack_name, type: :boolean, desc: "tack on random string at the end of the stack name", default: nil
   long_desc Help.create
   def create(name)
     Create.new(name, options).run

data/lib/lono/cfn/base.rb

@@ -1,17 +1,18 @@
 require "lono"
-require "byebug"
 
 class Lono::Cfn::Base
   include Lono::Cfn::AwsServices
   include Lono::Cfn::Util
 
+  attr_reader :randomize_stack_name
   def initialize(stack_name, options={})
-    @stack_name = stack_name
+    @randomize_stack_name = options[:randomize_stack_name]
+    @stack_name = randomize(stack_name)
     @options = options
     @project_root = options[:project_root] || '.'
     Lono::ProjectChecker.check(@project_root) unless options[:lono] # already ran checker in lono generate
 
-    @template_name = options[:template] || @stack_name
+    @template_name = options[:template] || derandomize(@stack_name)
     @param_name = options[:param] || @template_name
     @template_path = get_source_path(@template_name, :template)
     @param_path = get_source_path(@param_name, :param)
@@ -21,13 +22,42 @@ class Lono::Cfn::Base
 
   def run
     params = generate_all
-    save_stack(params) # defined in the sub class
+    begin
+      save_stack(params) # defined in the sub class
+    rescue Aws::CloudFormation::Errors::InsufficientCapabilitiesException => e
+      capabilities = e.message.match(/\[(.*)\]/)[1]
+      confirm = prompt_for_iam(capabilities)
+      if confirm =~ /^y/
+        @options.merge!(capabilities: [capabilities])
+        puts "Re-running: #{command_with_iam(capabilities).colorize(:green)}"
+        retry
+      else
+        puts "Exited"
+        exit 1
+      end
+    end
+  end
+
+  def prompt_for_iam(capabilities)
+    puts "This stack will create IAM resources.  Please approve to run the command again with #{capabilities} capabilities."
+    puts "  #{command_with_iam(capabilities)}"
+
+    puts "Please confirm (y/n)"
+    confirm = $stdin.gets
+  end
+
+  def command_with_iam(capabilities)
+    "#{File.basename($0)} #{ARGV.join(' ')} --capabilities #{capabilities}"
   end
 
   def generate_all
-    generate_templates if @options[:lono]
+    if @options[:lono]
+      generate_templates
+      upload_templates if @options[:s3_upload] and !@options[:noop]
+    end
+    params = generate_params(mute: @options[:mute_params])
     check_for_errors
-    generate_params(mute: @options[:mute_params])
+    params
   end
 
   def generate_templates
@@ -37,6 +67,17 @@ class Lono::Cfn::Base
     ).run
   end
 
+  def upload_templates
+    # only upload templates if s3.path configured in settings
+    settings = Lono::Settings.new(@project_root)
+    return unless settings.s3_path
+
+    Lono::Template::Upload.new(
+      project_root: @project_root,
+      pretty: true
+    ).run
+  end
+
   def generate_params(options={})
     generator_options = {
       project_root: @project_root,
@@ -66,8 +107,13 @@ class Lono::Cfn::Base
     unless File.exist?(@template_path)
       warns << "Template file missing: could not find #{@template_path}"
     end
-    if @options[:param] && !File.exist?(@param_path)
-      warns << "Parameters file missing: could not find #{@param_path}"
+    # Examples:
+    #   @param_path = params/prod/ecs.txt
+    #              => output/params/prod/ecs.json
+    output_param_path = @param_path.sub(/\.txt/, '.json')
+    output_param_path = "#{@project_root}/output/#{output_param_path}"
+    if @options[:param] && !File.exist?(output_param_path)
+      warns << "Parameters file missing: could not find #{output_param_path}"
     end
     [errors, warns]
   end
@@ -92,7 +138,7 @@ class Lono::Cfn::Base
       format = detect_format
       "#{@project_root}/output/#{name}.#{format}"
     when :param
-      "#{@project_root}/params/#{name}.txt"
+      "#{@project_root}/params/#{LONO_ENV}/#{name}.txt"
     else
       raise "hell: dont come here"
     end
@@ -143,4 +189,35 @@ class Lono::Cfn::Base
   def quit(signal)
     exit signal
   end
+
+  # Do nothing unless in Create class
+  def randomize(stack_name)
+    stack_name
+  end
+
+  # Strip the random string at end of the template name
+  def derandomize(template_name)
+    if randomize_stack_name?
+      template_name.sub(/-(\w{3})/,'') # strip the random part
+    else
+      template_name
+    end
+  end
+
+  def randomize_stack_name?
+    if !randomize_stack_name.nil?
+      return randomize_stack_name # CLI option takes highest precedence
+    end
+
+    # otherwise use the settings preference
+    settings = Lono::Settings.new(@project_root)
+    settings.data['randomize_stack_name']
+  end
+
+  def capabilities
+    return @options[:capabilities] if @options[:capabilities]
+    if @options[:iam]
+      ["CAPABILITY_IAM", "CAPABILITY_NAMED_IAM"]
+    end
+  end
 end