lono 0.5.2 → 1.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 95168f2761fc2cc741be8c7bd629397cda724e63
-  data.tar.gz: 4c1deb7dff498115fdaa348e139e9cab1368c9aa
+  metadata.gz: 6375e08e65be7c8a3a1eec9356454c1975a40400
+  data.tar.gz: 753b81d7bc5b3d1b5dd6fab7ed01858db6aeb7e0
 SHA512:
-  metadata.gz: f374bb0ef8c7bad94927613cd0497a7029ec607273132aeaa6a4a8829ef8a888354fba66cb9135a3dea133089de647c9f86135deec7f2cc463f4b6e085d00bdd
-  data.tar.gz: c1d292f6a35608743516b947ddd0763d98f07ed3554b91ab2b602bb9c9ec7609515efab2ced0c08c1a0a14d9ba55c1125d8939a631e3d6b26a03cb07c729978a
+  metadata.gz: c1e1f22813a7e5ab6444aafc53d21fc6dfec39b71ea2992d027e832429283a196360a4e9dabbc839b9b1d7e24800c322ffb49df9627662e6687e8c543008d2c7
+  data.tar.gz: 05f0f728a8e87110ee5641ce3f0da27ebbcb12719e3b619eccb2dc634b9df156536ebbccc1a95581be1f300280a291c3d332f23cc4bb3a2e3abb5e387926ff5a

data/.ruby-version

@@ -1 +1 @@
-2.3.1
+2.3.3

data/CHANGELOG.md

@@ -3,6 +3,11 @@
 All notable changes to this project will be documented in this file.
 This project *tries* to adhere to [Semantic Versioning](http://semver.org/), even before v1.0.
 
+## [1.0.0]
+- Yaml support added!  Makes for much more clean and concise templates.  The `lono new` command defaults to yaml format.
+- The starter project is app centric instead of env centric.  Example: blog-web-prod vs prod-blog-web.
+
+
 ## [0.5.2]
 - Add helper encode_base64 method in case you want to base64 encode a string in the ERB template and you are using lono outside of the context of CloudFormation where you will not have access to the FN::Base64 Function.
 

data/README.md

@@ -12,7 +12,7 @@
 [3]: https://codeclimate.com/repos/51d7f1407e00a4042c010ab4/badges/5273fe6cdb5a13e58554/gpa.png
 [4]: https://codeclimate.com/repos/51d7f1407e00a4042c010ab4/feed
 
-Lono is a CloudFormation Template ruby generator.  Lono generates CloudFormation templates based on ERB templates.
+Lono is a CloudFormation Template generator.  Lono generates CloudFormation templates based on ERB ruby templates.
 
 ## Usage
 
@@ -27,14 +27,14 @@ $ cd infra
 $ lono generate
 </pre>
 
-This generates the templates that have been defined in the config folder of the lono project to the output folder.
+This generates the templates in the `config` and `templates` folders to the `output` folder.
 
 The starter lono template project config files looks like [this](lib/starter_project/config/lono.rb) and [this](lib/starter_project/config/lono/api.rb).  Here's a snippet from one of the config files with the template call:
 
 ```ruby
-template "prod-api-app.json" do
-  env,app,role = name.sub('.json','').split('-')
-  source "app.json.erb"
+template "api-web-prod.yml" do
+  app,role,env = name.sub('.yml','').split('-')
+  source "web.yml.erb"
   variables(
     env: env,
     app: app,
@@ -55,20 +55,122 @@ template "prod-api-app.json" do
 end
 ```
 
-The corresponding ERB template example file is [here](lib/starter_project/templates/app.json.erb).
+The corresponding ERB template looks like the following.  Note that some of the output has been shorten for brevity.
+
+```yaml
+<% @app,@role,@env = name.sub('.yml','').split('-') %>
+---
+AWSTemplateFormatVersion: '2010-09-09'
+Description: <%= @app.capitalize %> Stack
+Mappings:
+...
+Outputs:
+...
+Parameters:
+  Application:
+    Default: <%= @app %>
+    Description: Application name
+    Type: String
+...
+Resources:
+  CPUAlarmHigh:
+    Properties:
+      AlarmActions:
+      - Ref: WebServerScaleUpPolicy
+      AlarmDescription: Scale-up if CPU > <%= @high_threshold %>% for <%= @high_mins %>
+...
+<%= partial("host_record.yml.erb", domain: "mydomain.com") %>
+  LaunchConfig:
+    Properties:
+      BlockDeviceMappings:
+      - DeviceName: "/dev/sdb"
+        VirtualName: ephemeral0
+      ImageId:
+...
+      UserData:
+        Fn::Base64: !Sub | # No more Fn::Join needed
+          #!/bin/bash -lexv
+          <% stack_name = "#{@env}-#{@app}-#{@role}" %>
+          exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
+          echo <%= stack_name %> > /tmp/stack_name
+          cat /proc/uptime | cut -f1 -d'.' > /tmp/time-to-boot
+    Type: AWS::AutoScaling::LaunchConfiguration
+
+```
+
+The output looks like this:
+
+```yaml
+---
+AWSTemplateFormatVersion: '2010-09-09'
+Description: Api Stack
+Mappings:
+...
+Outputs:
+...
+Parameters:
+  Application:
+    Default: api
+    Description: Application name
+    Type: String
+...
+Resources:
+  CPUAlarmHigh:
+    Properties:
+      AlarmActions:
+      - Ref: WebServerScaleUpPolicy
+      AlarmDescription: Scale-up if CPU > 15% for
+...
+  HostRecord:
+    Properties:
+      Comment: DNS name for mydomain.com
+      HostedZoneName: ".mydomain.net."
+      Name:
+        Fn::Join:
+        - ''
+        - - Ref: AWS::StackName
+          - mydomain.com
+      ResourceRecords:
+      - Fn::GetAtt:
+        - elb
+        - DNSName
+      TTL: '60'
+      Type: CNAME
+    Type: AWS::Route53::RecordSet
+  LaunchConfig:
+    Properties:
+      BlockDeviceMappings:
+      - DeviceName: "/dev/sdb"
+        VirtualName: ephemeral0
+      ImageId:
+...
+      UserData:
+        Fn::Base64: |
+          #!/bin/bash -lexv
+
+          exec > >(tee /var/log/user-data.log|logger -t user-data -s 2>/dev/console) 2>&1
+          echo api-web-prod > /tmp/stack_name
+          cat /proc/uptime | cut -f1 -d'.' > /tmp/time-to-boot
+    Type: AWS::AutoScaling::LaunchConfiguration
+```
 
 ## Template helper methods
 
 There are helper methods that are available in templates.
 
-* partial - can be use to embed other files in a template.  The partial should be placed in the templates/partial folder of the project.  So:
-  * partial('launch_config.json.erb') -> templates/partial/launch_config.json.erb
-  * partial('launch_config.json.erb', foo: "bar", hello: "world") - variables can be passed to the partial helper method and will be available to the partial as instance variables.  So, in this case @foo and @hello will be available in the launch_config.json.erb partial.
+* partial(relative_path, variables, options) - can be use to embed other files in a template.  The partial view should be placed in the `templates/partial` folder of the project.  So:
+  * partial('launch\_config.json.erb') -> `templates/partial/launch_config.json.erb`
+  * partial('launch\_config.json.erb', foo: "bar", hello: "world") - variables can be passed to the partial helper method are available to the partial as instance variables.  So, in this case `@foo` and `@hello` will be available in the `launch_config.json.erb` partial.
+  * partial('user_data/bootstrap.sh.erb', {}, indent: 10) - Indent the result partial by 10 spaces.  Useful for yaml format.
+
+## user\_data helper for json format
+
+The user\_data helper method is helpful for writting a script in bash form and having lono convert it to a json compatiable format. It is only really useful if you are using json as the CloudFormation format.  If you are using yaml as the format, which is recommended, then you should simply use raw yaml.
 
-* user_data - can be used to include a user data script which is written in bash script form.  The user data script should be placed in the templates/user_data folder of the project.  So:
-  * user_data('bootstrap.sh.erb') -> templates/user_data/bootstrap.sh.erb
-  * user_data('db.sh.erb') -> templates/user_data/db.sh.erb
-  * user_data('script1.sh.erb', foo: "bar", hello: "world") - variables can be passed to the user_data helper method and will be available to the partial as instance variables.  So, in this case @foo and @hello will be available in script1.sh.erb.
+* user\_data - can be used to include a user data script which is written in bash script form.  The user data script should be placed in the `templates/user_data` folder of the project.  So:
+  * user\_data('bootstrap.sh.erb') -> templates/user\_data/bootstrap.sh.erb
+  * user\_data('db.sh.erb') -> templates/user\_data/db.sh.erb
+  * user\_data('script1.sh.erb', foo: "bar", hello: "world") - variables can be passed to the user\_data helper method and will be available to the partial as instance variables.  So, in this case `@foo` and `@hello` will be available in `script1.sh.erb`.
 
 Here's how you would call it in the template.
 
@@ -84,9 +186,9 @@ Here's how you would call it in the template.
   }
 ```
 
-Within the user_data script you can use helper methods that correspond to CloudFormation [Instrinic Functions](http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/concept-intrinsic-functions.html).  Currently, base64, find_in_map, get_att, get_azs, join, and ref are supported.  Here's a short example of a user_data script using a helper method:
+Within the user\_data script you can use helper methods that correspond to CloudFormation [Instrinic Functions](http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/concept-intrinsic-functions.html).  Currently, `base64`, `find_in_map`, `get_att`, `get_azs`, `join`, and `ref` are supported.  Here's a short example of a user\_data script using a helper method:
 
-If you have a templates/user_data/db.sh.erb that looks like this:
+If you have a `templates/user_data/db.sh.erb` that looks like this:
 
 ```bash
 #!/bin/bash -lexv
@@ -118,7 +220,7 @@ chown -R redis:redis /media/redis
 /usr/local/bin/cfn-signal -e $? -r "Ready to rock" '<%= ref("WaitHandle") %>'
 ```
 
-The user_data helper will transform the bash script into a json array of elements for CloudFormation:
+The user\_data helper will transform the bash script into a json array of elements for CloudFormation:
 
 ```json
 [
@@ -174,7 +276,7 @@ The user_data helper will transform the bash script into a json array of element
 ]
 ```
 
-More examples of user_data and instrinic function helper method usage are found in the starter [project template](https://github.com/tongueroo/lono/blob/master/lib/starter_project/templates/user_data/db.sh.erb)
+More examples of user\_data and instrinic function helper method usage are found in the starter [project template](https://github.com/tongueroo/lono/blob/master/lib/starter_project_json/templates/user_data/db.sh.erb)
 
 ## Converting UserData scripts
 
@@ -186,11 +288,11 @@ $ lono bash cloud_formation_template.json # shorthand
 $ lono b https://s3.amazonaws.com/cloudformation-templates-us-east-1/LAMP_Single_Instance.template # shorthand and url
 </pre>
 
-This is useful if you want to take an existing [CloudFormation Template example](http://aws.amazon.com/cloudformation/aws-cloudformation-templates/) and quicklly change the UserData section into a bash script. The bashify command will generate a snippet that is meant to be copied and pasted into a bash script and used with user_data helper method.  The bash script should work right off the bat as lono will transform the generated CloudFormation object references to json objects, there's no need to manually change what is generated to the helper methods, though you can if you prefer the look of the helper methods.
+This is useful if you want to take an existing json [CloudFormation Template example](http://aws.amazon.com/cloudformation/aws-cloudformation-templates/) and quicklly change the UserData section into a bash script. The bashify command will generate a snippet that is meant to be copied and pasted into a bash script and used with user\_data helper method.  The bash script should work right off the bat as lono will transform the generated CloudFormation object references to json objects, there's no need to manually change what is generated to the helper methods, though you can if you prefer the look of the helper methods.
 
 ## Breaking up config/lono.rb
 
-If you have a lot of templates, the config/lono.rb file can get unwieldy long.  You can break up the lono.rb file and put template defintions in the config/lono directory.  Any file in this directory will be automatically loaded. An [example](lib/starter_project/config/lono/api.rb) is in the starter project.
+If you have a lot of templates, the config/lono.rb file can get unwieldy long.  You can break up the lono.rb file and put template defintions in the config/lono directory.  Any file in this directory will be automatically loaded. An [example](lib/starter_project_yaml/config/lono/api.rb) is in the starter project.
 
 
 ## Generate
@@ -208,10 +310,37 @@ The lono init command also sets up guard-lono.  Guard-lono continuously generate
 $ guard
 </pre>
 
-## Extras
+## lono-cfn and lono-params
 
-There are some helper lono gems also useful for working with CloudFormation:
+Running `lono generate` and building up the `aws cloudformation create-stack` command repeatedly gets old. The `lono-cfn` tool will automatically run `lono generate` and then launch the CloudFormation stack all in one command.  Example usage:
 
-* [lono-cfn](https://github.com/tongueroo/lono-cfn) - Wrapper cfn tool to quickly create CloudFormation stacks from lono templates and params files.
-* [lono-params](https://github.com/tongueroo/lono-params) - Tool to generate a CloudFormation params json formatted file from a simplier env like file.
+```
+$ lono-cfn create mystack-$(date +%Y%m%d%H%M%S) --template mystack --params mystack
+$ lono-cfn create mystack-$(date +%Y%m%d%H%M%S) # shorthand if template and params file matches.
+```
+
+More info about lono-cfn here: [lono-cfn](https://github.com/tongueroo/lono-cfn) - Wrapper cfn tool to quickly create CloudFormation stacks from lono templates and params files.
+
+The params file is formatted with a simple `key=value`, env-like file.  It is cleaner to have a `params/mystack.txt` file like so:
+
+```bash
+Param1=1
+Param2=2
+```
+
+Verus the rather verbose standard CloudFormation parameters json file:
+
+```json
+[
+  {
+    "ParameterKey": "Param1",
+    "ParameterValue": "1"
+  },
+  {
+    "ParameterKey": "Param2",
+    "ParameterValue": "2"
+  }
+]
+```
 
+More info about lono-parmas here: [lono-params](https://github.com/tongueroo/lono-params) - Tool to generate a CloudFormation parameters json formatted file from a simplier env like file.

data/lib/lono.rb

@@ -1,4 +1,5 @@
 require 'json'
+require 'yaml'
 require 'pp'
 require 'colorize'
 require 'fileutils'

data/lib/lono/cli.rb

@@ -8,6 +8,7 @@ module Lono
     Help.new_long_desc
     option :force, type: :boolean, aliases: "-f", desc: "override existing starter files"
     option :quiet, type: :boolean, aliases: "-q", desc: "silence the output"
+    option :format, type: :string, default: "yaml", desc: "starter project template format: json or yaml"
     def new(project_root)
       Lono::New.new(options.clone.merge(project_root: project_root)).run
     end
@@ -17,7 +18,7 @@ module Lono
     option :clean, type: :boolean, aliases: "-c", desc: "remove all output files before generating"
     option :project_root, default: ".", aliases: "-r", desc: "project root"
     option :quiet, type: :boolean, aliases: "-q", desc: "silence the output"
-    option :pretty, type: :boolean, default: true, desc: "json pretty the output"
+    option :pretty, type: :boolean, default: true, desc: "json pretty the output.  only applies with json format"
     def generate
       Lono::DSL.new(options.clone).run
     end
@@ -33,5 +34,5 @@ module Lono
       puts Lono::VERSION
     end
 
-  end  
+  end
 end

data/lib/lono/dsl.rb

@@ -6,11 +6,38 @@ module Lono
       @path = "#{@options[:project_root]}/config/lono.rb"
       @templates = []
       @results = {}
+      @detected_format = nil
+    end
+
+    def run(options={})
+      evaluate
+      build
+      output
     end
 
     def evaluate
       instance_eval(File.read(@path), @path)
       load_subfolder
+      @detected_format = detect_format
+    end
+
+    # Detects the format of the templates.  Simply checks the extension of all the
+    # templates files.
+    # All the templates must be of the same format, either all json or all yaml.
+    def detect_format
+      # @templates contains Array of Hashes. Example:
+      # [{name: ""blog-web-prod.json", block: ...},
+      #  {name: ""api-web-prod.json", block: ...}]
+      formats = @templates.map{ |t| File.extname(t[:name]) }.uniq
+      if formats.size > 1
+        puts "ERROR: Detected multiple formats: #{formats.join(", ")}".colorize(:red)
+        puts "All the source values in the template blocks in the config folder must have the same format extension."
+        exit 1
+      else
+        detected_format = formats.first.sub(/^\./,'')
+        detected_format = "yaml" if detected_format == "yml"
+      end
+      detected_format
     end
 
     # load any templates defined in project/config/lono/*
@@ -35,40 +62,60 @@ module Lono
       FileUtils.rm_rf(output_path) if @options[:clean]
       FileUtils.mkdir(output_path) unless File.exist?(output_path)
       puts "Generating CloudFormation templates:" unless @options[:quiet]
-      @results.each do |name,json|
+      @results.each do |name,text|
         path = "#{output_path}/#{name}"
         puts "  #{path}" unless @options[:quiet]
         ensure_parent_dir(path)
-        validate(json, path)
+        validate(text, path)
         File.open(path, 'w') do |f|
-          f.write(output_json(json))
+          f.write(output_format(text))
         end
       end
     end
 
-    def validate(json, path)
+    # TODO: set @detected_format upon DSL.new
+    def validate(text, path)
+      if @detected_format == "json"
+        validate_json(text, path)
+      else
+        validate_yaml(text, path)
+      end
+    end
+
+    def validate_yaml(yaml, path)
+      begin
+        YAML.load(yaml)
+      rescue Psych::SyntaxError => e
+        puts "Invalid yaml.  Output written to #{path} for debugging".colorize(:red)
+        puts "ERROR: #{e.message}".colorize(:red)
+        File.open(path, 'w') {|f| f.write(yaml) }
+        exit 1
+      end
+    end
+
+    def validate_json(json, path)
       begin
         JSON.parse(json)
       rescue JSON::ParserError => e
         puts "Invalid json.  Output written to #{path} for debugging".colorize(:red)
+        puts "ERROR: #{e.message}".colorize(:red)
         File.open(path, 'w') {|f| f.write(json) }
         exit 1
       end
     end
 
-    def output_json(json)
-      @options[:pretty] ? JSON.pretty_generate(JSON.parse(json)) : json
+    def output_format(text)
+      @options[:pretty] ? prettify(text) : text
+    end
+
+    # do not prettify yaml format because it removes the !Ref like CloudFormation notation
+    def prettify(text)
+      @detected_format == "json" ? JSON.pretty_generate(JSON.parse(text)) : text
     end
 
     def ensure_parent_dir(path)
       dir = File.dirname(path)
       FileUtils.mkdir_p(dir) unless File.exist?(dir)
     end
-
-    def run(options={})
-      evaluate
-      build
-      output
-    end
   end
 end

data/lib/lono/new.rb

@@ -4,15 +4,17 @@ module Lono
     def initialize(options)
       @options = options
       @project_root = options[:project_root] || '.'
+      @format = options[:format] || 'json'
     end
 
     def run
       puts "Setting up lono project" unless options[:quiet]
-      source_root = File.expand_path("../../starter_project", __FILE__)
+      source_root = File.expand_path("../../starter_project_#{@format}", __FILE__)
       paths = Dir.glob("#{source_root}/**/*").
                 select {|p| File.file?(p) }
       paths.each do |src|
-        dest = src.gsub(%r{.*starter_project/},'')
+        regexp = Regexp.new(".*starter_project_#{@format}/")
+        dest = src.gsub(regexp,'')
         dest = "#{@project_root}/#{dest}"
 
         if File.exist?(dest) and !options[:force]

data/lib/lono/template.rb

@@ -16,7 +16,7 @@ module Lono
     def build
       instance_eval(&@block)
       template = IO.read(@source)
-      ERB.new(template).result(binding)
+      erb_result(template)
     end
 
     def source(path)
@@ -29,18 +29,31 @@ module Lono
       end
     end
 
-    def partial(path,vars={})
+    def partial(path,vars={}, options={})
       path = "#{@options[:project_root]}/templates/partial/#{path}"
       template = IO.read(path)
       variables(vars)
-      ERB.new(template).result(binding)
+      result = erb_result(template)
+      result = indent(result, options[:indent]) if options[:indent]
+      result
+    end
+
+    # add indentation
+    def indent(result, indentation_amount)
+      result.split("\n").map do |line|
+        " " * indentation_amount + line
+      end.join("\n")
+    end
+
+    def erb_result(template)
+      ERB.new(template, nil, "-").result(binding)
     end
 
     def user_data(path, vars={})
       path = "#{@options[:project_root]}/templates/user_data/#{path}"
       template = IO.read(path)
       variables(vars)
-      result = ERB.new(template).result(binding)
+      result = erb_result(template)
       output = []
       result.split("\n").each do |line|
         output += transform(line)
@@ -87,7 +100,7 @@ module Lono
     def transform(data)
       data = evaluate(data)
       if data[-1].is_a?(String)
-        data[0..-2] + ["#{data[-1]}\n"] 
+        data[0..-2] + ["#{data[-1]}\n"]
       else
         data + ["\n"]
       end
@@ -98,7 +111,7 @@ module Lono
     # Output:
     #   Array of parse positions
     #
-    # The positions of tokens taking into account when brackets start and close, 
+    # The positions of tokens taking into account when brackets start and close,
     # handles nested brackets.
     def bracket_positions(line)
       positions,pair,count = [],[],0
@@ -106,7 +119,7 @@ module Lono
       line.split('').each_with_index do |char,i|
         pair << i if pair.empty?
 
-        first_pair_char = line[pair[0]]  
+        first_pair_char = line[pair[0]]
         if first_pair_char == '{' # object logic
           if char == '{'
             count += 1
@@ -203,8 +216,8 @@ module Lono
       recompose(decompose(line))
     end
 
-    # For simple just parameters files that can also be generated with lono, the CFN 
-    # Fn::Base64 function is not available and as lono is not being used in the context 
+    # For simple just parameters files that can also be generated with lono, the CFN
+    # Fn::Base64 function is not available and as lono is not being used in the context
     # of CloudFormation.  So this can be used in it's place.
     def encode_base64(text)
       Base64.strict_encode64(text).strip