sparkle_formation 0.4.0 → 1.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: f13e8abe07f3752130dc2655eba524647f390618
-  data.tar.gz: 874980adb868101566eb4756170280a5a1faec61
+  metadata.gz: 40ae36e7f30ce66c8f2866c246b873014effd6e1
+  data.tar.gz: c542ab9daceb4df19c8dfc4b50f1a1f8ac68090e
 SHA512:
-  metadata.gz: 5f1cb9bec94362ba83200fb0d9f47ce310799118d3903a565a9b1ac667bb28751a9fdfea0fb952fffce83e3eb6cfbd5af5120672ceb2332b5113547cacc3f724
-  data.tar.gz: 5b53d67cb905a968cab3932096e4603b324c68d133d76e66fea82982421b8c96654b4bf09df2db84149634981c14103c811972f44404e597d3e118d7e95e6971
+  metadata.gz: f75fa1dd08bc0438ffb2a92fa59a157c4d045a09423fd782ac33f92bdfaab3b10172b4697aa4bb31071d7901a7e99c86ea52789c238c3553b18b02278730e405
+  data.tar.gz: 2deadc43737f86a136dd7de4169d88cd7c5afefcf065685fb64c385b63835f379f6453c4d5a00993cd4f45aa50400b9fa99f6b98f64ba6fe14ec99c84df2b871

data/CHANGELOG.md

@@ -1,8 +1,13 @@
-## v0.4.0
-* Add support for compile time parameters
-* Backport SparkleStruct updates
-* Track parent stack within nested stacks
-* Fix nested stack check method which assumed resources hash
+## v1.0.0
+
+> NOTE: This is a major version release. It includes multiple
+> implementation updates that may cause breakage.
+
+* Add SparklePacks for isolation and distribution
+* Add SparkleFormation.component method for defining components
+* Support fully recursive nesting and parameter / output mapping
+* Support previous nesting style (shallow) and new style (deep)
+* Include support for in-line stack policy extraction
 
 ## v0.3.0
 * Update `or!` helper method to take multiple arguments

data/LICENSE

@@ -186,7 +186,7 @@ APPENDIX: How to apply the Apache License to your work.
    same "printed page" as the copyright notice for easier
    identification within third-party archives.
 
-Copyright [yyyy] [name of copyright owner]
+Copyright 2015 Chris Roberts
 
 Licensed under the Apache License, Version 2.0 (the "License");
 you may not use this file except in compliance with the License.

data/README.md

@@ -1,26 +1,40 @@
+![SparkleFormation](img/sparkle-formation.png)
+
 # SparkleFormation
 
-AWS CloudFormation template building tools for Ruby. Yay!
+Orchestration template building tools for Ruby.
 
 ## What's it do?
 
-Provides a very loose DSL to describe an AWS CloudFormation
-in Ruby.
+Provides a very loose DSL to describe orchestration API templates
+programmatically in Ruby.
 
 ## Is that it?
 
 Yes. Well, kinda. It also has some extra features, like defining
-components, dynamics, merging, AWS builtin function helpers, and
+building blocks to facilitate code reuse in template creation,
+helper functions for commonly generated data structures, builtin
+logic for handling template nesting, and most importantly:
 conjouring magic (to get unicorns).
 
-## Expanded User Docs
+## Documentation
+
+* [Library Documentation](https://sparkleformation.github.io/sparkle_formation)
+* [User Documentation](https://sparkleformation.github.io/sparkle_formation/UserDocs)
+
+## Overview
 
-New user documentation is now here! [User Documentation](http://sparkleformation.github.io/sparkle_formation/UserDocs/)
+Many template orchestration APIs accept serialized templates defining
+infrastructure resources and configurations. Interacting directly with
+these services via data serialization formats (JSON, YAML, etc) can be
+difficult for humans. SparkleFormation allows humans to programmatically
+define templates in Ruby. These are then exported into the desired
+serialization format, ready to send to the target orchestration API.
 
 ## What's it look like?
 
-Lets use one of the example CF templates that creates an EC2 instance. First
-we can just convert it into a single file (ec2_example.rb):
+Below is a simple example of an AWS CloudFormation template defined within
+SparkleFormation. It creates a single EC2 resource:
 
 ```ruby
 SparkleFormation.new('ec2_example') do
@@ -44,7 +58,7 @@ SparkleFormation.new('ec2_example') do
   dynamic!(:ec2_instance, :foobar) do
     properties do
       key_name ref!(:key_name)
-      image_id map!(:region_map, 'AWS::Region', :ami)
+      image_id map!(:region_map, region!, :ami)
       user_data base64!('80')
     end
   end
@@ -79,7 +93,7 @@ end
 ```
 
 And once compiled we get a nice Hash that we can then convert to JSON which
-is ready for AWS. To print:
+is ready to send to the AWS CloudFormation API:
 
 ```ruby
 require 'sparkle_formation'
@@ -92,184 +106,25 @@ puts JSON.pretty_generate(
 
 Easy!
 
-## Why not just write JSON?
-
-Because, who in their right mind would want to write all of that in JSON? Also,
-we can start applying some of the underlying features in `SparkleFormation` to
-make this easier to maintain.
-
-# Components
-
-Lets say we have a handful of CFN templates we want to maintain, and all of those
-templates use the same AMI. Instead of copying that information into all the
-templates, lets create an AMI component instead, and then load it into the actual
-templates.
-
-First, create the component (components/ami.rb):
-
-```ruby
-SparkleFormation.build do
-
-  mappings.region_map do
-    set!('us-east-1', :ami => 'ami-7f418316')
-    set!('us-east-1', :ami => 'ami-7f418316')
-    set!('us-west-1', :ami => 'ami-951945d0')
-    set!('us-west-2', :ami => 'ami-16fd7026')
-    set!('eu-west-1', :ami => 'ami-24506250')
-    set!('sa-east-1', :ami => 'ami-3e3be423')
-    set!('ap-southeast-1', :ami => 'ami-74dda626')
-    set!('ap-northeast-1', :ami => 'ami-dcfa4edd')
-  end
-
-end
-```
-
-Now, we can modify our initial example to use this component (ec2_example.rb):
-
-```ruby
-SparkleFormation.new('ec2_example').load(:ami).overrides do
-
-  description "AWS CloudFormation Sample Template ..."
-
-  parameters.key_name do
-    description 'Name of EC2 key pair'
-    type 'String'
-  end
-
-  dynamic!(:ec2_instance, :foobar) do
-    properties do
-      key_name ref!(:key_name)
-      image_id map!(:region_map, 'AWS::Region', :ami)
-      user_data base64!('80')
-    end
-  end
-
-  outputs do
-    instance_id do
-      description 'InstanceId of the newly created EC2 instance'
-      value ref!(:foobar_ec2_instance)
-    end
-    az do
-      description 'Availability Zone of the newly created EC2 instance'
-      value attr!(:foobar_ec2_instance, :availability_zone)
-    end
-    public_ip do
-      description 'Public IP address of the newly created EC2 instance'
-      value attr!(:foobar_ec2_instance, :public_ip)
-    end
-    private_ip do
-      description 'Private IP address of the newly created EC2 instance'
-      value attr!(:foobar_ec2_instance, :private_ip)
-    end
-    public_dns do
-      description 'Public DNSName of the newly created EC2 instance'
-      value attr!(:foobar_ec2_instance, :public_dns_name)
-    end
-    private_dns do
-      description 'Private DNSName of the newly created EC2 instance'
-      value attr!(:foobar_ec2_instance, :private_dns_name)
-    end
-  end
-end
-```
-
-Now a few things have changed. Instead of passing a block directly to the
-instance instantiation, we are loading a component (the `ami` component)
-into the formation, and then applying an override block on top of the `ami`
-component. The result is the same as the initial example, but now we have
-a DRY component to use. Great!
+## Reusability features
 
-## Dynamics
+SparkleFormation provides a number of features facilitating code reuse and
+logical structuring. These features help aid developers in applying DRY
+concepts to infrastructure codebases easing maintainability.
 
-Okay, so lets say we want to have two ec2 instances. We could duplicate the
-resource and outputs, renaming where required. This would get ugly quick,
-especially as more instances are added. Making a component for the ec2 resource
-won't really help since components are static, used to apply the same common
-parts to multiple templates. So what do we use?
+> [Learn more!](https://sparkleformation.github.io/sparkle_formation/UserDocs/building-blocks.html)
 
-Enter `dynamics`. These are much like components, except that instead of simply
-being merged, they allow passing of arguments which makes them reusable to create
-unique resources. So, from our last example, lets move the ec2 related items
-into a dynamic (dynamics/node.rb):
+## SparkleFormation Implementations
 
-```ruby
-SparkleFormation.dynamic(:node,
-  :parameters => {
-    :key_name => {
-      :type => 'String',
-      :description => 'Optionally make keypair static'
-    }
-  }
-) do |_name, _config|
-
-  if(_config[:key_name])
-    parameters.set!("#{_name}_key_name".to_sym) do
-      description 'Name of EC2 key pair'
-      type 'String'
-    end
-  end
-
-  dynamic!(:ec2_instance, _name) do
-    properties do
-      key_name _config.fetch(:key_name, ref!("#{_name}_key_name".to_sym))
-      image_id map!(:region_map, 'AWS::Region', :ami)
-      user_data baes64!('80')
-    end
-  end
-
-  outputs("#{_name}_instance_id".to_sym) do
-    description 'InstanceId of the newly created EC2 instance'
-    value ref!("#{_name}_ec2_instance".to_sym)
-  end
-  outputs("#{_name}_az".to_sym) do
-    description 'Availability Zone of the newly created EC2 instance'
-    value attr!("#{_name}_ec2_instance".to_sym, :availability_zone)
-  end
-  outputs("#{_name}_public_ip".to_sym) do
-    description 'Public IP address of the newly created EC2 instance'
-    value attr!("#{_name}_ec2_instance".to_sym, :public_ip)
-  end
-  outputs("#{_name}_private_ip".to_sym) do
-    description 'Private IP address of the newly created EC2 instance'
-    value attr!("#{_name}_ec2_instance".to_sym, :private_ip)
-  end
-  outputs("#{_name}_public_dns".to_sym) do
-    description 'Public DNSName of the newly created EC2 instance'
-    value attr!("#{_name}_ec2_instance".to_sym, :public_dns_name)
-  end
-  outputs("#{_name}_private_dns".to_sym) do
-    description 'Private DNSName of the newly created EC2 instance'
-    value attr!("#{_name}_ec2_instance".to_sym, :private_dns_name)
-  end
-end
-```
-
-Now we can put all of these together, and create multiple ec2 instance
-resource easily:
-
-```ruby
-SparkleFormation.new('ec2_example').load(:ami).overrides do
-
-  description "AWS CloudFormation Sample Template ..."
-
-  %w(node1 node2 node3).each do |_node_name|
-    dynamic!(:node, _node_name)
-  end
-
-  # and include one with predefined keypair
-
-  dynamic!(:node, 'snowflake', :key_pair => 'snowkeys')
-end
-```
+SparkleFormation itself is simply a library for building complex template
+documents in Ruby. It does not provide any integrations with remote API
+services. For interacting with remote services using the SparkleFormation
+library, see the SparkleFormation CLI application:
 
-## TODO
-* Add information about symbol importance
-* Add examples of camel case control
-* Add examples of complex merge strategies
-* Add examples of accessing parent hash elements
+* [SparkleFormation CLI (sfn)](https://github.com/sparkleformation/sfn)
 
 # Infos
 * Documentation: http://sparkleformation.github.io/sparkle_formation
 * User Documentation: http://sparkleformation.github.io/sparkle_formation/UserDocs/README.html
 * Repository: https://github.com/sparkleformation/sparkle_formation
-* IRC: Freenode @ #heavywater
+* IRC: Freenode @ #sparkleformation

data/lib/sparkle_formation/error.rb

@@ -0,0 +1,25 @@
+require 'sparkle_formation'
+
+class SparkleFormation
+
+  class Error < StandardError
+
+    class NotFound < KeyError
+      attr_reader :name
+
+      def initialize(*args)
+        opts = args.detect{|o| o.is_a?(Hash)}
+        args.delete(opts) if opts
+        super(args)
+        @name = opts[:name] if opts
+      end
+
+      class Dynamic < NotFound; end
+      class Component < NotFound; end
+      class Registry < NotFound; end
+      class Template < NotFound; end
+
+    end
+  end
+
+end

data/lib/sparkle_formation/sparkle.rb

@@ -0,0 +1,344 @@
+require 'sparkle_formation'
+
+class SparkleFormation
+  class Sparkle
+
+    class << self
+
+      @@_pack_registry = Smash.new
+
+      # Register a SparklePack for short name access
+      #
+      # @param name [String, Symbol] name of pack
+      # @param path [String] path to pack
+      # @return [Array<String:name, String:path>]
+      def register!(name=nil, path=nil)
+        unless(path)
+          idx = caller.index do |item|
+            item.end_with?("`register!'")
+          end
+          if(idx)
+            file = caller[idx.next].split(':', 2).first
+            path = File.join(File.dirname(file), 'sparkleformation')
+            unless(File.directory?(path))
+              path = nil
+            end
+            unless(name)
+              name = File.basename(caller[idx.next].split(':', 2).first)
+              name.sub!(File.extname(name), '')
+            end
+          end
+        end
+        unless(name)
+          if(path)
+            name = path.split(File::PATH_SEPARATOR)[-3].to_s
+          end
+        end
+        unless(path)
+          raise ArgumentError.new('No SparklePack path provided and failed to auto-detect!')
+        end
+        unless(name)
+          raise ArgumentError.new('No SparklePack name provided and failed to auto-detect!')
+        end
+        @@_pack_registry[name] = path
+        [name, path]
+      end
+
+      # Return the path to the SparkePack registered with the given
+      # name
+      #
+      # @param name [String, Symbol] name of pack
+      # @return [String] path
+      def path(name)
+        if(@@_pack_registry[name])
+          @@_pack_registry[name]
+        else
+          raise KeyError.new "No pack registered with requested name: #{name}!"
+        end
+      end
+
+    end
+
+    # Wrapper for evaluating sfn files to store within sparkle
+    # container and remove global application
+    def eval_wrapper
+      klass = Class.new(BasicObject)
+      klass.class_eval(<<-EOS
+        def require(*args)
+          ::Kernel.require *args
+        end
+
+        class SparkleFormation
+
+          attr_accessor :sparkle_path
+
+          class << self
+
+            def part_data(data=nil)
+              if(data)
+                @data = data
+              else
+                @data
+              end
+            end
+
+            def dynamic(name, args={}, &block)
+              part_data[:dynamic].push(
+                ::Smash.new(
+                  :name => name,
+                  :block => block,
+                  :args => Smash[
+                    args.map(&:to_a)
+                  ],
+                  :type => :dynamic
+                )
+              ).last
+            end
+
+            def build(&block)
+              part_data[:component].push(
+                ::Smash.new(
+                  :block => block,
+                  :type => :component
+                )
+              ).last
+            end
+
+            def component(name, &block)
+              part_data[:component].push(
+                ::Smash.new(
+                  :name => name,
+                  :block => block,
+                  :type => :component
+                )
+              ).last
+            end
+
+            def dynamic_info(*args)
+              Smash.new(:metadata => {}, :args => {})
+            end
+
+          end
+
+          def initialize(*args)
+            SparkleFormation.part_data[:template].push(
+              ::Smash.new(
+                :name => args.first
+              )
+            )
+            raise TypeError
+          end
+
+          class Registry
+
+            def self.register(name, &block)
+              SparkleFormation.part_data[:registry].push(
+                ::Smash.new(
+                  :name => name,
+                  :block => block,
+                  :type => :registry
+                )
+              ).last
+            end
+
+          end
+          SfnRegistry = Registry
+
+        end
+        ::Object.constants.each do |const|
+          unless(self.const_defined?(const))
+            next if const == :Config # prevent warning output
+            self.const_set(const, ::Object.const_get(const))
+          end
+        end
+
+        def part_data(arg)
+          SparkleFormation.part_data(arg)
+        end
+        EOS
+      )
+      klass
+    end
+
+    include Bogo::Memoization
+
+    # Valid directories from cwd to set as root
+    VALID_ROOT_DIRS = [
+      'sparkleformation',
+      'sfn',
+      'cloudformation',
+      'cfn',
+      '.'
+    ]
+
+    # Reserved directories
+    DIRS = [
+      'components',
+      'registry',
+      'dynamics'
+    ]
+
+    # Valid types
+    TYPES = Smash.new(
+      'component' => 'components',
+      'registry' => 'registries',
+      'dynamic' => 'dynamics',
+      'template' => 'templates'
+    )
+
+    # @return [String] path to sparkle directories
+    attr_reader :root
+    # @return [Smash] raw part data
+    attr_reader :raw_data
+
+    # Create new sparkle instance
+    #
+    # @param args [Hash]
+    # @option args [String] :root path to sparkle directories
+    # @option args [String, Symbol] :name registered pack name
+    # @return [self]
+    def initialize(args={})
+      if(args[:name])
+        @root = self.class.path(args[:name])
+      else
+        @root = args.fetch(:root, locate_root)
+      end
+      unless(File.directory?(@root))
+        raise Errno::ENOENT.new("No such directory - #{@root}")
+      end
+      @raw_data = Smash.new(
+        :dynamic => [],
+        :component => [],
+        :registry => []
+      )
+      @wrapper = eval_wrapper.new
+      wrapper.part_data(raw_data)
+      load_parts!
+    end
+
+    # @return [Smash<name:block>]
+    def components
+      memoize(:components) do
+        Smash.new
+      end
+    end
+
+    # @return [Smash<name:block>]
+    def dynamics
+      memoize(:dynamics) do
+        Smash.new
+      end
+    end
+
+    # @return [Smash<name:block>]
+    def registries
+      memoize(:registries) do
+        Smash.new
+      end
+    end
+
+    # @return [Smash<name:path>]
+    def templates
+      memoize(:templates) do
+        Smash.new.tap do |hash|
+          Dir.glob(File.join(root, '**', '**', '*.{json,rb}')) do |path|
+            slim_path = path.sub("#{root}/", '')
+            next if DIRS.include?(slim_path.split('/').first)
+            data = Smash.new(:template => [])
+            t_wrap = eval_wrapper.new
+            t_wrap.part_data(data)
+            begin
+              t_wrap.instance_eval(IO.read(path), path, 1)
+            rescue TypeError
+            end
+            data = data[:template].first
+            unless(data[:name])
+              data[:name] = slim_path.tr('/', '__')
+            end
+            hash[data[:name]] = data.merge(
+              Smash.new(
+                :type => :template,
+                :path => path
+              )
+            )
+          end
+        end
+      end
+    end
+
+    # Request item from the store
+    #
+    # @param type [String, Symbol] item type (see: TYPES)
+    # @param name [String, Symbol] name of item
+    # @return [Smash] requested item
+    # @raises [NameError, Error::NotFound]
+    def get(type, name)
+      unless(TYPES.keys.include?(type.to_s))
+        raise NameError.new "Invalid type requested (#{type})! Valid types: #{TYPES.join(', ')}"
+      end
+      result = send(TYPES[type])[name]
+      if(result.nil? && TYPES[type] == 'templates')
+        result = (
+          send(TYPES[type]).detect{|k,v|
+            name = name.to_s
+            short_name = v[:path].sub(/#{Regexp.escape(root)}\/?/, '')
+            v[:path] == name ||
+            short_name == name ||
+            short_name.sub('.rb', '').gsub(File::SEPARATOR, '__').tr('-', '_') == name ||
+            v[:path].end_with?(name)
+          } || []
+        ).last
+      end
+      unless(result)
+        klass = Error::NotFound.const_get(type.capitalize)
+        raise klass.new("No #{type} registered with requested name (#{name})!", :name => name)
+      end
+      result
+    end
+
+    # @return [String]
+    def inspect
+      "<SparkleFormation::Sparkle [root: #{root.inspect}]>"
+    end
+
+    private
+
+    attr_reader :wrapper
+
+    # Locate root directory. Defaults to current working directory if
+    # valid sub directory is not located
+    #
+    # @return [String] root path
+    def locate_root
+      VALID_ROOT_DIRS.map do |part|
+        path = File.expand_path(File.join(Dir.pwd, part))
+        if(File.exists?(path))
+          path
+        end
+      end.compact.first
+    end
+
+    # Load all sparkle parts
+    def load_parts!
+      memoize(:load_parts) do
+        Dir.glob(File.join(root, "{#{DIRS.join(',')}}", '*.rb')).each do |file|
+          wrapper.instance_eval(IO.read(file), file, 1)
+        end
+        raw_data.each do |key, items|
+          items.each do |item|
+            if(item[:name])
+              send(TYPES[key])[item.delete(:name)] = item
+            else
+              path = item[:block].source_location.first.sub('.rb', '').split(File::SEPARATOR)
+              type, name = path.slice(path.size - 2, 2)
+              send(type)[name] = item
+            end
+          end
+        end
+      end
+    end
+
+  end
+  # Alias for interfacing naming
+  SparklePack = Sparkle
+end