sf-hiera-aws 0.0.8 → 0.0.9

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 1afaefcd378030dc0861d15645e7f4500fcc2a15
-  data.tar.gz: bad9504fa7b8943bef3ce955f2b1e8c6cfc136ef
+  metadata.gz: 5b0f6bfe5a16cc429253eb75464f79de6cbbf1d3
+  data.tar.gz: 13f8fe61331c0a2b7399d6eb8fde29a70e07617e
 SHA512:
-  metadata.gz: 696ac0f3324b330ccd5d9d8302f779a679a86bad0ea2a1d86beaeefe39486d79619589937c7d58da9ffde3b7c450d5ff3082bd25e292f9e4e3ad5fc06692157f
-  data.tar.gz: 23ae91964ed63a0c24a25970fb08bf7bb13e66493535d3ca9284a272645feb6259540d96671df43d177348eb7c2e94502cdb0e6093207397970726c866512331
+  metadata.gz: 7db30954f794f1a0af1ac9484eb415f7813a902689465032200a65fcb745512935e35d779675547f27f4bed146fa0fd6f2a64565db8afeaea90545bf4e1f1741
+  data.tar.gz: 9fe5adbb420f3e17995943c71f5bd32497f00e33932775d70f69c098f98f1a346b7b9a3fc9d90b3f0b864b1ac11fec3ba8a8d44a8168accecec2f4beee26184a

data/CHANGELOG.md

@@ -1,5 +1,9 @@
 # sf-hiera-aws changelog
 
+## 0.0.9 (23 January 2017)
+
+ * Add autoscaling status information to EC2 instance
+
 ## 0.0.8 (11 January 2017)
 
  * Add handling of JSON Parse failures that occur behind squid when not in AWS.

data/README.md

@@ -2,7 +2,10 @@
 
 ## About
 
-This is a Hiera backend to provide access to the EC2 API for a small number of resource types. Its purpose is to prevent it from ever being necessary to copy and paste EC2, RDS and ElastiCache addresses from the AWS console into Puppet configs anywhere.
+This is a Hiera backend to provide access to the EC2 API for a small number of
+resource types. Its purpose is to prevent it from ever being necessary to copy
+and paste EC2, RDS, AutoScaling Instance members, and ElastiCache addresses from
+the AWS console into Puppet configs anywhere.
 
 ## Usage and Setup
 
@@ -14,9 +17,12 @@ To add this backend to hiera, edit `/etc/puppet/hiera.yaml`:
   - sf_hiera_aws
 ```
 
-This plugin will attempt to use a machine's IAM role to perform AWS lookups - this is the recommended method of operation. 
+This plugin will attempt to use a machine's IAM role to perform AWS lookups -
+this is the recommended method of operation. 
 
-Absent an IAM role, the plugin will fall back to looking up credentials in the environment. Use `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` and `AWS_REGION` variables.
+Absent an IAM role, the plugin will fall back to looking up credentials in the
+environment. Use `AWS_ACCESS_KEY_ID`, `AWS_SECRET_ACCESS_KEY` and `AWS_REGION`
+variables.
 
 The IAM role will need the following permissions:
 
@@ -28,7 +34,8 @@ The IAM role will need the following permissions:
                 "Action": [
                     "ec2:DescribeInstances",
                     "rds:DescribeDBInstances",
-                    "elasticache:DescribeCacheClusters"
+                    "elasticache:DescribeCacheClusters",
+                    "autoscaling:DescribeAutoScalingGroups"
                 ],
                 "Effect": "Allow",
                 "Resource": [
@@ -41,8 +48,15 @@ The IAM role will need the following permissions:
 
 ## Configuration
 
-The plugin expects to find a configuration file under `/etc/puppet/sf_hiera_aws.yaml`, defining how we look up named keys.  The keys at the top level of this file determine the names of the hiera keys the plugin will provide; the configuration determines how these are looked up.
-Additional configuration can be given in files under `/etc/puppet/sf_hiera_aws.d`, which are evaluated in alphanumerical order. If a duplicate key is encountered in files evaluated later, this will override the earlier config.
+The plugin expects to find a configuration file under
+`/etc/puppet/sf_hiera_aws.yaml`, defining how we look up named keys.  The keys
+at the top level of this file determine the names of the hiera keys the plugin
+will provide; the configuration determines how these are looked up.
+
+Additional configuration can be given in files under
+`/etc/puppet/sf_hiera_aws.d`, which are evaluated in alphanumerical order. If a
+duplicate key is encountered in files evaluated later, this will override the
+earlier config.
 
 ### Example - EC2 nodes by tag
 
@@ -60,9 +74,16 @@ aws_am_search_nodes:
     - :private_dns_name
 ```
 
-The value of `return` here is also the default, and so can be omitted. You can use any of the methods listed at http://docs.aws.amazon.com/sdkforruby/api/Aws/EC2/Instance.html to obtain other details from the Instance object.  Calls to this key will return a list of hashes, each containing `instace_id`, `private_ip_address` and `private_dns_name` keys.
+The value of `return` here is also the default, and so can be omitted. You can
+use any of the methods listed at
+http://docs.aws.amazon.com/sdkforruby/api/Aws/EC2/Instance.html to obtain other
+details from the Instance object.  Calls to this key will return a list of
+hashes, each containing `instace_id`, `private_ip_address` and
+`private_dns_name` keys.
 
-Note that by default all EC2 instances will be returned, including stopped instances. To return only the running instances, add a filter as shown in this example.
+Note that by default all EC2 instances will be returned, including stopped
+instances. To return only the running instances, add a filter as shown in this
+example.
 
 ### Example - EC2 nodes by tag, single item list
 
@@ -75,7 +96,9 @@ aws_am_search_nodes:
   return: :private_ip_address
 ```
 
-Here, we pass a single symbol to the `return` argument.  In this case, we'll get back a list of strings containing private ip addresses (rather than a list of hashes).
+Here, we pass a single symbol to the `return` argument.  In this case, we'll get
+back a list of strings containing private ip addresses (rather than a list of
+hashes).
 
 
 
@@ -87,9 +110,14 @@ aws_am_bullseye_rds:
   db_instance_identifier: "%{::sf_location}-%{::sf_environment}-db"
 ```
 
-Calls to `:rds_db_instance` type keys return the instance identifier, endpoint address and endpoint port in a hash.
-Pass a `return` key with value `:hostname` to have the hostname of the first matching instance returned.
-Pass a `return` key with value `:hostname_and_port` to have a `"<hostname>:<port>"` string of the first matching instance returned.
+Calls to `:rds_db_instance` type keys return the instance identifier, endpoint
+address and endpoint port in a hash.
+
+Pass a `return` key with value `:hostname` to have the hostname of the first
+matching instance returned.
+
+Pass a `return` key with value `:hostname_and_port` to have a
+`"<hostname>:<port>"` string of the first matching instance returned.
 
 ### Example - ElastiCache cluster by name
 
@@ -99,9 +127,14 @@ aws_am_bullseye_redis:
   cache_cluster_id: "%{::sf_location}-%{::sf_environment}-redis"
 ```
 
-Calls to `:elasticache_cache_cluster` type keys return a list of cache nodes, their IDs and endpoint address/ports.
-Pass a `return` key with value `:hostname` to have a list of hostnames of keys of all cache nodes matching the cache_cluster_id returned.
-Pass a `return` key with value `:hostname_and_port` to have a list of `"<hostname>:<port>"` strings returned.
+Calls to `:elasticache_cache_cluster` type keys return a list of cache nodes,
+their IDs and endpoint address/ports.
+
+Pass a `return` key with value `:hostname` to have a list of hostnames of keys
+of all cache nodes matching the cache_cluster_id returned.
+
+Pass a `return` key with value `:hostname_and_port` to have a list of
+`"<hostname>:<port>"` strings returned.
 
 ### Example - ElastiCache replication group by name
 
@@ -111,13 +144,52 @@ aws_app_redis:
   replication_group_id: "%{::sf_location}-%{::sf_environment}-redis"
 ```
 
-Calls to `:elasticache_replication_group` return a list of replication groups, their primary endpoints and node group members.
-Pass a `return` key with value `:primary_endpoint` to have the hostname for the primary end point of the node group returned.
-Pass a `return` key with value `:primary_endpoint_and_port` to have the hostname and port returned as a colon-separated string.
-Pass a `return` key with value `:read_endpoints` to return an array of read endpoint hostnames, if a `replication_group_id` is specified. Returns `nil` if `replication_group_id` is unspecified.
-Pass a `return` key with value `:read_endpoints_with_ports` to return an array of read endpoint hostnames and ports as colon delimted strongs. Returns `nil` if `replication_group_id` is unspecified.
+Calls to `:elasticache_replication_group` return a list of replication groups,
+their primary endpoints and node group members.
+
+Pass a `return` key with value `:primary_endpoint` to have the hostname for the
+primary end point of the node group returned.
+
+Pass a `return` key with value `:primary_endpoint_and_port` to have the hostname
+and port returned as a colon-separated string.
+
+Pass a `return` key with value `:read_endpoints` to return an array of read
+endpoint hostnames, if a `replication_group_id` is specified. Returns `nil` if
+`replication_group_id` is unspecified.
+
+Pass a `return` key with value `:read_endpoints_with_ports` to return an array
+of read endpoint hostnames and ports as colon delimted strongs. Returns `nil` if
+`replication_group_id` is unspecified.
+
+### Example - AutoScaling Instance members
+
+```
+---
+aws_asg_group:
+  type:                     :autoscaling_group
+  auto_scaling_group_names: ["euwest1-test-api"]
+  return:                   :instance_details_inservice_ip
+```
+
+Calls to `:autoscaling_group` return a list of autoscaling groups and
+instance-id.
+
+Pass a `return` key with value `:instance_details_inservice_ip` to have the
+instance IP's returned for any matching instances in those autoscaling groups
+that are in the 'InService' state. This prevents nodes which are coming online,
+or have been marked for termination as appearing in this list.
+
+You will need to setup an ASG Lifecycle hook to put the machine into a Waiting
+state for slightly more that your puppet run, e.g. 20 minutes.
 
 ## Notes
 
-* The order in which items are returned, for example EC2 nodes matching a tag, is undefined. If you are using an array of items in a configuration file template, for example, you are advised to sort the array in the template. This eliminates the likelihood of unnecessary configuration file changes, and the consequential unnecessary restart of dependent services.
-* By default, all EC2 instances are returned, including those in a non-running state. To return only running instances, add a filter on `name: instance-state-name` and `values: ['running']` as per the example above.
+* The order in which items are returned, for example EC2 nodes matching a tag,
+  is undefined. If you are using an array of items in a configuration file
+  template, for example, you are advised to sort the array in the template. This
+  eliminates the likelihood of unnecessary configuration file changes, and the
+  consequential unnecessary restart of dependent services.
+* By default, all EC2 instances are returned, including those in a non-running
+  state. To return only running instances, add a filter on
+  `name: instance-state-name` and `values: ['running']` as per the example
+  above.

data/lib/hiera/backend/sf_hiera_aws_backend.rb

@@ -37,6 +37,10 @@ class Hiera
                 Aws::ElastiCache::Client.new
             end
 
+            def get_autoscaling_client
+                Aws::AutoScaling::Client.new
+            end
+
             public
 
             def initialize
@@ -386,6 +390,89 @@ class Hiera
                end
 
             end
+
+            def type_autoscaling_group(options)
+                autoscaling = get_autoscaling_client
+
+                if options.key? 'auto_scaling_group_names'
+                    asgs = autoscaling.describe_auto_scaling_groups(
+                        auto_scaling_group_names: options['auto_scaling_group_names'],
+                    ).auto_scaling_groups
+                else
+                    asgs = autoscaling.describe_auto_scaling_groups.auto_scaling_groups
+                end
+
+                if !options.key? 'return'
+                    return asgs.collect do |asg|
+                        {
+                            'auto_scaling_group_name'   => asg.auto_scaling_group_name,
+                            'launch_configuration_name' => asg.launch_configuration_name,
+                            'load_balancer_names'       => asg.load_balancer_names,
+                            'instances'                 => asg.instances.collect do | instance |
+                                {
+                                    'instance_id' => instance.instance_id,
+                                    'availability_zone' => instance.availability_zone,
+                                    'lifecycle_state' => instance.lifecycle_state,
+                                    'health_status' => instance.health_status,
+                                    'launch_configuration_name' => instance.launch_configuration_name,
+                                    'protected_from_scale_in' => instance.protected_from_scale_in
+                                }
+                            end
+                        }
+                    end
+                else
+                    unless options.key? 'auto_scaling_group_names'
+                        Hiera.warn('Requested to return hash of instances for auto scaling group, but no auto_scaling_group_names specified')
+                        return nil
+                    end
+                end
+
+                if options['return'] == :instance_details_inservice_ip
+                    instances = []
+                    asgs = asgs.collect do |asg|
+                        {
+                            'instances' => asg.instances.select{|i| i.lifecycle_state == 'InService'}.map { |instance|
+                                { 'instance_id' => instance.instance_id } 
+                            }
+                        }
+                    end
+
+                    asgs.each do |asg|
+                        instances += asg['instances']
+                    end
+
+                    ec2_options = {
+                        'filters' => [{ 
+                            'name' => 'instance-id', 
+                            'values' => instances.map { |i| i['instance_id'] }
+                        }],
+                        'return'  => [ 'private_ip_address']
+                    }
+                    return type_ec2_instance(ec2_options)
+                else
+                    asgs.collect do |a|
+                        if options['return'].is_a?(Array)
+
+                            # If the 'return' option is a list, we treat these
+                            # as a list of desired hash keys, and return a hash
+                            # containing only those keys from the API call
+                            
+                            Hash[options['return'].map do |f|
+                                [f.to_s, a.key?(f) ? a[f] : nil]
+                            end]
+
+                        elsif options['return'].is_a?(Symbol)
+
+                            # If the 'return' option is a symbol, we treat that
+                            # as the one hash key we care about, and return a list
+                            # of that.
+
+                            a.key?(options['return']) ? a[options['return']] : nil
+
+                        end
+                    end
+                end
+            end
         end
     end
 end

data/sf-hiera-aws.gemspec

@@ -4,7 +4,7 @@ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
 
 Gem::Specification.new do |spec|
     spec.name          = 'sf-hiera-aws'
-    spec.version       = '0.0.8'
+    spec.version       = '0.0.9'
     spec.authors       = ['Jon Topper','Mike Griffiths','Jack Thomas']
     spec.email         = ['jon@scalefactory.com','mike@scalefactory.com','jack@scalefactory.com']
 

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: sf-hiera-aws
 version: !ruby/object:Gem::Version
-  version: 0.0.8
+  version: 0.0.9
 platform: ruby
 authors:
 - Jon Topper
@@ -10,7 +10,7 @@ authors:
 autorequire: 
 bindir: exe
 cert_chain: []
-date: 2017-01-13 00:00:00.000000000 Z
+date: 2017-02-10 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler