cfn-bridge 0.0.2 → 0.0.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: bb117f078381ae0cb260bfee9b04ea821b312091
-  data.tar.gz: ee0d543be32bf3ac7785b40dd1f2a1700afae8f0
+  metadata.gz: 07f63fba4fe361c558d340ea3c548e2c5e18536b
+  data.tar.gz: 63b98a3bc9025d48eb149f9559bd30440efc89b7
 SHA512:
-  metadata.gz: d95bf99fc45343d58c6ef18dc83e029d4c6af4b24119cf503694aa7c579a87b29aceccd51629f7ff5a5fec3ec5086ad921d969f9cd6e870e670974ea5556f81c
-  data.tar.gz: a8e052cf554b8c41dc73c298e8f90585371e29e6afb0b0cd564d1775597b3c41d6400d1594a874494b86750870f408fcbe80ee65e0bff2fa3e6ed95abdecf331
+  metadata.gz: 2026e5c2357a70ad1c6015dcb7253b283bd4df0a269ccba516c5ca58b8cde86a16fa9bfeb0bb2a618934e75ba3c05a7bd7cc57f189f72a57195300a1cc4e0ab5
+  data.tar.gz: cc9f7b83bca39812c6aede1ff96ca2c24bdc8098bb76d6869f0ba0a8919345b870738eea0911ed45ad0c81e538bc4b251b618e97d72dfd3f9848aded19b8471f

data/.rspec

@@ -0,0 +1,2 @@
+--color
+--require spec_helper

data/README.md

@@ -1,10 +1,33 @@
+<!-- START doctoc generated TOC please keep comment here to allow auto update -->
+<!-- DON'T EDIT THIS SECTION, INSTEAD RE-RUN doctoc TO UPDATE -->
+**Table of Contents**
+
+- [cfn-bridge](#cfn-bridge)
+	- [Installation](#installation)
+	- [Usage](#usage)
+	- [Building a custom resource](#building-a-custom-resource)
+		- [Implementing the `create` operation](#implementing-the-create-operation)
+		- [Implementing `update` is usually not a requirement](#implementing-update-is-usually-not-a-requirement)
+		- [Implementing the `delete` operation](#implementing-the-delete-operation)
+		- [Registering and using it](#registering-and-using-it)
+	- [Current custom resources](#current-custom-resources)
+		- [Custom::SubscribeSQSQueueToSNSTopic](#customsubscribesqsqueuetosnstopic)
+		- [Custom::CloudFormationOutputs](#customcloudformationoutputs)
+	- [Contributing](#contributing)
+
+<!-- END doctoc generated TOC please keep comment here to allow auto update -->
+
 # cfn-bridge
 
-A bridge to allow you to build custom AWS cloud formation resources.
+[This project is sponsored by Neat](http://www.neat.com/)
+
+A bridge to allow you to build custom AWS cloud formation resources in Ruby.
 
 Check Amazon's page [on custom cloud formation resources](http://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/crpg-walkthrough.html)
 to get more info on how and why you would like to have them.
 
+If you're into Python more than Ruby, there's an [AWS project for this as well](http://blogs.aws.amazon.com/application-management/post/Tx2FNAPE4YGYSRV/Customers-CloudFormation-and-Custom-Resources).
+
 ## Installation
 
 Add this line to your application's Gemfile:
@@ -25,7 +48,178 @@ Run:
 
     $ cfn-bridge start QUEUE_NAME
 
-This will start a worker consuming from `QUEUE_NAME` until the caller calls `CTRL-C` to stop it. `QUEUE_NAME` should be the SQS queue to where the SNS topic is publishing the custom resource messages. An example template that setups the topic and the queue [is available at the repo](spec/files/test-formation.json) and can be used to provide the base topic and queue to use this application.
+This will start a worker consuming from `QUEUE_NAME` until the caller calls `CTRL-C` to stop it. `QUEUE_NAME` should be the SQS queue to where the SNS topic is publishing the custom resource messages. An example template that setups the topic and the queue [is available at the repo](spec/files/test-formation.json) and can be used to provide the base topic and queue to use this gem.
+
+Since this gem uses the `aws-sdk` gem you have to setup your AWS keys as environment variables (or as an IAM profile if you're running on EC2) to make sure the gem has the right credentials to perform the operations. As usual, we recommend that you provide keys with access only to the operations you're going to perform, do not provide all access keys for this.
+
+## Building a custom resource
+
+Building a custom resource is simple, all it requires is a class with two methods, `create` and `delete` (you can also have an `update` one if it makes sense for your resource to be updateable) that will take the message parameter. Your custom resource should also inherit from `CloudFormation::Bridge::Resources::Base` to simplify your job, it provides a default `update` method that always fails and includes a couple constants you will have to use at your responses.
+
+All three operations take a [Request](lib/cloud_formation/bridge/request.rb) method as a parameter and you should check it to see the method the fields that are available there for you to use.
+
+Let's look at our `SubscribeQueueToTopic` to get a sense of how custom resources could be implemented, starting with the constants:
+
+```ruby
+ARN = 'Arn'
+ENDPOINT = 'Endpoint'
+PROTOCOL = 'Protocol'
+
+TOPIC_ARN = 'TopicArn'
+QUEUE_NAME = 'QueueName'
+RAW_MESSAGE_DELIVERY = 'RawMessageDelivery'
+
+REQUIRED_FIELDS = [
+  TOPIC_ARN,
+  QUEUE_NAME,
+]
+```
+
+This list of constants are first the outputs produced by the resource (`ARN`, `ENDPOINT` and `PROTOCOL`) and then the inputs that are used to create it, `TOPIC_ARN`, `QUEUE_NAME` and `RAW_MESSAGE_DELIVERY`. The inputs are the fields we use to create the resource, since this resource is meant to subscribe an SQS queue to an SNS topic, we accept the topic ARN, the queue name and an optional `raw message delivery` field that instructs the subscription to send the raw message only and not the message with the other SNS fields.
+
+### Implementing the `create` operation
+
+Now let's look at the first operation, the `create` method:
+
+```ruby
+def create(request)
+  require_fields(request, REQUIRED_FIELDS)
+
+  queue = queues.named(request.resource_properties[QUEUE_NAME])
+  topic = topics[request.resource_properties[TOPIC_ARN]]
+
+  subscription = topic.subscribe(queue)
+
+  if request.resource_properties[RAW_MESSAGE_DELIVERY]
+    subscription.raw_message_delivery = true
+  end
+
+  {
+    FIELDS::PHYSICAL_RESOURCE_ID => subscription.arn,
+    FIELDS::DATA => {
+      ARN => subscription.arn,
+      ENDPOINT => subscription.endpoint,
+      PROTOCOL => subscription.protocol,
+    },
+  }
+end
+```
+
+First, we validate that we have all the fields required to create the resource (topic ARN and queue name), if any one of these fields is empty, we raise an exception with the field that was empty. The code at the gem that does the messaging and executes your resources will catch any exception that inherits from `StandardError` and will forward it's message as a failure to the Cloud Formation service, this errors will be visible at the cloud formation events so make sure you raise exceptions with useful error messages whenever you have to.
+
+From that on, we just implement the operation, grab the queue, grab the topic, subscribe one to the other, set the raw message delivery field if it was set and then, here's the important part, return the response.
+
+Let's look at an example return response in raw JSON:
+
+```javascript
+{
+   "Status" : "SUCCESS",
+   "PhysicalResourceId" : "Tester1",
+   "StackId" : "arn:aws:cloudformation:us-east-1:EXAMPLE:stack/stack-name/guid",
+   "RequestId" : "unique id for this create request",
+   "LogicalResourceId" : "MySeleniumTester",
+   "Data" : {
+      "resultsPage" : "http://www.myexampledomain/test-results/guid",
+      "lastUpdate" : "2012-11-14T03:30Z",
+   }
+}
+```
+
+And let's look at the one we're returning at the current `create` method:
+
+```ruby
+{
+  FIELDS::PHYSICAL_RESOURCE_ID => subscription.arn,
+  FIELDS::DATA => {
+    ARN => subscription.arn,
+    ENDPOINT => subscription.endpoint,
+    PROTOCOL => subscription.protocol,
+  },
+}
+```
+
+The fields `Status`, `StackId`, `LogicalResourceId` and `RequestId` are not present here because the code that executes your resource already knows how to fill them, so you only have to care about `PhysicalResourceId` and `Data`.
+
+The physical resource id should be a unique value that you can use to find this resource later when you receive `update` and `delete` operations. When dealing with AWS resources your best bet is to always use the `ARN` for the resource you're creating, this guarantees it is unique for your cloud formation and that you can easily find it later on other requests.
+
+The `Data` field should contain fields that might be useful for the cloud formation where the resource is included, you don't actually have to provide one, but all fields returned here are available to `Fn::GetAtt` operations at your cloud formation template, so make sure you send back some useful data back if possible.
+
+### Implementing `update` is usually not a requirement
+
+Since it wouldn't make much sense to update a subscription (you're either subscribed to something or you're not) we don't have an `update` method here, but if it makes sense for your resource, the update method follows the same pattern as `create`, just make sure you're *not changing the physical resource id* as it will be ignored. It's usually much simpler not to implement updates and always require the resource to be deleted and created again.
+
+
+### Implementing the `delete` operation
+
+The `delete` method is much simpler than `create`:
+
+```ruby
+def delete(request)
+  subscription = subscriptions[request.physical_resource_id]
+  subscription.unsubscribe if subscription && subscription.exists?
+end
+```
+
+Here we find the subscription using it's physical resource id (we used the subscription's `ARN` for this) and, if it exists, it is destroyed. Checking for the existence here is important because if your resource fails to be created the cloud formation service will still issue a `delete` operation for it (in case it was created even after failing) so make sure your code ignores delete operations for resources that do not exist.
+
+### Registering and using it
+
+Once you resource is implemented, you must either register it at the [Executor::DEFAULT_REGISTRY hash](lib/cloud_formation/bridge/executor.rb) or manually create an executor providing it at the hash. Make sure the name you use starts with `Custom::` and that it doesn't clash with other resources already registered there.
+
+Once you have it registered, you can just declare your resource at any cloud formation template:
+
+```javascript
+"Resources": {
+    "FirstQueue": {
+        "Type": "AWS::SQS::Queue",
+        "Properties": {
+            "ReceiveMessageWaitTimeSeconds": 20,
+            "VisibilityTimeout": 60
+        }
+    },
+    "FirstTopic": {
+        "Type": "AWS::SNS::Topic"
+    },
+    "SubscribeResource": {
+        "Type": "Custom::SubscribeSQSQueueToSNSTopic",
+        "Properties": {
+            "ServiceToken": {
+                "Ref": "EntryTopic"
+            },
+            "TopicArn": {
+                "Ref": "FirstTopic"
+            },
+            "QueueName": {
+                "Fn::GetAtt": ["FirstQueue", "QueueName"]
+            }
+        }
+    }
+}
+```
+
+It is declared just like any other custom resource with the name you have registered at the executor (`Custom::SubscribeSQSQueueToSNSTopic` in this case) and then you can include your properties as needed. The `ServiceToken` property must always be there and should point to the SNS topic that is being watched for custom resource messages, the other properties are the ones your resource will use to implement it's actions.
+
+And with this you should be able to start creating your own custom cloud formation resources.
+
+## Current custom resources
+
+### Custom::SubscribeSQSQueueToSNSTopic
+
+Subscribes an SQS queue to an SNS topic.
+
+Parameters:
+
+* `TopicArn` - the SNS topic that will be subscribed to - *required*;
+* `QueueName` - the SQS queue that will receive the messages from the topic - *required*;
+* `RawMessageDelivery` - if set, the SNS message will not include the JSON envelope, it will send the raw message to the queue;
+
+### Custom::CloudFormationOutputs
+
+Makes all outputs from another cloud formation available to `Fn::GetAtt` calls.
+
+Parameters:
+
+* `Name` - the name of the cloud formation you want to get the outputs from - *required*;
 
 ## Contributing
 

data/lib/cloud_formation/bridge/exception_notifier.rb

@@ -1,3 +1,4 @@
+require 'cloud_formation/bridge/util'
 require 'singleton'
 require 'rollbar'
 
@@ -15,7 +16,7 @@ module CloudFormation
       include Singleton
 
       def report_exception(exception, custom_data = {}, user_data = {})
-        puts "#{exception.message} - #{custom_data.inspect} - #{user_data.inspect}\n#{exception.backtrace.join("\n")}"
+        Util::LOGGER.error("#{exception.message} - #{custom_data.inspect} - #{user_data.inspect}\n#{exception.backtrace.join("\n")}")
       end
 
     end

data/lib/cloud_formation/bridge/poller.rb

@@ -1,7 +1,7 @@
 require 'cloud_formation/bridge/executor'
 require 'cloud_formation/bridge/exception_notifier'
 require 'cloud_formation/bridge/request'
-require 'logger'
+require 'cloud_formation/bridge/util'
 
 module CloudFormation
   module Bridge
@@ -9,7 +9,7 @@ module CloudFormation
 
       attr_reader :logger, :running
 
-      def initialize(queue_name, executor = CloudFormation::Bridge::Executor.new, logger = Logger.new(STDOUT))
+      def initialize(queue_name, executor = CloudFormation::Bridge::Executor.new, logger = Util::LOGGER)
         @queue_name = queue_name
         @executor = executor
         @logger = logger
@@ -29,17 +29,20 @@ module CloudFormation
       def poll
         message = queue.receive_message
 
-        return unless message
+        unless message
+          logger.info("No messages found, looping again")
+          return
+        end
 
         begin
           logger.info("Received message #{message.id} - #{message.body}")
           body = JSON.parse(message.body)
-          request = CloudFormation::Bridge::Request.new(JSON.parse(body["Message"]))
+          request = CloudFormation::Bridge::Request.new(JSON.parse(body["Message"]), logger)
           @executor.execute(request)
           message.delete
           logger.info("Processed message #{message.id}")
           message
-        rescue => ex
+        rescue Exception => ex
           logger.info("Failed to process message #{message.id} - #{ex.message}")
           ExceptionNotifier.report_exception(ex,
             message: message.body,

data/lib/cloud_formation/bridge/request.rb

@@ -1,6 +1,7 @@
 require 'securerandom'
 require 'cloud_formation/bridge/http_bridge'
 require 'cloud_formation/bridge/names'
+require 'cloud_formation/bridge/util'
 
 module CloudFormation
   module Bridge
@@ -8,10 +9,11 @@ module CloudFormation
 
       include CloudFormation::Bridge::Names
 
-      attr_reader :request
+      attr_reader :request, :logger
 
-      def initialize(request)
+      def initialize(request, logger = Util::LOGGER)
         @request = request
+        @logger = logger
       end
 
       def update?
@@ -68,11 +70,17 @@ module CloudFormation
           FIELDS::STATUS => RESULTS::FAILED,
         )
 
+        logger.error("Failing request #{request_url} - #{response.inspect}")
+
         HttpBridge.put(request_url, response)
       end
 
       def succeed!(response)
-        HttpBridge.put(request_url, build_response(response || {}))
+        actual_response = build_response(response || {})
+
+        logger.info("Succeeding request #{request_url} - #{actual_response.inspect}")
+
+        HttpBridge.put(request_url, actual_response)
       end
 
       def build_response(response = {})

data/lib/cloud_formation/bridge/util.rb

@@ -0,0 +1,9 @@
+require 'logger'
+
+module CloudFormation
+  module Bridge
+    module Util
+      LOGGER = Logger.new(STDOUT)
+    end
+  end
+end

data/lib/cloud_formation/bridge/version.rb

@@ -1,5 +1,5 @@
 module CloudFormation
   module Bridge
-    VERSION = "0.0.2"
+    VERSION = "0.0.3"
   end
 end

data/spec/spec_helper.rb

@@ -88,5 +88,5 @@ RSpec.configure do |config|
 end
 
 ['cloud_formation_creator', 'file_support'].each do |file|
-  require File.join(__dir__, 'support', file)
+  require File.join(File.dirname(__FILE__), 'support', file)
 end

data/spec/support/file_support.rb

@@ -1,7 +1,7 @@
 module FileSupport
 
   def read_file(name)
-    IO.read(File.join(__dir__, '..', 'files', name))
+    IO.read(File.join(File.dirname(__FILE__), '..', 'files', name))
   end
 
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: cfn-bridge
 version: !ruby/object:Gem::Version
-  version: 0.0.2
+  version: 0.0.3
 platform: ruby
 authors:
 - Maurício Linhares
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2014-08-08 00:00:00.000000000 Z
+date: 2014-08-11 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
@@ -173,6 +173,7 @@ extensions: []
 extra_rdoc_files: []
 files:
 - ".gitignore"
+- ".rspec"
 - Gemfile
 - LICENSE.txt
 - README.md
@@ -191,6 +192,7 @@ files:
 - lib/cloud_formation/bridge/resources/base.rb
 - lib/cloud_formation/bridge/resources/cloud_formation_outputs.rb
 - lib/cloud_formation/bridge/resources/subscribe_queue_to_topic.rb
+- lib/cloud_formation/bridge/util.rb
 - lib/cloud_formation/bridge/version.rb
 - spec/files/outputs-formation.json
 - spec/files/sample-create-message.json