bbc-cosmos-tools 0.4.0 → 0.4.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: d312899466e33b1d4a65a7335be17846bd070f73
-  data.tar.gz: f83b250a15485d6d15ae407c4f3e2d11c89d484e
+  metadata.gz: 3cb7442d963df4f4ca2d403ba2f37e831616363f
+  data.tar.gz: add8398183ee75d4580df67175ee25c7a8835be9
 SHA512:
-  metadata.gz: abe8c0da8cdf40c91c63df435b7e858bb304830f73a7ceac96b335b15cdb704328462a959fc35b5c7c55af53465381446211daa91be15cb27575b0fc2631ba9a
-  data.tar.gz: d1b54bb031dc796f72a6692c491da4a3906a0b926e278f335ff3a2aefeb8714771decd417645f9b627f4f3b6cc8ca6ca2e388c356cd010e1c36dffdef993e27b
+  metadata.gz: 9e382b385c6ea1b38fc6a3f58fdf82a9c61650d19401f492f7df9537334006df9cd6bc64b5138a9180932043eae90a4426acd33325e97aab360305e392b61c95
+  data.tar.gz: 4f49a41d384d116bb140d5547552c4bace00a66cbd2b95e7533323457cea3b77a1e62a229a078310b2364a357f73be89916b7cb41169fadc2242b37586bb427d

data/README.md

@@ -1,105 +1,164 @@
 # BBC::Cosmos::Tools
 
-This gem provides tooling for interacting with cosmos. Currently it supports the following:
+- [Introduction](#introduction)
+- [Example commands](#example-commands)
+- [Project directory structure](#project-directory-structure)
+- [File structure explained](#file-structure-explained)
+- [Example repositories](#example-repositories)
 
-* Pushing component config to the cosmos API
-* Creating stacks for a component in cosmos
-* Updating stacks for a component in cosmos
+## Introduction
+
+The [BBC Cosmos Tool](https://github.com/BBC-News/bbc-cosmos-tools) is a Ruby gem that runs on the command line and helps make generating CloudFormation specifically for the BBC Cosmos infrastructure easier.
+
+The gem supports the following features:
+
+- Pushing component config to the cosmos API
+- Creating stacks for a component in cosmos
+- Updating stacks for a component in cosmos
+
+The gem also supports the following formats:
+
+- [CFNDSL](https://github.com/stevenjack/cfndsl)
+- YAML
+- JSON
 
 ## Installation
 
 Add this line to your application's Gemfile:
 
-    gem 'bbc-cosmos-tools'
+```
+gem 'bbc-cosmos-tools'
+```
 
 And then execute:
 
-    $ bundle
+```
+$ bundle
+```
 
 Or install it yourself as:
 
-    $ gem install bbc-cosmos-tools
+```
+$ gem install bbc-cosmos-tools
+```
 
-## Usage
+## Project directory structure
 
-The gem expects a certain folder structure for the config files:
+The cli tool needs to be used inside a project that has a specific directory structure. This structure looks something like the following (if using CFNDSL):
 
-#### Folder structure
+```
+.
+├── configs
+│   ├── app.yaml
+│   └── {component}.yaml.erb
+├── resources
+│   ├── int
+│   │   └── {component}.yaml
+│   ├── live
+│   │   └── {component}.yaml
+│   └── test
+│   │   └── {component}.yaml
+└── stacks
+    └── {component}
+        └── {stack}
+            ├── aws
+            │   ├── {resource}
+            │       ├── {type}.rb
+            │   └── {resource}
+            │       ├── {type}.rb
+            └── template.rb
+```
 
-```bash
- ├── configs
- │   ├── {project name}.yaml.erb
- │   ├── app.yam
- ├── resources
- │   ├── {int/test/stage/live}
- │   │   ├── {project-name}.yaml
- ├── stacks
- │   ├── {component id/group_id}
- │   │   ├── {stack name}
- │   │   │   ├── aws {CFNDSL templates}
- │   │   │   ├── template.rb
+Where by the `{x}` items equate to the following:
+
+- `{component}`  
+The name of your Cosmos component (e.g. `responsive-jmeter`)  
+
+- `{stack}`  
+Typically this will be `main` (as there *must* always be a `main` stack)  
+But be aware you can have multiple "stacks" for a component (it's just a structural convenience)  
+
+- `{resource}`  
+This could be (for example) a Scaling Group, IAM policy, SQS, or DynamoDB (any AWS service)  
+
+- `{type}`  
+This is a sub section of the resource  
+e.g. if the resource was `AWS::SQS::Queue` then this file would be named `queue.rb`  
+
+> Note: if you're using the YAML or JSON formats then the following structure is used (notice it's *almost* identical)
+
+```
+.
+├── configs
+│   ├── app.yaml
+│   └── {component}.yaml.erb
+├── resources
+│   ├── int
+│   │   └── {component}.yaml
+│   ├── live
+│   │   └── {component}.yaml
+│   └── test
+│   │   └── {component}.yaml
+└── stacks
+    └── {component}
+        └── {stack}.{yaml|json}
 ```
 
-an example of this would be:
+## File structure explained
 
-```bash
- ├── configs
- │   ├── my_project.yaml.erb
- │   ├── app.yam
- ├── resources
- │   ├── int
- │   │   ├── my_project.yaml
- ├── stacks
- │   ├── my_wicked_component
- │   │   ├── main
- │   │   │   ├── aws
- │   │   │   │   ├── auto_scaling
- │   │   │   │   │   ├── group.rb
- │   │   │   │   │   ├── launch_config.rb
- │   │   │   ├── template.rb
-```
-
-#### Project resources structure
+The file structure required by the BBC Cosmos CLI Tool can be confusing so let’s break it down into sections so we can better understand the purpose of each directory:
 
-The project resources yaml file is used to describe the resources used within the project. The reason for having a single file is that you can define resources that are used in the cloudformation templates and config, so when for example an ARN changes this will be picked up across the CF templates and the config.
+### Config
 
-The file is spilt up into two section:
+The files in this folder let you define custom configuration that will be baked into your EC2 instances.
 
-* The standard resource list, this is generally the ARNS for all the resources your component uses.
-* Cloudformation parameters (These are usually referenced from the aforementioned)
+You need to use the `bbc-cosmos-tools config push` command to upload the updated configuration data (run the help command `bbc-cosmos-tools help config push` to see more options and details).
 
-Below is an annotated example of a resource config:
+The content of this folder is usually:
+
+- `app.yaml` (defaults)
+- `{component}.yaml.erb` (custom)
+
+So the contents of `app.yaml` can look like the following:
 
 ```yaml
+bbc:
+  cosmos:
+    api: 'https://api.live.bbc.co.uk'
+    config_endpoint: '/cosmos/env/%s/component/%s/configuration'
+    stack_create_endpoint: '/cosmos/env/%s/component/%s/stacks/create'
+    stack_update_endpoint: '/cosmos/env/%s/component/%s/stack/%s/update'
+    deployments: '/cosmos/deployments/component/%s/env/%s'
+    release: '/cosmos/component/%s/releases'
+    deploy: '/cosmos/env/%s/component/%s/deploy_release'
+    snapshot_deploy: '/cosmos/env/%s/component/%s/deploy_snapshot'
+    stack_events: '/cosmos/env/%s/component/%s/stack/%s/events'
+    stacks: '/cosmos/env/%s/component/%s/stacks'
+    stack_delete: '/cosmos/env/%s/component/%s/stack/%s/delete'
+```
 
-sequencer:
-	name: 'example_name'
-	arn: 'example_arn'
+Where as the contents of `{component}.yaml.erb` can look like the following:
 
-roles:
-	broker:
-		name: &roles_broker_name 'some_iam_role_name'
+```erb
+components:
+  '{component}':
+    s3_bucket_id: "foo"
+    s3_bucket_path: "bar"
+```
 
-cloudformation:
-	shared: &component_shared #<- This node is ignored, it's used as a base for other config templates
-		ACFParameter: 'Value'
-		AnotherCFParameter: 'Another value'
+In YAML you can get fancy and make certain properties more re-usable by using `&` to create an "anchor" which you can then reference with `*`, along with `<<:` to inject content. See the following file for an example (although it’s not a very good example as there would be no need to use any of those features for such a small file, but hopefully it gives you an idea on how they are used)…
 
-	components:
-		'my-example-component': #<- The name of the component in cosmos
-			main: <- This is the name of the stack in cosmos
-				variants: <- This is so you can have a different variant of the same config, i.e for int/test one for durning the day and one for evening/weekends.
-					default: &my-example-component_default #<- We set this anchor so we can use this defaults in other variants
-						<<: *component_shared #<- We import the shared component params into these so we're not duplicating
-						ARoleParamForDefault: *roles_broker_name #<- We use the anchor for the roles set above to add them here, again stopping repitition
-					scheduled:
-						<<: *my-example-component_default
-						MinSize: 0
-						MaxSize: 0
-						
+```erb
+base: &shared
+  s3_bucket_id: "foo"
+  s3_object_path: "bar"
+
+components:
+  'responsive-jmeter':
+    <<: *shared
 ```
- 
-#### Project Config structure
+
+#### Using CFNDSL
 
 The project config is an ERB template that is renderer into a YAML file. it gets passed all resources from the resource config that can be used in the template. Below is an example of this:
 
@@ -131,7 +190,7 @@ the component config merged in afterwards.
 #### Application config
 
 There's also an application config that stores data used by the app, at the moment this
-is only the api endpoints.
+is only the API endpoints.
 
 ```yaml
 bbc:
@@ -141,7 +200,151 @@ bbc:
 	update_endpoint: "/some/restful/endpoint"
 ```
 
-The gem is a cli application, the commands are self documenting but below is a list of the commands for each set of sub commands
+### Resources
+
+The resources directory is used to provide custom values to the `Parameters` defined within your stacks (see the Stacks directory below). There is usually three folders inside this directory that reflect the Cosmos environments:
+
+- `int`
+- `test`
+- `live`
+
+Each of those folders will have a `{component}.yaml` file where you define the values for your Parameters (if you were using AWS directly, rather than via the Cosmos abstraction layer, then this is typically where you would specify the custom values either using the AWS CLI tool or by entering the values into form fields when using the AWS Console GUI).
+
+The format of this file looks something like the following:
+
+```yaml
+cloudformation:
+  components:
+    '{component_folder_name}':
+        {stack}:
+          variants:
+            default:
+              {parameter_a}: value
+              {parameter_b}: value
+              {parameter_c}: value
+      
+```
+
+#### Using CFNDSL
+
+The project resources yaml file is used to describe the resources used within the project. The reason for having a single file is that you can define resources that are used in the cloudformation templates and config, so when for example an ARN changes this will be picked up across the CF templates and the config.
+
+The file is spilt up into two section:
+
+* The standard resource list, this is generally the ARNS for all the resources your component uses.
+* Cloudformation parameters (These are usually referenced from the aforementioned)
+
+Below is an annotated example of a resource config:
+
+```yaml
+
+sequencer:
+	name: 'example_name'
+	arn: 'example_arn'
+
+roles:
+	broker:
+		name: &roles_broker_name 'some_iam_role_name'
+
+cloudformation:
+	shared: &component_shared #<- This node is ignored, it's used as a base for other config templates
+		ACFParameter: 'Value'
+		AnotherCFParameter: 'Another value'
+
+	components:
+		'my-example-component': #<- The name of the component in cosmos
+			main: <- This is the name of the stack in cosmos
+				variants: <- This is so you can have a different variant of the same config, i.e for int/test one for durning the day and one for evening/weekends.
+					default: &my-example-component_default #<- We set this anchor so we can use this defaults in other variants
+						<<: *component_shared #<- We import the shared component params into these so we're not duplicating
+						ARoleParamForDefault: *roles_broker_name #<- We use the anchor for the roles set above to add them here, again stopping repitition
+					scheduled:
+						<<: *my-example-component_default
+						MinSize: 0
+						MaxSize: 0
+						
+```
+
+### Stacks
+
+The stacks directory is the most important part of the configuration as it defines your infrastructure requirements (i.e. creates all the specified AWS resources as you’ve defined them using CloudFormation).
+
+The folder structure resembles the following:
+
+```
+.
+├── stacks
+    └── {component}
+        └── {stack}
+            ├── aws
+            │   ├── {resource}
+            │   │   ├── {type}
+```
+
+An example of this could be:
+
+```
+.
+├── stacks
+    └── responsive-jmeter
+        └── main
+            ├── aws
+            │   ├── sqs
+            │   │   ├── policy.rb
+            │   │   ├── queue.rb
+            │   ├── iam
+            │   │   ├── instance_profile.rb
+```
+
+> Note: when opening up the files `policy.rb` and `queue.rb` we’ll see the `Type` syntax and that should correspond to the folder structure described above. So aws/sqs/policy.rb will have a `Type "AWS::SQS::QueuePolicy”`, and aws/iam/instance_profile.rb will have a `Type "AWS::IAM::InstanceProfile”`
+
+## Example commands
+
+When you want to push up some new configuration into your instance (for your application to use) then the following command is what you need (this isn't done very often):
+
+```sh
+bbc-cosmos-tools config push {cosmos_component_name} \
+  --project={resources yaml filename} \
+  --key-path=/custom/path/to/your/pem/certificate
+```
+
+To update your stack:
+
+```sh
+bbc-cosmos-tools stack update {cosmos_component_name} \
+  --project={resources yaml filename} \
+  --key-path=/custom/path/to/your/pem/certificate
+```
+
+To generate CloudFormation that is sent to `stdout`:
+
+```sh
+bbc-cosmos-tools stack generate {cosmos_component_name} \
+  --project={resources yaml filename} \
+  --key-path=/custom/path/to/your/pem/certificate
+```
+
+> Note: in case you're not a CLI wizard, instead of specifying the `--key-path` option you can set a `$DEV_CERT_PATH` environment variable that the tool uses by default if it detects it. Inside your shell's configuration file (either `~/.bashrc` or `~/.zshrc`) you'll want to add `export DEV_CERT_PATH=/custom/path/to/your/pem/certificate`
+
+### Using YAML or JSON
+
+Use the same commands as above but make sure to include a `--type={yaml|json}`. If it's not specified then `cfndsl` is assumed as the default (almost as if you had written `--type=cfndsl`, but it's clearer to just remove the flag in that instance).
+
+## Example repositories
+
+There are a few example projects using this tool now:
+
+- [Election Data Config](https://github.com/BBC-News/election-data-config)
+- [Jmeter Config](https://github.com/BBC-News/jmeter-configs)
+- [Kaleidoscope Config](https://github.com/BBC-News/kaleidoscope-config)
+
+> Note: it's important to realise that you don't *need* to have your configuration in a separate repository. If anything it probably would be better maintained as part of your application repository (placed inside a `/config/` directory)
+
+> The reason some of the projects choose to locate their configuration in a separate repo is because they're made up of a number of different components.
+
+## API reference
+
+The gem is a cli application, and so the commands are self documenting; but below is a list of the commands for each set of sub commands
 
 ### Cosmos Config
 
@@ -163,10 +366,10 @@ bbc-cosmos-tools cf update
 
 You can use the built-in `help` command to see a list of parameters and options that need to be passed into the command you wish to execute. For example, if we were unsure of the options for the `stack` command then we would execute `bbc-cosmos-tools help stack` and this would display the following output:
 
-```sh
+```bash
 Commands:
   bbc-cosmos-tools stack help [COMMAND]                          # Describe subcommands or one specific subco...
-  bbc-cosmos-tools stack create [COMPONENT, MAIN_STACK = false]  # Generates and create the cloudformation te...
+  bbc-cosmos-tools stack create [COMPONENT, MAIN_STACK = true]  # Generates and create the cloudformation te...
   bbc-cosmos-tools stack delete [COMPONENT]                      # Deletes a stack
   bbc-cosmos-tools stack events [COMPONENT = nil]                # Shows the stack events for a component
   bbc-cosmos-tools stack generate [COMPONENT]                    # Generates a cloudformation template based ...

data/lib/bbc/cosmos/tools/commands/stack.rb

@@ -69,10 +69,7 @@ module BBC
             say "#{banner(component)}\n"
 
             type = Tools::Types::Factory.create(component, options, config)
-            data = type.generate_data.tap do |d|
-              @l.(d, component, options[:env])
-            end
-
+            data = type.generate_data
             say data, :white
 
           end
@@ -80,8 +77,8 @@ module BBC
           method_option :variant, :type => :string, :default => 'default', :desc => "The variant to use, i.e default|scheduled"
           method_option :group, :type => :string, :default => nil, :desc => "The group name to override the component id"
           method_option :tags, :type => :array, :default => nil, :desc => "The tags of the components you want to deploy (i.e 'renderer')"
-          desc "stack create [COMPONENT, MAIN_STACK = false]", "Generates and create the cloudformation template for the specific component"
-          def create(component = nil, main_stack = 'false')
+          desc "stack create [COMPONENT, MAIN_STACK = true]", "Generates and create the cloudformation template for the specific component"
+          def create(component = nil, main_stack = 'true')
             begin
               say "#{banner}\n"
 
@@ -89,10 +86,7 @@ module BBC
                 say get_key_value("\nComponent", id)
 
                 type = Tools::Types::Factory.create(id, options, config)
-                data = type.generate_data.tap do |d|
-                  @l.(d, id, options[:env])
-                end
-
+                data = type.generate_data
                 post_data = {
                   'name_suffix'   => options[:stack],
                   'service_stack' => main_stack.match(/(true|t|yes|y|1)$/i) != nil,

data/lib/bbc/cosmos/tools/version.rb

@@ -1,7 +1,7 @@
 module BBC
   module Cosmos
     module Tools
-        VERSION = "0.4.0"
+        VERSION = "0.4.1"
     end
   end
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: bbc-cosmos-tools
 version: !ruby/object:Gem::Version
-  version: 0.4.0
+  version: 0.4.1
 platform: ruby
 authors:
 - Steven Jack
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2014-09-24 00:00:00.000000000 Z
+date: 2014-10-01 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler