aws-must-templates 0.1.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (46) hide show
  1. checksums.yaml +7 -0
  2. data/README.md +390 -0
  3. data/lib/aws-must-templates.rb +2 -0
  4. data/lib/tasks/cross-ref.rb +78 -0
  5. data/lib/tasks/suite.rake +292 -0
  6. data/lib/test-suites/test_suites.rb +115 -0
  7. data/mustache/commonCfnSignal.mustache +34 -0
  8. data/mustache/commonCreationPolicy.mustache +33 -0
  9. data/mustache/commonDependsOn.mustache +25 -0
  10. data/mustache/commonInstanceType.mustache +27 -0
  11. data/mustache/commonKeyValue.mustache +28 -0
  12. data/mustache/commonStackRef.mustache +23 -0
  13. data/mustache/commonValue.mustache +34 -0
  14. data/mustache/initializeCFinit.mustache +27 -0
  15. data/mustache/initializeCFtools.mustache +27 -0
  16. data/mustache/initializeCfnInitHupFiles.mustache +85 -0
  17. data/mustache/initializeInstallAwsCli.mustache +32 -0
  18. data/mustache/initializeInstallChef.mustache +37 -0
  19. data/mustache/initializeProvisionChefZero.mustache +36 -0
  20. data/mustache/initializeStartCfnHup.mustache +25 -0
  21. data/mustache/mapping.mustache +27 -0
  22. data/mustache/mappingSubnetConfig.mustache +27 -0
  23. data/mustache/mappings.mustache +71 -0
  24. data/mustache/output.mustache +38 -0
  25. data/mustache/parameter.mustache +35 -0
  26. data/mustache/resource.mustache +51 -0
  27. data/mustache/resourceInstance.mustache +52 -0
  28. data/mustache/resourceInstanceInitialize.mustache +120 -0
  29. data/mustache/resourceInstanceMetadata.mustache +65 -0
  30. data/mustache/resourceInstanceProfile.mustache +34 -0
  31. data/mustache/resourceInternetGateway.mustache +87 -0
  32. data/mustache/resourcePolicy.mustache +50 -0
  33. data/mustache/resourceProvisionChef.mustache +38 -0
  34. data/mustache/resourceRole.mustache +40 -0
  35. data/mustache/resourceS3Bucket.mustache +39 -0
  36. data/mustache/resourceSecurityGroup.mustache +46 -0
  37. data/mustache/resourceStack.mustache +45 -0
  38. data/mustache/resourceSubnet.mustache +41 -0
  39. data/mustache/resourceUser.mustache +58 -0
  40. data/mustache/resourceVPC.mustache +44 -0
  41. data/mustache/resourceWait.mustache +39 -0
  42. data/mustache/resources.mustache +65 -0
  43. data/mustache/root.mustache +244 -0
  44. data/mustache/tag.mustache +27 -0
  45. data/pics/test-suites.jpg +0 -0
  46. metadata +146 -0
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA1:
3
+ metadata.gz: cf3a6e500d517d701c0215e0477fa9a8d595b979
4
+ data.tar.gz: 1b2ac0944b78a61036e66e217c39eb47623a2a5e
5
+ SHA512:
6
+ metadata.gz: 0c17f5daae17aabf52b9cd254f9cd13e5d5189c7cd8125f1299fc1e4ab15a30e28693da298a50447fc2cac4dacd8ba86f3db744a448f80bcdf9487ffce2cf9dd
7
+ data.tar.gz: 075de499bb571c08655a4a35df483ceaa4993bcae8c8ee75fe5a458a0420f8832df17a617c4b89a81ca3edf7656af809d5d72aff59361ce7c5898dad4bcf41eb
data/README.md ADDED
@@ -0,0 +1,390 @@
1
+ # aws-must-templates - cloudformation templates for aws-must - $Release:0.1.2$
2
+
3
+ Set of [extensible](#OVERRIDE)
4
+ [templates](https://rawgit.com/jarjuk/aws-must-templates/master/generated-docs/aws-must-templates.html)
5
+ for [aws-must](https://github.com/jarjuk/aws-must) tool to generate
6
+ CloudFormation JSON from a YAML configuration, and a
7
+ [Test Runner](#TESTING) for [validating correctness](#TEST_CASES) of
8
+ CloudFormation stacks provisioned.
9
+
10
+
11
+ ## The Problem
12
+
13
+ When using code generators, consider
14
+
15
+ 1. Code generators are much like a "Garden Party": Most of the things
16
+ that you need may be there, but some specific feature is quite
17
+ likely to be missing, or should be implemented in a different way.
18
+ "When you can't please everyone, you got to please yourself" - and
19
+ use the possibility to modify template generation process to meet
20
+ your own needs.
21
+
22
+ 2. You can't say that a "Day is Done" just by having an
23
+ implementation. Implementation without validating correctness is
24
+ asking for trouble.
25
+
26
+ 3. Avoid "Vanishing Mind" syndrome in testing, i.e. failing to reuse
27
+ existing tests. After all, we are relying on a generator, which
28
+ is hopefully comprehensively tested. The test mechanism and test
29
+ suites should be available also for template generator users.
30
+
31
+ 4. When reusing tests: "Do You Remember"/"It All Starts With One": the
32
+ tests that come along with the generator should also be open for
33
+ extensions.
34
+
35
+
36
+ ## The solution
37
+
38
+ **aws-must-templates** tries to address the above listed considerations
39
+
40
+ 1. by allowing users to [extend](#OVERRIDE) template generation
41
+ process
42
+
43
+ 2. by including a [configurable](#TEST-SUITES) [test runner](#TESTING) to lower
44
+ the threshold to write tests
45
+
46
+ 3. by allowing users [to reuse](#TEST_CASES)
47
+ [test cases](generated-docs/test-suites.md) used
48
+ in **aws-must-templates** quality assurance
49
+
50
+ 4. by having the possibility to [include own test](#TEST_CASES)
51
+ cases to test suites
52
+
53
+
54
+ ## Usage
55
+
56
+ ### Installation
57
+
58
+ Add the following lines to `Gemfile`
59
+
60
+ source 'https://rubygems.org'
61
+ gem 'aws-must-templates'
62
+
63
+ and run
64
+
65
+ bundle install
66
+
67
+ **Notice**: requires Ruby version ~> 2.0.
68
+
69
+ ### Configuration
70
+
71
+ Create a YAML configuration for a CloudFormation stack using
72
+ attributes referenced by
73
+ [aws-must-templates](https://rawgit.com/jarjuk/aws-must-templates/master/generated-docs/aws-must-templates.html).
74
+
75
+ For a configuration walk trough see blog post
76
+ [Announcing aws-must-templates - part 1](https://jarjuk.wordpress.com/2015/08/18/announcing-aws-must-templates-part1)
77
+
78
+ The easiest way to start, is to take a look at YAML configurations
79
+ used to validate **aws-must-templates** implementation. For, example
80
+
81
+ * [smoke.yaml](smoke.yaml): creates a S3 bucket
82
+
83
+ * [suite1.yaml](suite1.yaml): creates two EC2 instances, and one S3
84
+ bucket, one of the instances (`myInstance`) which is granted a read
85
+ access to the S3 bucket
86
+
87
+ See [test report](generated-docs/test-suites.md) of
88
+ **aws-must-templates** for more information.
89
+
90
+ ### Generate CloudFormation JSON templates
91
+
92
+ Assuming a YAML stack configuration in a file `mystack.yaml`, the
93
+ command
94
+
95
+ bundle exec aws-must.rb gen mystack.yaml -m aws-must-templates
96
+
97
+ prints the generated CloudFromation JSON template to STDOUT.
98
+
99
+ ### Overriding template implementation<a id="OVERRIDE"/>
100
+
101
+ For a better match of a particular need, user may modify template
102
+ generation by overriding
103
+ [templates](https://rawgit.com/jarjuk/aws-must-templates/master/generated-docs/aws-must-templates.html)
104
+ in **aws-must-templates** with own implementations.
105
+
106
+ For example, the default
107
+ [AIM mapping table](https://rawgit.com/jarjuk/aws-must-templates/master/generated-docs/aws-must-templates.html#mappings.mustache)
108
+ in **aws-must-templates** supports only `t2.micro` instance type for
109
+ type Ubuntu 14.04 LTS Trusty operating system version.
110
+
111
+ Saving the following template in `myextensions/mappings.mustache`
112
+
113
+
114
+ {{!
115
+ +++start+++
116
+
117
+ Use Ubuntu `utopic` v. 14.10 for `t2.micro` `instanceType`.
118
+
119
+ +++close+++
120
+ }}
121
+
122
+ {{! +++fold-on+++ }}
123
+
124
+ "AWSInstanceType2Arch" : {
125
+ "t2.micro" : { "Arch" : "64" }
126
+ },
127
+ "AWSRegionArch2AMI" : {
128
+ "ap-northeast-1" : { "64" : "ami-50c27450" },
129
+ "ap-southeast-1" : { "64" : "ami-8ae3e1d8" },
130
+ "ap-southeast-2" : { "64" : "ami-25eea81f" },
131
+ "cn-north-1" : { "64" : "ami-9671ecaf" },
132
+ "eu-central-1" : { "64" : "ami-84333699" },
133
+ "eu-west-1" : { "64" : "ami-b4a5eec3" },
134
+ "sa-east-1" : { "64" : "ami-0f199612" },
135
+ "us-east-1" : { "64" : "ami-d36cb0b8" },
136
+ "us-west-1" : { "64" : "ami-33fc9c10" },
137
+ "us-gov-west-1" : { "64" : "ami-77887533" },
138
+ "us-west-2" : { "64" : "ami-dd353aed" }
139
+ }
140
+
141
+ {{! +++fold-off+++ }}
142
+
143
+ and issuing the command
144
+
145
+ bundle exec aws-must.rb gen mystack.yaml -m myextensions/ aws-must-templates
146
+
147
+ uses Ubuntu `utopic` v. 14.10 for `t2.micro` instance types.
148
+
149
+ **NOTICE**: In `-m` option, directory names need to end with slash
150
+ character to distinguish them from Gem names.
151
+
152
+ ### Provision the stack on Amazon platform <a id="PROVISION"/>
153
+
154
+ **WARNING** Provisioning CloudFormation templates on Amazon will be
155
+ **charged according to Amazon pricing policies**.
156
+
157
+ Assuming that [Aws Command Line Utility](https://aws.amazon.com/cli)
158
+ is installed and
159
+ [setup correctly](http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-set-up.html),
160
+ the command
161
+
162
+ aws cloudformation create-stack --stack-name mystack --capabilities CAPABILITY_IAM --template-body "$(bundle exec aws-must.rb gen mystack.yaml -m myextensions/ aws-must-templates)" --disable-rollback
163
+
164
+ provisions stack `mystack` from YAML configuration in `mystack.yaml`.
165
+ Templates in `myextensions` directory override default implementations
166
+ in **aws-must-templates** Gem.
167
+
168
+ ## Testing CloudFormation stacks using **aws-must-templates** Test Runner<a id="TESTING"/>
169
+
170
+ CloudFormation stacks can be tested using Test Runner that comes along
171
+ with **aws-must-templates** Gem.
172
+
173
+ To use the **aws-must-templates** Test Runner
174
+
175
+ * [setup for Test Runner](#SETUP-TEST-RUNNER)
176
+ * [prepare Test Context](#TEST-CONTEXT)
177
+ * [implement Test Cases](#TEST_CASES)
178
+ * [configure Test Suites](#TEST-SUITES) in `test-suites.yaml` -file
179
+ * [run tests](#RUN-TESTS)
180
+
181
+
182
+ ### Setup for Test Runner<a id="SETUP-TEST-RUNNER"/>
183
+
184
+ Ensure that `Gemfile` includes
185
+
186
+ gem 'rake'
187
+ gem 'rspec'
188
+ gem 'serverspec'
189
+ gem 'aws-must-templates'
190
+
191
+ Run `bundle install`, if new gems were added to the `Gemfile`.
192
+
193
+ Add following lines to `Rakefile`
194
+
195
+ spec = Gem::Specification.find_by_name 'aws-must-templates'
196
+ load "#{spec.gem_dir}/lib/tasks/suite.rake"
197
+
198
+ and run
199
+
200
+ bundle exec rake -T suite
201
+
202
+ to show new tasks for the Test Runner.
203
+
204
+ ### Prepare Test Context <a id="TEST-CONTEXT"/>
205
+
206
+ Preparing Test Context includes
207
+
208
+ * Setting up AWS Account
209
+ * Preparing ssh-connection configuration
210
+
211
+ **Setup AWS Account**
212
+
213
+ In order to access an EC2 instances, to AWS needs to have a valid key
214
+ pair
215
+ [imported](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html#how-to-generate-your-own-key-and-import-it-to-aws).
216
+
217
+ **Prepare ssh-connection configuration**
218
+
219
+ Test Runner uses SSH connection to access an EC2 instances, and needs
220
+ an entry in `ssh/config` file defining user name, and ssh-key used in
221
+ authentication, etc.
222
+
223
+ For example, the SSH configuration below for an EC2 instance
224
+ `myInstance` defines `IdentityFile` parameter matching `demo-key` EC2
225
+ key used in **aws-must-templates** test suites.
226
+
227
+ host myInstance
228
+ StrictHostKeyChecking no
229
+ UserKnownHostsFile=/dev/null
230
+ user ubuntu
231
+ IdentityFile ~/.ssh/demo-key/demo-key
232
+
233
+ Parameters `UserKnownHostsFile` and `StrictHostKeyChecking` prevent
234
+ ssh from updating your default `.ssh/known_hosts` file with the
235
+ fingerprint of the (temporary) instance used in testing.
236
+
237
+ ### Implement Test Cases<a id="TEST_CASES"/>
238
+
239
+ Test Runner searches Test Cases first from sub-directories under
240
+ `spec/aws-must-templates` directory, and if not found, tries to locate
241
+ Test Cases in Gem **aws-must-templates**. This allows user to reuse
242
+ existing test in **aws-must-templates** and implement own Test Cases.
243
+
244
+ Test runner uses Rspec with [serverspec](http://serverspec.org/)
245
+ library. For example, `ValidOSVersion` Test Case in
246
+ **aws-must-templates**, validates operating system version
247
+ with the code shown:
248
+
249
+ require 'spec_helper'
250
+
251
+ current_test = "ValidOSVersion"
252
+
253
+ describe current_test do
254
+
255
+ # parameter 'Codename' defined in test-suites.yaml
256
+ codename = test_parameter( current_test, "Codename" )
257
+
258
+ describe "Operating system codename '#{codename.value}'" do
259
+ describe command('lsb_release --c -s') do
260
+ its( :stdout ) { should match /#{codename.value}/ }
261
+ end
262
+ end
263
+ end
264
+
265
+
266
+ See [test report](generated-docs/test-suites.md) created when running
267
+ suites defined in [test-suites.yaml](test-suites.yaml), and
268
+ [diagram](generated-docs/xref_suite_X_test.pdf) presenting Test Cases
269
+ used in Test Suites.
270
+
271
+ ### Configure Test Suites <a id="TEST-SUITES"/>
272
+
273
+ Test suites are configured in `test-suites.yaml`. Test runner search
274
+ this file in current working directory.
275
+
276
+ The picture below present main elements used in `test-suites.yaml`.
277
+
278
+ ![test-suites.yaml elements](./pics/test-suites.jpg)
279
+
280
+ A Test Suite validates the correctness of a CloudFormation Stack. One
281
+ Test Suite defines tests for multiple EC2 Instances. Each EC2 Instance
282
+ must have a corresponding SSH Connection prepared in
283
+ [ssh/config](#TEST-CONTEXT) -file. An EC2 Instance acts in many
284
+ Roles. A Role maps to a [Test Case](#TEST_CASES), and and defines
285
+ values for the Test Case Parameters. The parameter may be a constant,
286
+ or a reference to Stack Parameter, or to Stack Output.
287
+
288
+ An example Test Case for `mystack` is shown below
289
+
290
+ - mystack:
291
+ desc: Copy of suite1 EC2 instance with s3 access
292
+ instances:
293
+ - myInstance:
294
+ roles:
295
+ - ValidOSVersion:
296
+ Codename: utopic
297
+ - S3ReadAccessAllowed:
298
+ Bucket: "@Outputs.Bucket"
299
+
300
+ This configuration validates stack `mystack` by running two Test Cases
301
+ `ValidOSVersion` and `S3ReadAccessAllowed` on EC2 instance
302
+ `myInstance`.
303
+
304
+ Test Case ValidOSVersion was presented [earlier](#TEST_CASES). It uses
305
+ parameter `Codename` to validate operating system version. In this
306
+ configuration, the parameter value is constant `utopic`. Effectively
307
+ this validates that CloudFormation mapping used to
308
+ [override](#OVERRIDE) the default implementation in
309
+ **aws-must-templates** is successfully provisioned.
310
+
311
+ Test Case `S3ReadAccessAllowed` is passed a parameter, which gets its
312
+ value from stack output variable `Bucket`.
313
+
314
+ See [test-suites.yaml](test-suites.yaml) in **aws-must-templatest**
315
+ for more detailed explanation, and for more examples.
316
+
317
+ ### Running test suites <a id="RUN-TESTS"/>
318
+
319
+ **WARNING** Running tests provisions Amazon platform, and will be
320
+ **charged according to Amazon pricing policies**.
321
+
322
+ **NOTICE** It advisable to check on AWS console that all stack
323
+ resources are deleted successfully after running test suites.
324
+
325
+ To run test suite `mystack` defined in in `test-suites.yml` using
326
+ default templates in **aws-must-templates** use the command
327
+
328
+ bundle exec rake suite:mystack
329
+
330
+ To [override](#OVERRIDE) default implementation with templates in
331
+ directory `myextensions`, use the command
332
+
333
+ bundle exec rake suite:mystack['-m myextensions/ aws-must-templates']
334
+
335
+
336
+ For a Test Suite, Test Runner
337
+
338
+ * generates a CloudFormation JSON template,
339
+ * uses the JSON template to provision a stack on Amazon platform,
340
+ * and, once the `StackStatus` is `CREATE_COMPLETE`,
341
+ * iterates EC2 Instances and runs Test Cases in the EC2 Instance Role
342
+ * creates a test report to `generated-doc/suites` directory
343
+ * finally, after the test execution, the stack is deleted from Amazon platform.
344
+
345
+
346
+ To run all test suites defined in `test-suites.yaml`, use the command
347
+
348
+ bundle exec rake suite:all
349
+
350
+ or to override the default implementation
351
+
352
+ bundle exec rake suite:all['-m myextensions/ aws-must-templates']
353
+
354
+ Command
355
+
356
+ rake -T suite
357
+
358
+ list of tasks `rake suite:all` uses for implementation.
359
+
360
+ ## Development
361
+
362
+ See [DEVELOPMENT](DEVELOPMENT.md)
363
+
364
+ ## Changes
365
+
366
+ See [RELEASES](RELEASES.md)
367
+
368
+ ## TODO
369
+
370
+ Add more tests, e.g.
371
+
372
+ * VPC and subnets
373
+ * install Chef
374
+
375
+ Add more template support
376
+
377
+ * support for SNS notifications
378
+
379
+ Better documentation
380
+
381
+ * use RDoc to document Test Cases
382
+
383
+
384
+
385
+ ## License
386
+
387
+ MIT
388
+
389
+
390
+
@@ -0,0 +1,2 @@
1
+ require_relative "test-suites/test_suites.rb"
2
+
@@ -0,0 +1,78 @@
1
+
2
+ # extract cross refences from reports in 'test_reports' FileList
3
+ def build_cross_refs( test_reports )
4
+
5
+ xref_suite_X_test = {}
6
+ xref_test_X_suite = {}
7
+
8
+ # Iterate all test reports in 'generated-docs/suites'
9
+ test_reports.each do |test_report_file|
10
+
11
+ # puts "test_report_file=#{test_report_file}"
12
+
13
+ # Extract suite_id from test file name
14
+ # - generated-docs/suites/smoke.txt --> smoke
15
+ # - generated-docs/suites/suite1-myInstance.txt --> suite1
16
+ suite_id = File.basename(test_report_file)[ /([^.-]+)/,1]
17
+
18
+ # Extract test name from lines with non-inteded single word
19
+ tests = File.readlines( test_report_file ).select{ |line| line =~ /^\w\w*$/ }.map{ |line| line.strip }
20
+
21
+ # Build cross refrences
22
+
23
+ # suite seen for the first time
24
+ xref_suite_X_test[suite_id] = [] unless xref_suite_X_test[suite_id]
25
+ tests.each{ |test| xref_suite_X_test[suite_id].push( test ) unless xref_suite_X_test[suite_id].include?( test ) }
26
+
27
+ tests.each do |test|
28
+ # test seen for the first time
29
+ xref_test_X_suite[test] = [] unless xref_test_X_suite[test]
30
+ xref_test_X_suite[test].push( suite_id ) unless xref_test_X_suite[test].include?( suite_id )
31
+ end
32
+
33
+ # puts "suite_id = #{suite_id}, tests=#{tests}"
34
+
35
+ end
36
+
37
+ # Cross references resolved
38
+
39
+ # puts "xref_suite_X_test=#{xref_suite_X_test}"
40
+ # puts "xref_test_X_suite=#{xref_test_X_suite}"
41
+
42
+ return xref_suite_X_test, xref_test_X_suite
43
+
44
+ end
45
+
46
+ # output graphviz
47
+ def xref_to_dot( xref_suite_X_test, xref_test_X_suite )
48
+
49
+ puts <<-EOS
50
+ digraph {
51
+ rankdir=TB
52
+
53
+ node [ fontname = "Courier"
54
+ fontsize = 8
55
+ shape = "record"
56
+
57
+ ];
58
+ edge [
59
+ fontname = "Bitstream Vera Sans"
60
+ fontsize = 8
61
+ // arrowhead = "none"
62
+ ];
63
+ EOS
64
+
65
+ xref_test_X_suite.each do |test,suistes|
66
+ puts " #{test}"
67
+ end
68
+
69
+ xref_suite_X_test.each do |suite,tests|
70
+ puts " #{suite} [shape=\"ellipse\"]"
71
+ tests.each { |test| puts " #{suite} -> #{test}" }
72
+
73
+ end
74
+
75
+ puts "}"
76
+
77
+
78
+ end