aws-must-templates 0.1.2

Sign up to get free protection for your applications and to get access to all the features.
Files changed (46) hide show
  1. checksums.yaml +7 -0
  2. data/README.md +390 -0
  3. data/lib/aws-must-templates.rb +2 -0
  4. data/lib/tasks/cross-ref.rb +78 -0
  5. data/lib/tasks/suite.rake +292 -0
  6. data/lib/test-suites/test_suites.rb +115 -0
  7. data/mustache/commonCfnSignal.mustache +34 -0
  8. data/mustache/commonCreationPolicy.mustache +33 -0
  9. data/mustache/commonDependsOn.mustache +25 -0
  10. data/mustache/commonInstanceType.mustache +27 -0
  11. data/mustache/commonKeyValue.mustache +28 -0
  12. data/mustache/commonStackRef.mustache +23 -0
  13. data/mustache/commonValue.mustache +34 -0
  14. data/mustache/initializeCFinit.mustache +27 -0
  15. data/mustache/initializeCFtools.mustache +27 -0
  16. data/mustache/initializeCfnInitHupFiles.mustache +85 -0
  17. data/mustache/initializeInstallAwsCli.mustache +32 -0
  18. data/mustache/initializeInstallChef.mustache +37 -0
  19. data/mustache/initializeProvisionChefZero.mustache +36 -0
  20. data/mustache/initializeStartCfnHup.mustache +25 -0
  21. data/mustache/mapping.mustache +27 -0
  22. data/mustache/mappingSubnetConfig.mustache +27 -0
  23. data/mustache/mappings.mustache +71 -0
  24. data/mustache/output.mustache +38 -0
  25. data/mustache/parameter.mustache +35 -0
  26. data/mustache/resource.mustache +51 -0
  27. data/mustache/resourceInstance.mustache +52 -0
  28. data/mustache/resourceInstanceInitialize.mustache +120 -0
  29. data/mustache/resourceInstanceMetadata.mustache +65 -0
  30. data/mustache/resourceInstanceProfile.mustache +34 -0
  31. data/mustache/resourceInternetGateway.mustache +87 -0
  32. data/mustache/resourcePolicy.mustache +50 -0
  33. data/mustache/resourceProvisionChef.mustache +38 -0
  34. data/mustache/resourceRole.mustache +40 -0
  35. data/mustache/resourceS3Bucket.mustache +39 -0
  36. data/mustache/resourceSecurityGroup.mustache +46 -0
  37. data/mustache/resourceStack.mustache +45 -0
  38. data/mustache/resourceSubnet.mustache +41 -0
  39. data/mustache/resourceUser.mustache +58 -0
  40. data/mustache/resourceVPC.mustache +44 -0
  41. data/mustache/resourceWait.mustache +39 -0
  42. data/mustache/resources.mustache +65 -0
  43. data/mustache/root.mustache +244 -0
  44. data/mustache/tag.mustache +27 -0
  45. data/pics/test-suites.jpg +0 -0
  46. metadata +146 -0
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA1:
3
+ metadata.gz: cf3a6e500d517d701c0215e0477fa9a8d595b979
4
+ data.tar.gz: 1b2ac0944b78a61036e66e217c39eb47623a2a5e
5
+ SHA512:
6
+ metadata.gz: 0c17f5daae17aabf52b9cd254f9cd13e5d5189c7cd8125f1299fc1e4ab15a30e28693da298a50447fc2cac4dacd8ba86f3db744a448f80bcdf9487ffce2cf9dd
7
+ data.tar.gz: 075de499bb571c08655a4a35df483ceaa4993bcae8c8ee75fe5a458a0420f8832df17a617c4b89a81ca3edf7656af809d5d72aff59361ce7c5898dad4bcf41eb
data/README.md ADDED
@@ -0,0 +1,390 @@
1
+ # aws-must-templates - cloudformation templates for aws-must - $Release:0.1.2$
2
+
3
+ Set of [extensible](#OVERRIDE)
4
+ [templates](https://rawgit.com/jarjuk/aws-must-templates/master/generated-docs/aws-must-templates.html)
5
+ for [aws-must](https://github.com/jarjuk/aws-must) tool to generate
6
+ CloudFormation JSON from a YAML configuration, and a
7
+ [Test Runner](#TESTING) for [validating correctness](#TEST_CASES) of
8
+ CloudFormation stacks provisioned.
9
+
10
+
11
+ ## The Problem
12
+
13
+ When using code generators, consider
14
+
15
+ 1. Code generators are much like a "Garden Party": Most of the things
16
+ that you need may be there, but some specific feature is quite
17
+ likely to be missing, or should be implemented in a different way.
18
+ "When you can't please everyone, you got to please yourself" - and
19
+ use the possibility to modify template generation process to meet
20
+ your own needs.
21
+
22
+ 2. You can't say that a "Day is Done" just by having an
23
+ implementation. Implementation without validating correctness is
24
+ asking for trouble.
25
+
26
+ 3. Avoid "Vanishing Mind" syndrome in testing, i.e. failing to reuse
27
+ existing tests. After all, we are relying on a generator, which
28
+ is hopefully comprehensively tested. The test mechanism and test
29
+ suites should be available also for template generator users.
30
+
31
+ 4. When reusing tests: "Do You Remember"/"It All Starts With One": the
32
+ tests that come along with the generator should also be open for
33
+ extensions.
34
+
35
+
36
+ ## The solution
37
+
38
+ **aws-must-templates** tries to address the above listed considerations
39
+
40
+ 1. by allowing users to [extend](#OVERRIDE) template generation
41
+ process
42
+
43
+ 2. by including a [configurable](#TEST-SUITES) [test runner](#TESTING) to lower
44
+ the threshold to write tests
45
+
46
+ 3. by allowing users [to reuse](#TEST_CASES)
47
+ [test cases](generated-docs/test-suites.md) used
48
+ in **aws-must-templates** quality assurance
49
+
50
+ 4. by having the possibility to [include own test](#TEST_CASES)
51
+ cases to test suites
52
+
53
+
54
+ ## Usage
55
+
56
+ ### Installation
57
+
58
+ Add the following lines to `Gemfile`
59
+
60
+ source 'https://rubygems.org'
61
+ gem 'aws-must-templates'
62
+
63
+ and run
64
+
65
+ bundle install
66
+
67
+ **Notice**: requires Ruby version ~> 2.0.
68
+
69
+ ### Configuration
70
+
71
+ Create a YAML configuration for a CloudFormation stack using
72
+ attributes referenced by
73
+ [aws-must-templates](https://rawgit.com/jarjuk/aws-must-templates/master/generated-docs/aws-must-templates.html).
74
+
75
+ For a configuration walk trough see blog post
76
+ [Announcing aws-must-templates - part 1](https://jarjuk.wordpress.com/2015/08/18/announcing-aws-must-templates-part1)
77
+
78
+ The easiest way to start, is to take a look at YAML configurations
79
+ used to validate **aws-must-templates** implementation. For, example
80
+
81
+ * [smoke.yaml](smoke.yaml): creates a S3 bucket
82
+
83
+ * [suite1.yaml](suite1.yaml): creates two EC2 instances, and one S3
84
+ bucket, one of the instances (`myInstance`) which is granted a read
85
+ access to the S3 bucket
86
+
87
+ See [test report](generated-docs/test-suites.md) of
88
+ **aws-must-templates** for more information.
89
+
90
+ ### Generate CloudFormation JSON templates
91
+
92
+ Assuming a YAML stack configuration in a file `mystack.yaml`, the
93
+ command
94
+
95
+ bundle exec aws-must.rb gen mystack.yaml -m aws-must-templates
96
+
97
+ prints the generated CloudFromation JSON template to STDOUT.
98
+
99
+ ### Overriding template implementation<a id="OVERRIDE"/>
100
+
101
+ For a better match of a particular need, user may modify template
102
+ generation by overriding
103
+ [templates](https://rawgit.com/jarjuk/aws-must-templates/master/generated-docs/aws-must-templates.html)
104
+ in **aws-must-templates** with own implementations.
105
+
106
+ For example, the default
107
+ [AIM mapping table](https://rawgit.com/jarjuk/aws-must-templates/master/generated-docs/aws-must-templates.html#mappings.mustache)
108
+ in **aws-must-templates** supports only `t2.micro` instance type for
109
+ type Ubuntu 14.04 LTS Trusty operating system version.
110
+
111
+ Saving the following template in `myextensions/mappings.mustache`
112
+
113
+
114
+ {{!
115
+ +++start+++
116
+
117
+ Use Ubuntu `utopic` v. 14.10 for `t2.micro` `instanceType`.
118
+
119
+ +++close+++
120
+ }}
121
+
122
+ {{! +++fold-on+++ }}
123
+
124
+ "AWSInstanceType2Arch" : {
125
+ "t2.micro" : { "Arch" : "64" }
126
+ },
127
+ "AWSRegionArch2AMI" : {
128
+ "ap-northeast-1" : { "64" : "ami-50c27450" },
129
+ "ap-southeast-1" : { "64" : "ami-8ae3e1d8" },
130
+ "ap-southeast-2" : { "64" : "ami-25eea81f" },
131
+ "cn-north-1" : { "64" : "ami-9671ecaf" },
132
+ "eu-central-1" : { "64" : "ami-84333699" },
133
+ "eu-west-1" : { "64" : "ami-b4a5eec3" },
134
+ "sa-east-1" : { "64" : "ami-0f199612" },
135
+ "us-east-1" : { "64" : "ami-d36cb0b8" },
136
+ "us-west-1" : { "64" : "ami-33fc9c10" },
137
+ "us-gov-west-1" : { "64" : "ami-77887533" },
138
+ "us-west-2" : { "64" : "ami-dd353aed" }
139
+ }
140
+
141
+ {{! +++fold-off+++ }}
142
+
143
+ and issuing the command
144
+
145
+ bundle exec aws-must.rb gen mystack.yaml -m myextensions/ aws-must-templates
146
+
147
+ uses Ubuntu `utopic` v. 14.10 for `t2.micro` instance types.
148
+
149
+ **NOTICE**: In `-m` option, directory names need to end with slash
150
+ character to distinguish them from Gem names.
151
+
152
+ ### Provision the stack on Amazon platform <a id="PROVISION"/>
153
+
154
+ **WARNING** Provisioning CloudFormation templates on Amazon will be
155
+ **charged according to Amazon pricing policies**.
156
+
157
+ Assuming that [Aws Command Line Utility](https://aws.amazon.com/cli)
158
+ is installed and
159
+ [setup correctly](http://docs.aws.amazon.com/cli/latest/userguide/cli-chap-getting-set-up.html),
160
+ the command
161
+
162
+ aws cloudformation create-stack --stack-name mystack --capabilities CAPABILITY_IAM --template-body "$(bundle exec aws-must.rb gen mystack.yaml -m myextensions/ aws-must-templates)" --disable-rollback
163
+
164
+ provisions stack `mystack` from YAML configuration in `mystack.yaml`.
165
+ Templates in `myextensions` directory override default implementations
166
+ in **aws-must-templates** Gem.
167
+
168
+ ## Testing CloudFormation stacks using **aws-must-templates** Test Runner<a id="TESTING"/>
169
+
170
+ CloudFormation stacks can be tested using Test Runner that comes along
171
+ with **aws-must-templates** Gem.
172
+
173
+ To use the **aws-must-templates** Test Runner
174
+
175
+ * [setup for Test Runner](#SETUP-TEST-RUNNER)
176
+ * [prepare Test Context](#TEST-CONTEXT)
177
+ * [implement Test Cases](#TEST_CASES)
178
+ * [configure Test Suites](#TEST-SUITES) in `test-suites.yaml` -file
179
+ * [run tests](#RUN-TESTS)
180
+
181
+
182
+ ### Setup for Test Runner<a id="SETUP-TEST-RUNNER"/>
183
+
184
+ Ensure that `Gemfile` includes
185
+
186
+ gem 'rake'
187
+ gem 'rspec'
188
+ gem 'serverspec'
189
+ gem 'aws-must-templates'
190
+
191
+ Run `bundle install`, if new gems were added to the `Gemfile`.
192
+
193
+ Add following lines to `Rakefile`
194
+
195
+ spec = Gem::Specification.find_by_name 'aws-must-templates'
196
+ load "#{spec.gem_dir}/lib/tasks/suite.rake"
197
+
198
+ and run
199
+
200
+ bundle exec rake -T suite
201
+
202
+ to show new tasks for the Test Runner.
203
+
204
+ ### Prepare Test Context <a id="TEST-CONTEXT"/>
205
+
206
+ Preparing Test Context includes
207
+
208
+ * Setting up AWS Account
209
+ * Preparing ssh-connection configuration
210
+
211
+ **Setup AWS Account**
212
+
213
+ In order to access an EC2 instances, to AWS needs to have a valid key
214
+ pair
215
+ [imported](http://docs.aws.amazon.com/AWSEC2/latest/UserGuide/ec2-key-pairs.html#how-to-generate-your-own-key-and-import-it-to-aws).
216
+
217
+ **Prepare ssh-connection configuration**
218
+
219
+ Test Runner uses SSH connection to access an EC2 instances, and needs
220
+ an entry in `ssh/config` file defining user name, and ssh-key used in
221
+ authentication, etc.
222
+
223
+ For example, the SSH configuration below for an EC2 instance
224
+ `myInstance` defines `IdentityFile` parameter matching `demo-key` EC2
225
+ key used in **aws-must-templates** test suites.
226
+
227
+ host myInstance
228
+ StrictHostKeyChecking no
229
+ UserKnownHostsFile=/dev/null
230
+ user ubuntu
231
+ IdentityFile ~/.ssh/demo-key/demo-key
232
+
233
+ Parameters `UserKnownHostsFile` and `StrictHostKeyChecking` prevent
234
+ ssh from updating your default `.ssh/known_hosts` file with the
235
+ fingerprint of the (temporary) instance used in testing.
236
+
237
+ ### Implement Test Cases<a id="TEST_CASES"/>
238
+
239
+ Test Runner searches Test Cases first from sub-directories under
240
+ `spec/aws-must-templates` directory, and if not found, tries to locate
241
+ Test Cases in Gem **aws-must-templates**. This allows user to reuse
242
+ existing test in **aws-must-templates** and implement own Test Cases.
243
+
244
+ Test runner uses Rspec with [serverspec](http://serverspec.org/)
245
+ library. For example, `ValidOSVersion` Test Case in
246
+ **aws-must-templates**, validates operating system version
247
+ with the code shown:
248
+
249
+ require 'spec_helper'
250
+
251
+ current_test = "ValidOSVersion"
252
+
253
+ describe current_test do
254
+
255
+ # parameter 'Codename' defined in test-suites.yaml
256
+ codename = test_parameter( current_test, "Codename" )
257
+
258
+ describe "Operating system codename '#{codename.value}'" do
259
+ describe command('lsb_release --c -s') do
260
+ its( :stdout ) { should match /#{codename.value}/ }
261
+ end
262
+ end
263
+ end
264
+
265
+
266
+ See [test report](generated-docs/test-suites.md) created when running
267
+ suites defined in [test-suites.yaml](test-suites.yaml), and
268
+ [diagram](generated-docs/xref_suite_X_test.pdf) presenting Test Cases
269
+ used in Test Suites.
270
+
271
+ ### Configure Test Suites <a id="TEST-SUITES"/>
272
+
273
+ Test suites are configured in `test-suites.yaml`. Test runner search
274
+ this file in current working directory.
275
+
276
+ The picture below present main elements used in `test-suites.yaml`.
277
+
278
+ ![test-suites.yaml elements](./pics/test-suites.jpg)
279
+
280
+ A Test Suite validates the correctness of a CloudFormation Stack. One
281
+ Test Suite defines tests for multiple EC2 Instances. Each EC2 Instance
282
+ must have a corresponding SSH Connection prepared in
283
+ [ssh/config](#TEST-CONTEXT) -file. An EC2 Instance acts in many
284
+ Roles. A Role maps to a [Test Case](#TEST_CASES), and and defines
285
+ values for the Test Case Parameters. The parameter may be a constant,
286
+ or a reference to Stack Parameter, or to Stack Output.
287
+
288
+ An example Test Case for `mystack` is shown below
289
+
290
+ - mystack:
291
+ desc: Copy of suite1 EC2 instance with s3 access
292
+ instances:
293
+ - myInstance:
294
+ roles:
295
+ - ValidOSVersion:
296
+ Codename: utopic
297
+ - S3ReadAccessAllowed:
298
+ Bucket: "@Outputs.Bucket"
299
+
300
+ This configuration validates stack `mystack` by running two Test Cases
301
+ `ValidOSVersion` and `S3ReadAccessAllowed` on EC2 instance
302
+ `myInstance`.
303
+
304
+ Test Case ValidOSVersion was presented [earlier](#TEST_CASES). It uses
305
+ parameter `Codename` to validate operating system version. In this
306
+ configuration, the parameter value is constant `utopic`. Effectively
307
+ this validates that CloudFormation mapping used to
308
+ [override](#OVERRIDE) the default implementation in
309
+ **aws-must-templates** is successfully provisioned.
310
+
311
+ Test Case `S3ReadAccessAllowed` is passed a parameter, which gets its
312
+ value from stack output variable `Bucket`.
313
+
314
+ See [test-suites.yaml](test-suites.yaml) in **aws-must-templatest**
315
+ for more detailed explanation, and for more examples.
316
+
317
+ ### Running test suites <a id="RUN-TESTS"/>
318
+
319
+ **WARNING** Running tests provisions Amazon platform, and will be
320
+ **charged according to Amazon pricing policies**.
321
+
322
+ **NOTICE** It advisable to check on AWS console that all stack
323
+ resources are deleted successfully after running test suites.
324
+
325
+ To run test suite `mystack` defined in in `test-suites.yml` using
326
+ default templates in **aws-must-templates** use the command
327
+
328
+ bundle exec rake suite:mystack
329
+
330
+ To [override](#OVERRIDE) default implementation with templates in
331
+ directory `myextensions`, use the command
332
+
333
+ bundle exec rake suite:mystack['-m myextensions/ aws-must-templates']
334
+
335
+
336
+ For a Test Suite, Test Runner
337
+
338
+ * generates a CloudFormation JSON template,
339
+ * uses the JSON template to provision a stack on Amazon platform,
340
+ * and, once the `StackStatus` is `CREATE_COMPLETE`,
341
+ * iterates EC2 Instances and runs Test Cases in the EC2 Instance Role
342
+ * creates a test report to `generated-doc/suites` directory
343
+ * finally, after the test execution, the stack is deleted from Amazon platform.
344
+
345
+
346
+ To run all test suites defined in `test-suites.yaml`, use the command
347
+
348
+ bundle exec rake suite:all
349
+
350
+ or to override the default implementation
351
+
352
+ bundle exec rake suite:all['-m myextensions/ aws-must-templates']
353
+
354
+ Command
355
+
356
+ rake -T suite
357
+
358
+ list of tasks `rake suite:all` uses for implementation.
359
+
360
+ ## Development
361
+
362
+ See [DEVELOPMENT](DEVELOPMENT.md)
363
+
364
+ ## Changes
365
+
366
+ See [RELEASES](RELEASES.md)
367
+
368
+ ## TODO
369
+
370
+ Add more tests, e.g.
371
+
372
+ * VPC and subnets
373
+ * install Chef
374
+
375
+ Add more template support
376
+
377
+ * support for SNS notifications
378
+
379
+ Better documentation
380
+
381
+ * use RDoc to document Test Cases
382
+
383
+
384
+
385
+ ## License
386
+
387
+ MIT
388
+
389
+
390
+
@@ -0,0 +1,2 @@
1
+ require_relative "test-suites/test_suites.rb"
2
+
@@ -0,0 +1,78 @@
1
+
2
+ # extract cross refences from reports in 'test_reports' FileList
3
+ def build_cross_refs( test_reports )
4
+
5
+ xref_suite_X_test = {}
6
+ xref_test_X_suite = {}
7
+
8
+ # Iterate all test reports in 'generated-docs/suites'
9
+ test_reports.each do |test_report_file|
10
+
11
+ # puts "test_report_file=#{test_report_file}"
12
+
13
+ # Extract suite_id from test file name
14
+ # - generated-docs/suites/smoke.txt --> smoke
15
+ # - generated-docs/suites/suite1-myInstance.txt --> suite1
16
+ suite_id = File.basename(test_report_file)[ /([^.-]+)/,1]
17
+
18
+ # Extract test name from lines with non-inteded single word
19
+ tests = File.readlines( test_report_file ).select{ |line| line =~ /^\w\w*$/ }.map{ |line| line.strip }
20
+
21
+ # Build cross refrences
22
+
23
+ # suite seen for the first time
24
+ xref_suite_X_test[suite_id] = [] unless xref_suite_X_test[suite_id]
25
+ tests.each{ |test| xref_suite_X_test[suite_id].push( test ) unless xref_suite_X_test[suite_id].include?( test ) }
26
+
27
+ tests.each do |test|
28
+ # test seen for the first time
29
+ xref_test_X_suite[test] = [] unless xref_test_X_suite[test]
30
+ xref_test_X_suite[test].push( suite_id ) unless xref_test_X_suite[test].include?( suite_id )
31
+ end
32
+
33
+ # puts "suite_id = #{suite_id}, tests=#{tests}"
34
+
35
+ end
36
+
37
+ # Cross references resolved
38
+
39
+ # puts "xref_suite_X_test=#{xref_suite_X_test}"
40
+ # puts "xref_test_X_suite=#{xref_test_X_suite}"
41
+
42
+ return xref_suite_X_test, xref_test_X_suite
43
+
44
+ end
45
+
46
+ # output graphviz
47
+ def xref_to_dot( xref_suite_X_test, xref_test_X_suite )
48
+
49
+ puts <<-EOS
50
+ digraph {
51
+ rankdir=TB
52
+
53
+ node [ fontname = "Courier"
54
+ fontsize = 8
55
+ shape = "record"
56
+
57
+ ];
58
+ edge [
59
+ fontname = "Bitstream Vera Sans"
60
+ fontsize = 8
61
+ // arrowhead = "none"
62
+ ];
63
+ EOS
64
+
65
+ xref_test_X_suite.each do |test,suistes|
66
+ puts " #{test}"
67
+ end
68
+
69
+ xref_suite_X_test.each do |suite,tests|
70
+ puts " #{suite} [shape=\"ellipse\"]"
71
+ tests.each { |test| puts " #{suite} -> #{test}" }
72
+
73
+ end
74
+
75
+ puts "}"
76
+
77
+
78
+ end