pupistry 0.0.7 → 0.0.8

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: a2438d4ff627cb0c9719ae22d33a026833365769
4
- data.tar.gz: ac00b03d3b81ad8bcac5f45d6d3bbc01fcfae733
3
+ metadata.gz: 2d77b057ae7f6a793146456653336f4d8518ad9c
4
+ data.tar.gz: 883a456f63a65d166c6c9fb27d94b249ff64d10d
5
5
  SHA512:
6
- metadata.gz: 91bcf3aecc457d6794b9b3f1c52cec5551ac185d8c69e84d295c38a4b0565a91b810601e121159e9476a16009b16bbf3f7969a84ee79b990a9fef878fcb47594
7
- data.tar.gz: c42a271c5b29e972b1295f9e4891332472340065a14b9327d70670e31be3fba08463c78b6cfebcf73d8818fa23b3ccae3f153c995659dd109c12698acdc4093b
6
+ metadata.gz: f502b9aea495831ccfa58e14ade4d146cf272c23aa039f94421ed1ec135ad42e8d4459302afbc66e975d0cc15dc5b4241724c339c7f9f6aecedcbb0100a017bb
7
+ data.tar.gz: aa02112d087ff67fe57b075c02571beaa607a194bfef37cd042cbaeb2ad54a7c6a8f8076ef68b8cede0fdf22e2b1ebbb22745cc4454ac1c5ee827744d4828877
data/README.md CHANGED
@@ -20,14 +20,15 @@ servers.
20
20
  Pupistry builds on the functionality offered by the r10k workflow but rather
21
21
  than requiring the implementing of site-specific custom bootstrap and custom
22
22
  workflow mechanisms, Pupistry executes r10k, assembles the combined modules
23
- and then generates a compress artifact file. It then signs the artifact with
24
- GPG and uploads it into an Amazon S3 bucket along with a manifest file.
23
+ and then generates a compress artifact file. It then optionally signs the
24
+ artifact with GPG and finally uploads it into an Amazon S3 bucket along with a
25
+ manifest file.
25
26
 
26
27
  The masterless Puppet machines then just run a Pupistry job which checks for a
27
28
  new version of the manifest file. If there is, it downloads the new artifact
28
- and does a GPG validation before applying it and running Puppet. To make life
29
- even easier, Pupistry will even spit out bootstrap files for your platform
30
- which sets up each server from scratch to pull and run the artifacts.
29
+ and does an optional GPG validation before applying it and running Puppet. To
30
+ make life even easier, Pupistry will even spit out bootstrap files for your
31
+ platform which sets up each server from scratch to pull and run the artifacts.
31
32
 
32
33
  Essentially Pupistry is intended to be a robust solution for masterless Puppet
33
34
  deployments and makes it trivial for beginners to get started with Puppet.
@@ -210,7 +211,6 @@ Alternatively if you like living on the edge, download this repository and run:
210
211
  gem install pupistry-VERSION.gem
211
212
  pupistry setup
212
213
 
213
- TODO: Currently setup not implemented, copy the sample file provided.
214
214
 
215
215
  ## 2. S3 Bucket
216
216
 
@@ -343,6 +343,31 @@ and running masterless Puppet environment using Pupistry. It covers the very
343
343
  basics of setting up your r10k environment.
344
344
 
345
345
 
346
+ # GPG Notes
347
+
348
+ GPG can be a bit of a beast to setup and get used to. Pupistry tries to make
349
+ the signing and key sharing process as simple as possible, but setting up GPG
350
+ on your platform and creating your key is beyond the scope of this document.
351
+
352
+ If you are being asked for your GPG password for every `pupistry push` even in
353
+ rapid succession, then you may need to setup gpg-agent so it can keep you
354
+ logged in for short durations to get a better balance of security vs usability.
355
+
356
+ Currently Pupistry supports a 1:1 approach, where the key used to sign the
357
+ artifact is the key used to verify it. Pull requests to add support for signing
358
+ and verifying against a keyring list would be welcome to make it easier for
359
+ teams to use GPG without having everyone with a single master key.
360
+
361
+ Note that GPG isn't vital for security - you still have end-to-end transport
362
+ security between your build machine and your servers via HTTPS/TLS to and from
363
+ the S3 bucket, all that GPG does is prevent anyone who managed to break into
364
+ your S3 bucket from pushing their own Puppet manifests out.
365
+
366
+ Generally S3 is secure (assuming no bugs in AWS itself), any likely exploit
367
+ would be from you accidentally sharing your IAM credentials in the wrong place,
368
+ or an exploited build server.
369
+
370
+
346
371
  # Caveats & Future Plans
347
372
 
348
373
  ## Use r10k
@@ -444,6 +469,10 @@ issue tracker is fine, but pull requests speak louder than words. :-)
444
469
  If you find a bug or need support, please use the issue tracker rather than
445
470
  personal emails to the author.
446
471
 
472
+ Feel free to grep the source for "TODO" comments on various tasks that
473
+ need doing.
474
+
475
+
447
476
 
448
477
  # Author
449
478
 
@@ -376,7 +376,11 @@ class CLI < Thor
376
376
  raise e
377
377
  end
378
378
 
379
- # Tell the user to edit it
379
+ # TODO: This is where I'd like to do things more cleverly. Currently we
380
+ # just tell the user to edit the configuration file, but would be really
381
+ # cool to write a setup wizard that helps them complete the configuration
382
+ # file and validates the input (eg AWS keys).
383
+
380
384
  if ENV['EDITOR']
381
385
  $logger.info "Now open the config file with `#{ENV['EDITOR']} #{config_dest}` and set your configuration values before running Pupistry."
382
386
  else
@@ -1,8 +1,8 @@
1
- require 'pupistry/config'
2
- require 'pupistry/bootstrap'
3
1
  require 'pupistry/artifact'
2
+ require 'pupistry/bootstrap'
3
+ require 'pupistry/config'
4
+ require 'pupistry/gpg'
4
5
  require 'pupistry/storage_aws'
5
6
 
6
7
 
7
-
8
8
  # vim:shiftwidth=2:tabstop=2:softtabstop=2:expandtab:smartindent
@@ -3,6 +3,7 @@ require 'yaml'
3
3
  require 'time'
4
4
  require 'digest'
5
5
  require 'fileutils'
6
+ require 'base64'
6
7
 
7
8
  module Pupistry
8
9
  # Pupistry::Artifact
@@ -215,9 +216,28 @@ module Pupistry
215
216
  $logger.warn "You have GPG signing *disabled*, whilst not critical it does weaken your security."
216
217
  $logger.warn "Skipping signing step..."
217
218
  else
218
- $logger.info "GPG signing the artifact with configured key"
219
219
 
220
- # TODO: should probably write this bit!
220
+ gpgsig = Pupistry::GPG.new @checksum
221
+
222
+ # Sign the artifact
223
+ unless gpgsig.artifact_sign
224
+ $logger.fatal "Unable to proceed with an unsigned artifact"
225
+ exit 0
226
+ end
227
+
228
+ # Verify the signature - we want to make sure what we've just signed
229
+ # can actually be validated properly :-)
230
+ unless gpgsig.artifact_verify
231
+ $logger.fatal "Whilst a signature was generated, it was unable to be validated. This would suggest a bug of some kind."
232
+ exit 0
233
+ end
234
+
235
+ # Save the signature to the manifest
236
+ unless gpgsig.signature_save
237
+ $logger.fatal "Unable to write the signature into the manifest file for the artifact."
238
+ exit 0
239
+ end
240
+
221
241
  end
222
242
 
223
243
 
@@ -313,7 +333,7 @@ module Pupistry
313
333
  "version" => @checksum,
314
334
  "date" => Time.new.inspect,
315
335
  "builduser" => ENV['USER'] || 'unlabled',
316
- "gpgsig" => 'unsighed',
336
+ "gpgsig" => 'unsigned',
317
337
  }
318
338
 
319
339
  begin
@@ -387,6 +407,22 @@ module Pupistry
387
407
  raise "Application bug, trying to install no artifact"
388
408
  end
389
409
 
410
+ # Validate the artifact if GPG is enabled.
411
+ if $config["general"]["gpg_disable"] == true
412
+ $logger.warn "You have GPG validation *disabled*, whilst not critical it does weaken your security."
413
+ $logger.warn "Skipping validation step..."
414
+ else
415
+
416
+ gpgsig = Pupistry::GPG.new @checksum
417
+
418
+ unless gpgsig.artifact_verify
419
+ $logger.fatal "The GPG signature could not be validated for the artifact. This could be a bug, a file corruption or a POSSIBLE SECURITY ISSUE such as maliciously modified content."
420
+ raise "Fatal unexpected error"
421
+ end
422
+
423
+ end
424
+
425
+
390
426
  # Make sure the artifact has been unpacked
391
427
  unless Dir.exists?($config["general"]["app_cache"] + "/artifacts/unpacked.#{@checksum}")
392
428
  $logger.error "The unpacked directory expected for #{@checksum} does not appear to exist or is not readable"
@@ -0,0 +1,283 @@
1
+ require 'rubygems'
2
+ require 'yaml'
3
+ require 'fileutils'
4
+ require 'base64'
5
+
6
+ module Pupistry
7
+ # Pupistry::GPG
8
+
9
+ class GPG
10
+ # All the functions needed for manipulating the GPG signatures
11
+ attr_accessor :checksum
12
+ attr_accessor :signature
13
+
14
+ def initialize checksum
15
+
16
+ # Need a checksum to do signing for
17
+ if checksum
18
+ @checksum = checksum
19
+ else
20
+ $logger.fatal "Probable bug, need a checksum provided with GPG validation"
21
+ exit 0
22
+ end
23
+
24
+ # Make sure that we have GPG available
25
+ unless system("gpg --version >> /dev/null 2>&1")
26
+ $logger.fatal "'gpg' command is not available, unable to do any signature creation or verification."
27
+ exit 0
28
+ end
29
+
30
+ end
31
+
32
+
33
+ # Sign the artifact and return the signature. Does not validation of the signature.
34
+ #
35
+ # false Failure
36
+ # base64 Encoded signature
37
+ #
38
+ def artifact_sign
39
+ @signature = 'unsigned'
40
+
41
+ # Clean up the existing signature file
42
+ signature_cleanup
43
+
44
+
45
+ Dir.chdir("#{$config["general"]["app_cache"]}/artifacts/") do
46
+
47
+ # Generate the signature file and pick up the signature data
48
+ unless system "gpg --use-agent --detach-sign artifact.#{@checksum}.tar.gz"
49
+ $logger.error "Unable to sign the artifact, an unexpected failure occured. No file uploaded."
50
+ return false
51
+ end
52
+
53
+ if File.exists?("artifact.#{@checksum}.tar.gz.sig")
54
+ $logger.info "A signature file was successfully generated."
55
+ else
56
+ $logger.error "A signature file was NOT generated."
57
+ return false
58
+ end
59
+
60
+ # Convert the signature into base64. It's easier to bundle all the
61
+ # metadata into a single file and extracting it out when needed, than
62
+ # having to keep track of yet-another-file. Because we encode into
63
+ # ASCII here, no need to call GPG with --armor either.
64
+
65
+ @signature = Base64.encode64(File.read("artifact.#{@checksum}.tar.gz.sig"))
66
+
67
+ unless @signature
68
+ $logger.error "An unexpected issue occured and no signature was generated"
69
+ return false
70
+ end
71
+ end
72
+
73
+
74
+ # Make sure the public key has been uploaded if it hasn't already
75
+ pubkey_upload
76
+
77
+ return @signature
78
+ end
79
+
80
+
81
+ # Verify the signature for a particular artifact.
82
+ #
83
+ # true Signature is legit
84
+ # false Signature is invalid (security issue!)
85
+ #
86
+ def artifact_verify
87
+
88
+ Dir.chdir("#{$config["general"]["app_cache"]}/artifacts/") do
89
+
90
+ if File.exists?("artifact.#{@checksum}.tar.gz.sig")
91
+ $logger.debug "Signature already extracted on disk, running verify...."
92
+ else
93
+ $logger.debug "Extracting signature from manifest data..."
94
+ signature_extract
95
+ end
96
+
97
+ # Verify the signature
98
+ unless pubkey_exists?
99
+ pubkey_install
100
+ end
101
+
102
+ output_verify = `gpg --quiet --status-fd 1 --verify artifact.#{@checksum}.tar.gz.sig 2>&1`
103
+
104
+ # Cleanup on disk file
105
+ signature_cleanup
106
+
107
+ # Was it valid?
108
+ output_verify.each_line do |line|
109
+
110
+ if /\[GNUPG:\]\sGOODSIG\s[A-Z0-9]*#{$config["general"]["gpg_signing_key"]}\s/.match(line)
111
+ $logger.info "Artifact #{@checksum} has a valid signature belonging to #{$config["general"]["gpg_signing_key"]}"
112
+ return true
113
+ end
114
+
115
+ if /\[GNUPG:\]\sBADSIG\s/.match(line)
116
+ $logger.fatal "Artifact #{@checksum} has AN INVALID GPG SECURITY SIGNATURE and could be CORRUPT or TAMPERED with."
117
+ exit 0
118
+ end
119
+
120
+ end
121
+
122
+ # Unexpected error
123
+ $logger.error "An unexpected validation issue occured, see below debug information:"
124
+
125
+ output_verify.each_line do |line|
126
+ $logger.error "GPG: #{line}"
127
+ end
128
+
129
+ end
130
+
131
+ # Something went wrong
132
+ $logger.fatal "Artifact #{@checksum} COULD NOT BE GPG VALIDATED and could be CORRUPT or TAMPERED with."
133
+ exit 0
134
+
135
+ end
136
+
137
+
138
+ # Generally we should clean up old signature files before and after using them
139
+ #
140
+ def signature_cleanup
141
+ FileUtils.rm("#{$config["general"]["app_cache"]}/artifacts/artifact.#{@checksum}.tar.gz.sig", :force => true)
142
+ end
143
+
144
+
145
+ # Extract the signature from the manifest file and write it to file in native binary format.
146
+ #
147
+ # false Unable to extract
148
+ # unsigned Manifest shows that the artifact is not signed
149
+ # base64 Encoded signature
150
+ #
151
+ def signature_extract
152
+ begin
153
+ manifest = YAML::load(File.open($config["general"]["app_cache"] + "/artifacts/manifest.#{@checksum}.yaml"))
154
+
155
+ if manifest['gpgsig']
156
+ # We have the base64 version
157
+ @signature = manifest['gpgsig']
158
+
159
+ # Decode the base64 and write the signature file
160
+ File.write("#{$config["general"]["app_cache"]}/artifacts/artifact.#{@checksum}.tar.gz.sig", Base64.decode64(@signature))
161
+
162
+ return @signature
163
+ else
164
+ return false
165
+ end
166
+
167
+ rescue Exception => e
168
+ $logger.error "Something unexpected occured when reading the manifest file"
169
+ raise e
170
+ end
171
+
172
+ end
173
+
174
+
175
+ # Save the signature into the manifest file
176
+ #
177
+ def signature_save
178
+ begin
179
+ manifest = YAML::load(File.open($config["general"]["app_cache"] + "/artifacts/manifest.#{@checksum}.yaml"))
180
+ manifest['gpgsig'] = @signature
181
+
182
+ File.open("#{$config["general"]["app_cache"]}/artifacts/manifest.#{@checksum}.yaml",'w') do |fh|
183
+ fh.write YAML::dump(manifest)
184
+ end
185
+
186
+ return true
187
+
188
+ rescue Exception => e
189
+ $logger.error "Something unexpected occured when updating the manifest file with GPG signature"
190
+ return false
191
+ end
192
+
193
+ end
194
+
195
+
196
+ # Check if the public key is installed on this machine?
197
+ #
198
+ def pubkey_exists?
199
+
200
+ # We prefix with 0x to avoid matching on strings in key names
201
+ if system "gpg --status-fd a --list-keys 0x#{$config["general"]["gpg_signing_key"]} 2>&1 >> /dev/null"
202
+ $logger.debug "Public key exists on this system"
203
+ return true
204
+ else
205
+ $logger.debug "Public key does not exist on this system"
206
+ return false
207
+ end
208
+ end
209
+
210
+
211
+ # Extract & upload the public key to the s3 bucket for other users
212
+ #
213
+ def pubkey_upload
214
+ unless File.exists?("#{$config["general"]["app_cache"]}/artifacts/#{$config["general"]["gpg_signing_key"]}.publickey")
215
+
216
+ # GPG key does not exist locally, we therefore assume it's not in the S3
217
+ # bucket either, so we should export out and upload. Technically this may
218
+ # result in a few extra uploads (once for any new machine using Pupistry)
219
+ # but it doesn't cause any issue and saves me writing more code ;-)
220
+
221
+ $logger.info "Exporting GPG key #{$config["general"]["gpg_signing_key"]} and uploading to S3 bucket..."
222
+
223
+ # If it doesn't exist on this machine, then we're a bit stuck!
224
+ unless pubkey_exists?
225
+ $logger.error "The public key #{$config["general"]["gpg_signing_key"]} does not exist on this system, so unable to export it out"
226
+ return false
227
+ end
228
+
229
+ # Export out key
230
+ unless system "gpg --export --armour 0x#{$config["general"]["gpg_signing_key"]} > #{$config["general"]["app_cache"]}/artifacts/#{$config["general"]["gpg_signing_key"]}.publickey"
231
+ $logger.error "A fault occured when trying to export the GPG key"
232
+ return false
233
+ end
234
+
235
+
236
+ # Upload
237
+ s3 = Pupistry::Storage_AWS.new 'build'
238
+
239
+ unless s3.upload "#{$config["general"]["app_cache"]}/artifacts/#{$config["general"]["gpg_signing_key"]}.publickey", "#{$config["general"]["gpg_signing_key"]}.publickey"
240
+ $logger.error "Unable to upload GPG key to S3 bucket"
241
+ return false
242
+ end
243
+
244
+ end
245
+
246
+ end
247
+
248
+
249
+ # Install the public key. This is a potential avenue for exploit, if a
250
+ # machine is being built for the first time, it has no existing trust of
251
+ # the GPG key, other than transit encryption to the S3 bucket. To protect
252
+ # against attacks at the bootstrap time, you should pre-load your machine
253
+ # images with the public GPG key.
254
+ #
255
+ # For those users who trade off some security for convienence, we install
256
+ # the GPG public key for them direct from the S3 repo.
257
+ #
258
+ def pubkey_install
259
+ begin
260
+ $logger.warn "Installing GPG key #{$config["general"]["gpg_signing_key"]}..."
261
+
262
+ s3 = Pupistry::Storage_AWS.new 'agent'
263
+
264
+ unless s3.download "#{$config["general"]["gpg_signing_key"]}.publickey", "#{$config["general"]["app_cache"]}/artifacts/#{$config["general"]["gpg_signing_key"]}.publickey"
265
+ $logger.error "Unable to download GPG key from S3 bucket, this will prevent validation of signature"
266
+ return false
267
+ end
268
+
269
+ unless system "gpg --import < #{$config["general"]["app_cache"]}/artifacts/#{$config["general"]["gpg_signing_key"]}.publickey"
270
+ $logger.error "A fault occured when trying to import the GPG key"
271
+ return false
272
+ end
273
+
274
+ rescue Exception => e
275
+ $logger.error "Something unexpected occured when installing the GPG public key"
276
+ return false
277
+ end
278
+ end
279
+
280
+ end
281
+ end
282
+
283
+ # vim:shiftwidth=2:tabstop=2:softtabstop=2:expandtab:smartindent
@@ -74,6 +74,13 @@ module Pupistry
74
74
  rescue AWS::S3::Errors::SignatureDoesNotMatch => e
75
75
  $logger.error "IAM signature error when accessing #{$config["general"]["s3_bucket"]}, probably invalid IAM credentials"
76
76
  raise e
77
+
78
+ rescue AWS::S3::Errors::MissingCredentialsError => e
79
+ $logger.error "AWS credentials not supplied. You must either:"
80
+ $logger.error "a) Specify them in the config file for Pupistry"
81
+ $logger.error "b) Use IAM roles with an EC2 instance."
82
+ $logger.error "c) Set them in ENV as AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY"
83
+ return false
77
84
 
78
85
  rescue Exception => e
79
86
  raise e
@@ -124,6 +131,13 @@ module Pupistry
124
131
  $logger.error "IAM signature error when accessing #{$config["general"]["s3_bucket"]}, probably invalid IAM credentials"
125
132
  raise e
126
133
 
134
+ rescue AWS::S3::Errors::MissingCredentialsError => e
135
+ $logger.error "AWS credentials not supplied. You must either:"
136
+ $logger.error "a) Specify them in the config file for Pupistry"
137
+ $logger.error "b) Use IAM roles with an EC2 instance."
138
+ $logger.error "c) Set them in ENV as AWS_ACCESS_KEY_ID and AWS_SECRET_ACCESS_KEY"
139
+ return false
140
+
127
141
  rescue Exception => e
128
142
  raise e
129
143
  end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: pupistry
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.0.7
4
+ version: 0.0.8
5
5
  platform: ruby
6
6
  authors:
7
7
  - Jethro Carr
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2015-04-22 00:00:00.000000000 Z
11
+ date: 2015-04-24 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: aws-sdk-v1
@@ -78,6 +78,7 @@ files:
78
78
  - lib/pupistry/artifact.rb
79
79
  - lib/pupistry/bootstrap.rb
80
80
  - lib/pupistry/config.rb
81
+ - lib/pupistry/gpg.rb
81
82
  - lib/pupistry/storage_aws.rb
82
83
  - README.md
83
84
  - settings.example.yaml