pupistry 0.0.1 → 0.0.2
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +117 -16
- data/bin/pupistry +81 -1
- data/lib/pupistry/artifact.rb +452 -0
- data/lib/pupistry/bootstrap.rb +22 -0
- data/lib/pupistry/config.rb +80 -0
- data/lib/pupistry/storage_aws.rb +134 -0
- data/settings.example.yaml +70 -0
- metadata +6 -1
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 732d292847fab362692d6118e33d286e1c50f4bb
|
4
|
+
data.tar.gz: 5992f7c0698262d35b2a357b978821c23da7cbb8
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 2591da8ee00435744fa2cb97357f88cba20e7bd3f1ab7b7148a158f4dcaea8d60f806dcb93a320995c7738b7a41684d225688cc60ab4cd00a5a1c72fdb709c3b
|
7
|
+
data.tar.gz: bda64095f11c6ce61ceb6fffa42238f9d029ab07d829805a91d207f8efb86cdda39496252dd2c869f02b92a8f48c68cd9a907294ede2fbfc57906d8d2ebbcf67
|
data/README.md
CHANGED
@@ -47,7 +47,26 @@ a number of issues with it.
|
|
47
47
|
|
48
48
|
Build a new artifact:
|
49
49
|
|
50
|
-
pupistry build
|
50
|
+
$ pupistry build
|
51
|
+
I, [2015-04-08T22:19:30.419392 #52534] INFO -- : Using r10k utility to fetch the latest Puppet code
|
52
|
+
[R10K::Action::Deploy::Environment - INFO] Deploying environment /Users/jethro/.pupistry/cache/puppetcode/master
|
53
|
+
[R10K::Action::Deploy::Environment - INFO] Deploying module /Users/jethro/.pupistry/cache/puppetcode/master/modules/stdlib
|
54
|
+
[R10K::Action::Deploy::Environment - INFO] Deploying module /Users/jethro/.pupistry/cache/puppetcode/master/modules/ruby
|
55
|
+
[R10K::Action::Deploy::Environment - INFO] Deploying module /Users/jethro/.pupistry/cache/puppetcode/master/modules/gcc
|
56
|
+
[R10K::Action::Deploy::Environment - INFO] Deploying module /Users/jethro/.pupistry/cache/puppetcode/master/modules/inifile
|
57
|
+
[R10K::Action::Deploy::Environment - INFO] Deploying module /Users/jethro/.pupistry/cache/puppetcode/master/modules/vcsrepo
|
58
|
+
[R10K::Action::Deploy::Environment - INFO] Deploying module /Users/jethro/.pupistry/cache/puppetcode/master/modules/git
|
59
|
+
[R10K::Action::Deploy::Environment - INFO] Deploying module /Users/jethro/.pupistry/cache/puppetcode/master/modules/ntp
|
60
|
+
[R10K::Action::Deploy::Environment - INFO] Deploying module /Users/jethro/.pupistry/cache/puppetcode/master/modules/firewall
|
61
|
+
[R10K::Action::Deploy::Environment - INFO] Deploying module /Users/jethro/.pupistry/cache/puppetcode/master/modules/soe
|
62
|
+
I, [2015-04-08T22:21:21.705315 #52534] INFO -- : r10k run completed
|
63
|
+
I, [2015-04-08T22:21:21.706023 #52534] INFO -- : Creating artifact...
|
64
|
+
I, [2015-04-08T22:21:21.999753 #52534] INFO -- : Compressing artifact...
|
65
|
+
I, [2015-04-08T22:21:22.103131 #52534] INFO -- : Building manifest information for artifact...
|
66
|
+
I, [2015-04-08T22:21:22.107012 #52534] INFO -- : New artifact version 3f29c324aab076cd81667f9031a675e7 ready for pushing
|
67
|
+
--
|
68
|
+
Tip: Run pupistry diff to see what changed since the last artifact version
|
69
|
+
|
51
70
|
|
52
71
|
Note that artifact builds are done from the upstream git repos, so if you
|
53
72
|
have made changes, remember to git push first before generating. The tool will
|
@@ -56,13 +75,29 @@ remind you if it detects nothing has changed since the last run.
|
|
56
75
|
Once your artifact is built, you can double check what has changed in the
|
57
76
|
Puppet modules since the last run with:
|
58
77
|
|
59
|
-
pupistry diff
|
78
|
+
$ pupistry diff
|
79
|
+
diff -Nuar unpacked.3f29c324aab076cd81667f9031a675e7/puppetcode/master/README.md unpacked.4a522dd22c0453e1e3ec3d17dfed151b/puppetcode/master/README.md
|
80
|
+
--- unpacked.3f29c324aab076cd81667f9031a675e7/puppetcode/master/README.md 2015-04-08 22:19:42.000000000 +1200
|
81
|
+
+++ unpacked.4a522dd22c0453e1e3ec3d17dfed151b/puppetcode/master/README.md 2015-04-08 23:01:14.000000000 +1200
|
82
|
+
@@ -1 +1,4 @@
|
83
|
+
Personal Puppet Repo
|
84
|
+
+
|
85
|
+
+Example of a changed file in a module somewhere, nice and visible for all to see.
|
86
|
+
+
|
87
|
+
--
|
88
|
+
Tip: Run pupistry push to GPG sign & upload if happy to go live
|
60
89
|
|
61
90
|
|
62
91
|
Finally when you're happy, push it to S3 to be delivered to all your servers.
|
63
|
-
If you have gpg signing enabled, it will ask you to sign here
|
92
|
+
If you have gpg signing enabled, it will ask you to sign here... or tell you
|
93
|
+
off if you have it disabled. :-)
|
94
|
+
|
95
|
+
$ pupistry push
|
96
|
+
I, [2015-04-08T22:52:01.020865 #53037] INFO -- : Uploading artifact version latest (3f29c324aab076cd81667f9031a675e7)
|
97
|
+
W, [2015-04-08T22:52:01.888356 #53037] WARN -- : You have GPG signing *disabled*, whilst not critical it does weaken your security.
|
98
|
+
W, [2015-04-08T22:52:01.888418 #53037] WARN -- : Skipping signing step...
|
99
|
+
I, [2015-04-08T22:52:03.043886 #53037] INFO -- : Upload of artifact version 3f29c324aab076cd81667f9031a675e7 completed and is now latest
|
64
100
|
|
65
|
-
pupistry push
|
66
101
|
|
67
102
|
|
68
103
|
## Bootstrapping nodes
|
@@ -81,26 +116,39 @@ data field of most cloud providers like AWS or Digital Ocean.
|
|
81
116
|
|
82
117
|
## Running Puppet on target nodes
|
83
118
|
|
84
|
-
|
119
|
+
Pupistry replaces the need to call Puppet directly. Instead, call Pupistry with
|
120
|
+
and it will handle getting the artifact and then executing Puppet for you. It
|
121
|
+
respects some parameters like --environment and --noop for easy testing of new
|
122
|
+
manifests and modules.
|
85
123
|
|
86
|
-
|
124
|
+
At it's simpliest, to apply the current Puppet manifests:
|
87
125
|
|
126
|
+
$ pupistry apply
|
127
|
+
I, [2015-04-10T00:44:40.623101 #6726] INFO -- : Pulling latest artifact....
|
128
|
+
I, [2015-04-10T00:44:42.700540 #6726] INFO -- : Executing Puppet...
|
129
|
+
Notice: Compiled catalog for testhost1 in environment master in 2.21 seconds
|
130
|
+
Notice: Finished catalog run in 3.07 seconds
|
88
131
|
|
89
|
-
Apply the current Puppet manifests:
|
90
132
|
|
91
|
-
|
133
|
+
Check what is going to be applied (Puppet in --noop mode)
|
134
|
+
|
135
|
+
pupistry apply --noop
|
92
136
|
|
93
137
|
Specify an alternative environment:
|
94
138
|
|
95
139
|
pupistry apply --environment staging
|
96
140
|
|
97
|
-
|
98
141
|
Run pupistry as a system daemon. When you use the companion Puppet module, a
|
99
142
|
system init file gets installed that sets this daemon up for you automatically.
|
100
143
|
|
101
144
|
pupistry apply --daemon
|
102
145
|
|
103
146
|
|
147
|
+
Alternatively, if you don't wish to use Pupistry to run the nodes, you don't
|
148
|
+
have to. You can use Pupistry to build the artifacts and then pull them down
|
149
|
+
and unpack via any means you find appropiate. It's just standard S3 + tar with
|
150
|
+
some YAML and optional GPG signing.
|
151
|
+
|
104
152
|
|
105
153
|
# Installation
|
106
154
|
|
@@ -118,6 +166,7 @@ Alternatively if you like living on the edge, download this repository and run:
|
|
118
166
|
gem install pupistry-VERSION.gem
|
119
167
|
pupistry setup
|
120
168
|
|
169
|
+
TODO: Currently setup not implemented, copy the sample file provided.
|
121
170
|
|
122
171
|
## 2. S3 Bucket
|
123
172
|
|
@@ -138,7 +187,53 @@ needing to create explicit IAM credentials for the agents/servers.
|
|
138
187
|
|
139
188
|
|
140
189
|
|
141
|
-
## 3.
|
190
|
+
## 3. Puppet Manifests & Configuration
|
191
|
+
|
192
|
+
### Puppet Code Structure
|
193
|
+
|
194
|
+
The following is the expected minmum structure of the Puppetcode repository to
|
195
|
+
enable it to work with Pupistry:
|
196
|
+
/Puppetfile
|
197
|
+
/hiera.yaml
|
198
|
+
/manifests/site.pp
|
199
|
+
|
200
|
+
Puppetfile is standard r10k and site.pp is standard Puppet. The Hiera config
|
201
|
+
is generally normal, but you do need to define a datadir to tell Puppet to look
|
202
|
+
where the puppetcode gets unpacked to. Generally the following sample Hiera
|
203
|
+
will do the trick:
|
204
|
+
---
|
205
|
+
:backends: yaml
|
206
|
+
:yaml:
|
207
|
+
:datadir: "%{::settings::confdir}/environments/%{::environment}/hieradata"
|
208
|
+
:hierarchy:
|
209
|
+
- "environments/%{::environment}"
|
210
|
+
- "nodes/%{::hostname}"
|
211
|
+
- common
|
212
|
+
|
213
|
+
Then in Pupistry, the following configuration should be used for the agent (or
|
214
|
+
subsitute /etc/puppet/ for wherever your platform has %{::settings::confdir}
|
215
|
+
set to).
|
216
|
+
agent:
|
217
|
+
puppetcode: /etc/puppet/environments
|
218
|
+
|
219
|
+
Pupistry will default to applying the "master" branch if one is not listed, if
|
220
|
+
you are doing branch-based environments, you can specifiy when bootstrapping
|
221
|
+
and override on a per-execution basis.
|
222
|
+
|
223
|
+
You'll notice pretty quickly if something is broken when doing `puppet apply`
|
224
|
+
|
225
|
+
|
226
|
+
|
227
|
+
TODO: Longer term intend to add support for various popular structure, but
|
228
|
+
for now it is what it is. It's not hard, check out bin/puppistry and send
|
229
|
+
pull requests.
|
230
|
+
|
231
|
+
TODO: Provide an example repo to build from.... or maybe have smarter
|
232
|
+
assistance in the app to check for files and find them or generate defaults
|
233
|
+
if missing (eg hiera)
|
234
|
+
|
235
|
+
|
236
|
+
### Helper Module
|
142
237
|
|
143
238
|
Whilst you can use Pupistry to roll out any particular design of Puppet
|
144
239
|
manifests, you will save yourself a lot of pain by also including the Pupistry
|
@@ -149,12 +244,13 @@ up the system service and configuring Puppet and Hiera correctly for masterless
|
|
149
244
|
operation.
|
150
245
|
|
151
246
|
You can fetch the module from:
|
152
|
-
https://github.com/jethrocarr/pupistry
|
247
|
+
https://github.com/jethrocarr/puppet-pupistry
|
153
248
|
|
154
249
|
If you're doing r10k and Puppet masterless from scratch, this is probably
|
155
250
|
something you want to make life easy.
|
156
251
|
|
157
252
|
|
253
|
+
|
158
254
|
## 4. Bootstrapping Nodes
|
159
255
|
|
160
256
|
No need for manual configuration of your servers/nodes, you just need to build
|
@@ -162,6 +258,7 @@ your first artifact with Pupistry (`pupistry build && pupistry push`) and then
|
|
162
258
|
generate a bootstrap script for your particular OS with `pupistry bootstrap`
|
163
259
|
|
164
260
|
The bootstrap script will:
|
261
|
+
|
165
262
|
1. Install Puppet and Pupistry for the particular OS.
|
166
263
|
2. Download the latest artifact
|
167
264
|
3. Trigger a Puppet run to build your server.
|
@@ -169,6 +266,11 @@ The bootstrap script will:
|
|
169
266
|
Once done, it's up to your Puppet manifests to build your machine how you want
|
170
267
|
it - enjoy!
|
171
268
|
|
269
|
+
TODO: Currently being worked on, for now the following is a rough example of
|
270
|
+
what you can do to bootstrap a RHEL/CentOS7 box:
|
271
|
+
|
272
|
+
|
273
|
+
|
172
274
|
|
173
275
|
|
174
276
|
# Tutorials
|
@@ -183,8 +285,6 @@ and running masterless Puppet environment using Pupistry. It covers the very
|
|
183
285
|
basics of setting up your r10k environment.
|
184
286
|
|
185
287
|
|
186
|
-
|
187
|
-
|
188
288
|
# Caveats & Future Plans
|
189
289
|
|
190
290
|
## Use r10k
|
@@ -265,10 +365,11 @@ When developing Pupistry, you can run the git repo copy with:
|
|
265
365
|
ruby -Ilib/ -r rubygems bin/pupistry
|
266
366
|
|
267
367
|
By default Pupistry will try to load a settings.yaml file in the current
|
268
|
-
working directory, before then trying
|
269
|
-
finally
|
368
|
+
working directory, before then trying `~/.pupistry/settings.yaml` and then
|
369
|
+
finally `/etc/pupistry/settings.yaml`. You can also override with `--config`.
|
270
370
|
|
271
|
-
Add
|
371
|
+
Add `--verbose` for additional debugging information. If you have a bug this
|
372
|
+
is the first thing you should run to get more context for reports.
|
272
373
|
|
273
374
|
|
274
375
|
# Contributions
|
data/bin/pupistry
CHANGED
@@ -19,6 +19,87 @@ class CLI < Thor
|
|
19
19
|
class_option :verbose, :type => :boolean
|
20
20
|
class_option :config, :type => :string
|
21
21
|
|
22
|
+
|
23
|
+
## Agent Commands
|
24
|
+
|
25
|
+
desc "apply", "Apply the latest Puppet artifact"
|
26
|
+
method_option :noop, :type => :boolean
|
27
|
+
method_option :daemon, :type => :boolean
|
28
|
+
method_option :environment, :type => :string
|
29
|
+
def apply
|
30
|
+
|
31
|
+
# Thor seems to force class options to be defined repeatedly? :-/
|
32
|
+
if options[:verbose]
|
33
|
+
$logger.level = Logger::DEBUG
|
34
|
+
else
|
35
|
+
$logger.level = Logger::INFO
|
36
|
+
end
|
37
|
+
|
38
|
+
if options[:config]
|
39
|
+
Pupistry::Config.load(options[:config])
|
40
|
+
else
|
41
|
+
Pupistry::Config.find_and_load
|
42
|
+
end
|
43
|
+
|
44
|
+
# Muppet Check
|
45
|
+
if options[:noop] and options[:daemon]
|
46
|
+
$logger.warn "A daemon running in noop will do nothing except log what changes it could apply"
|
47
|
+
end
|
48
|
+
|
49
|
+
# Muppet dev check
|
50
|
+
if options[:daemon]
|
51
|
+
$logger.fatal "Daemon mode not implemented yet"
|
52
|
+
exit 0
|
53
|
+
end
|
54
|
+
|
55
|
+
# Install the artifact
|
56
|
+
# TODO: Check if we've already installed it or not... :-)
|
57
|
+
$logger.info "Pulling latest artifact...."
|
58
|
+
|
59
|
+
artifact = Pupistry::Artifact.new
|
60
|
+
artifact.checksum = artifact.fetch_latest
|
61
|
+
artifact.fetch_artifact
|
62
|
+
artifact.unpack
|
63
|
+
unless artifact.install
|
64
|
+
$logger.fatal "An unexpected error happened when installing the latest artifact, cancelling Puppet run"
|
65
|
+
exit 0
|
66
|
+
end
|
67
|
+
|
68
|
+
# Remove temporary unpacked files
|
69
|
+
artifact.clean_unpack
|
70
|
+
|
71
|
+
|
72
|
+
# Execute Puppet. At this point
|
73
|
+
puppet_cmd = "puppet apply"
|
74
|
+
|
75
|
+
if options[:noop]
|
76
|
+
puppet_cmd += " --noop"
|
77
|
+
end
|
78
|
+
|
79
|
+
if options[:environment]
|
80
|
+
environment = options[:environment]
|
81
|
+
else
|
82
|
+
environment = 'master'
|
83
|
+
end
|
84
|
+
|
85
|
+
puppet_cmd += " --environment #{environment}"
|
86
|
+
puppet_cmd += " --modulepath #{$config["agent"]["puppetcode"]}/#{environment}/modules/"
|
87
|
+
puppet_cmd += " --hiera_config #{$config["agent"]["puppetcode"]}/#{environment}/hiera.yaml"
|
88
|
+
puppet_cmd += " #{$config["agent"]["puppetcode"]}/#{environment}/manifests/site.pp"
|
89
|
+
|
90
|
+
$logger.info "Executing Puppet..."
|
91
|
+
$logger.debug "With: #{puppet_cmd}"
|
92
|
+
|
93
|
+
unless system puppet_cmd
|
94
|
+
$logger.fatal "An unexpected issue occured when running puppet"
|
95
|
+
end
|
96
|
+
|
97
|
+
end
|
98
|
+
|
99
|
+
|
100
|
+
|
101
|
+
## Workstation Commands
|
102
|
+
|
22
103
|
desc "build", "Build a new archive file"
|
23
104
|
def build
|
24
105
|
|
@@ -51,7 +132,6 @@ class CLI < Thor
|
|
51
132
|
raise e
|
52
133
|
end
|
53
134
|
|
54
|
-
|
55
135
|
end
|
56
136
|
|
57
137
|
|
@@ -0,0 +1,452 @@
|
|
1
|
+
require 'rubygems'
|
2
|
+
require 'yaml'
|
3
|
+
require 'time'
|
4
|
+
require 'digest'
|
5
|
+
require 'fileutils'
|
6
|
+
|
7
|
+
module Pupistry
|
8
|
+
# Pupistry::Artifact
|
9
|
+
|
10
|
+
class Artifact
|
11
|
+
# All the functions needed for manipulating the artifats
|
12
|
+
attr_accessor :checksum
|
13
|
+
|
14
|
+
|
15
|
+
def fetch_r10k
|
16
|
+
$logger.info "Using r10k utility to fetch the latest Puppet code"
|
17
|
+
|
18
|
+
unless defined? $config["build"]["puppetcode"]
|
19
|
+
$logger.fatal "You must configure the build:puppetcode config option in settings.yaml"
|
20
|
+
raise "Invalid Configuration"
|
21
|
+
end
|
22
|
+
|
23
|
+
# https://github.com/puppetlabs/r10k
|
24
|
+
#
|
25
|
+
# r10k does a fantastic job with all the git stuff and we want to use it
|
26
|
+
# to download the Puppet code from all the git modules (based on following
|
27
|
+
# the master one provided), then we can steal the Puppet code from the
|
28
|
+
# artifact generated.
|
29
|
+
#
|
30
|
+
# TODO: We should re-write this to hook directly into r10k's libraries,
|
31
|
+
# given that both Pupistry and r10k are Ruby, presumably it should be
|
32
|
+
# doable and much more polished approach. For now the MVP is to just run
|
33
|
+
# it via system, pull requests/patches to fix very welcome!
|
34
|
+
|
35
|
+
|
36
|
+
# Build the r10k config to instruct it to use our cache path for storing
|
37
|
+
# it's data and exporting the finished result.
|
38
|
+
$logger.debug "Generating an r10k configuration file..."
|
39
|
+
r10k_config = {
|
40
|
+
"cachedir" => "#{$config["general"]["app_cache"]}/r10kcache",
|
41
|
+
"sources" => {
|
42
|
+
"puppet" => {
|
43
|
+
"remote" => $config["build"]["puppetcode"],
|
44
|
+
"basedir" => $config["general"]["app_cache"] + "/puppetcode",
|
45
|
+
}
|
46
|
+
}
|
47
|
+
}
|
48
|
+
|
49
|
+
begin
|
50
|
+
File.open("#{$config["general"]["app_cache"]}/r10kconfig.yaml",'w') do |fh|
|
51
|
+
fh.write YAML::dump(r10k_config)
|
52
|
+
end
|
53
|
+
rescue Exception => e
|
54
|
+
$logger.fatal "Unexpected error when trying to write the r10k configuration file"
|
55
|
+
raise e
|
56
|
+
end
|
57
|
+
|
58
|
+
|
59
|
+
# Execute R10k with the provided configuration
|
60
|
+
$logger.debug "Executing r10k"
|
61
|
+
|
62
|
+
if system "r10k deploy environment -c #{$config["general"]["app_cache"]}/r10kconfig.yaml -pv"
|
63
|
+
$logger.info "r10k run completed"
|
64
|
+
else
|
65
|
+
$logger.error "r10k run failed, unable to generate artifact"
|
66
|
+
raise "r10k run did not complete, unable to generate artifact"
|
67
|
+
end
|
68
|
+
|
69
|
+
end
|
70
|
+
|
71
|
+
def fetch_latest
|
72
|
+
# Fetch the latest S3 YAML file and check the version metadata without writing
|
73
|
+
# it to disk. Returns the version. Useful for quickly checking for updates :-)
|
74
|
+
|
75
|
+
$logger.debug "Checking latest artifact version..."
|
76
|
+
|
77
|
+
s3 = Pupistry::Storage_AWS.new 'agent'
|
78
|
+
contents = s3.download 'manifest.latest.yaml'
|
79
|
+
|
80
|
+
if contents
|
81
|
+
manifest = YAML::load(contents)
|
82
|
+
|
83
|
+
if defined? manifest['version']
|
84
|
+
return manifest['version']
|
85
|
+
else
|
86
|
+
return false
|
87
|
+
end
|
88
|
+
|
89
|
+
else
|
90
|
+
# download did not work
|
91
|
+
return false
|
92
|
+
end
|
93
|
+
|
94
|
+
|
95
|
+
end
|
96
|
+
|
97
|
+
|
98
|
+
def fetch_current
|
99
|
+
# Fetch the latest on-disk YAML file and check the version metadata, used
|
100
|
+
# to determine the latest artifact that has not yet been pushed to S3.
|
101
|
+
# Returns the version.
|
102
|
+
|
103
|
+
# Read the symlink information to get the latest version
|
104
|
+
if File.exists?($config["general"]["app_cache"] + "/artifacts/manifest.latest.yaml")
|
105
|
+
manifest = YAML::load(File.open($config["general"]["app_cache"] + "/artifacts/manifest.latest.yaml"))
|
106
|
+
@checksum = manifest['version']
|
107
|
+
else
|
108
|
+
$logger.error "No artifact has been built yet. You need to run pupistry build first?"
|
109
|
+
return 0
|
110
|
+
end
|
111
|
+
end
|
112
|
+
|
113
|
+
|
114
|
+
def fetch_artifact
|
115
|
+
|
116
|
+
# Figure out which version to fetch (if not explicitly defined)
|
117
|
+
if defined? @checksum
|
118
|
+
$logger.debug "Downloading artifact version #{@checksum}"
|
119
|
+
else
|
120
|
+
@checksum = fetch_latest
|
121
|
+
|
122
|
+
if defined? @checksum
|
123
|
+
$logger.debug "Downloading latest artifact (#{@checksum})"
|
124
|
+
else
|
125
|
+
$logger.error "There is not current artifact that can be fetched"
|
126
|
+
return false
|
127
|
+
end
|
128
|
+
|
129
|
+
end
|
130
|
+
|
131
|
+
# Make sure the download dir/cache exists
|
132
|
+
unless Dir.exists?($config["general"]["app_cache"] + "/artifacts/")
|
133
|
+
FileUtils.mkdir_p $config["general"]["app_cache"] + "/artifacts/"
|
134
|
+
end
|
135
|
+
|
136
|
+
# Download files if they don't already exist
|
137
|
+
if File.exists?($config["general"]["app_cache"] + "/artifacts/manifest.#{@checksum}.yaml") and File.exists?($config["general"]["app_cache"] + "/artifacts/artifact.#{@checksum}.tar.gz")
|
138
|
+
$logger.debug "This artifact is already present, no download required."
|
139
|
+
else
|
140
|
+
s3 = Pupistry::Storage_AWS.new 'agent'
|
141
|
+
s3.download "manifest.#{@checksum}.yaml", $config["general"]["app_cache"] + "/artifacts/manifest.#{@checksum}.yaml"
|
142
|
+
s3.download "artifact.#{@checksum}.tar.gz", $config["general"]["app_cache"] + "/artifacts/artifact.#{@checksum}.tar.gz"
|
143
|
+
end
|
144
|
+
|
145
|
+
end
|
146
|
+
|
147
|
+
|
148
|
+
|
149
|
+
def push_artifact
|
150
|
+
# The push step involves 2 steps:
|
151
|
+
# 1. GPG sign the artifact and write it into the manifest file
|
152
|
+
# 2. Upload the manifest and archive files to S3.
|
153
|
+
# 3. Upload a copy as the "latest" manifest file which will be hit by clients.
|
154
|
+
|
155
|
+
|
156
|
+
# Determine which version we are uploading. Either one specifically
|
157
|
+
# selected, otherwise find the latest one to push
|
158
|
+
|
159
|
+
if defined? @checksum
|
160
|
+
$logger.info "Uploading artifact version #{@checksum}."
|
161
|
+
else
|
162
|
+
@checksum = fetch_current
|
163
|
+
|
164
|
+
if @checksum
|
165
|
+
$logger.info "Uploading artifact version latest (#{@checksum})"
|
166
|
+
else
|
167
|
+
# If there is no current version, we can't do much....
|
168
|
+
exit 0
|
169
|
+
end
|
170
|
+
end
|
171
|
+
|
172
|
+
|
173
|
+
# Do we even need to upload? If nothing has changed....
|
174
|
+
if @checksum == fetch_latest
|
175
|
+
$logger.error "You've already pushed this artifact version, nothing to do."
|
176
|
+
exit 0
|
177
|
+
end
|
178
|
+
|
179
|
+
|
180
|
+
# Make sure the files actually exist...
|
181
|
+
unless File.exists?($config["general"]["app_cache"] + "/artifacts/manifest.#{@checksum}.yaml")
|
182
|
+
$logger.error "The files expected for #{@checksum} do not appear to exist or are not readable"
|
183
|
+
raise "Fatal unexpected error"
|
184
|
+
end
|
185
|
+
|
186
|
+
unless File.exists?($config["general"]["app_cache"] + "/artifacts/artifact.#{@checksum}.tar.gz")
|
187
|
+
$logger.error "The files expected for #{@checksum} do not appear to exist or are not readable"
|
188
|
+
raise "Fatal unexpected error"
|
189
|
+
end
|
190
|
+
|
191
|
+
|
192
|
+
# GPG sign the files
|
193
|
+
if $config["general"]["gpg_disable"] == true
|
194
|
+
$logger.warn "You have GPG signing *disabled*, whilst not critical it does weaken your security."
|
195
|
+
$logger.warn "Skipping signing step..."
|
196
|
+
else
|
197
|
+
$logger.info "GPG signing the artifact with configured key"
|
198
|
+
|
199
|
+
# TODO: should probably write this bit!
|
200
|
+
end
|
201
|
+
|
202
|
+
|
203
|
+
# Upload the artifact & manifests to S3. We also make an additional copy
|
204
|
+
# as the "latest" file which will be downloaded by all the agents checking
|
205
|
+
# for new updates.
|
206
|
+
|
207
|
+
s3 = Pupistry::Storage_AWS.new 'build'
|
208
|
+
s3.upload $config["general"]["app_cache"] + "/artifacts/artifact.#{@checksum}.tar.gz", "artifact.#{@checksum}.tar.gz"
|
209
|
+
s3.upload $config["general"]["app_cache"] + "/artifacts/manifest.#{@checksum}.yaml", "manifest.#{@checksum}.yaml"
|
210
|
+
s3.upload $config["general"]["app_cache"] + "/artifacts/manifest.#{@checksum}.yaml", "manifest.latest.yaml"
|
211
|
+
|
212
|
+
|
213
|
+
# Test a read of the manifest, we do this to make sure the S3 ACLs setup
|
214
|
+
# allow downloading of the uploaded files - helps avoid user headaches if
|
215
|
+
# they misconfigure and then blindly trust their bootstrap config.
|
216
|
+
#
|
217
|
+
# Only worth doing this step if they've explicitly set their AWS IAM credentials
|
218
|
+
# for the agent, which should be everyone except for IAM role users.
|
219
|
+
|
220
|
+
if $config["agent"]["aws_access_id"]
|
221
|
+
fetch_artifact
|
222
|
+
else
|
223
|
+
$logger.warn "The agent's AWS credentials are unset on this machine, unable to do download test to check permissions for you."
|
224
|
+
$logger.warn "Assuming you know what you're doing, please set if unsure."
|
225
|
+
end
|
226
|
+
|
227
|
+
$logger.info "Upload of artifact version #{@checksum} completed and is now latest"
|
228
|
+
end
|
229
|
+
|
230
|
+
|
231
|
+
def build_artifact
|
232
|
+
# r10k has done all the heavy lifting for us, we just need to generate a
|
233
|
+
# tarball from the app_cache /puppetcode directory. There are some Ruby
|
234
|
+
# native libraries, but really we might as well just use the native tools
|
235
|
+
# since we don't want to do anything clever like in-memory assembly of
|
236
|
+
# the file. Like r10k, if you want to convert to a nicely polished native
|
237
|
+
# Ruby solution, patches welcome.
|
238
|
+
|
239
|
+
$logger.info "Creating artifact..."
|
240
|
+
|
241
|
+
Dir.chdir($config["general"]["app_cache"]) do
|
242
|
+
|
243
|
+
# Make sure there is a directory to write artifacts into
|
244
|
+
FileUtils.mkdir_p('artifacts')
|
245
|
+
|
246
|
+
# Build the tar file - we delibertly don't compress in a single step
|
247
|
+
# so that we can grab the checksum, since checksum will always differ
|
248
|
+
# post-compression.
|
249
|
+
unless system "tar -c --exclude '.git' -f artifacts/artifact.temp.tar puppetcode/*"
|
250
|
+
$logger.error "Unable to create tarball"
|
251
|
+
raise "An unexpected error occured when executing tar"
|
252
|
+
end
|
253
|
+
|
254
|
+
# The checksum is important, we use it as our version for each artifact
|
255
|
+
# so we can tell them apart in a unique way.
|
256
|
+
@checksum = Digest::MD5.file($config["general"]["app_cache"] + "/artifacts/artifact.temp.tar").hexdigest
|
257
|
+
|
258
|
+
# Now we have the checksum, check if it's the same as any existing
|
259
|
+
# artifacts. If so, drop out here, good to give feedback to the user
|
260
|
+
# if nothing has changed since it's easy to forget to git push a single
|
261
|
+
# module/change.
|
262
|
+
|
263
|
+
if File.exists?($config["general"]["app_cache"] + "/artifacts/manifest.#{@checksum}.yaml")
|
264
|
+
$logger.error "This artifact version (#{@checksum}) has already been built, nothing todo."
|
265
|
+
$logger.error "Did you remember to \"git push\" your module changes?"
|
266
|
+
|
267
|
+
# Cleanup temp file
|
268
|
+
FileUtils.rm($config["general"]["app_cache"] + "/artifacts/artifact.temp.tar")
|
269
|
+
exit 0
|
270
|
+
end
|
271
|
+
|
272
|
+
# Compress the artifact now that we have taken it's checksum
|
273
|
+
$logger.info "Compressing artifact..."
|
274
|
+
|
275
|
+
if system "gzip artifacts/artifact.temp.tar"
|
276
|
+
else
|
277
|
+
$logger.error "An unexpected error occured during compression of the artifact"
|
278
|
+
raise "An unexpected error occured during compression of the artifact"
|
279
|
+
end
|
280
|
+
end
|
281
|
+
|
282
|
+
|
283
|
+
# We have the checksum, so we can now rename the artifact file
|
284
|
+
FileUtils.mv($config["general"]["app_cache"] + "/artifacts/artifact.temp.tar.gz", $config["general"]["app_cache"] + "/artifacts/artifact.#{@checksum}.tar.gz")
|
285
|
+
|
286
|
+
|
287
|
+
$logger.info "Building manifest information for artifact..."
|
288
|
+
|
289
|
+
# Create the manifest file, this is used by clients for pulling details about
|
290
|
+
# the latest artifacts. We don't GPG sign here, but we do put in a placeholder.
|
291
|
+
manifest = {
|
292
|
+
"version" => @checksum,
|
293
|
+
"date" => Time.new.inspect,
|
294
|
+
"builduser" => ENV['USER'] || 'unlabled',
|
295
|
+
"gpgsig" => 'unsighed',
|
296
|
+
}
|
297
|
+
|
298
|
+
begin
|
299
|
+
File.open("#{$config["general"]["app_cache"]}/artifacts/manifest.#{@checksum}.yaml",'w') do |fh|
|
300
|
+
fh.write YAML::dump(manifest)
|
301
|
+
end
|
302
|
+
rescue Exception => e
|
303
|
+
$logger.fatal "Unexpected error when trying to write the manifest file"
|
304
|
+
raise e
|
305
|
+
end
|
306
|
+
|
307
|
+
# This is the latest artifact, create some symlinks pointing the latest to it
|
308
|
+
begin
|
309
|
+
FileUtils.ln_s("manifest.#{@checksum}.yaml", "#{$config["general"]["app_cache"]}/artifacts/manifest.latest.yaml", :force => true)
|
310
|
+
FileUtils.ln_s("artifact.#{@checksum}.tar.gz", "#{$config["general"]["app_cache"]}/artifacts/artifact.latest.tar.gz", :force => true)
|
311
|
+
rescue Exception => e
|
312
|
+
$logger.fatal "Something weird went really wrong trying to symlink the latest artifacts"
|
313
|
+
raise e
|
314
|
+
end
|
315
|
+
|
316
|
+
|
317
|
+
$logger.info "New artifact version #{@checksum} ready for pushing"
|
318
|
+
end
|
319
|
+
|
320
|
+
|
321
|
+
def unpack
|
322
|
+
# Unpack the currently selected artifact to the archives directory.
|
323
|
+
|
324
|
+
# An application version must be specified
|
325
|
+
unless defined? @checksum
|
326
|
+
raise "Application bug, trying to unpack no artifact"
|
327
|
+
end
|
328
|
+
|
329
|
+
# Make sure the files actually exist...
|
330
|
+
unless File.exists?($config["general"]["app_cache"] + "/artifacts/manifest.#{@checksum}.yaml")
|
331
|
+
$logger.error "The files expected for #{@checksum} do not appear to exist or are not readable"
|
332
|
+
raise "Fatal unexpected error"
|
333
|
+
end
|
334
|
+
|
335
|
+
unless File.exists?($config["general"]["app_cache"] + "/artifacts/artifact.#{@checksum}.tar.gz")
|
336
|
+
$logger.error "The files expected for #{@checksum} do not appear to exist or are not readable"
|
337
|
+
raise "Fatal unexpected error"
|
338
|
+
end
|
339
|
+
|
340
|
+
# Clean up an existing unpacked copy - in *theory* it should be same, but
|
341
|
+
# a mistake like running out of disk could have left it in an unclean state
|
342
|
+
# so let's make sure it's gone
|
343
|
+
clean_unpack
|
344
|
+
|
345
|
+
# Unpack the archive file
|
346
|
+
FileUtils.mkdir_p($config["general"]["app_cache"] + "/artifacts/unpacked.#{@checksum}")
|
347
|
+
Dir.chdir($config["general"]["app_cache"] + "/artifacts/unpacked.#{@checksum}") do
|
348
|
+
|
349
|
+
unless system "tar -xf ../artifact.#{@checksum}.tar.gz"
|
350
|
+
$logger.error "Unable to unpack artifact files to #{Dir.pwd}"
|
351
|
+
raise "An unexpected error occured when executing tar"
|
352
|
+
else
|
353
|
+
$logger.debug "Successfully unpacked artifact #{@checksum}"
|
354
|
+
end
|
355
|
+
end
|
356
|
+
|
357
|
+
end
|
358
|
+
|
359
|
+
|
360
|
+
def install
|
361
|
+
# Copy the unpacked artifact into the agent's configured location. Generally all the
|
362
|
+
# heavy lifting is done by fetch_latest and unpack methods.
|
363
|
+
|
364
|
+
# An application version must be specified
|
365
|
+
unless defined? @checksum
|
366
|
+
raise "Application bug, trying to install no artifact"
|
367
|
+
end
|
368
|
+
|
369
|
+
# Make sure the artifact has been unpacked
|
370
|
+
unless Dir.exists?($config["general"]["app_cache"] + "/artifacts/unpacked.#{@checksum}")
|
371
|
+
$logger.error "The unpacked directory expected for #{@checksum} does not appear to exist or is not readable"
|
372
|
+
raise "Fatal unexpected error"
|
373
|
+
end
|
374
|
+
|
375
|
+
# Purge any currently installed files in the directory. See clean_install
|
376
|
+
# TODO notes for how this could be improved.
|
377
|
+
unless clean_install
|
378
|
+
$logger.error "Installation not proceeduing due to issues cleaning/prepping destination dir"
|
379
|
+
end
|
380
|
+
|
381
|
+
# Make sure the destination directory exists
|
382
|
+
unless Dir.exists?($config["agent"]["puppetcode"])
|
383
|
+
$logger.error "The destination path of #{$config["agent"]["puppetcode"]} does not appear to exist or is not readable"
|
384
|
+
raise "Fatal unexpected error"
|
385
|
+
end
|
386
|
+
|
387
|
+
# Clone unpacked contents to the installation directory
|
388
|
+
begin
|
389
|
+
FileUtils.cp_r $config["general"]["app_cache"] + "/artifacts/unpacked.#{@checksum}/puppetcode/.", $config["agent"]["puppetcode"]
|
390
|
+
return true
|
391
|
+
rescue
|
392
|
+
$logger.fatal "An unexpected error occured when copying the unpacked artifact to #{$config["agent"]["puppetcode"]}"
|
393
|
+
raise e
|
394
|
+
end
|
395
|
+
|
396
|
+
end
|
397
|
+
|
398
|
+
|
399
|
+
def clean_install
|
400
|
+
# Cleanup the destination installation directory before we unpack the artifact
|
401
|
+
# into it, otherwise long term we will end up with old deprecated files hanging
|
402
|
+
# around.
|
403
|
+
#
|
404
|
+
# TODO: Do this smarter, we should track what files we drop in, and then remove
|
405
|
+
# any that weren't touched. Need to avoid rsync and stick with native to make
|
406
|
+
# support easier for weird/minimilistic distributions.
|
407
|
+
|
408
|
+
if defined? $config["agent"]["puppetcode"]
|
409
|
+
if $config["agent"]["puppetcode"].empty?
|
410
|
+
$logger.error "You must configure a location for the agent's Puppet code to be deployed to"
|
411
|
+
return false
|
412
|
+
else
|
413
|
+
$logger.debug "Cleaning up #{$config["agent"]["puppetcode"]} directory"
|
414
|
+
|
415
|
+
if Dir.exists?($config["agent"]["puppetcode"])
|
416
|
+
FileUtils.rm_r Dir.glob($config["agent"]["puppetcode"] + "/*"), :secure => true
|
417
|
+
else
|
418
|
+
FileUtils.mkdir_p $config["agent"]["puppetcode"]
|
419
|
+
end
|
420
|
+
|
421
|
+
return true
|
422
|
+
end
|
423
|
+
end
|
424
|
+
|
425
|
+
end
|
426
|
+
|
427
|
+
|
428
|
+
def clean_unpack
|
429
|
+
# Cleanup/remove any unpacked archive directories. Requires that the
|
430
|
+
# checksum be set to the version to be purged.
|
431
|
+
|
432
|
+
unless defined? @checksum
|
433
|
+
raise "Application bug, trying to unpack no artifact"
|
434
|
+
end
|
435
|
+
|
436
|
+
if Dir.exists?($config["general"]["app_cache"] + "/artifacts/unpacked.#{@checksum}/")
|
437
|
+
$logger.debug "Cleaning up #{$config["general"]["app_cache"]}/artifacts/unpacked.#{@checksum}..."
|
438
|
+
FileUtils.rm_r $config["general"]["app_cache"] + "/artifacts/unpacked.#{@checksum}", :secure => true
|
439
|
+
return true
|
440
|
+
else
|
441
|
+
$logger.debug "Nothing to cleanup (selected artifact is not currently unpacked)"
|
442
|
+
return true
|
443
|
+
end
|
444
|
+
|
445
|
+
return false
|
446
|
+
|
447
|
+
end
|
448
|
+
|
449
|
+
end
|
450
|
+
end
|
451
|
+
|
452
|
+
# vim:shiftwidth=2:tabstop=2:softtabstop=2:expandtab:smartindent
|
@@ -0,0 +1,22 @@
|
|
1
|
+
require 'rubygems'
|
2
|
+
|
3
|
+
module Pupistry
|
4
|
+
# Pupistry::Bootstrap
|
5
|
+
|
6
|
+
class Bootstrap
|
7
|
+
attr_accessor :template_dir
|
8
|
+
|
9
|
+
def initalize
|
10
|
+
template_dir = "dir"
|
11
|
+
end
|
12
|
+
|
13
|
+
def self.templates_list
|
14
|
+
# glob all the templates
|
15
|
+
puts "Template"
|
16
|
+
end
|
17
|
+
|
18
|
+
|
19
|
+
end
|
20
|
+
end
|
21
|
+
|
22
|
+
# vim:shiftwidth=2:tabstop=2:softtabstop=2:expandtab:smartindent
|
@@ -0,0 +1,80 @@
|
|
1
|
+
require 'rubygems'
|
2
|
+
require 'fileutils'
|
3
|
+
require 'tempfile'
|
4
|
+
require 'yaml'
|
5
|
+
|
6
|
+
module Pupistry
|
7
|
+
# Pupistry::Config
|
8
|
+
#
|
9
|
+
# Provides loading of configuration.
|
10
|
+
#
|
11
|
+
|
12
|
+
class Config
|
13
|
+
|
14
|
+
def self.load file
|
15
|
+
$logger.debug "Loading configuration file #{file}"
|
16
|
+
|
17
|
+
unless File.exists?(file)
|
18
|
+
$logger.fatal "The configuration file provided does not exist, or cannot be accessed"
|
19
|
+
exit 0
|
20
|
+
end
|
21
|
+
|
22
|
+
$config = YAML::load(File.open(file))
|
23
|
+
|
24
|
+
# Make sure cache directory exists, create it otherwise
|
25
|
+
$config["general"]["app_cache"] = File.expand_path($config["general"]["app_cache"]).chomp('/')
|
26
|
+
|
27
|
+
unless Dir.exists?($config["general"]["app_cache"])
|
28
|
+
begin
|
29
|
+
FileUtils.mkdir_p($config["general"]["app_cache"])
|
30
|
+
rescue Exception => e
|
31
|
+
$logger.fatal "Unable to create cache directory at \"#{$config["general"]["app_cache"]}\"."
|
32
|
+
raise e
|
33
|
+
end
|
34
|
+
end
|
35
|
+
|
36
|
+
# Write test file to confirm writability
|
37
|
+
begin
|
38
|
+
FileUtils.touch($config["general"]["app_cache"] + "/testfile")
|
39
|
+
FileUtils.rm($config["general"]["app_cache"] + "/testfile")
|
40
|
+
rescue Exception => e
|
41
|
+
$logger.fatal "Unexpected exception when creating testfile in cache directory at \"#{$config["general"]["app_cache"]}\", is the directory writable?"
|
42
|
+
raise e
|
43
|
+
end
|
44
|
+
|
45
|
+
end
|
46
|
+
|
47
|
+
def self.find_and_load
|
48
|
+
$logger.debug "Looking for configuration file in common locations"
|
49
|
+
|
50
|
+
# Locations in order of preference:
|
51
|
+
# settings.yaml (current dir)
|
52
|
+
# ~/.pupistry/settings.yaml
|
53
|
+
# /etc/pupistry/settings.yaml
|
54
|
+
|
55
|
+
config = ''
|
56
|
+
local_dir = Dir.pwd
|
57
|
+
|
58
|
+
if File.exists?("#{local_dir}/settings.yaml")
|
59
|
+
config = "#{local_dir}/settings.yaml"
|
60
|
+
|
61
|
+
elsif File.exists?( File.expand_path "~/.pupistry/settings.yaml" )
|
62
|
+
config = File.expand_path "~/.pupistry/settings.yaml"
|
63
|
+
|
64
|
+
elsif File.exists?("/etc/pupistry/settings.yaml")
|
65
|
+
config = "/etc/pupistry/settings.yaml"
|
66
|
+
|
67
|
+
else
|
68
|
+
$logger.error "No configuration file provided."
|
69
|
+
$logger.error "See pupistry help for information on configuration"
|
70
|
+
exit 0
|
71
|
+
end
|
72
|
+
|
73
|
+
self.load(config)
|
74
|
+
|
75
|
+
end
|
76
|
+
|
77
|
+
|
78
|
+
end
|
79
|
+
end
|
80
|
+
# vim:shiftwidth=2:tabstop=2:softtabstop=2:expandtab:smartindent
|
@@ -0,0 +1,134 @@
|
|
1
|
+
require 'rubygems'
|
2
|
+
require 'yaml'
|
3
|
+
require 'aws-sdk-v1'
|
4
|
+
|
5
|
+
module Pupistry
|
6
|
+
# Pupistry::Storage_AWS
|
7
|
+
|
8
|
+
class Storage_AWS
|
9
|
+
attr_accessor :s3
|
10
|
+
attr_accessor :bucket
|
11
|
+
|
12
|
+
def initialize mode
|
13
|
+
# mode is either "build" or "agent", depending which we load a different
|
14
|
+
# set of permissions. Awareness of both is intentional, since we want the
|
15
|
+
# build machines to known the agent creds so we can generate bootstrap
|
16
|
+
# template files.
|
17
|
+
|
18
|
+
unless defined? $config["general"]["s3_bucket"]
|
19
|
+
$logger.fatal "You must set the AWS s3_bucket"
|
20
|
+
exit 0
|
21
|
+
end
|
22
|
+
|
23
|
+
# Define AWS configuration
|
24
|
+
if defined? $config[mode]["access_key_id"]
|
25
|
+
unless $config[mode]["access_key_id"] == ''
|
26
|
+
$logger.debug "Loading AWS credentials from configuration file"
|
27
|
+
|
28
|
+
AWS.config(
|
29
|
+
:access_key_id => $config[mode]["access_key_id"],
|
30
|
+
:secret_access_key => $config[mode]["secret_access_key"],
|
31
|
+
:region => $config[mode]["region"],
|
32
|
+
:proxy_uri => $config[mode]["proxy_uri"],
|
33
|
+
)
|
34
|
+
else
|
35
|
+
$logger.debug "No AWS IAM credentials specified, defaulting to environmental discovery"
|
36
|
+
$logger.debug "If you get weird permissions errors, try setting the credentials explicity in config first."
|
37
|
+
end
|
38
|
+
else
|
39
|
+
$logger.debug "No AWS IAM credentials specified, defaulting to environmental discovery"
|
40
|
+
$logger.debug "If you get weird permissions errors, try setting the credentials explicity in config first."
|
41
|
+
end
|
42
|
+
|
43
|
+
# Setup S3 bucket
|
44
|
+
@s3 = AWS::S3.new
|
45
|
+
@bucket = @s3.buckets[ $config[mode]["s3_bucket"] ]
|
46
|
+
|
47
|
+
end
|
48
|
+
|
49
|
+
|
50
|
+
|
51
|
+
def upload src, dest
|
52
|
+
$logger.debug "Pushing file #{src} to s3://#{$config["general"]["s3_bucket"]}/#{$config["general"]["s3_prefix"]}#{dest}"
|
53
|
+
|
54
|
+
begin
|
55
|
+
# Generate the object name/key based on the relative file name and path.
|
56
|
+
s3_obj_name = "#{$config["general"]["s3_prefix"]}#{dest}"
|
57
|
+
s3_obj = @s3.buckets[$config["general"]["s3_bucket"]].objects[s3_obj_name]
|
58
|
+
|
59
|
+
# Perform S3 upload
|
60
|
+
s3_obj.write(:file => src)
|
61
|
+
|
62
|
+
rescue AWS::S3::Errors::NoSuchBucket => e
|
63
|
+
$logger.fatal "S3 bucket #{$config["general"]["s3_bucket"]} does not exist"
|
64
|
+
exit 0
|
65
|
+
|
66
|
+
rescue AWS::S3::Errors::AccessDenied => e
|
67
|
+
$logger.fatal "Access to S3 bucket #{$config["general"]["s3_bucket"]} denied"
|
68
|
+
exit 0
|
69
|
+
|
70
|
+
rescue AWS::S3::Errors::PermanentRedirect => e
|
71
|
+
$logger.error "The wrong endpoint has been specified (or autodetected) for #{$config["general"]["s3_bucket"]}."
|
72
|
+
raise e
|
73
|
+
|
74
|
+
rescue AWS::S3::Errors::SignatureDoesNotMatch => e
|
75
|
+
$logger.error "IAM signature error when accessing #{$config["general"]["s3_bucket"]}, probably invalid IAM credentials"
|
76
|
+
raise e
|
77
|
+
|
78
|
+
rescue Exception => e
|
79
|
+
raise e
|
80
|
+
end
|
81
|
+
end
|
82
|
+
|
83
|
+
|
84
|
+
|
85
|
+
def download src, dest = 'stream'
|
86
|
+
$logger.debug "Downloading file s3://#{$config["general"]["s3_bucket"]}/#{$config["general"]["s3_prefix"]}#{src} to #{dest}"
|
87
|
+
|
88
|
+
begin
|
89
|
+
# Generate the object name/key based on the relative file name and path.
|
90
|
+
s3_obj_name = "#{$config["general"]["s3_prefix"]}#{src}"
|
91
|
+
s3_obj = @s3.buckets[$config["general"]["s3_bucket"]].objects[s3_obj_name]
|
92
|
+
|
93
|
+
# Download the file
|
94
|
+
if dest == 'stream'
|
95
|
+
# Return the contents rather than writing to disk. We assume stream mode
|
96
|
+
# if the dest filename was unspecified
|
97
|
+
return s3_obj.read
|
98
|
+
else
|
99
|
+
# Download to an ondisk file
|
100
|
+
File.open(dest, 'wb') do |file|
|
101
|
+
s3_obj.read do |chunk|
|
102
|
+
file.write(chunk)
|
103
|
+
end
|
104
|
+
end
|
105
|
+
end
|
106
|
+
|
107
|
+
rescue AWS::S3::Errors::NoSuchKey => e
|
108
|
+
$logger.debug "No such file exists for download, this is normal at times."
|
109
|
+
return false
|
110
|
+
|
111
|
+
rescue AWS::S3::Errors::NoSuchBucket => e
|
112
|
+
$logger.fatal "S3 bucket #{$config["general"]["s3_bucket"]} does not exist"
|
113
|
+
exit 0
|
114
|
+
|
115
|
+
rescue AWS::S3::Errors::AccessDenied => e
|
116
|
+
$logger.fatal "Access to S3 bucket #{$config["general"]["s3_bucket"]} denied"
|
117
|
+
exit 0
|
118
|
+
|
119
|
+
rescue AWS::S3::Errors::PermanentRedirect => e
|
120
|
+
$logger.error "The wrong endpoint has been specified (or autodetected) for #{$config["general"]["s3_bucket"]}."
|
121
|
+
raise e
|
122
|
+
|
123
|
+
rescue AWS::S3::Errors::SignatureDoesNotMatch => e
|
124
|
+
$logger.error "IAM signature error when accessing #{$config["general"]["s3_bucket"]}, probably invalid IAM credentials"
|
125
|
+
raise e
|
126
|
+
|
127
|
+
rescue Exception => e
|
128
|
+
raise e
|
129
|
+
end
|
130
|
+
end
|
131
|
+
|
132
|
+
end
|
133
|
+
end
|
134
|
+
# vim:shiftwidth=2:tabstop=2:softtabstop=2:expandtab:smartindent
|
@@ -0,0 +1,70 @@
|
|
1
|
+
## Configuration file for Pupistry.
|
2
|
+
|
3
|
+
|
4
|
+
# The following settings apply for all use cases of Pupistry and must be set
|
5
|
+
general:
|
6
|
+
|
7
|
+
# We need somewhere to cache files like archives and git repos. This will
|
8
|
+
# be as big as the total size of all your git repos when being used to
|
9
|
+
# build artifacts. Agent-only systems will be far smaller as it only includes
|
10
|
+
# the latest version of the artifacts.
|
11
|
+
app_cache: ~/.pupistry/cache
|
12
|
+
|
13
|
+
# The S3 bucket must be set in order to have a place to push and
|
14
|
+
# pull artifact and manifests from. This bucket should be PRIVATE, we
|
15
|
+
# only want your servers accessing the files!
|
16
|
+
#
|
17
|
+
# REMEMBER - S3 buckets are a global namespace, other people might have
|
18
|
+
# already picked the name you want. Make sure you update this default
|
19
|
+
# with something you actually own :-)
|
20
|
+
s3_bucket: example
|
21
|
+
|
22
|
+
# S3 prefix is entirely optional, useful if you're reusing/sharing an S3
|
23
|
+
# bucket with other applications. Leave blank if not needed.
|
24
|
+
s3_prefix:
|
25
|
+
|
26
|
+
# GPG key to use for signing & validating the artifacts. It is possible to
|
27
|
+
# run pupistry in an unsigned mode, but you will lose the protection against
|
28
|
+
# someone with access to the S3 bucket tampering with the files and pushing
|
29
|
+
# malicious puppet manifests to your servers
|
30
|
+
gpg_disable: true
|
31
|
+
gpg_signing_key: XXXXXX
|
32
|
+
|
33
|
+
|
34
|
+
# Settings for agents, these are required on the machines that will be
|
35
|
+
# downloading and applying artifacts but also need to be set for build
|
36
|
+
# machines so we can popular bootstrap templates for you and automatically
|
37
|
+
# check stuff like IAM permissions before you roll your hosts.
|
38
|
+
agent:
|
39
|
+
puppetcode: /etc/puppet/environents/
|
40
|
+
|
41
|
+
# The AWS credentials with READ permission to the S3 bucket for downloading
|
42
|
+
# artifact files. If unset, we try to figure it out from any AWS creds
|
43
|
+
access_key_id:
|
44
|
+
secret_access_key:
|
45
|
+
region: ap-southeast-2
|
46
|
+
proxy_uri:
|
47
|
+
|
48
|
+
|
49
|
+
|
50
|
+
# The following settings are only needed on the build machines (people building
|
51
|
+
# new artifacts) and are not needed on the actual agent servers that will be
|
52
|
+
# downloading and applying them.
|
53
|
+
build:
|
54
|
+
|
55
|
+
# Define the Git repo for the Puppet manifest & r10k data
|
56
|
+
# (ie repo where your Puppetfile & site.pp is)
|
57
|
+
puppetcode: git@bitbucket.org:user/example.git
|
58
|
+
|
59
|
+
|
60
|
+
# The AWS credentials with write permission to the S3 bucket for uploading
|
61
|
+
# new artifact files. If unset, we try to figure it out from any AWS creds
|
62
|
+
# set in the environmnt, but you're best to make it explicit here to avoid
|
63
|
+
# surprises....
|
64
|
+
#
|
65
|
+
access_key_id:
|
66
|
+
secret_access_key:
|
67
|
+
region: ap-southeast-2
|
68
|
+
proxy_uri:
|
69
|
+
|
70
|
+
|
metadata
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: pupistry
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.0.
|
4
|
+
version: 0.0.2
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Jethro Carr
|
@@ -61,7 +61,12 @@ extra_rdoc_files: []
|
|
61
61
|
files:
|
62
62
|
- bin/pupistry
|
63
63
|
- lib/pupistry.rb
|
64
|
+
- lib/pupistry/artifact.rb
|
65
|
+
- lib/pupistry/bootstrap.rb
|
66
|
+
- lib/pupistry/config.rb
|
67
|
+
- lib/pupistry/storage_aws.rb
|
64
68
|
- README.md
|
69
|
+
- settings.example.yaml
|
65
70
|
homepage: https://github.com/jethrocarr/pupistry
|
66
71
|
licenses:
|
67
72
|
- Apache
|