cartage-remote 1.1 → 2.0.rc1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
data/.autotest DELETED
@@ -1,27 +0,0 @@
1
- # -*- ruby -*-
2
-
3
- require "autotest/restart"
4
-
5
- Autotest.add_hook :initialize do |at|
6
- # .minitest.rb ensures that the gem version of minitest is used.
7
- at.testlib = ".minitest.rb"
8
- # at.testlib = "minitest/unit"
9
- #
10
- # at.extra_files << "../some/external/dependency.rb"
11
- #
12
- # at.libs << ":../some/external"
13
- #
14
- # at.add_exception "vendor"
15
- #
16
- # at.add_mapping(/dependency.rb/) do |f, _|
17
- # at.files_matching(/test_.*rb$/)
18
- # end
19
- #
20
- # %w(TestA TestB).each do |klass|
21
- # at.extra_class_map[klass] = "test/test_misc.rb"
22
- # end
23
- end
24
-
25
- # Autotest.add_hook :run_command do |at|
26
- # system "rake build"
27
- # end
data/.gemtest DELETED
@@ -1 +0,0 @@
1
-
@@ -1,2 +0,0 @@
1
- gem "minitest"
2
- require "minitest/autorun"
@@ -1,63 +0,0 @@
1
- == Contributing
2
-
3
- We value any contribution to cartage-remote you can provide: a bug report, a
4
- feature request, or code contributions.
5
-
6
- cartage-remote has a few contribution guidelines:
7
-
8
- * Changes *will* *not* be accepted without tests. The test suite is written
9
- with {Minitest}[https://github.com/seattlerb/minitest].
10
- * Match our coding style.
11
- * Use a thoughtfully-named topic branch that contains your change. Rebase your
12
- commits into logical chunks as necessary.
13
- * Use {quality commit messages}[http://tbaggery.com/2008/04/19/a-note-about-git-commit-messages.html].
14
- * Do not change the version number; when your patch is accepted and a release
15
- is made, the version will be updated at that point.
16
- * Submit a GitHub pull request with your changes.
17
-
18
- === Test Dependencies
19
-
20
- cartage-remote uses Ryan Davis’s {Hoe}[https://github.com/seattlerb/hoe] to manage
21
- the release process, and it adds a number of rake tasks. You will mostly be
22
- interested in:
23
-
24
- $ rake
25
-
26
- which runs the tests the same way that:
27
-
28
- $ rake test
29
- $ rake travis
30
-
31
- will do.
32
-
33
- To assist with the installation of the development dependencies for cartage-remote,
34
- I have provided the simplest possible Gemfile pointing to the (generated)
35
- +cartage-remote.gemspec+ file. This will permit you to do:
36
-
37
- $ bundle install
38
-
39
- to get the development dependencies. If you aleady have +hoe+ installed, you
40
- can accomplish the same thing with:
41
-
42
- $ rake newb
43
-
44
- This task will install any missing dependencies, run the tests/specs, and
45
- generate the RDoc.
46
-
47
- === Workflow
48
-
49
- Here's the most direct way to get your work merged into the project:
50
-
51
- * Fork the project.
52
- * Clone down your fork (<tt>git clone git://github.com/KineticCafe/cartage-remote.git</tt>).
53
- * Create a topic branch to contain your change (<tt>git checkout -b my\_awesome\_feature</tt>).
54
- * Hack away, add tests. Not necessarily in that order.
55
- * Make sure everything still passes by running +rake+.
56
- * If necessary, rebase your commits into logical chunks, without errors.
57
- * Push the branch up (<tt>git push origin my\_awesome\_feature</tt>).
58
- * Create a pull request against KineticCafe/cartage-remote and describe
59
- what your change does and the why you think it should be merged.
60
-
61
- === Contributors
62
-
63
- * Austin Ziegler created cartage-remote.
data/Gemfile DELETED
@@ -1,9 +0,0 @@
1
- # -*- ruby -*-
2
-
3
- # NOTE: This file is not the canonical source of dependencies. Edit the
4
- # Rakefile, instead.
5
-
6
- source "https://rubygems.org/"
7
- gemspec
8
-
9
- # vim: syntax=ruby
@@ -1,419 +0,0 @@
1
- begin
2
- require 'psych'
3
- rescue LoadError
4
- end
5
- require 'tempfile'
6
- require 'yaml'
7
- require 'erb'
8
- require 'micromachine'
9
- require 'cartage/plugin'
10
-
11
- class Cartage
12
- # Connect to a remote machine and build a package remotely. Cartage::Remote
13
- # uses Fog::SSH with key-based, not password-based authentication to connect
14
- # to a remote server.
15
- #
16
- # Cartage::Remote assumes a relatively stable build server, but does not
17
- # require one (custom +prebuild+ and +postbuild+ scripts could be used to
18
- # manage that).
19
- #
20
- # == Remote Build Isolation
21
- #
22
- # Cartage::Remote allows for safe builds across multiple projects and
23
- # branches with path-based isolation. The pattern for the build path is shown
24
- # below, where the last part of the path is where the code will be cloned to.
25
- #
26
- # ~<remote_user>/cartage/<project-name>/<timestamp>/<project-name>
27
- # | | | | |
28
- # v | | | |
29
- # build root v | | |
30
- # cartage | | |
31
- # path v | |
32
- # project v |
33
- # path isolation |
34
- # path v
35
- # build
36
- # path
37
- #
38
- # So that if I am deploying on a project called +calliope+ and my remote user
39
- # is +build+, my isolated build path might be:
40
- #
41
- # ~build/cartage/calliope/20150321091432/calliope
42
- #
43
- # == Remote Build Steps
44
- #
45
- # The steps for a remote build are:
46
- #
47
- # 1. Configure Cartage and save the active Cartage configuration as a
48
- # temporary file that will be copied to the remote server.
49
- # 2. Configure the Fog::SSH adapters with the keys to connect to the remote
50
- # system.
51
- # 3. Create the +prebuild+ script and run it locally.
52
- # 4. Connect to the remote server, put the Cartage configuration file in the
53
- # isolation path, and clone the repository. Check the repo out to the
54
- # appropriate +release_hashref+.
55
- # 5. Create the +build+ script, copy it remotely, and run it from the build
56
- # isolation path (+build_path+). This is effectively:
57
- # cd "$build_path" && $build_script
58
- # 6. Clean up the remote server
59
- #
60
- # == Configuration
61
- #
62
- # Cartage::Remote is configured in the +plugins.remote+ section of the
63
- # Cartage configuration file. The following keys are *required*:
64
- #
65
- # +server+:: A server string in the form <tt>[user@]host[:port]</tt> *or* a
66
- # dictionary with +user+, +host+, and +port+. In either form, this
67
- # will set @remote_user, @remote_host, and @remote_port. If
68
- # @remote_user is not provided, it will be set from
69
- # <tt>$USER</tt>.
70
- #
71
- # The following keys are optional:
72
- #
73
- # +keys+:: The SSH key(s) used to connect to the server. There are two basic
74
- # ways that keys can be provided:
75
- #
76
- # * If provided as a string or an array of strings, the value(s)
77
- # will be applied as glob patterns to find key files on disk.
78
- # * If provided as a dictionary, the keys are irrelevant but the
79
- # values are the key data.
80
- #
81
- # If keys are not provided, a default pattern of
82
- # <tt>~/.ssh/*id_[rd]sa</tt> will be used to find keys on the local
83
- # machine.
84
- # +build+:: A multiline YAML string that is copied to the remote machine and
85
- # executed as a script there. If not provided, the following script
86
- # will be run:
87
- #
88
- # #!/bin/bash
89
- # set -e
90
- # if [ -f Gemfile ]; then
91
- # bundle install --path %<remote_bundle>s
92
- # bundle exec cartage build \
93
- # --config-file %<config_file>s \
94
- # --target %<project_path>s
95
- # else
96
- # cartage build --config-file %<config_file>s \
97
- # --target %<project_path>s
98
- # fi
99
- # +prebuild+:: A multiline YAML string that is run as a script on the local
100
- # machine to prepare for running remotely. If not provided, the
101
- # following script will be run:
102
- #
103
- # #!/bin/bash
104
- # ssh-keyscan -H %<remote_host>s >> ~/.ssh/known_hosts
105
- # +postbuild+:: A multiline YAML string that is run as a script on the local
106
- # machine to finish the build process locally. If not
107
- # provided, nothing will run. The script will be passed the
108
- # stage (+config+, +ssh_config+, +prebuild+, +remote_clone+,
109
- # +remote_build+, +cleanup+, or +finished+) and, if the stage
110
- # is not +finished+, the error message.
111
- #
112
- # == Script Substitution
113
- #
114
- # The +build+, +prebuild+, and +postbuild+ scripts require information from
115
- # the Cartage and Cartage::Remote instances. When these scripts are rendered
116
- # to disk, they will be run through Kernel#sprintf with the following
117
- # substitution parameters specified as strings
118
- # (<tt>%<<em>parameter-name</em>>s</tt>). All of these values are computed
119
- # from the local Cartage configuration.
120
- #
121
- # +repo_url+:: The repository URL.
122
- # +name+:: The package name.
123
- # +release_hashref+:: The release hashref to build.
124
- # +timestamp+:: The build timestamp.
125
- # +remote_host+:: The remote build host.
126
- # +remote_port+:: The remote build host SSH port (may be empty).
127
- # +remote_user+:: The remote build user.
128
- # +build_root+:: The remote build root, (usually
129
- # <tt>~<em>remote_user</em></tt>).
130
- # +cartage_path+:: <tt><em>build_root</em>/cartage</tt>.
131
- # +project_path+:: <tt><em>cartage_path</em>/<em>name</em></tt>.
132
- # +isolation_path+:: <tt><em>project_path</em>/<em>timestamp</em></tt>.
133
- # +build_path+:: The remote build path (contains the code to package).
134
- # <tt><em>isolation_path</em>/<em>name</em></tt>
135
- # +remote_bundle+:: A place where dependencies for the build can be installed
136
- # locally. <tt><em>isolation_path</em>/deps</tt>. Typically
137
- # used in the +build+ script.
138
- # bundle install --path %<remote_bundle>s
139
- # +bundle_cache+:: The +bundle_cache+ for the remote server. Set the same as
140
- # +project_path+.
141
- # +config_file+:: The remote filename of the computed Cartage configuration.
142
- # Must be provided to the remote run of +cartage+.
143
- # bundle exec cartage build --config-file %<config_file>s
144
- # +build_script+:: The full path to the remote build script.
145
- # <tt><em>isolation_path</em>/cartage-build-remote</tt>.
146
- #
147
- # == Configuration Example
148
- #
149
- # ---
150
- # plugins:
151
- # remote:
152
- # server:
153
- # host: build-server
154
- # script: |
155
- # #! /bin/bash
156
- # bundle install --path %<remote_bundle>s
157
- # bundle exec cartage s3 --config-file %<config_file>s
158
- #
159
- class Remote < Cartage::Plugin
160
- VERSION = '1.1' #:nodoc:
161
-
162
- def initialize(*) #:nodoc:
163
- super
164
- @tmpfiles = []
165
- @fsm = MicroMachine.new(:new).tap do |fsm|
166
- fsm.when(:local_setup, new: :config)
167
- fsm.when(:ssh_setup, config: :ssh_config)
168
- fsm.when(:run_prebuild, ssh_config: :prebuild)
169
- fsm.when(:clone_remote, prebuild: :remote_clone)
170
- fsm.when(:build_remote, remote_clone: :remote_build)
171
- fsm.when(:clean_remote, remote_build: :cleanup)
172
- fsm.when(:complete, cleanup: :finished)
173
- fsm.on(:any) { |event| dispatch(event, @fsm.state) }
174
- end
175
- end
176
-
177
- # Build on the remote server.
178
- def build
179
- @fsm.trigger!(:local_setup)
180
- @fsm.trigger!(:ssh_setup)
181
- @fsm.trigger!(:run_prebuild)
182
- @fsm.trigger!(:clone_remote)
183
- @fsm.trigger!(:build_remote)
184
- @fsm.trigger!(:clean_remote)
185
- @fsm.trigger!(:complete)
186
- rescue Cartage::StatusError
187
- raise
188
- rescue Exception => e
189
- error = e.exception("Remote error in stage #{@fsm.state}: #{e.message}")
190
- error.set_backtrace(e.backtrace)
191
- raise error
192
- ensure
193
- if @postbuild
194
- @cartage.display 'Running postbuild script...'
195
- system(make_tmpscript('postbuild', @postbuild, @subs).path,
196
- @fsm.state.to_s, error.to_s)
197
- end
198
-
199
- @tmpfiles.each { |tmpfile|
200
- tmpfile.close
201
- tmpfile.unlink
202
- }
203
- @tmpfiles.clear
204
- end
205
-
206
- private
207
-
208
- def local_setup
209
- @cartage.display 'Pre-build configuration...'
210
- @paths = OpenStruct.new(build_root: @build_root)
211
- @paths.cartage_path = @paths.build_root.join('cartage')
212
- @paths.project_path = @paths.cartage_path.join(@cartage.name)
213
- @paths.isolation_path = @paths.project_path.join(@cartage.timestamp)
214
- @paths.build_path = @paths.isolation_path.join(@cartage.name)
215
- @paths.remote_bundle = @paths.isolation_path.join('deps')
216
- @paths.bundle_cache = @paths.project_path
217
- @paths.config_file = @paths.isolation_path.join('cartage.yml')
218
- @paths.build_script = @paths.isolation_path.join('cartage-build-remote')
219
-
220
- @config_file = make_config(@paths).path
221
-
222
- @subs = OpenStruct.new(
223
- @paths.to_h.merge(repo_url: @cartage.repo_url,
224
- name: @cartage.name,
225
- release_hashref: @cartage.release_hashref,
226
- timestamp: @cartage.timestamp,
227
- remote_host: @remote_host,
228
- remote_port: @remote_port,
229
- remote_user: @remote_user)
230
- )
231
-
232
- end
233
-
234
- def ssh_setup
235
- require 'fog'
236
- options = {
237
- paranoid: true,
238
- keys: @keys,
239
- key_data: @key_data
240
- }
241
-
242
- options[:port] = @remote_port if @remote_port
243
-
244
- @ssh = Fog::SSH.new(@remote_host, @remote_user, options)
245
- @scp = Fog::SCP.new(@remote_host, @remote_user, options)
246
- end
247
-
248
- def run_prebuild
249
- @cartage.display 'Running prebuild script...'
250
- system(make_tmpscript('prebuild', @prebuild, @subs).path)
251
- end
252
-
253
- def clone_remote
254
- @cartage.display <<-message
255
- Checking out #{@cartage.repo_url} at #{@cartage.release_hashref} remotely...
256
- message
257
-
258
- ssh %Q(mkdir -p #{@paths.isolation_path})
259
- @scp.upload(@config_file, user_path(@paths.config_file))
260
- ssh %Q(git clone #{@cartage.repo_url} #{@paths.build_path})
261
- ssh <<-command
262
- cd #{@paths.build_path} && git checkout #{@cartage.release_hashref}
263
- command
264
- end
265
-
266
- def build_remote
267
- @cartage.display 'Running build script...'
268
- script = make_tmpscript('build', @build, @subs).path
269
- @scp.upload(script, user_path(@paths.build_script))
270
- ssh %Q(cd #{@paths.build_path} && #{@paths.build_script})
271
- end
272
-
273
- def clean_remote
274
- @cartage.display 'Cleaning up after the build...'
275
- ssh %Q(rm -rf #{@paths.isolation_path})
276
- end
277
-
278
- def dispatch(event, state)
279
- send(event) if respond_to?(event, true)
280
- send(state) if respond_to?(state, true)
281
- end
282
-
283
- def resolve_config!(remote_config)
284
- unless remote_config
285
- raise ArgumentError, 'Cartage remote has no configuration.'
286
- end
287
-
288
- @remote_user = @remote_host = @remote_port = nil
289
-
290
- case server = remote_config.server
291
- when OpenStruct
292
- @remote_user = server.user
293
- @remote_host = server.host
294
- @remote_port = server.port
295
- when %r{\A(?:(?<user>[^@]+)@)?(?<host>[^@:]+)(?::(?<port>[^:]+))?\z}
296
- @remote_user = $~[:user]
297
- @remote_host = $~[:host]
298
- @remote_port = $~[:port]
299
- end
300
-
301
- @remote_user ||= ENV['USER']
302
-
303
- if @remote_host.nil? or @remote_host.empty?
304
- raise ArgumentError, 'Cannot connect to remote; no server specified.'
305
- end
306
-
307
- @remote_server = @remote_host
308
- @remote_server = "#{@remote_user}@#{@remote_server}" if @remote_user
309
- @remote_server = "#{@remote_server}:#{@remote_port}" if @remote_port
310
-
311
- @build_root = Pathname(remote_config.build_root || '~')
312
-
313
- @build = remote_config.build
314
- raise ArgumentError, <<-exception if @build.nil? or @build.empty?
315
- No build script to run on remote #{@remote_server}.
316
- exception
317
-
318
- @key_data = @keys = nil
319
-
320
- case keys = remote_config.keys
321
- when OpenStruct
322
- @key_data = keys.to_h.values
323
- when Array
324
- @keys = keys
325
- when String
326
- @keys = [ keys ]
327
- when nil
328
- @keys = %w(~/.ssh/*id_[rd]sa)
329
- end
330
-
331
- @keys &&= @keys.map { |key|
332
- Pathname.glob(Pathname(key).expand_path)
333
- }.flatten
334
-
335
- @prebuild = remote_config.prebuild || DEFAULT_PREBUILD_SCRIPT
336
- @postbuild = remote_config.postbuild
337
-
338
- # Force lazy values to be present during execution.
339
- @cartage.repo_url
340
- @cartage.root_path
341
- @cartage.release_hashref
342
- @cartage.timestamp
343
- @cartage
344
- end
345
-
346
-
347
- def ssh(*commands)
348
- results = @ssh.run(commands) do |stdout, stderr|
349
- $stdout.print stdout unless stdout.nil?
350
- $stderr.print stderr unless stderr.nil?
351
- end
352
-
353
- results.each do |result|
354
- if result.status.nonzero?
355
- message = <<-msg
356
- Remote error in stage #{@fsm.state}:
357
- SSH command failed with status (#{result.status}):
358
- #{result.command}
359
- msg
360
- fail Cartage::StatusError.new(result.status, message)
361
- end
362
- end
363
- end
364
-
365
- def make_tmpfile(basename, content = nil)
366
- Tempfile.new("#{basename}.").tap { |f|
367
- f.write content || yield
368
- f.close
369
- @tmpfiles << f
370
- }
371
- end
372
-
373
- def make_tmpscript(basename, content, subs)
374
- make_tmpfile(basename, content % subs.to_h).tap { |f|
375
- File.chmod(0700, f.path)
376
- }
377
- end
378
-
379
- def make_config(paths)
380
- make_tmpfile('config.yml') do
381
- config = Cartage::Config.new(@cartage.config)
382
- config.name = @cartage.name
383
- config.release_hashref = @cartage.release_hashref
384
- config.timestamp = @cartage.timestamp
385
- config.root_path = paths.build_path.to_s
386
- config.bundle_cache = paths.bundle_cache.to_s
387
- config.to_yaml
388
- end
389
- end
390
-
391
- def user_path(path)
392
- path.to_s.sub(%r{\A~/}, '')
393
- end
394
-
395
- def self.commands #:nodoc:
396
- require_relative 'remote/command'
397
- [ Cartage::Remote::Command ]
398
- end
399
-
400
- DEFAULT_PREBUILD_SCRIPT = <<-script #:nodoc:
401
- #!/bin/bash
402
-
403
- ssh-keyscan -H %<remote_host>s >> ~/.ssh/known_hosts
404
- script
405
-
406
- DEFAULT_BUILD_SCRIPT = <<-script #:nodoc:
407
- #!/bin/bash
408
-
409
- set -e
410
-
411
- if [ -f Gemfile ]; then
412
- bundle install --path %<remote_bundle>s
413
- bundle exec cartage build --config-file %<config_file>s --target %<project_path>s
414
- else
415
- cartage build --config-file %<config_file>s --target %<project_path>s
416
- fi
417
- script
418
- end
419
- end