bosh-bootstrap 0.5.0

Sign up to get free protection for your applications and to get access to all the features.
Files changed (47) hide show
  1. data/.gitignore +18 -0
  2. data/Gemfile +4 -0
  3. data/LICENSE.txt +22 -0
  4. data/README.md +318 -0
  5. data/Rakefile +1 -0
  6. data/bin/bosh-bootstrap +8 -0
  7. data/bosh-bootstrap.gemspec +34 -0
  8. data/lib/bosh-bootstrap.rb +10 -0
  9. data/lib/bosh-bootstrap/cli.rb +1024 -0
  10. data/lib/bosh-bootstrap/commander.rb +9 -0
  11. data/lib/bosh-bootstrap/commander/README.md +47 -0
  12. data/lib/bosh-bootstrap/commander/command.rb +25 -0
  13. data/lib/bosh-bootstrap/commander/commands.rb +80 -0
  14. data/lib/bosh-bootstrap/commander/local_server.rb +68 -0
  15. data/lib/bosh-bootstrap/commander/remote_script_command.rb +48 -0
  16. data/lib/bosh-bootstrap/commander/remote_server.rb +121 -0
  17. data/lib/bosh-bootstrap/commander/upload_command.rb +17 -0
  18. data/lib/bosh-bootstrap/helpers.rb +2 -0
  19. data/lib/bosh-bootstrap/helpers/fog_setup.rb +50 -0
  20. data/lib/bosh-bootstrap/helpers/settings.rb +36 -0
  21. data/lib/bosh-bootstrap/stages.rb +8 -0
  22. data/lib/bosh-bootstrap/stages/stage_micro_bosh_delete.rb +90 -0
  23. data/lib/bosh-bootstrap/stages/stage_micro_bosh_delete/bosh_micro_delete +19 -0
  24. data/lib/bosh-bootstrap/stages/stage_micro_bosh_deploy.rb +135 -0
  25. data/lib/bosh-bootstrap/stages/stage_micro_bosh_deploy/bosh_micro_deploy +36 -0
  26. data/lib/bosh-bootstrap/stages/stage_micro_bosh_deploy/download_micro_bosh_stemcell +132 -0
  27. data/lib/bosh-bootstrap/stages/stage_micro_bosh_deploy/install_key_pair_for_user +23 -0
  28. data/lib/bosh-bootstrap/stages/stage_prepare_inception_vm.rb +52 -0
  29. data/lib/bosh-bootstrap/stages/stage_prepare_inception_vm/convert_salted_password +9 -0
  30. data/lib/bosh-bootstrap/stages/stage_prepare_inception_vm/create_vcap_user +79 -0
  31. data/lib/bosh-bootstrap/stages/stage_prepare_inception_vm/install_base_packages +13 -0
  32. data/lib/bosh-bootstrap/stages/stage_prepare_inception_vm/install_bosh +54 -0
  33. data/lib/bosh-bootstrap/stages/stage_prepare_inception_vm/install_ruby +33 -0
  34. data/lib/bosh-bootstrap/stages/stage_prepare_inception_vm/install_useful_gems +24 -0
  35. data/lib/bosh-bootstrap/stages/stage_prepare_inception_vm/validate_bosh_deployer +21 -0
  36. data/lib/bosh-bootstrap/stages/stage_setup_new_bosh.rb +52 -0
  37. data/lib/bosh-bootstrap/stages/stage_setup_new_bosh/cleanup_permissions +14 -0
  38. data/lib/bosh-bootstrap/stages/stage_setup_new_bosh/setup_bosh_user +29 -0
  39. data/lib/bosh-bootstrap/stages/stage_validate_inception_vm.rb +39 -0
  40. data/lib/bosh-bootstrap/stages/stage_validate_inception_vm/validate_ubuntu +6 -0
  41. data/lib/bosh-bootstrap/version.rb +5 -0
  42. data/lib/bosh/providers.rb +21 -0
  43. data/lib/bosh/providers/README.md +5 -0
  44. data/lib/bosh/providers/aws.rb +77 -0
  45. data/lib/bosh/providers/base_provider.rb +20 -0
  46. data/lib/bosh/providers/openstack.rb +40 -0
  47. metadata +239 -0
data/.gitignore ADDED
@@ -0,0 +1,18 @@
1
+ *.gem
2
+ *.rbc
3
+ .bundle
4
+ .config
5
+ .yardoc
6
+ Gemfile.lock
7
+ InstalledFiles
8
+ _yardoc
9
+ coverage
10
+ doc/
11
+ lib/bundler/man
12
+ pkg
13
+ rdoc
14
+ spec/reports
15
+ test/tmp
16
+ test/version_tmp
17
+ tmp
18
+ .DS_Store
data/Gemfile ADDED
@@ -0,0 +1,4 @@
1
+ source 'https://rubygems.org'
2
+
3
+ # Specify your gem's dependencies in bosh-bootstrap.gemspec
4
+ gemspec
data/LICENSE.txt ADDED
@@ -0,0 +1,22 @@
1
+ Copyright (c) 2012 Stark & Wayne LLC
2
+
3
+ MIT License
4
+
5
+ Permission is hereby granted, free of charge, to any person obtaining
6
+ a copy of this software and associated documentation files (the
7
+ "Software"), to deal in the Software without restriction, including
8
+ without limitation the rights to use, copy, modify, merge, publish,
9
+ distribute, sublicense, and/or sell copies of the Software, and to
10
+ permit persons to whom the Software is furnished to do so, subject to
11
+ the following conditions:
12
+
13
+ The above copyright notice and this permission notice shall be
14
+ included in all copies or substantial portions of the Software.
15
+
16
+ THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
17
+ EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
18
+ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
19
+ NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
20
+ LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
21
+ OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
22
+ WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
data/README.md ADDED
@@ -0,0 +1,318 @@
1
+ # Stark & Wayne's Bosh Bootstrapper
2
+
3
+ In order to deploy CloudFoundry, and a growing number of other complex systems, you will need a BOSH. BOSH provides a complete lifecycle manager/deployer for complex systems. CloudFoundry is a very complex system when it comes to deployment/upgrades.
4
+
5
+ The Stark & Wayne Bosh Bootstrapper is the simplest way to get a Micro BOSH running, to upgrade an existing Micro BOSH, and to delete it if you change your mind.
6
+
7
+ Bootstrap a Micro BOSH universe from one CLI command. Also allows SSH access and the ability to delete created Micro BOSHes.
8
+
9
+ ```
10
+ $ bosh-bootstrap deploy --latest-stemcell
11
+ Creating inception VM...
12
+ Creating micro BOSH VM...
13
+
14
+ $ bosh-bootstrap ssh
15
+ Open SSH tunnel to inception VM...
16
+
17
+ $ bosh-bootstrap delete
18
+ Deleting micro BOSH VM...
19
+ ```
20
+
21
+ It is now very simple to bootstrap a micro BOSH from a single, local CLI. The bootstrapper first creates an inception VM and then uses the `bosh_deployer` (`bosh micro deploy`) to deploy micro BOSH.
22
+
23
+ To be cute about it, the Stark & Wayne Bosh Bootstrapper aims to provide lifecycle management for the BOSH lifecycle manager. Zing! See the "Deep dive into deploy command" section below for greater understanding why the Stark & Wayne Bosh Bootstrapper is very useful.
24
+
25
+ ## Installation
26
+
27
+ This bootstrapper for BOSH is distributed as a RubyGem for Ruby 1.8+.
28
+
29
+ ```
30
+ $ gem install bosh-bootstrap
31
+ ```
32
+
33
+ ## Usage
34
+
35
+ ### First time usage
36
+
37
+ The first time you use `bosh-bootstrap` it will create everything necessary. The example output below includes user prompts.
38
+
39
+ ```
40
+ $ bosh-bootstrap deploy --latest-stemcell
41
+
42
+ Stage 1: Choose infrastructure
43
+
44
+ Found infrastructure API credentials at ~/.fog (override with --fog)
45
+ 1. AWS (default)
46
+ 2. AWS (bosh)
47
+ 3. Rackspace (default)
48
+ Choose infrastructure: 1
49
+
50
+ Confirming: using AWS infrastructure.
51
+
52
+ 1. ap-northeast-1
53
+ 2. ap-southeast-1
54
+ 3. eu-west-1
55
+ 4. us-east-1
56
+ 5. us-west-1
57
+ 6. us-west-2
58
+ 7. sa-east-1
59
+ Choose AWS region: 6
60
+ Confirming: Using AWS us-west-2 region.
61
+
62
+
63
+ Stage 2: Configuration
64
+
65
+ Confirming: Micro BOSH will be named microbosh_aws_us_east_1
66
+
67
+ BOSH username: drnic
68
+ BOSH password: ********
69
+ Confirming: After BOSH is created, your username will be drnic
70
+
71
+ Confirming: Micro BOSH will be assigned IP address 174.129.227.124
72
+
73
+ Confirming: Micro BOSH protected by security group named microbosh_aws_us_east_1, with ports [22, 6868, 25555, 25888]
74
+
75
+ Confirming: Micro BOSH accessible via key pair named microbosh_aws_us_east_1
76
+
77
+ Confirming: Micro BOSH will be created with stemcell micro-bosh-stemcell-aws-0.6.4.tgz
78
+
79
+
80
+ Stage 3: Create/Allocate the Inception VM
81
+
82
+ 1. create new inception VM
83
+ 2. use an existing Ubuntu server
84
+ 3. use this server (must be ubuntu & on same network as bosh)
85
+ Create or specify an Inception VM: 1
86
+
87
+ Confirming: Inception VM has been created
88
+
89
+ Stage 4: Preparing the Inception VM
90
+
91
+ Successfully created vcap user
92
+ Successfully installed base packages
93
+ Successfully installed ruby 1.9.3
94
+ Successfully installed useful ruby gems
95
+ Successfully installed bosh
96
+ Successfully captured value of salted password
97
+ Successfully validated bosh deployer
98
+
99
+ Stage 5: Deploying micro BOSH
100
+
101
+ Successfully downloaded micro-bosh stemcell
102
+ Successfully uploaded micro-bosh deployment manifest file
103
+ Successfully installed key pair for user
104
+ Successfully deploy micro bosh
105
+ ```
106
+
107
+ ### Local usage
108
+
109
+ During the `bosh-bootstrap deploy` sequence above, you could choose to use the local VM as the Inception VM.
110
+
111
+ For AWS, it is important that you only use a VM that is on the same infrastructure region. The process of creating a micro-bosh VM is to use a local
112
+ EBS volume to create an AMI. The target region for the micro-bosh VM must therefore be in the same region.
113
+
114
+ ### Repeat usage
115
+
116
+ The `deploy` command can be re-run and it will not prompt again for inputs. It aims to be idempotent. This means that if you see any errors when running `deploy`, such as unavailability of VMs or IP addresses, then when you resolve those issues you can re-run the `deploy` command and it will resume the bootstrap of micro-bosh (and the optional inception VM).
117
+
118
+ ## SSH access
119
+
120
+ You can open an SSH shell with the Inception VM:
121
+
122
+ ```
123
+ $ bosh-bootstrap ssh
124
+ ```
125
+
126
+ You can also pass a COMMAND argument and that command will be run instead of the shell being opened.
127
+
128
+ ```
129
+ $ bosh-bootstrap ssh 'whoami'
130
+ ubuntu
131
+ ```
132
+
133
+ ## Deleting micro BOSH
134
+
135
+ The `bosh-bootstrap delete` command will delete the target micro-bosh.
136
+
137
+ ```
138
+ $ bosh-bootstrap delete
139
+ Stage 1: Target inception VM to use to delete micro-bosh
140
+
141
+ Confirming: Using inception VM ubuntu@ec2-184-73-231-239.compute-1.amazonaws.com
142
+
143
+ Stage 2: Deleting micro BOSH
144
+ Delete micro BOSH
145
+ stopping agent services (00:00:01)
146
+ unmount disk (00:00:10)
147
+ detach disk (00:00:13)
148
+ delete disk (00:02:35)
149
+ delete VM (00:00:37)
150
+ delete stemcell (00:00:00)
151
+ Done 6/6 00:03:37
152
+ Deleted deployment 'microbosh-aws-us-east-1', took 00:03:37 to complete
153
+ ```
154
+
155
+ ## Deep dive into the BOSH Bootstrap deploy command
156
+
157
+ What is actually happening when you run `bosh-bootstrap deploy`?
158
+
159
+ At the heart of `bosh-bootstrap deploy` is the execution of the BOSH Deployer, a command provided with BOSH to bootstrap a single VM with all the parts of BOSH running on it. If you ran this command yourself you would run:
160
+
161
+ ```
162
+ $ gem install bosh-deployer
163
+ $ bosh download public stemcell some-microbosh-stemcell.tgz
164
+ $ bosh micro deploy some-microbosh-stemcell.tgz
165
+ ```
166
+
167
+ Unfortunately for this simple scenario, there are many little prerequisite steps before those three commands. The Stark & Wayne Bosh Bootstrapper replaces pages and pages of step-by-step instructions with a single command line that does everything. It even allows you to upgrade your Micro BOSH with newer BOSH releases: both publicly available stemcells and custom stemcells generated from the BOSH source code.
168
+
169
+ To understand exactly what the `bosh-bootstrap deploy` command is doing, let's start with what the running parts of BOSH are and how `bosh micro deploy` deploys them.
170
+
171
+ ### What is in BOSH?
172
+
173
+ A running BOSH, whether it is running on a single server or a cluster of servers, is a collection of processes. The core of BOSH is the Director and the Blobstore. The remaining processes provide support, storage or messaging.
174
+
175
+ * The Director, the public API for the bosh CLI and coordinator of BOSH behavior
176
+ * The Blobstore, to store and retrieve precompiled packages
177
+ * Agents, run on each server within deployments
178
+ * The Health Manager, to track the state of deployed systems (the infrastructure and running jobs)
179
+ * Internal DNS, called PowerDNS, for internal unique naming of servers within BOSH deployments
180
+ * Registry, for example AWS Registry, for tracking the infrastructure that has been provisioned (servers, persistent disks)
181
+ * PostgreSQL
182
+ * Redis
183
+
184
+ When you deploy a BOSH using the BOSH Deployer (`bosh micro deploy`) or indirectly via the BOSH Bootstrapper, you are actually deploying a BOSH release that describes a BOSH called [bosh-release](https://github.com/cloudfoundry/bosh-release). The processes listed above are called "jobs" and you can see the full list of jobs inside a BOSH within the [jobs/ directory](https://github.com/cloudfoundry/bosh-release/jobs) of `bosh-release`.
185
+
186
+ But you don't yet have a BOSH to deploy another BOSH.
187
+
188
+ ### How to get your first BOSH?
189
+
190
+ The BOSH Deployer (`bosh micro deploy`) exists to spin you up a pre-baked server with all the packages and jobs running.
191
+
192
+ When you run the BOSH Deployer on a server, it does not convert that server into a BOSH. Rather, it provisions a single brand new server, with all the required packages, configuration and startup scripts. We call this pre-baked server a Micro BOSH.
193
+
194
+ A Micro BOSH server is a normal running server built from a base OS image that already contains all the packages, configuration and startup scripts for the jobs listed above.
195
+
196
+ In BOSH terminology, call these pre-packaged base OS images "stemcells".
197
+
198
+ For AWS, vSphere and OpenStack there are publicly available stemcells that can bootstrap a Micro BOSH for that infrastructure. To see the current list of all public Micro BOSH stemcells for all infrastructure providers; and to download one of them:
199
+
200
+ ```
201
+ $ bosh public stemcells --tag micro
202
+ $ bosh download public stemcell micro-bosh-stemcell-aws-0.6.4.tgz
203
+ ```
204
+
205
+ The CloudFoundry BOSH team will release new public stemcells overtime. The BOSH Deployer allows you to upgrade to newer stemcells as easily as it is to deploy a Micro BOSH initially.
206
+
207
+ ```
208
+ $ bosh micro deploy micro-bosh-stemcell-aws-0.6.4.tgz
209
+ $ bosh micro deploy micro-stemcell-aws-0.7.0.tgz --update
210
+ ```
211
+
212
+ ### Configuring a Micro BOSH
213
+
214
+ The command above will not work without first providing BOSH Deployer with configuration details. The stemcell file alone is not sufficient information. When we deploy or update a Micro BOSH we need to provide the following:
215
+
216
+ * A static IP address - this IP address will be bound to the initial Micro BOSH server, and when the Micro BOSH is updated in future and the server is thrown away and replaced, then it is bound to the replacement servers
217
+ * Server properties - the instance type (such as m1.large on AWS) or RAM/CPU combination (on vSphere)
218
+ * Server persistent disk - a single persistent, attached disk volume will be provisioned and mounted at `/var/vcap/store`; when the Micro BOSH is updated is is unmounted, unattached from the current server and then reattached and remounted to the upgraded server
219
+ * Infrastructure API credentials - the magic permissions for the Micro BOSH to provision servers and persistent disks for its BOSH deployments
220
+
221
+ This information is to go into a file called `/path/to/deployments/NAME/micro_bosh.yml`. Before `bosh micro deploy` is run, we first need to tell BOSH Deployer which file contains the Micro BOSH deployment manifest.
222
+
223
+ In the Stark & Wayne Bosh Bootstrapper, the manifests are stored at `/var/vcap/store/microboshes/deployments/NAME/micro_bosh.yml`.
224
+
225
+ So the BOSH Deployer command that is run to specify the deployment manifest and run the deployment is:
226
+
227
+ ```
228
+ $ bosh micro deployment `/var/vcap/store/microboshes/deployments/NAME/micro_bosh.yml`
229
+ $ bosh micro deploy /var/vcap/store/stemcells/micro-bosh-stemcell-aws-0.6.4.tgz
230
+ ```
231
+
232
+ ### Why does it take so long to deploy Micro BOSH on AWS?
233
+
234
+ On AWS it can take over 20 minutes to deploy or upgrade a Micro BOSH from a public stemcell. The majority of this time is taken with converting the stemcell file (such as `micro-bosh-stemcell-aws-0.6.4.tgz`) into an Amazon AMI.
235
+
236
+ When you boot a new server on AWS you provide the base machine image for the root filesystem. This is called the Amazon Machine Image (AMI). For our Micro BOSH, we need an AMI that contains all the packages, process configuration and startup scripts. That is, we need to convert our stemcell into an AMI; then use the AMI to boot the Micro BOSH server.
237
+
238
+ The BOSH Deployer performs all the hard work to create an AMI. Believe me, it is a lot of hard work.
239
+
240
+ The summary of the process of creating the Micro BOSH AMI is:
241
+
242
+ 1. Create a new EBS volume (an attached disk) on the server running BOSH Deployer
243
+ 2. Unpack/upload the stemcell onto the EBS volume
244
+ 3. Create a snapshot of the EBS volume
245
+ 4. Register the snapshot as an AMI
246
+
247
+ This process takes the majority of the time to deploy a new/replacement Micro BOSH server.
248
+
249
+ ### Why can't I run BOSH Deployer from my laptop?
250
+
251
+ One of the feature of the BOSH Bootstrapper is that you can run it from your local laptop. BOSH Deployer itself cannot be run from your laptop. The reason is hidden in the step-by-step AMI example above. In AWS, to create an EBS volume, create a snapshot and register it as an AMI, you need to be running the commands on an AWS server in the same target region as your future Micro BOSH server.
252
+
253
+ The server that runs the BOSH Deployer is commonly called the Inception VM. For AWS you need an Inception VM in the same AWS region that you will provision your Micro BOSH server. Since a BOSH also manages a stemcell process similar to the above, your BOSH must be in the same AWS region that you a deploying BOSH releases.
254
+
255
+ ### How does BOSH Bootstrapper get around this requirement?
256
+
257
+ The BOSH Bootstrapper can run from your laptop or locally from an Inception VM.
258
+
259
+ If you run it from your laptop, then it will prompt you to create a new Inception VM or for the host/username of a pre-existing Ubuntu server. The BOSH Bootstrapper will then use SSH to command the Inception VM to perform all the deployment steps discussed above.
260
+
261
+ That is the core of the service being provided by the BOSH Bootstrapper - to prepare an Inception VM and to command it to deploy and upgrade Micro BOSHes.
262
+
263
+ ## Internal configuration/settings
264
+
265
+ Once you've used the CLI it stores your settings for your BOSH, so that you can re-run the tool for upgrades or other future functionality.
266
+
267
+ By default, the settings file is stored at `~/.bosh_bootstrap/manifest.yml`.
268
+
269
+ For an AWS BOSH it looks like:
270
+
271
+ ``` yaml
272
+ ---
273
+ fog_path: /Users/drnic/.fog
274
+ fog_credentials:
275
+ provider: AWS
276
+ aws_access_key_id: ACCESS_KEY
277
+ aws_secret_access_key: SECRET_KEY
278
+ region: us-east-1
279
+ bosh_cloud_properties:
280
+ aws:
281
+ access_key_id: ACCESS_KEY
282
+ secret_access_key: SECRET_KEY
283
+ default_key_name: microbosh
284
+ default_security_groups:
285
+ - microbosh
286
+ ec2_private_key: /home/vcap/.ssh/microbosh.pem
287
+ bosh_resources_cloud_properties:
288
+ instance_type: m1.medium
289
+ bosh_provider: aws
290
+ region_code: us-east-1
291
+ bosh_username: drnic
292
+ bosh_password: PASSWORD
293
+ bosh:
294
+ password: PASSWORD
295
+ salted_password: 'sdfkjhadsjkadsfjhdsf'
296
+ persistent_disk: 16384
297
+ ip_address: 107.22.247.45
298
+ micro_bosh_stemcell_name: "micro-bosh-stemcell-aws-0.6.4.tgz"
299
+ ```
300
+
301
+ ## Contributing
302
+
303
+ 1. Fork it
304
+ 2. Create your feature branch (`git checkout -b my-new-feature`)
305
+ 3. Commit your changes (`git commit -am 'Add some feature'`)
306
+ 4. Push to the branch (`git push origin my-new-feature`)
307
+ 5. Create new Pull Request
308
+
309
+ ## Copyright
310
+
311
+ All documentation and source code is copyright of Stark & Wayne LLC.
312
+
313
+ ## Subscription and Support
314
+
315
+ This documentation & tool is freely available to all people and companies coming to CloudFoundry and BOSH.
316
+
317
+ If you decide to run CloudFoundry and BOSH in production, please purchase a Subscription and Support Agreement with Stark & Wayne so we can continue to create and maintain top quality documentation and tools; and also provide you with bespoke support for your deployments. We want to help you be successfully.
318
+
data/Rakefile ADDED
@@ -0,0 +1 @@
1
+ require "bundler/gem_tasks"
@@ -0,0 +1,8 @@
1
+ #!/usr/bin/env ruby
2
+ $:.push File.dirname(__FILE__) + '/../lib'
3
+
4
+ require "rubygems"
5
+ require "bosh-bootstrap"
6
+ require "bosh-bootstrap/cli"
7
+
8
+ Bosh::Bootstrap::Cli.start
@@ -0,0 +1,34 @@
1
+ # -*- encoding: utf-8 -*-
2
+ lib = File.expand_path('../lib', __FILE__)
3
+ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
4
+ require 'bosh-bootstrap/version'
5
+
6
+ Gem::Specification.new do |gem|
7
+ gem.name = "bosh-bootstrap"
8
+ gem.version = Bosh::Bootstrap::VERSION
9
+ gem.authors = ["Dr Nic Williams"]
10
+ gem.email = ["drnicwilliams@gmail.com"]
11
+ gem.description = %q{Bootstrap a micro BOSH universe from one CLI}
12
+ gem.summary = <<-EOS
13
+ Now very simple to bootstrap a micro BOSH from a single, local CLI.
14
+ The bootstrapper first creates an inception VM and then uses
15
+ bosh_deployer (bosh micro deploy) to deploy micro BOSH from
16
+ an available stemcell.
17
+ EOS
18
+ gem.homepage = "https://github.com/StarkAndWayne/bosh-bootstrap"
19
+
20
+ gem.files = `git ls-files`.split($/)
21
+ gem.executables = gem.files.grep(%r{^bin/}).map{ |f| File.basename(f) }
22
+ gem.test_files = gem.files.grep(%r{^(test|spec|features)/})
23
+ gem.require_paths = ["lib"]
24
+
25
+ gem.add_dependency "thor"
26
+ gem.add_dependency "highline"
27
+ gem.add_dependency "settingslogic"
28
+ gem.add_dependency "POpen4"
29
+ gem.add_dependency "net-scp"
30
+ gem.add_dependency "fog", "~>1.8.0"
31
+ gem.add_dependency "escape"
32
+ gem.add_dependency "bosh_cli"
33
+ gem.add_development_dependency "bosh_deployer"
34
+ end
@@ -0,0 +1,10 @@
1
+ module Bosh
2
+ module Bootstrap
3
+ end
4
+ end
5
+
6
+ require "bosh-bootstrap/version"
7
+ require "bosh-bootstrap/commander"
8
+ require "bosh-bootstrap/stages"
9
+
10
+ require "bosh/providers"
@@ -0,0 +1,1024 @@
1
+ require "thor"
2
+ require "highline"
3
+ require "fileutils"
4
+
5
+ # for the #sh helper
6
+ require "rake"
7
+ require "rake/file_utils"
8
+
9
+ require "escape"
10
+
11
+ require "bosh-bootstrap/helpers"
12
+
13
+ module Bosh::Bootstrap
14
+ class Cli < Thor
15
+ include Thor::Actions
16
+ include Bosh::Bootstrap::Helpers::FogSetup
17
+ include Bosh::Bootstrap::Helpers::Settings
18
+ include FileUtils
19
+
20
+ attr_reader :fog_credentials
21
+ attr_reader :server
22
+
23
+ desc "deploy", "Bootstrap Micro BOSH, and optionally an Inception VM"
24
+ method_option :fog, :type => :string, :desc => "fog config file (default: ~/.fog)"
25
+ method_option :"private-key", :type => :string, :desc => "Local passphrase-less private key path"
26
+ method_option :"upgrade-deps", :type => :boolean, :desc => "Force upgrade dependencies, packages & gems"
27
+ method_option :"edge-deployer", :type => :boolean, :desc => "Install bosh deployer from git instead of rubygems"
28
+ method_option :"latest-stemcell", :type => :boolean, :desc => "Use latest micro-bosh stemcell; possibly not tagged stable"
29
+ method_option :"edge-stemcell", :type => :boolean, :desc => "Create custom stemcell from BOSH git source"
30
+ def deploy
31
+ load_deploy_options # from method_options above
32
+
33
+ deploy_stage_1_choose_infrastructure_provider
34
+ deploy_stage_2_bosh_configuration
35
+ deploy_stage_3_create_allocate_inception_vm
36
+ deploy_stage_4_prepare_inception_vm
37
+ deploy_stage_5_deploy_micro_bosh
38
+ deploy_stage_6_setup_new_bosh
39
+ end
40
+
41
+ # desc "delete", "Delete Micro BOSH"
42
+ # method_option :all, :type => :boolean, :desc => "Delete all micro-boshes and inception VM [coming soon]"
43
+ # def delete
44
+ # delete_stage_1_target_inception_vm
45
+ #
46
+ # if options[:all]
47
+ # error "I'm sorry; the awesome --all flag is not yet implemented"
48
+ # delete_all_stage_2_delete_micro_boshes
49
+ # delete_all_stage_3_delete_inception_vm
50
+ # else
51
+ # delete_one_stage_2_delete_micro_bosh
52
+ # end
53
+ # end
54
+ #
55
+ desc "ssh [COMMAND]", "Open an ssh session to the inception VM [do nothing if local machine is inception VM]"
56
+ long_desc <<-DESC
57
+ If a command is supplied, it will be run, otherwise a session will be
58
+ opened.
59
+ DESC
60
+ def ssh(cmd=nil)
61
+ run_ssh_command_or_open_tunnel(cmd)
62
+ end
63
+
64
+ no_tasks do
65
+ DEFAULT_INCEPTION_VOLUME_SIZE = 32 # Gb
66
+
67
+ def deploy_stage_1_choose_infrastructure_provider
68
+ header "Stage 1: Choose infrastructure"
69
+ unless settings[:fog_credentials]
70
+ choose_fog_provider
71
+ end
72
+ confirm "Using infrastructure provider #{settings.fog_credentials.provider}"
73
+
74
+ unless settings[:region_code]
75
+ choose_provider_region
76
+ end
77
+ if settings[:region_code]
78
+ confirm "Using #{settings.fog_credentials.provider} region #{settings.region_code}"
79
+ else
80
+ confirm "No specific region/data center for #{settings.fog_credentials.provider}"
81
+ end
82
+ end
83
+
84
+ def deploy_stage_2_bosh_configuration
85
+ header "Stage 2: BOSH configuration"
86
+ unless settings[:bosh_name]
87
+ provider, region = settings.bosh_provider, settings.region_code
88
+ if region
89
+ default_name = "microbosh-#{provider}-#{region}".gsub(/\W+/, '-')
90
+ else
91
+ default_name = "microbosh-#{provider}".gsub(/\W+/, '-')
92
+ end
93
+ bosh_name = hl.ask("Useful name for Micro BOSH? ") { |q| q.default = default_name }
94
+ settings[:bosh_name] = bosh_name
95
+ save_settings!
96
+ end
97
+ confirm "Micro BOSH will be named #{settings.bosh_name}"
98
+
99
+ unless settings[:bosh_username]
100
+ prompt_for_bosh_credentials
101
+ end
102
+ confirm "After BOSH is created, your username will be #{settings.bosh_username}"
103
+
104
+ unless settings[:bosh]
105
+ say "Defaulting to 16Gb persistent disk for BOSH"
106
+ password = settings.bosh_password # FIXME dual use of password?
107
+ settings[:bosh] = {}
108
+ settings[:bosh][:password] = password
109
+ settings[:bosh][:persistent_disk] = 16384
110
+ save_settings!
111
+ end
112
+ unless settings[:bosh]["ip_address"]
113
+ say "Acquiring IP address for micro BOSH..."
114
+ ip_address = acquire_ip_address
115
+ settings[:bosh]["ip_address"] = ip_address
116
+ end
117
+ unless settings[:bosh]["ip_address"]
118
+ error "IP address not available/provided currently"
119
+ else
120
+ confirm "Micro BOSH will be assigned IP address #{settings[:bosh]['ip_address']}"
121
+ end
122
+ save_settings!
123
+
124
+ unless settings[:bosh_security_group]
125
+ security_group_name = settings.bosh_name
126
+ create_security_group(security_group_name)
127
+ end
128
+ ports = settings.bosh_security_group.ports.values
129
+ confirm "Micro BOSH protected by security group " +
130
+ "named #{settings.bosh_security_group.name}, with ports #{ports}"
131
+
132
+ unless settings[:bosh_key_pair]
133
+ key_pair_name = settings.bosh_name
134
+ create_key_pair(key_pair_name)
135
+ end
136
+ confirm "Micro BOSH accessible via key pair named #{settings.bosh_key_pair.name}"
137
+
138
+ unless settings[:micro_bosh_stemcell_name]
139
+ settings[:micro_bosh_stemcell_name] = micro_bosh_stemcell_name
140
+ save_settings!
141
+ end
142
+
143
+ confirm "Micro BOSH will be created with stemcell #{settings.micro_bosh_stemcell_name}"
144
+ end
145
+
146
+ def deploy_stage_3_create_allocate_inception_vm
147
+ header "Stage 3: Create/Allocate the Inception VM"
148
+ unless settings["inception"] && settings["inception"]["host"]
149
+ hl.choose do |menu|
150
+ menu.prompt = "Create or specify an Inception VM: "
151
+ if aws? || openstack?
152
+ menu.choice("create new inception VM") do
153
+ aws? ? boot_aws_inception_vm : boot_openstack_inception_vm
154
+ end
155
+ end
156
+ menu.choice("use an existing Ubuntu server") do
157
+ settings["inception"] = {}
158
+ settings["inception"]["host"] = \
159
+ hl.ask("Host address (IP or domain) to inception VM? ")
160
+ settings["inception"]["username"] = \
161
+ hl.ask("Username that you have SSH access to? ") {|q| q.default = "ubuntu"}
162
+ end
163
+ menu.choice("use this server (must be ubuntu & on same network as bosh)") do
164
+ # dummy data for settings.inception
165
+ settings["inception"] = {}
166
+ settings["inception"]["username"] = `whoami`.strip
167
+ end
168
+ end
169
+ end
170
+ # If successfully validate inception VM, then save those settings.
171
+ save_settings!
172
+
173
+ if settings["inception"]["host"]
174
+ @server = Commander::RemoteServer.new(settings.inception.host)
175
+ confirm "Using inception VM #{settings.inception.username}@#{settings.inception.host}"
176
+ else
177
+ @server = Commander::LocalServer.new
178
+ confirm "Using this server as the inception VM"
179
+ end
180
+ unless settings["inception"]["validated"]
181
+ unless server.run(Bosh::Bootstrap::Stages::StageValidateInceptionVm.new(settings).commands)
182
+ error "Failed to complete Stage 3: Create/Allocate the Inception VM"
183
+ end
184
+ settings["inception"]["validated"] = true
185
+ end
186
+ # If successfully validate inception VM, then save those settings.
187
+ save_settings!
188
+ end
189
+
190
+ def deploy_stage_4_prepare_inception_vm
191
+ unless settings["inception"]["prepared"] && !settings["upgrade_deps"]
192
+ header "Stage 4: Preparing the Inception VM"
193
+ unless server.run(Bosh::Bootstrap::Stages::StagePrepareInceptionVm.new(settings).commands)
194
+ error "Failed to complete Stage 4: Preparing the Inception VM"
195
+ end
196
+ # Settings are updated by this stage
197
+ # it generates a salted password from settings.bosh.password
198
+ # and stores it in settings.bosh.salted_password
199
+ settings["inception"]["prepared"] = true
200
+ save_settings!
201
+ else
202
+ header "Stage 4: Preparing the Inception VM", :skipping => "Already prepared inception VM."
203
+ end
204
+ end
205
+
206
+ def deploy_stage_5_deploy_micro_bosh
207
+ header "Stage 5: Deploying micro BOSH"
208
+ unless server.run(Bosh::Bootstrap::Stages::MicroBoshDeploy.new(settings).commands)
209
+ error "Failed to complete Stage 5: Deploying micro BOSH"
210
+ end
211
+
212
+ confirm "Successfully built micro BOSH"
213
+ end
214
+
215
+ def deploy_stage_6_setup_new_bosh
216
+ # TODO change to a polling test of director being available
217
+ say "Pausing to wait for BOSH Director..."
218
+ sleep 5
219
+
220
+ header "Stage 6: Setup bosh"
221
+ unless server.run(Bosh::Bootstrap::Stages::SetupNewBosh.new(settings).commands)
222
+ error "Failed to complete Stage 6: Setup bosh"
223
+ end
224
+
225
+ say "Locally targeting and login to new BOSH..."
226
+ sh "bosh -u #{settings.bosh_username} -p #{settings.bosh_password} target #{settings.bosh.ip_address}"
227
+ sh "bosh login #{settings.bosh_username} #{settings.bosh_password}"
228
+
229
+ save_settings!
230
+
231
+ confirm "You are now targeting and logged in to your BOSH"
232
+ end
233
+
234
+ def delete_stage_1_target_inception_vm
235
+ header "Stage 1: Target inception VM to use to delete micro-bosh"
236
+ if settings["inception"]["host"]
237
+ @server = Commander::RemoteServer.new(settings.inception.host)
238
+ confirm "Using inception VM #{settings.inception.username}@#{settings.inception.host}"
239
+ else
240
+ @server = Commander::LocalServer.new
241
+ confirm "Using this server as the inception VM"
242
+ end
243
+ end
244
+
245
+ def delete_one_stage_2_delete_micro_bosh
246
+ header "Stage 2: Deleting micro BOSH"
247
+ unless server.run(Bosh::Bootstrap::Stages::MicroBoshDelete.new(settings).commands)
248
+ error "Failed to complete Stage 1: Delete micro BOSH"
249
+ end
250
+ save_settings!
251
+ end
252
+
253
+ def delete_all_stage_2_delete_micro_boshes
254
+
255
+ end
256
+
257
+ def delete_all_stage_3_delete_inception_vm
258
+
259
+ end
260
+
261
+ def run_ssh_command_or_open_tunnel(cmd)
262
+ unless settings[:inception]
263
+ say "No inception VM being used", :yellow
264
+ exit 0
265
+ end
266
+ unless host = settings.inception[:host]
267
+ exit "Inception VM has not finished launching; run to complete: #{self.class.banner_base} deploy"
268
+ end
269
+ username = 'vcap'
270
+ exit system Escape.shell_command(['ssh', "#{username}@#{host}", cmd].compact)
271
+
272
+ # TODO how to use the specific private_key_path as configured in settings
273
+ # _, private_key_path = local_ssh_key_paths
274
+ # exit system Escape.shell_command(['ssh', "-i #{private_key_path}", "#{username}@#{host}", cmd].compact)
275
+ #
276
+ # Currently this shows:
277
+ # Warning: Identity file /Users/drnic/.ssh/id_rsa not accessible: No such file or directory.
278
+ end
279
+
280
+ # Display header for a new section of the bootstrapper
281
+ def header(title, options={})
282
+ say "" # golden whitespace
283
+ if skipping = options[:skipping]
284
+ say "Skipping #{title}", [:yellow, :bold]
285
+ say skipping
286
+ else
287
+ say title, [:green, :bold]
288
+ end
289
+ say "" # more golden whitespace
290
+ end
291
+
292
+ def error(message)
293
+ say message, :red
294
+ exit 1
295
+ end
296
+
297
+ def confirm(message)
298
+ say "Confirming: #{message}", green
299
+ say "" # bonus golden whitespace
300
+ end
301
+
302
+ def load_deploy_options
303
+ settings["fog_path"] = File.expand_path(options[:fog] || "~/.fog")
304
+
305
+ settings["bosh_git_source"] = options[:"edge-deployer"] # use bosh git repo instead of rubygems
306
+
307
+ # determine which micro-bosh stemcell to download/create
308
+ if options[:"latest-stemcell"]
309
+ settings["micro_bosh_stemcell_type"] = "latest"
310
+ settings["micro_bosh_stemcell_name"] = nil # force name to be refetched
311
+ elsif options[:"edge-stemcell"]
312
+ settings["micro_bosh_stemcell_type"] = "custom"
313
+ settings["micro_bosh_stemcell_name"] = "custom"
314
+ else
315
+ # may have already been set from previous deploy run
316
+ settings["micro_bosh_stemcell_type"] ||= "stable"
317
+ end
318
+
319
+ # once a stemcell is downloaded or created; these fields above should
320
+ # be uploaded with values such as:
321
+ # -> settings["micro_bosh_stemcell_name"] = "micro-bosh-stemcell-aws-0.8.1.tgz"
322
+
323
+ if options["private-key"]
324
+ private_key_path = File.expand_path(options["private-key"])
325
+ unless File.exists?(private_key_path)
326
+ error "Cannot find a file at #{private_key_path}"
327
+ end
328
+ public_key_path = "#{private_key_path}.pub"
329
+ unless File.exists?(public_key_path)
330
+ error "Cannot find a file at #{public_key_path}"
331
+ end
332
+
333
+ settings["local"] ||= {}
334
+ settings["local"]["private_key_path"] = private_key_path
335
+ settings["local"]["public_key_path"] = public_key_path
336
+ end
337
+
338
+ if options["upgrade-deps"]
339
+ settings["upgrade_deps"] = options["upgrade-deps"]
340
+ else
341
+ settings.delete("upgrade_deps")
342
+ end
343
+ save_settings!
344
+ end
345
+
346
+ # Displays a prompt for known IaaS that are configured
347
+ # within .fog config file.
348
+ #
349
+ # For example:
350
+ #
351
+ # 1. AWS (default)
352
+ # 2. AWS (bosh)
353
+ # 3. Alternate credentials
354
+ # Choose infrastructure: 1
355
+ #
356
+ # If .fog config only contains one provider, do not prompt.
357
+ #
358
+ # fog config file looks like:
359
+ # :default:
360
+ # :aws_access_key_id: PERSONAL_ACCESS_KEY
361
+ # :aws_secret_access_key: PERSONAL_SECRET
362
+ # :bosh:
363
+ # :aws_access_key_id: SPECIAL_IAM_ACCESS_KEY
364
+ # :aws_secret_access_key: SPECIAL_IAM_SECRET_KEY
365
+ #
366
+ # Convert this into:
367
+ # { "AWS (default)" => {:aws_access_key_id => ...}, "AWS (bosh)" => {...} }
368
+ #
369
+ # Then display options to user to choose.
370
+ #
371
+ # Currently detects following fog providers:
372
+ # * AWS
373
+ # * OpenStack
374
+ #
375
+ # If "Alternate credentials" is selected, then user is prompted for fog
376
+ # credentials:
377
+ # * provider?
378
+ # * access keys?
379
+ # * API URI or region?
380
+ #
381
+ # At the end, settings.fog_credentials contains the credentials for target IaaS
382
+ # and :provider key for the IaaS name.
383
+ #
384
+ # {:provider=>"AWS",
385
+ # :aws_access_key_id=>"PERSONAL_ACCESS_KEY",
386
+ # :aws_secret_access_key=>"PERSONAL_SECRET"}
387
+ #
388
+ # settings.fog_credentials.provider is the provider name
389
+ # settings.bosh_provider is the BOSH name for the provider (aws,vsphere,openstack)
390
+ # so as to local stemcells (see +micro_bosh_stemcell_name+)
391
+ def choose_fog_provider
392
+ @fog_providers = {}
393
+ # Prepare menu options:
394
+ # each provider/profile name gets a menu choice option
395
+ fog_config.inject({}) do |iaas_options, fog_profile|
396
+ profile_name, profile = fog_profile
397
+ if profile[:aws_access_key_id]
398
+ # TODO does fog have inbuilt detection algorithm?
399
+ @fog_providers["AWS (#{profile_name})"] = {
400
+ "provider" => "AWS",
401
+ "aws_access_key_id" => profile[:aws_access_key_id],
402
+ "aws_secret_access_key" => profile[:aws_secret_access_key]
403
+ }
404
+ end
405
+ if profile[:openstack_username]
406
+ # TODO does fog have inbuilt detection algorithm?
407
+ @fog_providers["OpenStack (#{profile_name})"] = {
408
+ "provider" => "OpenStack",
409
+ "openstack_username" => profile[:openstack_username],
410
+ "openstack_api_key" => profile[:openstack_api_key],
411
+ "openstack_tenant" => profile[:openstack_tenant],
412
+ "openstack_auth_url" => profile[:openstack_auth_url]
413
+ }
414
+ end
415
+ end
416
+ # Display menu
417
+ # Include "Alternate credentials" as the last option
418
+ if @fog_providers.keys.size > 0
419
+ hl.choose do |menu|
420
+ menu.prompt = "Choose infrastructure: "
421
+ @fog_providers.each do |label, credentials|
422
+ menu.choice(label) { @fog_credentials = credentials }
423
+ end
424
+ menu.choice("Alternate credentials") { prompt_for_alternate_fog_credentials }
425
+ end
426
+ else
427
+ prompt_for_alternate_fog_credentials
428
+ end
429
+ settings[:fog_credentials] = {}
430
+ @fog_credentials.each do |key, value|
431
+ settings[:fog_credentials][key] = value
432
+ end
433
+ setup_bosh_cloud_properties
434
+ settings[:bosh_resources_cloud_properties] = bosh_resources_cloud_properties
435
+ settings[:bosh_provider] = settings.bosh_cloud_properties.keys.first # aws, vsphere...
436
+ save_settings!
437
+ end
438
+
439
+ # If no .fog file is found, or if user chooses "Alternate credentials",
440
+ # then this method prompts the user:
441
+ # * provider?
442
+ # * access keys?
443
+ # * API URI or region?
444
+ #
445
+ # Populates +@fog_credentials+ with a Hash that includes :provider key
446
+ # For example:
447
+ # {
448
+ # :provider => "AWS",
449
+ # :aws_access_key_id => ACCESS_KEY,
450
+ # :aws_secret_access_key => SECRET_KEY
451
+ # }
452
+ def prompt_for_alternate_fog_credentials
453
+ say "" # glorious whitespace
454
+ creds = {}
455
+ hl.choose do |menu|
456
+ menu.prompt = "Choose infrastructure: "
457
+ menu.choice("AWS") do
458
+ creds[:provider] = "AWS"
459
+ creds[:aws_access_key_id] = hl.ask("Access key: ")
460
+ creds[:aws_secret_access_key] = hl.ask("Secret key: ")
461
+ end
462
+ menu.choice("OpenStack") do
463
+ creds[:provider] = "OpenStack"
464
+ creds[:openstack_username] = hl.ask("Username: ")
465
+ creds[:openstack_api_key] = hl.ask("API key: ")
466
+ creds[:openstack_tenant] = hl.ask("Tenant: ")
467
+ creds[:openstack_auth_url] = hl.ask("Authorization URL: ")
468
+ end
469
+ end
470
+ @fog_credentials = creds
471
+ end
472
+
473
+ def setup_bosh_cloud_properties
474
+ if aws?
475
+ settings[:bosh_cloud_properties] = {}
476
+ settings[:bosh_cloud_properties][:aws] = {}
477
+ props = settings[:bosh_cloud_properties][:aws]
478
+ props[:access_key_id] = settings.fog_credentials.aws_access_key_id
479
+ props[:secret_access_key] = settings.fog_credentials.aws_secret_access_key
480
+ # props[:ec2_endpoint] = "ec2.REGION.amazonaws.com" - via +choose_aws_region+
481
+ # props[:region] = REGION - via +choose_aws_region+
482
+ # props[:default_key_name] = "microbosh" - via +create_aws_key_pair+
483
+ # props[:ec2_private_key] = "/home/vcap/.ssh/microbosh.pem" - via +create_aws_key_pair+
484
+ # props[:default_security_groups] = ["microbosh"], - via +create_aws_security_group+
485
+ elsif openstack?
486
+ settings[:bosh_cloud_properties] = {}
487
+ settings[:bosh_cloud_properties][:openstack] = {}
488
+ props = settings[:bosh_cloud_properties][:openstack]
489
+ props[:username] = settings.fog_credentials.openstack_username
490
+ props[:api_key] = settings.fog_credentials.openstack_api_key
491
+ props[:tenant] = settings.fog_credentials.openstack_tenant
492
+ props[:auth_url] = settings.fog_credentials.openstack_auth_url
493
+ # props[:default_key_name] = "microbosh" - via +create_openstack_key_pair+
494
+ # props[:private_key] = "/home/vcap/.ssh/microbosh.pem" - via +create_openstack_key_pair+
495
+ # props[:default_security_groups] = ["microbosh"], - via +create_openstack_security_group+
496
+ else
497
+ raise "implement #bosh_cloud_properties for #{settings.fog_credentials.provider}"
498
+ end
499
+ end
500
+
501
+ def bosh_resources_cloud_properties
502
+ if aws?
503
+ {"instance_type" => "m1.medium"}
504
+ elsif openstack?
505
+ # TODO: Ask for instance type
506
+ {"instance_type" => "m1.medium"}
507
+ else
508
+ raise "implement #bosh_resources_cloud_properties for #{settings.fog_credentials.provider}"
509
+ end
510
+ end
511
+
512
+ # Ask user to provide region information (URI)
513
+ # or choose from a known list of regions (e.g. AWS)
514
+ # Return true if region selected (@region_code is set)
515
+ # Else return false
516
+ def choose_provider_region
517
+ if aws?
518
+ choose_aws_region
519
+ else
520
+ settings["region_code"] = nil
521
+ false
522
+ end
523
+ end
524
+
525
+ def choose_aws_region
526
+ aws_regions = provider.region_labels
527
+ default_aws_region = provider.default_region_label
528
+
529
+ hl.choose do |menu|
530
+ menu.prompt = "Choose AWS region (default: #{default_aws_region}): "
531
+ aws_regions.each do |region|
532
+ menu.choice(region) do
533
+ settings["region_code"] = region
534
+ settings["fog_credentials"]["region"] = region
535
+ settings["bosh_cloud_properties"]["aws"]["region"] = region
536
+ settings["bosh_cloud_properties"]["aws"]["ec2_endpoint"] = "ec2.#{region}.amazonaws.com"
537
+ save_settings!
538
+ end
539
+ menu.default = default_aws_region
540
+ end
541
+ end
542
+ reset_fog_compute
543
+ true
544
+ end
545
+
546
+ # Creates a security group.
547
+ # Also sets up the bosh_cloud_properties for the remote server
548
+ #
549
+ # Adds settings:
550
+ # * bosh_security_group.name
551
+ # * bosh_security_group.ports
552
+ # * bosh_cloud_properties.<bosh_provider>.default_security_groups
553
+ def create_security_group(security_group_name)
554
+ ports = {
555
+ ssh_access: 22,
556
+ nats_server: 4222,
557
+ message_bus: 6868,
558
+ blobstore: 25250,
559
+ bosh_director: 25555
560
+ }
561
+ if aws?
562
+ ports[:aws_registry] = 25777
563
+ elsif openstack?
564
+ ports[:openstack_registry] = 25889
565
+ end
566
+
567
+ provider.create_security_group(security_group_name, "microbosh", ports)
568
+
569
+ settings["bosh_cloud_properties"][provider_name]["default_security_groups"] = [security_group_name]
570
+ settings["bosh_security_group"] = {}
571
+ settings["bosh_security_group"]["name"] = security_group_name
572
+ settings["bosh_security_group"]["ports"] = {}
573
+ ports.each { |name, port| settings["bosh_security_group"]["ports"][name.to_s] = port }
574
+ save_settings!
575
+ end
576
+
577
+ # Creates a key pair.
578
+ def create_key_pair(key_pair_name)
579
+ if aws?
580
+ create_aws_key_pair(key_pair_name)
581
+ elsif openstack?
582
+ create_openstack_key_pair(key_pair_name)
583
+ else
584
+ raise "implement #create_key_pair for #{settings.fog_credentials.provider}"
585
+ end
586
+ end
587
+
588
+ # Creates an AWS key pair, and stores the private key
589
+ # in settings manifest.
590
+ # Also sets up the bosh_cloud_properties for the remote server
591
+ # to have the .pem key installed.
592
+ #
593
+ # Adds settings:
594
+ # * bosh_key_pair.name
595
+ # * bosh_key_pair.private_key
596
+ # * bosh_key_pair.fingerprint
597
+ # * bosh_cloud_properties.aws.default_key_name
598
+ # * bosh_cloud_properties.aws.ec2_private_key
599
+ def create_aws_key_pair(key_pair_name)
600
+ unless fog_compute.key_pairs.get(key_pair_name)
601
+ say "creating key pair #{key_pair_name}..."
602
+ kp = fog_compute.key_pairs.create(:name => key_pair_name)
603
+ settings[:bosh_key_pair] = {}
604
+ settings[:bosh_key_pair][:name] = key_pair_name
605
+ settings[:bosh_key_pair][:private_key] = kp.private_key
606
+ settings[:bosh_key_pair][:fingerprint] = kp.fingerprint
607
+ settings["bosh_cloud_properties"]["aws"]["default_key_name"] = key_pair_name
608
+ settings["bosh_cloud_properties"]["aws"]["ec2_private_key"] = "/home/vcap/.ssh/#{key_pair_name}.pem"
609
+ save_settings!
610
+ else
611
+ error "AWS key pair '#{key_pair_name}' already exists. Rename BOSH or delete old key pair manually and re-run CLI."
612
+ end
613
+ end
614
+
615
+ # Creates an OpenStack key pair, and stores the private key
616
+ # in settings manifest.
617
+ # Also sets up the bosh_cloud_properties for the remote server
618
+ # to have the .pem key installed.
619
+ #
620
+ # Adds settings:
621
+ # * bosh_key_pair.name
622
+ # * bosh_key_pair.private_key
623
+ # * bosh_key_pair.fingerprint
624
+ # * bosh_cloud_properties.openstack.default_key_name
625
+ # * bosh_cloud_properties.openstack.ec2_private_key
626
+ def create_openstack_key_pair(key_pair_name)
627
+ unless fog_compute.key_pairs.get(key_pair_name)
628
+ say "creating key pair #{key_pair_name}..."
629
+ kp = fog_compute.key_pairs.create(:name => key_pair_name)
630
+ settings[:bosh_key_pair] = {}
631
+ settings[:bosh_key_pair][:name] = key_pair_name
632
+ settings[:bosh_key_pair][:private_key] = kp.private_key
633
+ settings[:bosh_key_pair][:fingerprint] = kp.fingerprint
634
+ settings["bosh_cloud_properties"]["openstack"]["default_key_name"] = key_pair_name
635
+ settings["bosh_cloud_properties"]["openstack"]["private_key"] = "/home/vcap/.ssh/#{key_pair_name}.pem"
636
+ save_settings!
637
+ else
638
+ error "OpenStack key pair '#{key_pair_name}' already exists. Rename BOSH or delete old key pair manually and re-run CLI."
639
+ end
640
+ end
641
+
642
+ # Provisions an AWS m1.small VM as the inception VM
643
+ # Updates settings.inception.host/username
644
+ #
645
+ # NOTE: if any stage fails, when the CLI is re-run
646
+ # and "create new server" is selected again, the process should
647
+ # complete
648
+ #
649
+ # Assumes that local CLI user has public/private keys at ~/.ssh/id_rsa.pub
650
+ def boot_aws_inception_vm
651
+ say "" # glowing whitespace
652
+
653
+ public_key_path, private_key_path = local_ssh_key_paths
654
+ unless settings["inception"] && settings["inception"]["server_id"]
655
+ username = "ubuntu"
656
+ size = "m1.small"
657
+ say "Provisioning #{size} for inception VM..."
658
+ server = fog_compute.servers.bootstrap({
659
+ :public_key_path => public_key_path,
660
+ :private_key_path => private_key_path,
661
+ :flavor_id => size,
662
+ :bits => 64,
663
+ :username => "ubuntu"
664
+ })
665
+ unless server
666
+ error "Something mysteriously cloudy happened and fog could not provision a VM. Please check your limits."
667
+ end
668
+
669
+ settings["inception"] = {}
670
+ settings["inception"]["server_id"] = server.id
671
+ settings["inception"]["username"] = username
672
+ save_settings!
673
+ end
674
+
675
+ server ||= fog_compute.servers.get(settings["inception"]["server_id"])
676
+
677
+ unless settings["inception"]["ip_address"]
678
+ say "Provisioning IP address for inception VM..."
679
+ ip_address = acquire_ip_address
680
+ associate_ip_address_with_server(ip_address, server)
681
+ host = server.dns_name
682
+
683
+ settings["inception"]["ip_address"] = ip_address
684
+ save_settings!
685
+ end
686
+
687
+ unless settings["inception"]["disk_size"]
688
+ disk_size = DEFAULT_INCEPTION_VOLUME_SIZE # Gb
689
+ device = "/dev/sdi"
690
+ provision_and_mount_volume(server, disk_size, device)
691
+
692
+ settings["inception"]["disk_size"] = disk_size
693
+ settings["inception"]["disk_device"] = device
694
+ save_settings!
695
+ end
696
+
697
+ # settings["inception"]["host"] is used externally to determine
698
+ # if an inception VM has been assigned already; so we leave it
699
+ # until last in this method to set this setting.
700
+ # This way we can always rerun the CLI and rerun this method
701
+ # and idempotently get an inception VM
702
+ unless settings["inception"]["host"]
703
+ settings["inception"]["host"] = server.dns_name
704
+ save_settings!
705
+ end
706
+
707
+ confirm "Inception VM has been created"
708
+ display_inception_ssh_access
709
+ end
710
+
711
+ # Provisions an OpenStack m1.small VM as the inception VM
712
+ # Updates settings.inception.host/username
713
+ #
714
+ # NOTE: if any stage fails, when the CLI is re-run
715
+ # and "create new server" is selected again, the process should
716
+ # complete
717
+ #
718
+ # Assumes that local CLI user has public/private keys at ~/.ssh/id_rsa.pub
719
+ def boot_openstack_inception_vm
720
+ say "" # glowing whitespace
721
+
722
+ public_key_path, private_key_path = local_ssh_key_paths
723
+
724
+ # make sure we've a fog key pair
725
+ key_pair_name = Fog.respond_to?(:credential) && Fog.credential || :default
726
+ unless key_pair = fog_compute.key_pairs.get("fog_#{key_pair_name}")
727
+ say "creating key pair fog_#{key_pair_name}..."
728
+ public_key = File.open(public_key_path, 'rb') { |f| f.read }
729
+ key_pair = fog_compute.key_pairs.create(
730
+ :name => "fog_#{key_pair_name}",
731
+ :public_key => public_key
732
+ )
733
+ end
734
+ confirm "Using key pair #{key_pair.name} for Inception VM"
735
+
736
+ # make sure port 22 is open in the default security group
737
+ security_group = fog_compute.security_groups.find { |sg| sg.name == 'default' }
738
+ authorized = security_group.rules.detect do |ip_permission|
739
+ ip_permission['ip_range'].first && ip_permission['ip_range']['cidr'] == '0.0.0.0/0' &&
740
+ ip_permission['from_port'] == 22 &&
741
+ ip_permission['ip_protocol'] == 'tcp' &&
742
+ ip_permission['to_port'] == 22
743
+ end
744
+ unless authorized
745
+ security_group.create_security_group_rule(22, 22)
746
+ end
747
+ confirm "Inception VM port 22 open"
748
+
749
+ unless settings["inception"] && settings["inception"]["server_id"]
750
+ username = "ubuntu"
751
+ say "Provisioning server for inception VM..."
752
+ settings["inception"] = {}
753
+
754
+ # Select OpenStack flavor
755
+ unless settings["inception"]["flavor_id"]
756
+ say ""
757
+ hl.choose do |menu|
758
+ menu.prompt = "Choose OpenStack flavor: "
759
+ fog_compute.flavors.each do |flavor|
760
+ menu.choice(flavor.name) do
761
+ settings["inception"]["flavor_id"] = flavor.id
762
+ save_settings!
763
+ end
764
+ end
765
+ end
766
+ end
767
+
768
+ # Select OpenStack image
769
+ unless settings["inception"]["image_id"]
770
+ say ""
771
+ hl.choose do |menu|
772
+ menu.prompt = "Choose OpenStack image (Ubuntu): "
773
+ fog_compute.images.each do |image|
774
+ menu.choice(image.name) do
775
+ settings["inception"]["image_id"] = image.id
776
+ save_settings!
777
+ end
778
+ end
779
+ end
780
+ end
781
+
782
+ # Boot OpenStack server
783
+ server = fog_compute.servers.create(
784
+ :name => "Inception VM",
785
+ :key_name => key_pair.name,
786
+ :public_key_path => public_key_path,
787
+ :private_key_path => private_key_path,
788
+ :flavor_ref => settings["inception"]["flavor_id"],
789
+ :image_ref => settings["inception"]["image_id"],
790
+ :username => username
791
+ )
792
+ unless server
793
+ error "Something mysteriously cloudy happened and fog could not provision a VM. Please check your limits."
794
+ end
795
+ server.wait_for { ready? }
796
+
797
+ settings["inception"]["server_id"] = server.id
798
+ settings["inception"]["username"] = username
799
+ save_settings!
800
+ end
801
+
802
+ server ||= fog_compute.servers.get(settings["inception"]["server_id"])
803
+
804
+ unless settings["inception"]["ip_address"]
805
+ say "Provisioning IP address for inception VM..."
806
+ ip_address = acquire_ip_address
807
+ associate_ip_address_with_server(ip_address, server)
808
+
809
+ settings["inception"]["ip_address"] = ip_address
810
+ save_settings!
811
+ end
812
+
813
+ unless settings["inception"]["disk_size"]
814
+ disk_size = DEFAULT_INCEPTION_VOLUME_SIZE # Gb
815
+ device = "/dev/vdc"
816
+ provision_and_mount_volume(server, disk_size, device)
817
+
818
+ settings["inception"]["disk_size"] = disk_size
819
+ settings["inception"]["disk_device"] = device
820
+ save_settings!
821
+
822
+ # TODO use provision_and_mount_volume
823
+
824
+ disk_size = 16 # Gb
825
+ va = fog_compute.get_server_volumes(server.id).body['volumeAttachments']
826
+ unless vol = va.find { |v| v["device"] == "/dev/vdc" }
827
+ say "Provisioning #{disk_size}Gb persistent disk for inception VM..."
828
+ volume = fog_compute.volumes.create(:name => "Inception Disk",
829
+ :description => "",
830
+ :size => disk_size,
831
+ :availability_zone => server.availability_zone)
832
+ volume.wait_for { volume.status == 'available' }
833
+ volume.attach(server.id, "/dev/vdc")
834
+ volume.wait_for { volume.status == 'in-use' }
835
+ end
836
+
837
+ # Format and mount the volume
838
+ # TODO: Hack
839
+ unless server.public_ip_address
840
+ server.addresses["public"] = [settings["inception"]["ip_address"]]
841
+ end
842
+ unless server.public_key_path
843
+ server.public_key_path = public_key_path
844
+ end
845
+ unless server.private_key_path
846
+ server.private_key_path = private_key_path
847
+ end
848
+ server.username = settings["inception"]["username"]
849
+ server.sshable?
850
+
851
+ say "Mounting persistent disk as volume on inception VM..."
852
+ # TODO if any of these ssh calls fail; retry
853
+ server.ssh(['sudo mkfs.ext4 /dev/vdc -F'])
854
+ server.ssh(['sudo mkdir -p /var/vcap/store'])
855
+ server.ssh(['sudo mount /dev/vdc /var/vcap/store'])
856
+
857
+ settings["inception"]["disk_size"] = disk_size
858
+ save_settings!
859
+ end
860
+
861
+ # settings["inception"]["host"] is used externally to determine
862
+ # if an inception VM has been assigned already; so we leave it
863
+ # until last in this method to set this setting.
864
+ # This way we can always rerun the CLI and rerun this method
865
+ # and idempotently get an inception VM
866
+ unless settings["inception"]["host"]
867
+ settings["inception"]["host"] = settings["inception"]["ip_address"]
868
+ save_settings!
869
+ end
870
+
871
+ confirm "Inception VM has been created"
872
+ display_inception_ssh_access
873
+ end
874
+
875
+ # Provision or provide an IP address to use
876
+ # For AWS, it will dynamically provision an elastic IP
877
+ # For OpenStack, it will dynamically provision a floating IP
878
+ def acquire_ip_address
879
+ unless public_ip = provider.provision_public_ip_address
880
+ say "Unable to acquire a public IP. Please check your account for capacity or service issues.".red
881
+ exit 1
882
+ end
883
+ public_ip
884
+ end
885
+
886
+ def associate_ip_address_with_server(ip_address, server)
887
+ address = fog_compute.addresses.get(ip_address)
888
+ address.server = server
889
+ server.reload
890
+ end
891
+
892
+ # Provision a volume for a specific device (unless already provisioned)
893
+ # Request that the +server+ mount the volume at the +device+ location.
894
+ #
895
+ # Requires that we can SSH into +server+.
896
+ def provision_and_mount_volume(server, disk_size, device)
897
+ unless volume = server.volumes.all.find {|v| v.device == device}
898
+ say "Provisioning #{disk_size}Gb persistent disk for inception VM..."
899
+ volume = fog_compute.volumes.create(
900
+ size: disk_size,
901
+ name: "Inception Disk",
902
+ description: '',
903
+ device: "/dev/sdi",
904
+ availability_zone: server.availability_zone)
905
+ # TODO: the following works in fog 1.9.0+ (but which has a bug in bootstrap)
906
+ # https://github.com/fog/fog/issues/1516
907
+ #
908
+ # volume.wait_for { volume.status == 'available' }
909
+ # volume.attach(server.id, "/dev/vdc")
910
+ # volume.wait_for { volume.status == 'in-use' }
911
+ #
912
+ # Instead, using:
913
+ volume.server = server
914
+ end
915
+
916
+ # Format and mount the volume
917
+ say "Mounting persistent disk as volume on inception VM..."
918
+ disk_mounted = false
919
+ until disk_mounted
920
+ begin
921
+ # TODO catch Errno::ETIMEDOUT and re-run ssh commands
922
+ server.ssh(["sudo mkfs.ext4 #{device} -F"])
923
+ server.ssh(["sudo mkdir -p /var/vcap/store"])
924
+ server.ssh(["sudo mount #{device} /var/vcap/store"])
925
+ disk_mounted = true
926
+ rescue Errno::ETIMEDOUT => e
927
+ say "Timeout error/warning mounting volume, retrying...", yellow
928
+ end
929
+ end
930
+ end
931
+
932
+ def display_inception_ssh_access
933
+ _, private_key_path = local_ssh_key_paths
934
+ say "SSH access: ssh -i #{private_key_path} #{settings["inception"]["username"]}@#{settings["inception"]["host"]}"
935
+ end
936
+
937
+ # Discover/create local passphrase-less SSH keys to allow
938
+ # communication with Inception VM
939
+ #
940
+ # Returns [public_key_path, private_key_path]
941
+ def local_ssh_key_paths
942
+ unless settings["local"] && settings["local"]["private_key_path"]
943
+ settings["local"] = {}
944
+ public_key_path = File.expand_path("~/.ssh/id_rsa.pub")
945
+ private_key_path = File.expand_path("~/.ssh/id_rsa")
946
+ raise "Please create public keys at ~/.ssh/id_rsa.pub or use --private-key flag" unless File.exists?(public_key_path)
947
+
948
+ settings["local"]["public_key_path"] = public_key_path
949
+ settings["local"]["private_key_path"] = private_key_path
950
+ save_settings!
951
+ end
952
+ [settings.local.public_key_path, settings.local.private_key_path]
953
+ end
954
+
955
+ def aws?
956
+ settings.fog_credentials.provider == "AWS"
957
+ end
958
+
959
+ def openstack?
960
+ settings.fog_credentials.provider == "OpenStack"
961
+ end
962
+
963
+ def prompt_for_bosh_credentials
964
+ prompt = hl
965
+ say "Please enter a user/password for the BOSH that will be created."
966
+ settings[:bosh_username] = prompt.ask("BOSH username: ") { |q| q.default = `whoami`.strip }
967
+ settings[:bosh_password] = prompt.ask("BOSH password: ") { |q| q.echo = "x" }
968
+ save_settings!
969
+ end
970
+
971
+ # Returns the latest micro-bosh stemcell
972
+ # for the target provider (aws, vsphere, openstack)
973
+ def micro_bosh_stemcell_name
974
+ @micro_bosh_stemcell_name ||= begin
975
+ provider = settings.bosh_provider.downcase # aws, vsphere, openstack
976
+ stemcell_filter_tags = ['micro', provider]
977
+ if openstack?
978
+ # FIXME remove this if when openstack has its first stable
979
+ else
980
+ if settings["micro_bosh_stemcell_type"] == "stable"
981
+ stemcell_filter_tags << "stable" # latest stable micro-bosh stemcell by default
982
+ end
983
+ end
984
+ tags = stemcell_filter_tags.join(",")
985
+ bosh_stemcells_cmd = "bosh public stemcells --tags #{tags}"
986
+ say "Locating micro-bosh stemcell, running '#{bosh_stemcells_cmd}'..."
987
+ #
988
+ # The +bosh_stemcells_cmd+ has an output that looks like:
989
+ # +-----------------------------------+--------------------+
990
+ # | Name | Tags |
991
+ # +-----------------------------------+--------------------+
992
+ # | micro-bosh-stemcell-aws-0.6.4.tgz | aws, micro, stable |
993
+ # | micro-bosh-stemcell-aws-0.7.0.tgz | aws, micro, test |
994
+ # +-----------------------------------+--------------------+
995
+ #
996
+ # So to get the latest version for the filter tags,
997
+ # get the Name field, reverse sort, and return the first item
998
+ `#{bosh_stemcells_cmd} | grep micro | awk '{ print $2 }' | sort -r | head -n 1`.strip
999
+ end
1000
+ end
1001
+
1002
+ def provider_name
1003
+ settings.bosh_provider
1004
+ end
1005
+
1006
+ # a helper object for the target BOSH provider
1007
+ def provider
1008
+ @provider ||= Bosh::Providers.for_bosh_provider_name(settings.bosh_provider, fog_compute)
1009
+ end
1010
+
1011
+ def cyan; "\033[36m" end
1012
+ def clear; "\033[0m" end
1013
+ def bold; "\033[1m" end
1014
+ def red; "\033[31m" end
1015
+ def green; "\033[32m" end
1016
+ def yellow; "\033[33m" end
1017
+
1018
+ # Helper to access HighLine for ask & menu prompts
1019
+ def hl
1020
+ @hl ||= HighLine.new
1021
+ end
1022
+ end
1023
+ end
1024
+ end