logical-construct 0.0.5 → 0.1.0
Sign up to get free protection for your applications and to get access to all the features.
- data/bin/flight-deck +3 -0
- data/doc/DESIGN +48 -0
- data/doc/EC2-baking-notes +70 -0
- data/doc/ExampleStartupRakefile +152 -0
- data/doc/ExampleTargetRakefile +4 -0
- data/doc/TODO +148 -0
- data/doc/Vb-EC2-translation-notes +96 -0
- data/doc/hating-chef +32 -0
- data/lib/logical-construct/archive-tasks.rb +307 -0
- data/lib/logical-construct/ground-control.rb +4 -1
- data/lib/logical-construct/ground-control/build-plan.rb +95 -0
- data/lib/logical-construct/ground-control/core.rb +1 -1
- data/lib/logical-construct/ground-control/generate-manifest.rb +67 -0
- data/lib/logical-construct/ground-control/provision.rb +73 -168
- data/lib/logical-construct/ground-control/run-on-target.rb +1 -1
- data/lib/logical-construct/ground-control/setup.rb +1 -4
- data/lib/logical-construct/ground-control/setup/copy-files.rb +2 -2
- data/lib/logical-construct/ground-control/tools.rb +66 -0
- data/lib/logical-construct/node-client.rb +112 -0
- data/lib/logical-construct/plan.rb +2 -0
- data/lib/logical-construct/plan/core.rb +45 -0
- data/lib/logical-construct/plan/standalone-bundle.rb +80 -0
- data/lib/logical-construct/port-open-check.rb +41 -0
- data/lib/logical-construct/protocol.rb +2 -0
- data/lib/logical-construct/protocol/plan-validation.rb +46 -0
- data/lib/logical-construct/protocol/ssh-tunnel.rb +127 -0
- data/lib/logical-construct/protocol/vocabulary.rb +8 -0
- data/lib/logical-construct/target/Implement.rake +8 -0
- data/lib/logical-construct/target/command-line.rb +90 -0
- data/lib/logical-construct/target/flight-deck.rb +341 -0
- data/lib/logical-construct/target/implementation.rb +33 -0
- data/lib/logical-construct/target/plan-records.rb +317 -0
- data/lib/logical-construct/target/resolution-server.rb +153 -0
- data/lib/logical-construct/target/{unpack-cookbook.rb → unpack-plan.rb} +8 -4
- data/lib/logical-construct/template-file.rb +41 -0
- data/lib/templates/Rakefile.erb +8 -0
- data/spec/ground-control/smoke-test.rb +8 -7
- data/spec/node_resolution.rb +62 -0
- data/spec/target/plan-records.rb +142 -0
- data/spec/target/provisioning.rb +21 -0
- data/spec_help/file-sandbox.rb +12 -6
- data/spec_help/fixtures/Manifest +1 -0
- data/spec_help/fixtures/source/one.tbz +1 -0
- data/spec_help/fixtures/source/three.tbz +1 -0
- data/spec_help/fixtures/source/two.tbz +1 -0
- data/spec_help/spec_helper.rb +5 -7
- metadata +165 -72
- data/lib/logical-construct/ground-control/setup/build-files.rb +0 -93
- data/lib/logical-construct/ground-control/setup/create-construct-directory.rb +0 -22
- data/lib/logical-construct/ground-control/setup/install-init.rb +0 -32
- data/lib/logical-construct/resolving-task.rb +0 -141
- data/lib/logical-construct/satisfiable-task.rb +0 -87
- data/lib/logical-construct/target.rb +0 -4
- data/lib/logical-construct/target/chef-solo.rb +0 -37
- data/lib/logical-construct/target/platforms.rb +0 -51
- data/lib/logical-construct/target/platforms/aws.rb +0 -8
- data/lib/logical-construct/target/platforms/default/chef-config.rb +0 -134
- data/lib/logical-construct/target/platforms/default/resolve-configuration.rb +0 -44
- data/lib/logical-construct/target/platforms/default/volume.rb +0 -11
- data/lib/logical-construct/target/platforms/virtualbox.rb +0 -8
- data/lib/logical-construct/target/platforms/virtualbox/volume.rb +0 -15
- data/lib/logical-construct/target/provision.rb +0 -36
- data/lib/logical-construct/target/sinatra-resolver.rb +0 -99
- data/lib/logical-construct/testing/resolve-configuration.rb +0 -32
- data/lib/logical-construct/testing/resolving-task.rb +0 -15
- data/lib/templates/chef.rb.erb +0 -9
- data/lib/templates/construct.init.d.erb +0 -18
- data/lib/templates/resolver/finished.html.erb +0 -1
- data/lib/templates/resolver/index.html.erb +0 -17
- data/lib/templates/resolver/task-file-form.html.erb +0 -6
- data/lib/templates/resolver/task-form.html.erb +0 -6
- data/spec/resolution.rb +0 -147
- data/spec/target/chef-config.rb +0 -67
- data/spec/target/chef-solo.rb +0 -55
- data/spec/target/platforms.rb +0 -36
- data/spec/target/smoke-test.rb +0 -45
- data/spec_help/ungemmer.rb +0 -36
data/bin/flight-deck
ADDED
data/doc/DESIGN
ADDED
@@ -0,0 +1,48 @@
|
|
1
|
+
After consideration:
|
2
|
+
|
3
|
+
This is not a cloud server manager. Those exist already.
|
4
|
+
|
5
|
+
This is a cloud deployment manager. Write one description of how your cloud is built, deploy it whereever.
|
6
|
+
|
7
|
+
Puppet/chef handle like 90% of that, but before either can run, network has to
|
8
|
+
be configured, and persistent/extra volumes need to be mounted.
|
9
|
+
|
10
|
+
"Blessing" deploys is another thing - snapshot/repackage a currently running
|
11
|
+
instance so that we can shortcut deploy next time.
|
12
|
+
|
13
|
+
Things Ground control should be able to do:
|
14
|
+
|
15
|
+
* Survey Target hosts and make sure they have the configs they need.
|
16
|
+
|
17
|
+
* Given a target server with Ruby and SSH, set it up as a particular role.
|
18
|
+
* (I.e: SSH and deliver files, install gems, locked to GC version)
|
19
|
+
|
20
|
+
Three basic tools:
|
21
|
+
|
22
|
+
Construct: delivers and executes provisioning plans (which themselves may be "use Chef to...")
|
23
|
+
(coupling point: construct manifest)
|
24
|
+
|
25
|
+
AWS toolkit: scripts for handling a bunch of instances, including:
|
26
|
+
* starting with user data
|
27
|
+
* expanding credentials (i.e. from u/p to full set of keypairs and
|
28
|
+
certs and things)
|
29
|
+
* baking and migrating images
|
30
|
+
* putting instances into and out of LB.
|
31
|
+
|
32
|
+
(coupling point: server database - consider (initialially) SQLite)
|
33
|
+
|
34
|
+
Remote management: run (command) with (set of servers) as targets. Including
|
35
|
+
curl/rsync or ssh -c "(command)" - cap or vlad may be Good Enough for this
|
36
|
+
already?
|
37
|
+
* servers: an address + metadata
|
38
|
+
* metadata:
|
39
|
+
- hosting environment (AWS/vbox)
|
40
|
+
- deployment type (prod/staging/etc)
|
41
|
+
- deployment role (app/db/etc)
|
42
|
+
- ...
|
43
|
+
* metadata used to:
|
44
|
+
- select servers to run commands against
|
45
|
+
- restrict command that can be run against a server
|
46
|
+
- included into command template
|
47
|
+
(coupling point (to construct): construct <role>:provision[<server from
|
48
|
+
set>])
|
@@ -0,0 +1,70 @@
|
|
1
|
+
ALWAYS check the ec2-api and ec2-ami tools versions
|
2
|
+
|
3
|
+
Outstanding issues:
|
4
|
+
Where is the ephemeral storage mounted? Probably needs to start as a config, maybe detect later
|
5
|
+
|
6
|
+
Data required:
|
7
|
+
Three categories: arbitrary (merely shared), task parameters, configuations
|
8
|
+
|
9
|
+
Task params:
|
10
|
+
Target machine
|
11
|
+
AMI name
|
12
|
+
|
13
|
+
Configurations:
|
14
|
+
Ephemeral storage mount
|
15
|
+
Private Key File
|
16
|
+
Certificate File
|
17
|
+
User ID
|
18
|
+
(bundle options, like includes)
|
19
|
+
Upload bucket
|
20
|
+
S3 access key
|
21
|
+
S3 secret key
|
22
|
+
|
23
|
+
Arbitrary:
|
24
|
+
Manifest path
|
25
|
+
Credential file paths
|
26
|
+
|
27
|
+
From the EC2 user guide:
|
28
|
+
|
29
|
+
[Provisioned]
|
30
|
+
Upload EC2 credentials to ephemeral storage (i.e. pk.pem + cert.pem, AWS creds)
|
31
|
+
[end provisioned]
|
32
|
+
|
33
|
+
[Local to baked machine - should be in target rakefile]
|
34
|
+
Write access needed on instance store (/mnt or /media/ephemeral0)
|
35
|
+
ec2-bundle-vol -k <private_keyfile> -c <certificate_file> -u <user_id> --destination <somewhere ephemeral> --prefix <something unique - not 'image'> --arch x86_64 -i /etc/ec2/amitools/cert-ec2.pem -i $(ls /etc/ssl/certs/*.pem | tr \\n ,) --ec2cert /etc/ec2/amitools/cert-ec2.pem
|
36
|
+
ec2-upload-bundle -b <bucket> -m <manifest_path> -a <access_key> -s <secret_key> --retry
|
37
|
+
ec2-register <your-s3-bucket>/<prefix>.manifest.xml -n image_name --aws-access-key <access_key> --aws-secret-key <secret_key>
|
38
|
+
[end local]
|
39
|
+
|
40
|
+
(Commands that have worked:)
|
41
|
+
ec2-bundle-vol -k /mnt/pk.pem -c /mnt/cert.pem -u 180593873119 -d /mnt/bundling/ -r x86_64 -p nascent-042513 -i /etc/ec2/amitools/cert-ec2.pem -i $(ls /etc/ssl/certs/*.pem | tr \\n ,) --ec2cert /etc/ec2/amitools/cert-ec2.pem
|
42
|
+
|
43
|
+
ec2-upload-bundle -b sbmp-instances -m /mnt/bundling/nascent-042513.manifest.xml -a <access key> -s <secret key>
|
44
|
+
|
45
|
+
(For some reason this didn't:)
|
46
|
+
ec2-register sbmp-instances/nascent-042513.manifest.xml -n Nascent042513 #but web console worked fine
|
47
|
+
|
48
|
+
|
49
|
+
So, if provision, needs to be something like:
|
50
|
+
|
51
|
+
GC:
|
52
|
+
rake bake[target,ami_name]
|
53
|
+
-> ssh target rake bake (can return 13:"Target incapable", but if 0:...)
|
54
|
+
Important here - on success, there needs to be a long-running process on target
|
55
|
+
So: background self? Fork new process and return "We're good to go?"
|
56
|
+
-> build json configs
|
57
|
+
-> ssh tunnel'd provision (target wants creds, configs)
|
58
|
+
"Baking initialized"
|
59
|
+
|
60
|
+
(Pattern to be repeated in remote re-provisioning, too)
|
61
|
+
|
62
|
+
How to tell when done, where it's at?
|
63
|
+
|
64
|
+
Target rake task needs to log process, so reviewing/tailing logs lets us answer
|
65
|
+
the question "is it done yet." SNS/SES/other email when done? Maybe something
|
66
|
+
simple like "mail" command - if set up, bully, otherwise, you're on your own
|
67
|
+
|
68
|
+
|
69
|
+
Of note: there is a ec2-migrate-manifest command that has the basis of regenerating a manifest for a bundle (it's Ruby)
|
70
|
+
The right solution to multiple-client AMIs is LRD hosting the bundles on our S3, give permissions on them, and let clients register the AMIs that way. -- I think. Sorted this out with Locaverse, but the details are fuzzy atm.
|
@@ -0,0 +1,152 @@
|
|
1
|
+
# vim: set ft=ruby:
|
2
|
+
=begin
|
3
|
+
|
4
|
+
Of note:
|
5
|
+
|
6
|
+
/etc/chef/solo.rb:
|
7
|
+
file_cache_path "/var/chef-solo"
|
8
|
+
cookbook_path "/var/chef-solo/cookbooks"
|
9
|
+
json_attribs "http://www.example.com/node.json"
|
10
|
+
recipe_url "http://www.example.com/chef-solo.tar.gz
|
11
|
+
|
12
|
+
Means that "chef-solo" will work ootb
|
13
|
+
|
14
|
+
So: packaging cookbooks, and putting them places (VM dir + unpack, S3) is a
|
15
|
+
thing
|
16
|
+
|
17
|
+
Leaning towards: not using S3 as a webserver, since it makes cookbooks public
|
18
|
+
(ish)
|
19
|
+
Also, adds variation to VM/AMI cases - S3get with perms needs to pull/unpack
|
20
|
+
tgz. Then the solo runs on unpacked cookbooks in both cases
|
21
|
+
|
22
|
+
First feature after "it works": rollback. It was just working - make it work again.
|
23
|
+
|
24
|
+
Running specs on VM. Auto-mount of code directory a la Vagrant...
|
25
|
+
|
26
|
+
Checkpointing stuff - snapshot VM, bundling EC2 - after setup, after provision
|
27
|
+
|
28
|
+
|
29
|
+
Fundamental goal:
|
30
|
+
|
31
|
+
Two orthagonal configurations:
|
32
|
+
|
33
|
+
1) What my hosting environment looks like
|
34
|
+
2) Where my hosting environment lives.
|
35
|
+
|
36
|
+
I should be able to describe 1 and make it happen on different 2s.
|
37
|
+
|
38
|
+
Scenarios:
|
39
|
+
|
40
|
+
Dev in VM, deploy to EC2 (or ideally, any Fog target)
|
41
|
+
|
42
|
+
Deploy a monolith ->
|
43
|
+
deploy simple cluster ->
|
44
|
+
deploy autoscaling worldbeater
|
45
|
+
|
46
|
+
Clear already that there are cases where a particular action needs to
|
47
|
+
happen in different places - Maybe special cases ignore configs they don't
|
48
|
+
handle?
|
49
|
+
|
50
|
+
Only trouble is the possiblity of ball-dropping: "I don't do that, and neither
|
51
|
+
do I"
|
52
|
+
|
53
|
+
|
54
|
+
Version 1:
|
55
|
+
Everything possible is "chef'll do it"
|
56
|
+
"Target Configuration" is: write your Rakefile that way.
|
57
|
+
"Env Configuration" is: write your cookbook/attrs
|
58
|
+
Ideally: that's enough.
|
59
|
+
|
60
|
+
=end
|
61
|
+
|
62
|
+
#Parent/host system
|
63
|
+
module Construction
|
64
|
+
|
65
|
+
#setup => rake bootstrap[address]
|
66
|
+
setup = Setup.new
|
67
|
+
|
68
|
+
setup.in_namespace do
|
69
|
+
#create chef config dir on server
|
70
|
+
dir = CreateDirectory.new(setup) #needs server, dir
|
71
|
+
|
72
|
+
#template Rakefile
|
73
|
+
rakefile = BuildRakefile.new(setup)
|
74
|
+
|
75
|
+
#scp Gemfile to server
|
76
|
+
#scp Rakefile to server
|
77
|
+
copyfiles = CopyFiles.new(setup, rakefile, dir)
|
78
|
+
|
79
|
+
#bundle setup config dir
|
80
|
+
bundler = BundleSetup.new(setup, dir)
|
81
|
+
end
|
82
|
+
|
83
|
+
#Construct bootstrap:
|
84
|
+
#
|
85
|
+
#VM mode:
|
86
|
+
#
|
87
|
+
#
|
88
|
+
#EC2
|
89
|
+
|
90
|
+
configs = ChefConfigs.new #data for precursor
|
91
|
+
|
92
|
+
vbox = Launch::VirtualBox.new(configs) #vbox instance?
|
93
|
+
|
94
|
+
#launch VM
|
95
|
+
vbox.in_namespace do
|
96
|
+
#scp json precursor to VM
|
97
|
+
#scp chef cookbook to config dir
|
98
|
+
scp_files = CopyFiles.new(vbox) #cookbook, attributes
|
99
|
+
#ssh rake constuct:provision
|
100
|
+
run_provision = RemoteProvision.new(vbox)
|
101
|
+
end
|
102
|
+
|
103
|
+
#launch AMI w/ user metadata - json precusor
|
104
|
+
ec2 = Launch::EC2.new(server_configs)
|
105
|
+
end
|
106
|
+
|
107
|
+
=begin
|
108
|
+
rake launch =>
|
109
|
+
|
110
|
+
AMI mode:
|
111
|
+
|
112
|
+
run on start
|
113
|
+
/etc/rc.d/local => rake startup
|
114
|
+
|
115
|
+
get chef.json precursor from instance metadata
|
116
|
+
s3 get chef cookbook into config dir
|
117
|
+
|
118
|
+
rake construct:provision
|
119
|
+
|
120
|
+
Launch tasks (rake construct:provision)
|
121
|
+
|
122
|
+
build chef json config
|
123
|
+
build chef rb config
|
124
|
+
run chef solo
|
125
|
+
|
126
|
+
=end
|
127
|
+
#Target/child
|
128
|
+
module Construction
|
129
|
+
provision = Provision.new do |prov|
|
130
|
+
prov.attr_source = ""
|
131
|
+
prov.cookbook_path = ""
|
132
|
+
prov.config_path = ""
|
133
|
+
end
|
134
|
+
|
135
|
+
ec2_start = EC2Boot.new(provision)
|
136
|
+
ec2_start.in_namespace do
|
137
|
+
metadata = RetrieveMetadata.new(ec2_start) #url path => filesystem path
|
138
|
+
cookbook = GetCookbook.new(ec2_start) do
|
139
|
+
source_url = "s3://..."
|
140
|
+
end
|
141
|
+
end
|
142
|
+
|
143
|
+
provision.in_namespace do
|
144
|
+
attrs = BuildAttributes.new(provision) do |attrs|
|
145
|
+
attrs.destination_path = ""
|
146
|
+
end
|
147
|
+
unpack = UnpackCookbook.new(provision)
|
148
|
+
#location of precursor
|
149
|
+
chef = RunChef.new(attrs, provision) #loc of json, rc, cookbook
|
150
|
+
end
|
151
|
+
task :launch => [ec2_start[:run], provision[:run]]
|
152
|
+
end
|
data/doc/TODO
ADDED
@@ -0,0 +1,148 @@
|
|
1
|
+
--- Current showstoppers (pre-share with Evan/LRD)
|
2
|
+
|
3
|
+
Current provisioning can't handle symlink for cookbooks dir
|
4
|
+
|
5
|
+
[written] SSH tunnelling for provision - ideally including a local HTTP proxy of all servers in need
|
6
|
+
|
7
|
+
Chef: multiple cookbooks (e.g. LRD base + Client deploy)
|
8
|
+
Manifest patching (the Stanford NLP problem)
|
9
|
+
|
10
|
+
[written] Bless(/bake) instances
|
11
|
+
|
12
|
+
[written] Logging provisioning stuff - especially the chef output. rake provision outputs nothing until done. Logging or output would be handy.
|
13
|
+
|
14
|
+
--- 1= known showstoppers fixed. Other bugs:
|
15
|
+
|
16
|
+
Check fixed: Bug: /etc/init.d/logical-construct is generated once and never rebuilt, so the LC_DP variable isn't set per rake setup.
|
17
|
+
|
18
|
+
[written] Unpack needs (at least the option) to clobber target directory - otherwise
|
19
|
+
"bad" files that are just deleted poison the directory.
|
20
|
+
|
21
|
+
[written] Pack needs (something) to handle deleted files in the source directory - so
|
22
|
+
file-style + archive index - we need to recreate an archive if a file has been
|
23
|
+
deleted
|
24
|
+
|
25
|
+
The ephemeral mounts task mounts and remounts (... and remounts) directories in
|
26
|
+
the /mnt directory. Should be able to re-run chef without duplicating mounts
|
27
|
+
|
28
|
+
--- Features
|
29
|
+
|
30
|
+
Quick target management tasks:
|
31
|
+
Collect server IPs into file(s)
|
32
|
+
template shell scripts in related files/dirs
|
33
|
+
rake target_management:restart_sidekiq <- auto from script name
|
34
|
+
|
35
|
+
Also: Some tasks might amount to "run same task name on target"
|
36
|
+
(Most should migrate there.) So: see if it's there, and run it if it is.
|
37
|
+
(Instead of local?) Hm. Would be exposed as form on the LC web service
|
38
|
+
|
39
|
+
VBox provisioning test mode - normally vbox should be === normal deploy. Also
|
40
|
+
good would be a mode where cookbooks dirs from host machine are mounted at
|
41
|
+
client so that they can be edited locally vagrant style
|
42
|
+
Once you start down that path, might as well look into mounting code
|
43
|
+
directory as well...
|
44
|
+
|
45
|
+
Like to do smaller provisioning chunks. SB has 100+MB files in their cookbooks,
|
46
|
+
and transmitting the whole thing is torturous.
|
47
|
+
One solution would be multiple cookbook-things that get fused somehow A tool
|
48
|
+
to produce those things (basically an install pack) would be helpful.
|
49
|
+
I think git could be used to produce them without having to be the distribution
|
50
|
+
channel - basically diff the last one etc.
|
51
|
+
|
52
|
+
Separate provision volumes makes sense from the perspective of bridging
|
53
|
+
projects (i.e. the LRD base volume, the LRD NLP volume, the SB special
|
54
|
+
volume) and then patches for each.
|
55
|
+
|
56
|
+
Distinct packages (filelists intersections are empty sets)
|
57
|
+
Ensure a "rakelib" dir exists.
|
58
|
+
Target: `rake provision` pulls in package rake tasks - standard hooks
|
59
|
+
Means: a "chef" package to make sure gems are installed, and set up chef hooks
|
60
|
+
Then "cookbook" packages hook into the chef stuff to configure chef in memory before the chef tasks write configs and run
|
61
|
+
|
62
|
+
|
63
|
+
Non-file platform provisioning requirements of the server-
|
64
|
+
notably volumes: perhaps "I need X GB with on (device/label)"
|
65
|
+
The userdata solution ... change the instance device mapping config
|
66
|
+
At which point, a chef recipe could mount them on the right place
|
67
|
+
maybe extra NICs? GPU?
|
68
|
+
"this OS installed. portage version=..."
|
69
|
+
listed on the provisioning web service, and PUT means "check again"
|
70
|
+
|
71
|
+
Arrange deployment vs. application code... related at all?
|
72
|
+
related to:
|
73
|
+
"Compile" (rails) app into deployable - assets:precompile on deployment servers is silly.
|
74
|
+
|
75
|
+
AWS resolutions:
|
76
|
+
from user_data (or other instance metadata)
|
77
|
+
from S3?
|
78
|
+
|
79
|
+
Resolution chains - "look at instance metadata, then S3, then start Sinatra"
|
80
|
+
Needs to also be "loops" - "Sinatra got a resolution, but that changed the needs list - recheck"
|
81
|
+
|
82
|
+
Resolution manifests - all Satisfiables "needed" unless they match the manifest (if the manifest task is present...)
|
83
|
+
|
84
|
+
"Promote" VBox instances to AWS (generally: convert instances between platforms)
|
85
|
+
(Therefore likewise: translate AWS to VBox)
|
86
|
+
|
87
|
+
Something to handle private keys - an -i option to the SSH commands, basically,
|
88
|
+
but it needs to be per-project, and not in the Rakefile. There's the
|
89
|
+
user-configuration, which'll make sense for a lot of things
|
90
|
+
(Time being: Judson is using Host *.compute-1.amazonaws.com in .ssh/config)
|
91
|
+
|
92
|
+
Switchable identities - there's some baking needs to happen for LRD, and then some for SB
|
93
|
+
|
94
|
+
GC provision WebConfigure: output re: uploads
|
95
|
+
|
96
|
+
Integrate with ... something for instance management
|
97
|
+
Should completely replace current workflow of: start Instance, record IP, later:
|
98
|
+
for s in $(cat <some ip files>); do <management>; done
|
99
|
+
|
100
|
+
Update LC on provisioned box (aot go back to nascent and re-setup)
|
101
|
+
|
102
|
+
Cascading rakelib dirs - especially for the case of adding commands that loop on/pick a server.
|
103
|
+
|
104
|
+
git management for plan dirs: commit to a local branch-name (e.g.
|
105
|
+
"jdl-deployed") with a tag of the Manifest ID, push to repo, so that others
|
106
|
+
able to reproduce the details of a deploy. Obviously would need a "don't git
|
107
|
+
publish" for "secret plans" e.g. github secret keys etc.
|
108
|
+
|
109
|
+
--- Nice to have (usually easy)
|
110
|
+
|
111
|
+
[written] Decompose into 2 tasklibs the ChefConfig tasklib - something like UnpackTarballs and ConfigureChef
|
112
|
+
|
113
|
+
Bug: why do we emit several "tar" commands for the cookbook directory?
|
114
|
+
|
115
|
+
Descriptions: need for setup, provision tasks - not sub tasks of setup
|
116
|
+
|
117
|
+
[written] Move /var/logical-construct to /var/run/lc or /opt/bin/lc or ...
|
118
|
+
|
119
|
+
provision namespace needs a "list roles" task
|
120
|
+
|
121
|
+
LC should echo local IP (at least) on failure
|
122
|
+
|
123
|
+
LC init task should start a "status/report" server on complete - maybe just single page of "success/fail" and chef log
|
124
|
+
|
125
|
+
--- Gentoo related
|
126
|
+
|
127
|
+
After initial deploy, /usr/src/linux not needed
|
128
|
+
|
129
|
+
sqlite version of md5cache
|
130
|
+
|
131
|
+
/usr/portage should be an external mount (not just distfiles/packages) because the md5cache is huge.
|
132
|
+
|
133
|
+
Consider S3 as a PORTAGE_BINHOST - should be possible to do installs on a machine, then mirror /usr/portage/packages to S3, especially if the metadata in S3 is such that we don't re-upload packages we pulled down to do the build.
|
134
|
+
|
135
|
+
Binhost needs to be related to arch, CHOST, CFLAGS, processor USE flags (MXX, SSE, ???)
|
136
|
+
|
137
|
+
How to deal with portage? Open questions:
|
138
|
+
eix-sync
|
139
|
+
& syncing /usr/portage
|
140
|
+
& updating portage
|
141
|
+
I think the real answer here is that "version of portage" is a non-file provisioning requirement. U2D system is really a not a touchless problem, IMO. Does imply a smaller/staged deployment, if only to get mounts.
|
142
|
+
Maybe: "how long since last sync? and if more than (days) sync, update portage"
|
143
|
+
man emerge: "emerge-webrsync pulls one tbz which is faster for first-time"
|
144
|
+
How long to last: emerge --info includes "Timestamp of tree"
|
145
|
+
|
146
|
+
--- Bluesky wishlist
|
147
|
+
|
148
|
+
Hypermedia client in Ruby for provisioning
|
@@ -0,0 +1,96 @@
|
|
1
|
+
VERY IMPORTANT:
|
2
|
+
Version of the ec2-ami-tools MUST BE UP TO DATE.
|
3
|
+
Otherwise bundling will be buggy and waste lots of time.
|
4
|
+
|
5
|
+
|
6
|
+
Determine size of Vbox instance
|
7
|
+
exclude some dirs?
|
8
|
+
df/du
|
9
|
+
Create disk image of appropriate size (+wiggle room - 10-20%)
|
10
|
+
`dd if=/dev/zero of=aws-nascent.img bs=1M count=5000`
|
11
|
+
Format image
|
12
|
+
gotcha: tunefs to never fsck
|
13
|
+
gotcha: keep sparse
|
14
|
+
`/sbin/mke2fs -j aws-nascent.img`
|
15
|
+
`/sbin/tune2fs -c 0 -e continue -i 0 aws-nascent.img`
|
16
|
+
Mount image
|
17
|
+
sudo mkdir /mnt/nascent_vm
|
18
|
+
sudo mount aws-nascent.img /mnt/nascent_vm/
|
19
|
+
Copy data from vm
|
20
|
+
cd /mnt/nascent_vm
|
21
|
+
(ensure /boot is mounted on vbox)
|
22
|
+
ssh root@lrd-aws-nascent 'find / -regextype posix-egrep -depth \! -regex "/(proc|sys|dev).*" -print0 | cpio -o -0a' | sudo cpio -i -dumv --no-absolute-filenames
|
23
|
+
|
24
|
+
FYI
|
25
|
+
ec2-bundle-vol excludes:
|
26
|
+
/proc
|
27
|
+
/proc/xen
|
28
|
+
/sys
|
29
|
+
/sys/kernel/debug
|
30
|
+
/sys/fs/cgroup/cpuset
|
31
|
+
/sys/fs/cgroup/cpu
|
32
|
+
/sys/fs/cgroup/cpuacct
|
33
|
+
/sys/fs/cgroup/freezer
|
34
|
+
/dev/pts
|
35
|
+
/proc/sys/fs/binfmt_misc
|
36
|
+
/dev
|
37
|
+
/media
|
38
|
+
/mnt
|
39
|
+
/proc
|
40
|
+
/sys
|
41
|
+
/mnt/ami/lc-setup
|
42
|
+
/mnt/img-mnt
|
43
|
+
|
44
|
+
|
45
|
+
Additions to EC2 pre-bundle
|
46
|
+
Update /etc/ssh/sshd_config - RootLogin without-password
|
47
|
+
Add lrd_rsa.pub to root authorized_keys
|
48
|
+
Add ec2-init script to /etc/init.d
|
49
|
+
add ec2-init to boot runlevel with softlinks
|
50
|
+
check /etc/fstab
|
51
|
+
Change fstab and /boot/grub/menu.lst to refer to /dev/xvda
|
52
|
+
ensure /proc /sys /dev exist
|
53
|
+
---
|
54
|
+
|
55
|
+
Unmount bundle image
|
56
|
+
umount /mnt/nascent_vm
|
57
|
+
|
58
|
+
Bundle for EC2
|
59
|
+
#This does happen at volume bundle time
|
60
|
+
block mapping: /dev/sda2=swap /dev/sda3=ephemeral0 #if fails, /etc/fstab
|
61
|
+
Minimal options on bundle
|
62
|
+
Must include arch - don't include kernel
|
63
|
+
|
64
|
+
Upload to EC2
|
65
|
+
Remember --retry
|
66
|
+
Register AMI
|
67
|
+
Register with instance options (kernel, etc)
|
68
|
+
|
69
|
+
*********
|
70
|
+
|
71
|
+
Set up AWS credentials:
|
72
|
+
|
73
|
+
openssl genrsa 1024 > aws-creds/pk.pem
|
74
|
+
openssl req -new -x509 -nodes -sha1 -days 3650 -key aws-creds/pk.pem -outform PEM > aws-creds/cert.pem
|
75
|
+
|
76
|
+
vim aws-creds/AwsCredentialFile
|
77
|
+
"""
|
78
|
+
AWSAccessKeyId=<Write your AWS access key ID>
|
79
|
+
AWSSecretKey=<Write your AWS secret key>
|
80
|
+
"""
|
81
|
+
vim aws-creds/establish-certs
|
82
|
+
"""
|
83
|
+
export AWS_CREDENTIAL_FILE=aws-creds/AwsCredentialFile
|
84
|
+
export EC2_CERT=aws-certs/cert.pem
|
85
|
+
export EC2_PRIVATE_KEY=aws-certs/pk.pem
|
86
|
+
"""
|
87
|
+
|
88
|
+
source aws-creds/establish-certs
|
89
|
+
iam-useraddcert -f aws-creds/cert.pem
|
90
|
+
|
91
|
+
|
92
|
+
**** related to baking EC2 images:
|
93
|
+
default ec2-bundle-vol excludes all .pem files by default. Including files required to re-bundle. Explicit include required.
|
94
|
+
includes required on:
|
95
|
+
/opt/ec2-ami-tools/etc/ec2/amitools/cert-ec2.pem
|
96
|
+
/etc/ssl/
|