logical-construct 0.0.5 → 0.1.0

data/bin/flight-deck

@@ -0,0 +1,3 @@
+require 'logical-construct/target/command-line'
+
+LogicalConstruct::Target::CommandLine.new(ARGV).go

data/doc/DESIGN

@@ -0,0 +1,48 @@
+After consideration:
+
+This is not a cloud server manager.  Those exist already.
+
+This is a cloud deployment manager.  Write one description of how your cloud is built, deploy it whereever.
+
+Puppet/chef handle like 90% of that, but before either can run, network has to
+be configured, and persistent/extra volumes need to be mounted.
+
+"Blessing" deploys is another thing - snapshot/repackage a currently running
+instance so that we can shortcut deploy next time.
+
+Things Ground control should be able to do:
+
+* Survey Target hosts and make sure they have the configs they need.
+
+* Given a target server with Ruby and SSH, set it up as a particular role.
+* (I.e: SSH and deliver files, install gems, locked to GC version)
+
+Three basic tools:
+
+Construct: delivers and executes provisioning plans (which themselves may be "use Chef to...")
+  (coupling point: construct manifest)
+
+AWS toolkit: scripts for handling a bunch of instances, including:
+  * starting with user data
+  * expanding credentials (i.e. from u/p to full set of keypairs and
+    certs and things)
+  * baking and migrating images
+  * putting instances into and out of LB.
+
+  (coupling point: server database - consider (initialially) SQLite)
+
+Remote management: run (command) with (set of servers) as targets. Including
+curl/rsync or ssh -c "(command)" - cap or vlad may be Good Enough for this
+already?
+  * servers: an address + metadata
+  * metadata:
+    - hosting environment (AWS/vbox)
+    - deployment type (prod/staging/etc)
+    - deployment role (app/db/etc)
+    - ...
+  * metadata used to:
+    - select servers to run commands against
+    - restrict command that can be run against a server
+    - included into command template
+  (coupling point (to construct): construct <role>:provision[<server from
+  set>])

data/doc/EC2-baking-notes

@@ -0,0 +1,70 @@
+ALWAYS check the ec2-api and ec2-ami tools versions
+
+Outstanding issues:
+Where is the ephemeral storage mounted?  Probably needs to start as a config, maybe detect later
+
+Data required:
+Three categories: arbitrary (merely shared), task parameters, configuations
+
+Task params:
+Target machine
+AMI name
+
+Configurations:
+Ephemeral storage mount
+Private Key File
+Certificate File
+User ID
+(bundle options, like includes)
+Upload bucket
+S3 access key
+S3 secret key
+
+Arbitrary:
+Manifest path
+Credential file paths
+
+From the EC2 user guide:
+
+[Provisioned]
+Upload EC2 credentials to ephemeral storage (i.e. pk.pem + cert.pem, AWS creds)
+[end provisioned]
+
+[Local to baked machine - should be in target rakefile]
+Write access needed on instance store (/mnt or /media/ephemeral0)
+ec2-bundle-vol -k <private_keyfile> -c <certificate_file> -u <user_id> --destination <somewhere ephemeral> --prefix <something unique - not 'image'> --arch x86_64 -i /etc/ec2/amitools/cert-ec2.pem -i $(ls /etc/ssl/certs/*.pem | tr \\n ,) --ec2cert /etc/ec2/amitools/cert-ec2.pem
+ec2-upload-bundle -b <bucket> -m <manifest_path> -a <access_key> -s <secret_key> --retry
+ec2-register <your-s3-bucket>/<prefix>.manifest.xml -n image_name --aws-access-key <access_key> --aws-secret-key <secret_key>
+[end local]
+
+(Commands that have worked:)
+ec2-bundle-vol -k /mnt/pk.pem -c /mnt/cert.pem -u 180593873119 -d /mnt/bundling/ -r x86_64 -p nascent-042513 -i /etc/ec2/amitools/cert-ec2.pem -i $(ls /etc/ssl/certs/*.pem | tr \\n ,) --ec2cert /etc/ec2/amitools/cert-ec2.pem
+
+ec2-upload-bundle -b sbmp-instances -m /mnt/bundling/nascent-042513.manifest.xml -a <access key> -s <secret key>
+
+(For some reason this didn't:)
+ec2-register sbmp-instances/nascent-042513.manifest.xml -n Nascent042513  #but web console worked fine
+
+
+So, if provision, needs to be something like:
+
+GC:
+rake bake[target,ami_name]
+ -> ssh target rake bake (can return 13:"Target incapable", but if 0:...)
+    Important here - on success, there needs to be a long-running process on target
+    So: background self?  Fork new process and return "We're good to go?"
+ -> build json configs
+ -> ssh tunnel'd provision (target wants creds, configs)
+ "Baking initialized"
+
+(Pattern to be repeated in remote re-provisioning, too)
+
+How to tell when done, where it's at?
+
+Target rake task needs to log process, so reviewing/tailing logs lets us answer
+the question "is it done yet." SNS/SES/other email when done?  Maybe something
+simple like "mail" command - if set up, bully, otherwise, you're on your own
+
+
+Of note: there is a ec2-migrate-manifest command that has the basis of regenerating a manifest for a bundle (it's Ruby)
+The right solution to multiple-client AMIs is LRD hosting the bundles on our S3, give permissions on them, and let clients register the AMIs that way. -- I think. Sorted this out with Locaverse, but the details are fuzzy atm.

data/doc/ExampleStartupRakefile

@@ -0,0 +1,152 @@
+# vim: set ft=ruby:
+=begin
+
+Of note:
+
+/etc/chef/solo.rb:
+file_cache_path "/var/chef-solo"
+cookbook_path "/var/chef-solo/cookbooks"
+json_attribs "http://www.example.com/node.json"
+recipe_url "http://www.example.com/chef-solo.tar.gz
+
+Means that "chef-solo" will work ootb
+
+So: packaging cookbooks, and putting them places (VM dir + unpack, S3) is a
+thing
+
+Leaning towards: not using S3 as a webserver, since it makes cookbooks public
+(ish)
+Also, adds variation to VM/AMI cases - S3get with perms needs to pull/unpack
+tgz.  Then the solo runs on unpacked cookbooks in both cases
+
+First feature after "it works": rollback.  It was just working - make it work again.
+
+Running specs on VM.  Auto-mount of code directory a la Vagrant...
+
+Checkpointing stuff - snapshot VM, bundling EC2 - after setup, after provision
+
+
+Fundamental goal:
+
+Two orthagonal configurations:
+
+1) What my hosting environment looks like
+2) Where my hosting environment lives.
+
+I should be able to describe 1 and make it happen on different 2s.
+
+Scenarios:
+
+Dev in VM, deploy to EC2 (or ideally, any Fog target)
+
+Deploy a monolith ->
+ deploy simple cluster ->
+  deploy autoscaling worldbeater
+
+Clear already that there are cases where a particular action needs to
+happen in different places - Maybe special cases ignore configs they don't
+handle?
+
+Only trouble is the possiblity of ball-dropping: "I don't do that, and neither
+do I"
+
+
+Version 1:
+Everything possible is "chef'll do it"
+"Target Configuration" is: write your Rakefile that way.
+"Env Configuration" is: write your cookbook/attrs
+Ideally: that's enough.
+
+=end
+
+#Parent/host system
+module Construction
+
+  #setup => rake bootstrap[address]
+  setup = Setup.new
+
+  setup.in_namespace do
+    #create chef config dir on server
+    dir = CreateDirectory.new(setup) #needs server, dir
+
+    #template Rakefile
+    rakefile = BuildRakefile.new(setup)
+
+    #scp Gemfile to server
+    #scp Rakefile to server
+    copyfiles = CopyFiles.new(setup, rakefile, dir)
+
+    #bundle setup config dir
+    bundler = BundleSetup.new(setup, dir)
+  end
+
+  #Construct bootstrap:
+  #
+  #VM mode:
+  #
+  #
+  #EC2
+
+  configs = ChefConfigs.new #data for precursor
+
+  vbox = Launch::VirtualBox.new(configs) #vbox instance?
+
+  #launch VM
+  vbox.in_namespace do
+  #scp json precursor to VM
+  #scp chef cookbook to config dir
+    scp_files = CopyFiles.new(vbox) #cookbook, attributes
+  #ssh rake constuct:provision
+    run_provision = RemoteProvision.new(vbox)
+  end
+
+  #launch AMI w/ user metadata - json precusor
+  ec2 = Launch::EC2.new(server_configs)
+end
+
+=begin
+rake launch =>
+
+AMI mode:
+
+run on start
+/etc/rc.d/local => rake startup
+
+get chef.json precursor from instance metadata
+s3 get chef cookbook into config dir
+
+rake construct:provision
+
+Launch tasks (rake construct:provision)
+
+build chef json config
+build chef rb config
+run chef solo
+
+=end
+#Target/child
+module Construction
+  provision = Provision.new do |prov|
+    prov.attr_source = ""
+    prov.cookbook_path = ""
+    prov.config_path = ""
+  end
+
+  ec2_start = EC2Boot.new(provision)
+  ec2_start.in_namespace do
+    metadata = RetrieveMetadata.new(ec2_start) #url path => filesystem path
+    cookbook = GetCookbook.new(ec2_start) do
+      source_url = "s3://..."
+    end
+  end
+
+  provision.in_namespace do
+    attrs = BuildAttributes.new(provision) do |attrs|
+      attrs.destination_path = ""
+    end
+    unpack = UnpackCookbook.new(provision)
+    #location of precursor
+    chef = RunChef.new(attrs, provision) #loc of json, rc, cookbook
+  end
+  task :launch => [ec2_start[:run], provision[:run]]
+end

data/doc/ExampleTargetRakefile

@@ -0,0 +1,4 @@
+# vim: set ft=ruby :
+require 'logical-construct/target'
+
+include LogicalConstruct

data/doc/TODO

@@ -0,0 +1,148 @@
+--- Current showstoppers (pre-share with Evan/LRD)
+
+Current provisioning can't handle symlink for cookbooks dir
+
+[written] SSH tunnelling for provision - ideally including a local HTTP proxy of all servers in need
+
+Chef: multiple cookbooks (e.g. LRD base + Client deploy)
+Manifest patching (the Stanford NLP problem)
+
+[written] Bless(/bake) instances
+
+[written] Logging provisioning stuff - especially the chef output.  rake provision outputs nothing until done.  Logging or output would be handy.
+
+--- 1= known showstoppers fixed.  Other bugs:
+
+Check fixed: Bug: /etc/init.d/logical-construct is generated once and never rebuilt, so the LC_DP variable isn't set per rake setup.
+
+[written] Unpack needs (at least the option) to clobber target directory - otherwise
+"bad" files that are just deleted poison the directory.
+
+[written] Pack needs (something) to handle deleted files in the source directory - so
+file-style + archive index - we need to recreate an archive if a file has been
+deleted
+
+The ephemeral mounts task mounts and remounts (... and remounts) directories in
+the /mnt directory.  Should be able to re-run chef without duplicating mounts
+
+--- Features
+
+Quick target management tasks:
+  Collect server IPs into file(s)
+  template shell scripts in related files/dirs
+  rake target_management:restart_sidekiq <- auto from script name
+
+  Also: Some tasks might amount to "run same task name on target"
+  (Most should migrate there.) So: see if it's there, and run it if it is.
+  (Instead of local?) Hm. Would be exposed as form on the LC web service
+
+VBox provisioning test mode - normally vbox should be === normal deploy. Also
+good would be a mode where cookbooks dirs from host machine are mounted at
+client so that they can be edited locally vagrant style
+  Once you start down that path, might as well look into mounting code
+  directory as well...
+
+Like to do smaller provisioning chunks. SB has 100+MB files in their cookbooks,
+and transmitting the whole thing is torturous.
+  One solution would be multiple cookbook-things that get fused somehow A tool
+  to produce those things (basically an install pack) would be helpful.
+  I think git could be used to produce them without having to be the distribution
+  channel - basically diff the last one etc.
+
+  Separate provision volumes makes sense from the perspective of bridging
+  projects (i.e. the LRD base volume, the LRD NLP volume, the SB special
+  volume) and then patches for each.
+
+  Distinct packages (filelists intersections are empty sets)
+  Ensure a "rakelib" dir exists.
+  Target: `rake provision` pulls in package rake tasks - standard hooks
+  Means: a "chef" package to make sure gems are installed, and set up chef hooks
+    Then "cookbook" packages hook into the chef stuff to configure chef in memory before the chef tasks write configs and run
+
+
+Non-file platform provisioning requirements of the server-
+  notably volumes: perhaps "I need X GB with on (device/label)"
+    The userdata solution ... change the instance device mapping config
+    At which point, a chef recipe could mount them on the right place
+  maybe extra NICs? GPU?
+  "this OS installed. portage version=..."
+  listed on the provisioning web service, and PUT means "check again"
+
+Arrange deployment vs. application code... related at all?
+  related to:
+"Compile" (rails) app into deployable - assets:precompile on deployment servers is silly.
+
+AWS resolutions:
+  from user_data (or other instance metadata)
+  from S3?
+
+Resolution chains - "look at instance metadata, then S3, then start Sinatra"
+  Needs to also be "loops" - "Sinatra got a resolution, but that changed the needs list - recheck"
+
+Resolution manifests - all Satisfiables "needed" unless they match the manifest (if the manifest task is present...)
+
+"Promote" VBox instances to AWS (generally: convert instances between platforms)
+(Therefore likewise: translate AWS to VBox)
+
+Something to handle private keys - an -i option to the SSH commands, basically,
+but it needs to be per-project, and not in the Rakefile.  There's the
+user-configuration, which'll make sense for a lot of things
+(Time being: Judson is using Host *.compute-1.amazonaws.com in .ssh/config)
+
+Switchable identities - there's some baking needs to happen for LRD, and then some for SB
+
+GC provision WebConfigure: output re: uploads
+
+Integrate with ... something for instance management
+  Should completely replace current workflow of: start Instance, record IP, later:
+  for s in $(cat <some ip files>); do <management>; done
+
+Update LC on provisioned box (aot go back to nascent and re-setup)
+
+Cascading rakelib dirs - especially for the case of adding commands that loop on/pick a server.
+
+git management for plan dirs: commit to a local branch-name (e.g.
+"jdl-deployed") with a tag of the Manifest ID, push to repo, so that others
+able to reproduce the details of a deploy. Obviously would need a "don't git
+publish" for "secret plans" e.g. github secret keys etc.
+
+--- Nice to have (usually easy)
+
+[written] Decompose into 2 tasklibs the ChefConfig tasklib - something like UnpackTarballs and ConfigureChef
+
+Bug: why do we emit several "tar" commands for the cookbook directory?
+
+Descriptions: need for setup, provision tasks - not sub tasks of setup
+
+[written] Move /var/logical-construct to /var/run/lc or /opt/bin/lc or ...
+
+provision namespace needs a "list roles" task
+
+LC should echo local IP (at least) on failure
+
+LC init task should start a "status/report" server on complete - maybe just single page of "success/fail" and chef log
+
+--- Gentoo related
+
+After initial deploy, /usr/src/linux not needed
+
+sqlite version of md5cache
+
+/usr/portage should be an external mount (not just distfiles/packages) because the md5cache is huge.
+
+Consider S3 as a PORTAGE_BINHOST - should be possible to do installs on a machine, then mirror /usr/portage/packages to S3, especially if the metadata in S3 is such that we don't re-upload packages we pulled down to do the build.
+
+Binhost needs to be related to arch, CHOST, CFLAGS, processor USE flags (MXX, SSE, ???)
+
+How to deal with portage?  Open questions:
+  eix-sync
+  & syncing /usr/portage
+  & updating portage
+    I think the real answer here is that "version of portage" is a non-file provisioning requirement. U2D system is really a not a touchless problem, IMO. Does imply a smaller/staged deployment, if only to get mounts.
+  Maybe: "how long since last sync? and if more than (days) sync, update portage"
+  man emerge: "emerge-webrsync pulls one tbz which is faster for first-time"
+  How long to last: emerge --info includes "Timestamp of tree"
+
+--- Bluesky wishlist
+
+Hypermedia client in Ruby for provisioning

data/doc/Vb-EC2-translation-notes

@@ -0,0 +1,96 @@
+VERY IMPORTANT:
+Version of the ec2-ami-tools MUST BE UP TO DATE.
+Otherwise bundling will be buggy and waste lots of time.
+
+
+Determine size of Vbox instance
+  exclude some dirs?
+  df/du
+Create disk image of appropriate size (+wiggle room - 10-20%)
+  `dd if=/dev/zero of=aws-nascent.img bs=1M count=5000`
+Format image
+  gotcha: tunefs to never fsck
+  gotcha: keep sparse
+  `/sbin/mke2fs -j aws-nascent.img`
+  `/sbin/tune2fs -c 0 -e continue -i 0 aws-nascent.img`
+Mount image
+  sudo mkdir /mnt/nascent_vm
+  sudo mount aws-nascent.img /mnt/nascent_vm/
+Copy data from vm
+  cd /mnt/nascent_vm
+  (ensure /boot is mounted on vbox)
+  ssh root@lrd-aws-nascent 'find / -regextype posix-egrep -depth \! -regex "/(proc|sys|dev).*" -print0 | cpio -o -0a' | sudo cpio -i -dumv --no-absolute-filenames
+
+FYI
+ec2-bundle-vol excludes:
+   /proc
+   /proc/xen
+   /sys
+   /sys/kernel/debug
+   /sys/fs/cgroup/cpuset
+   /sys/fs/cgroup/cpu
+   /sys/fs/cgroup/cpuacct
+   /sys/fs/cgroup/freezer
+   /dev/pts
+   /proc/sys/fs/binfmt_misc
+   /dev
+   /media
+   /mnt
+   /proc
+   /sys
+   /mnt/ami/lc-setup
+   /mnt/img-mnt
+
+
+Additions to EC2 pre-bundle
+  Update /etc/ssh/sshd_config - RootLogin without-password
+  Add lrd_rsa.pub to root authorized_keys
+  Add ec2-init script to /etc/init.d
+  add ec2-init to boot runlevel with softlinks
+  check /etc/fstab
+  Change fstab and /boot/grub/menu.lst to refer to /dev/xvda
+  ensure /proc /sys /dev exist
+---
+
+Unmount bundle image
+  umount /mnt/nascent_vm
+
+Bundle for EC2
+  #This does happen at volume bundle time
+  block mapping: /dev/sda2=swap /dev/sda3=ephemeral0 #if fails, /etc/fstab
+  Minimal options on bundle
+  Must include arch - don't include kernel
+
+Upload to EC2
+  Remember --retry
+Register AMI
+  Register with instance options (kernel, etc)
+
+*********
+
+Set up AWS credentials:
+
+openssl genrsa 1024 > aws-creds/pk.pem
+openssl req -new -x509 -nodes -sha1 -days 3650 -key aws-creds/pk.pem  -outform PEM > aws-creds/cert.pem
+
+vim aws-creds/AwsCredentialFile
+"""
+AWSAccessKeyId=<Write your AWS access key ID>
+AWSSecretKey=<Write your AWS secret key>
+"""
+vim aws-creds/establish-certs
+"""
+export AWS_CREDENTIAL_FILE=aws-creds/AwsCredentialFile
+export EC2_CERT=aws-certs/cert.pem
+export EC2_PRIVATE_KEY=aws-certs/pk.pem
+"""
+
+source aws-creds/establish-certs
+iam-useraddcert -f aws-creds/cert.pem
+
+
+**** related to baking EC2 images:
+default ec2-bundle-vol excludes all .pem files by default.  Including files required to re-bundle.  Explicit include required.
+includes required on:
+/opt/ec2-ami-tools/etc/ec2/amitools/cert-ec2.pem
+/etc/ssl/