bosh-gen 0.9.0 → 0.10.0

data/ChangeLog.md

@@ -1,5 +1,15 @@
 # Change Log
 
+## v0.10.0
+
+Added:
+
+* `micro` - create a "micro" job that packages all/some jobs into a single VM
+
+Improved:
+
+* `new` - read/write credentials now in private.yml; more useful default README
+
 ## v0.9.0
 
 Major news

data/README.md

@@ -60,7 +60,112 @@ $ bosh-gen template some-ruby-job config/some-config.ini
      force  jobs/some-ruby-job/spec
 ```
 
+### Micro jobs - all-in-one VM
 
+If your release includes two or more jobs you might want to offer a "micro" job that includes all/some jobs into a single VM.
+
+To achieve this, there is a special `micro` generator.
+
+```
+$ bosh-gen micro
+      create  jobs/micro
+      create  jobs/micro/prepare
+       chmod  jobs/micro/prepare
+      create  jobs/micro/prepare_spec
+      append  .gitignore
+
+Edit jobs/micro/prepare_spec with ordered list of jobs to include
+in micro job. The order of jobs implicitly specifies the order in
+which they are started.
+```
+
+As above, now edit `prepare_spec` to order/restrict the list of jobs to be included in the micro VM.
+
+Now create a new bosh release and a "micro/0.1-dev" job will be included:
+
+```
+$ bosh create release --force
+...
+Jobs
++----------+---------+-------+------------------------------------------+
+| Name     | Version | Notes | Fingerprint                              |
++----------+---------+-------+------------------------------------------+
+...
+| micro    | 0.1-dev |       | 6eb2f98644ef7f61a0399c015cbe062987dfd498 |
++----------+---------+-------+------------------------------------------+
+```
+
+## Tutorial
+
+To see how the various commands work together, let's create a new bosh release for [Cassandra](http://cassandra.apache.org/ "The Apache Cassandra Project").
+
+```
+$ bosh-gen new cassandra --s3
+$ cd cassandra
+$ bosh-gen extract-pkg ../cf-release/packages/dea_jvm7
+      create  packages/dea_jvm7
+      create  packages/dea_jvm7/packaging
+      create  packages/dea_jvm7/spec
+      create  blobs/java/jre-7u4-linux-i586.tar.gz
+      create  blobs/java/jre-7u4-linux-x64.tar.gz
+      readme  Upload blobs with 'bosh upload blobs'
+$ mv packages/dea_jvm7 packages/java7
+```
+
+In `packages/java7/spec`, rename it to `java7`.
+
+```
+$ bosh-gen package cassandra -d java7 -f ~/Downloads/apache-cassandra-1.0.11-bin.tar.gz
+      create  packages/cassandra/packaging
+      create  packages/cassandra/pre_packaging
+      create  blobs/cassandra/apache-cassandra-1.0.11-bin.tar.gz
+      create  packages/cassandra/spec
+```
+
+Change `packages/cassandra/packaging` to:
+
+```
+tar xfv cassandra/apache-cassandra-1.0.11-bin.tar.gz
+cp -a apache-cassandra-1.0.11/* $BOSH_INSTALL_TARGET
+```
+
+Now create a stub for running cassandra as a job:
+
+```
+$ bosh-gen job cassandra -d java7 cassandra
+      create  jobs/cassandra
+      create  jobs/cassandra/monit
+      create  jobs/cassandra/templates/bin/cassandra_ctl
+      create  jobs/cassandra/templates/bin/monit_debugger
+      create  jobs/cassandra/templates/data/properties.sh.erb
+      create  jobs/cassandra/templates/helpers/ctl_setup.sh
+      create  jobs/cassandra/templates/helpers/ctl_utils.sh
+      create  jobs/cassandra/spec
+      create  examples/cassandra_simple
+      create  examples/cassandra_simple/default.yml
+```
+
+Look at all that goodness!
+
+A quick summary of these files:
+
+* The `monit` script uses `bin/monit_debugger` to help you debug any glitches in starting/stopping processes.
+* `ctl_setup.sh` setups up lots of common folders and env vars.
+* `ctl_utils.sh` comes from cf-release's common/utils.sh with some extra helper functions
+* `data/properties.sh.erb` is where you extract any `<%= properties.cassandra... %>` values from the deployment manifest.
+* `bin/cassandra_ctl` no longer needs to be an unreadable ERb template! Use the env variables you create in `data/properties.sh.erb` and normal bash if statements instead of ERb `<% if ... %>` templates.
+* `examples/...` is a folder for documenting example, valid deployment manifest properties for the release.
+
+In `bin/cassandra_ctl` you now change "TODO" to `cassandra` and the rest of the tutorial is left to you, dear cassandra lover.
+
+Your release is now ready to build, test and deploy:
+
+```
+bosh create release --force
+bosh upload release
+```
+
+When you create a final release, you will first need to setup your AWS credentials in `config/final.yml`
 
 ## Contributing
 

data/lib/bosh/gen/cli.rb

@@ -67,7 +67,13 @@ module Bosh
         require 'bosh/gen/generators/job_generator'
         Bosh::Gen::Generators::JobGenerator.start([name, dependencies, 'simple'])
       end
-      
+
+      desc "micro", "Create a micro job - a collection of all jobs and packages"
+      def micro
+        require 'bosh/gen/generators/micro_job_generator'
+        Bosh::Gen::Generators::MicroJobGenerator.start([])
+      end
+
       desc "template JOB FILE_PATH", 
         "Add a Job template (example FILE_PATH: config/httpd.conf)"
       def template(job_name, file_path)

data/lib/bosh/gen/generators/job_generator/templates/jobs/%job_name%_simple/templates/bin/%job_name%_ctl.tt

@@ -15,7 +15,7 @@ case $1 in
     pid_guard $PIDFILE $JOB_NAME
 
     # TODO: Run some process
-    exec TODO
+    exec TODO \
          >>$LOG_DIR/$JOB_NAME.stdout.log \
          2>>$LOG_DIR/$JOB_NAME.stderr.log
 

data/lib/bosh/gen/generators/micro_job_generator/templates/jobs/micro/prepare

@@ -0,0 +1,130 @@
+#!/usr/bin/env ruby
+#
+require 'fileutils'
+require 'yaml'
+require 'erb'
+
+TEMPLATES = "templates"
+OVERRIDE = "override"
+
+MICRO_JOB_NAME="micro"
+
+def job_dir(job)
+  File.expand_path("../../#{job}", __FILE__)
+end
+
+PREPARE_SPEC = YAML.load_file(File.expand_path("../prepare_spec", __FILE__))
+
+# Order of jobs, implicitly specifies the order in which they are started
+jobs = PREPARE_SPEC["jobs"]
+
+spec = {
+  "name" => MICRO_JOB_NAME,
+  "templates" => {
+    "post_install" => "bin/post_install"
+  },
+  "packages" => []
+}
+
+monit = []
+
+# make sure we start with an empty directory, as it will stop if
+# there are any conflicts when populating the template dir
+FileUtils.rm_rf(TEMPLATES)
+FileUtils.mkdir(TEMPLATES)
+
+jobs.each do |job|
+  # concatenate monit file
+  override = File.join(OVERRIDE, "monit", job)
+  if File.exist?(override)
+    monit_file = override
+  else
+    monit_file = "#{job_dir(job)}/monit"
+  end
+  File.open(monit_file) do |file|
+    monit += file.readlines
+    monit << "\n"
+  end
+
+  # copy templates from:
+  # * templates/<template_name>
+  # * templates/<template_dir>/<template_name>
+  Dir.glob("#{job_dir(job)}/templates/**/*").each do |file|
+    next if File.directory?(file)
+    
+    template_name = File.basename(file)
+    template_dir = File.basename(File.dirname(file))
+    # check if template file is in a nested dir
+    # if 'templates' then its not
+    if template_dir == "templates"
+      override = File.join(OVERRIDE, job, template_name)
+      dest = File.join(TEMPLATES, job)
+    else
+      override = File.join(OVERRIDE, job, template_dir, template_name)
+      dest = File.join(TEMPLATES, job, template_dir)
+    end
+    FileUtils.mkdir_p(dest) unless File.directory?(dest)
+  
+    if File.exist?(File.join(dest, template_name))
+      puts "File conflict for job #{job}: #{template_name}"
+      exit 1
+    elsif File.exist?(override)
+      FileUtils.cp(override, dest)
+    else
+      FileUtils.cp(file, dest)
+    end
+  end
+
+  # build spec
+  s = YAML.load_file("#{job_dir(job)}/spec")
+  spec["packages"] += s["packages"]
+  s["templates"].each do |src, dst|
+    spec["templates"]["#{job}/#{src}"] = "#{job}/#{dst}"
+  end
+end
+
+# write out the concatenated monit file
+File.open("monit", "w") do |file|
+  file.puts monit.join
+end
+
+
+# write out spec file
+spec["packages"] = spec["packages"].uniq
+File.open("spec", "w") do |file|
+  file.puts spec.to_yaml
+end
+
+
+
+# generate post_install file with all jobs
+# it creates links from
+# jobs/micro/<job>/bin/foo to jobs/<job>/bin/foo
+# jobs/micro/<job>/config/bar to jobs/<job>/config/bar
+post_install = <<POST_INSTALL
+#!/bin/sh
+
+for DIR in /var/vcap/jobs /var/vcap/data/jobs; do
+  cd $DIR
+  for JOB in <%= jobs.join(" ") %>; do
+    mkdir -p ${JOB}/bin ${JOB}/config ${JOB}/data ${JOB}/helpers
+    for FILE in <%= MICRO_JOB_NAME %>/$JOB/bin/*; do
+      ln -nsf /var/vcap/jobs/$FILE $JOB/bin
+    done
+    for FILE in <%= MICRO_JOB_NAME %>/$JOB/config/*; do
+      ln -nsf /var/vcap/jobs/$FILE $JOB/config
+    done
+    for FILE in <%= MICRO_JOB_NAME %>/$JOB/data/*; do
+      ln -nsf /var/vcap/jobs/$FILE $JOB/data
+    done
+    for FILE in <%= MICRO_JOB_NAME %>/$JOB/helpers/*; do
+      ln -nsf /var/vcap/jobs/$FILE $JOB/helpers
+    done
+  done
+done
+POST_INSTALL
+
+File.open("templates/post_install", "w") do |file|
+  template = ERB.new(post_install)
+  file.puts template.result
+end

data/lib/bosh/gen/generators/micro_job_generator.rb

@@ -0,0 +1,50 @@
+require 'yaml'
+require 'thor/group'
+
+module Bosh::Gen
+  module Generators
+    class MicroJobGenerator < Thor::Group
+      include Thor::Actions
+
+      def self.source_root
+        File.join(File.dirname(__FILE__), "micro_job_generator", "templates")
+      end
+      
+      def check_root_is_release
+        unless File.exist?("jobs") && File.exist?("packages")
+          raise Thor::Error.new("run inside a BOSH release project")
+        end
+      end
+      
+      def create_job
+        directory "jobs/micro"
+        chmod "jobs/micro/prepare", 0755
+      end
+      
+      def prepare_spec_defaults_all_jobs
+        jobs = Dir[File.expand_path("jobs/*")].map {|job| File.basename(job) } - ['micro']
+        spec = { "jobs" => jobs }
+        create_file "jobs/micro/prepare_spec", YAML.dump(spec)
+      end
+      
+      def gitignore
+        append_file ".gitignore", <<-IGNORE.gsub(/^\s{8}/, '')
+        jobs/micro*/monit
+        jobs/micro*/spec
+        jobs/micro*/templates/
+        IGNORE
+      end
+      
+      def readme
+        say ""
+        say "Edit "; say "jobs/micro/prepare_spec ", :yellow
+          say "with ordered list of jobs to include"
+        say "in micro job. The order of jobs implicitly specifies the order in"
+        say "which they are started."
+        say ""
+        say ""
+      end
+      
+    end
+  end
+end

data/lib/bosh/gen/generators/new_release_generator/templates/README.md.tt

@@ -1,4 +1,50 @@
-# <%= name %> - BOSH Release
+# BOSH Release for <%= name %>
 
-This project is a BOSH release for `<%= name %>`.
+## Release to your BOSH
+
+To create and upload this release to your BOSH:
+
+```
+bosh target BOSH_URL
+git clone git@github.com:<%= `whoami`.strip %>/<%= name %>.git
+cd <%= name %>
+bosh create release
+# blobs are automatically downloaded
+# name it 'cassandra-dev' or something unique to your bosh
+bosh upload release
+```
+
+### Finalizing a release
+
+If you create a final release `bosh create release --final`, you must immediately create a new development release. Yeah, this is a bug I guess.
+
+```
+[outside vagrant]
+bosh create release --final
+bosh create release
+
+[inside vagrant as vcap user]
+/vagrant/scripts/update examples/default.yml
+```
+
+
+### Alternate configurations
+
+This BOSH release is configurable during deployment with properties. 
+
+Please maintain example scenarios in the `examples/` folder.
+
+To switch between example scenarios, run `sm bosh-solo update examples/FILE.yml` with a different example scenario.
+
+## Uploading to BOSH
+
+Once you have a BOSH release that you like, you can upload it to BOSH and deploy it.
+
+```
+bosh upload release
+bosh deployment path/to/manifest.yml
+bosh deploy
+```
+
+Example `properties` for your `manifest.yml` can be taken from the examples in `examples\`
 

data/lib/bosh/gen/generators/new_release_generator.rb

@@ -42,7 +42,34 @@ module Bosh::Gen
         config_dev = { "dev_name" => name }
         create_file "config/dev.yml", YAML.dump(config_dev)
 
-        config_private = { "blobstore_secret" => 'BLOBSTORE_SECRET' }
+        case blobstore_type
+        when :local
+          config_private = { 
+            "blobstore" => {
+              "simple" => {
+                "user" => "USER",
+                "password" => "PASSWORD"
+              }
+            }
+          }
+        when :s3
+          config_private = { 
+            "blobstore" => {
+              "s3" => {
+                "access_key_id" => "READWRITE_AWS_ACCESS_KEY",
+                "secret_access_key" => "READWRITE_AWS_SECRET_ACCESS_KEY"
+              }
+            }
+          }
+        when :atmos
+          config_private = { 
+            "blobstore" => {
+              "atmos" => {
+                "secret" => "SECRET"
+              }
+            }
+          }
+        end
         create_file "config/private.yml", YAML.dump(config_private)
 
         case blobstore_type
@@ -54,13 +81,12 @@ module Bosh::Gen
             }
           }
         when :s3
-          config_private = { "blobstore_secret" => 'BLOBSTORE_SECRET' }
           config_final = { "blobstore" => {
               "provider" => "s3",
               "options" => {
                 "bucket_name" => "BOSH",
-                "access_key_id" => "AWS_ACCESS_KEY",
-                "secret_access_key" => "AWS_SECRET_ACCESS_KEY",
+                "access_key_id" => "READONLY_AWS_ACCESS_KEY",
+                "secret_access_key" => "READONLY_AWS_SECRET_ACCESS_KEY",
                 "encryption_key" => "PERSONAL_RANDOM_KEY",
               }
             }

data/lib/bosh/gen/version.rb

@@ -1,5 +1,5 @@
 module Bosh
   module Gen
-    VERSION = "0.9.0"
+    VERSION = "0.10.0"
   end
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: bosh-gen
 version: !ruby/object:Gem::Version
-  version: 0.9.0
+  version: 0.10.0
   prerelease: 
 platform: ruby
 authors:
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2012-08-11 00:00:00.000000000 Z
+date: 2012-08-20 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: thor
@@ -157,6 +157,8 @@ files:
 - lib/bosh/gen/generators/job_generator/templates/jobs/%job_name%_simple/templates/helpers/ctl_setup.sh
 - lib/bosh/gen/generators/job_generator/templates/jobs/%job_name%_simple/templates/helpers/ctl_utils.sh
 - lib/bosh/gen/generators/job_template_generator.rb
+- lib/bosh/gen/generators/micro_job_generator.rb
+- lib/bosh/gen/generators/micro_job_generator/templates/jobs/micro/prepare
 - lib/bosh/gen/generators/new_release_generator.rb
 - lib/bosh/gen/generators/new_release_generator/templates/README.md.tt
 - lib/bosh/gen/generators/new_release_generator/templates/Rakefile
@@ -235,7 +237,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
       version: '0'
       segments:
       - 0
-      hash: 2628842034372834781
+      hash: -108278516028601848
 required_rubygems_version: !ruby/object:Gem::Requirement
   none: false
   requirements:
@@ -244,7 +246,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
       segments:
       - 0
-      hash: 2628842034372834781
+      hash: -108278516028601848
 requirements: []
 rubyforge_project: 
 rubygems_version: 1.8.24