bosh-cloudfoundry 0.6.1 → 0.7.0.alpha.1

data/config/defaults.yml

@@ -1,6 +0,0 @@
----
-
-logging:
-  level: INFO
-
-base_systems_dir: /var/vcap/store/systems

data/docs/concept.md

@@ -1,141 +0,0 @@
-If you're new to BOSH, Cloud Foundry and PaaS in general you might find this as confusing as I did. So I'll try to explain some of the concepts.
-
-# The parts
-There are several parts in action here. Let me take you for a quick tour.
-
-## Cloud Foundry
-
-[Cloud Foundry](http://cloudfoundry.org) is a Open Source PaaS solution. It is available as a public cloud at [http://cloudfoundry.com](http://cloudfoundry.com) operated by VMWare, but it is also a solution you can bring in-house and install on your own servers, or even operate on Amazon like [AppFog](http://appfog.com) does.
-
-* __Cloud Controller__ is the main component in Cloud Foundry and the server you point your vmc command to. It can be split up into separate VMs, but this bootstrap keeps it all on one instance.
-* __DEA__ is the compute instance. In the initial setup the Cloud Controller is also a DEA. But you can basically view it as VMs that will handle and serve your applications.
-* __Service__ is the concept Cloud Foundry uses to describe added functionality to your application like Postgres or MongoDB.
-
-
-## BOSH 
-
-[BOSH](https://github.com/cloudfoundry/bosh) is a cloud oriented tool chain for releasing, deployment and lifecycle management of applications. It abstracts away the underlying IaaS solution (AWS, OpenStack, VMWare) and performs requests for new virtual machines etc. It also handles the distribution of "Jobs" to the virtual machines in the BOSH setup. In some ways you can say that it eliminates the need for Puppet/Chef, but it uses some different concepts. It also covers more, like the provisioning of virtual machines.
-
-* __Job__ is BOSH's concept of grouping functionality together. In the normal instance you can think of a Job as a composition of installed software and configuration that a virtual machine will have. So two virtual machines with the same Job attached will be identical.
-* __Template__ is BOSH's concept for defining a software setup. CloudController is one such template, and dea is another one.
-* __Stemcell__ is BOSH's concept for a virtual machine image. For Amazon it equals and AMI. It is a template that is booted and becoms an instance of an virtual machine.
-* __Resource Pool__ is basically a collection of virtual machines, that you can reference/assign jobs to. The have the same stemcell and configuration (eg. AWS size).
-* __Compilation instance__ is instances that BOSH uses when deploying a new part. At first it is used to compile everything that is needed for VCAP and DEA. If you add a template like Postgres you'll see that it compiles the components used for that. I think this is something that resembles rpm or deb packages, and that they get compiled on your Stemcells to ensure compatibility when running on your cloud.
-
-## Inception VM
-Inception VM is a Virtual Machine used for bootstrapping BOSH.
-
-# Building a runnable Cloud Foundry
-This Bootstrap takes you throug the steps needed to have Cloud Foundry running and serving requests. It launches BOSH which then is used to deploy Cloud Foundry. BOSH is the only officially supported way of deploying Cloud Foundry, even though Chef scripts is maintained by the community.
-
-The way it operates is illustrated in the figure below:
-
-![BOSH-CF initial deploy](BOSH-cf-initial-create.png)
-
-1. The bosh-bootstrap gem on "My Computer" creates an Inception virtual machine from an Ubuntu image.
-2. The Inception VM creates the BOSH controller
-3. The BOSH controller creates virtual machines for Cloud Foundry
-4. The BOSH controller deploys Jobs to the separate instances
-
-In the initial configuration the Cloud Controller acts as both a controller and a compute instance where your applications run.
-
-# Scaling
-Now if the above description was crystal clear, you'll know where the scaling happens. ;) It happens to the far right. The figure below shows how everything looks when I've added a couple of compute instances.
-
-![BOSH-CF scaled deploy](BOSH-cf-scaled.png)
-
-In the case above we have three resource pools. They are called core, DEA and postgres. This is that standard way that the BOSH cf plugin does it. The following has been done here:
-
-* ```bosh cf change deas 3``` creates a pool with 3 virtual machines and assigns the DEA job to them. It also removes the DEA template from the core job.
-* ```bosh cf add service postgresql 2``` creates a pool with 2 virtual machines and assigns the postgresql job to them.
-* The dotted boxes above are compile instances that BOSH will create temporarily to compile the necessary components. See the BOSH description above.
-
-With BOSH it is possible to allocate the postgres templates on the already existing DEA instances and thus save costs on Amazon. It all comes down to what kind of scale you need, and also separation from load issues.
-
-It is the responsibility of the separate application to handle clustering and communication between the instances.
-
-## Implementation
-
-```bosh cf change deas 1``` will add a resource pool and set up jobs. This is from the cf plugin to BOSH.
-
-It adds a resource pool:
-
-```
-- name: dea
-  network: default
-  size: 2
-  stemcell:
-    name: bosh-stemcell
-    version: 0.7.0
-  cloud_properties:
-    region: eu-west-1
-    availability_zone: eu-west-1b
-    instance_type: m1.large
-```
-
-Size 2 specifies to launch two virtual machines for this pool.
-
-It then sets up a job:
-
-```
-- name: dea
-  template:
-  - dea
-  instances: 2
-  resource_pool: dea
-  networks:
-  - name: default
-    default:
-    - dns
-    - gateway
-```
-
-The instances setting tells it to use two instances in the dea resource pool. For our case this matches the size of the pool exactly.    
-
-After the changes has been done a ```bosh deploy``` will perform the needed changes to your BOSH setup.
-
-### Using your Cloud Controller as a compute instance
-
-If you take a look at the job named "core" before you do the adding of DEAs you will see that it has a template named dea. If you leave that in the list your Cloud Controller will also be a compute instance serving deployed applications.
-
-# Tips
-
-* Provision an Elastic IP from Amazon as early as possible and assign a DNS entry with wildcard. Example: *.cf.mydomain.com . It will take some time for the DNS to propagate (actually your old *.mydomain.com record to time out if you have one).
-* If you're deploying to a different region than us-east-1 check out [this bug.](https://github.com/StarkAndWayne/bosh-cloudfoundry/issues/100) Make the changes to the configuration just before you do ```bosh deploy```.
-* If you want to scale out by adding more instances you need to tweak the FW rules. Check [this bug.](https://github.com/StarkAndWayne/bosh-cloudfoundry/issues/112)
-* You might need to reduce the number of compile instances to avoid Amazon API errors. Check [this bug.](https://github.com/StarkAndWayne/bosh-cloudfoundry/issues/111)
-* If you're having problems with ```vmc push``` check out [this bug.](https://github.com/StarkAndWayne/bosh-cloudfoundry/issues/49)
-* Once you're up and running [test your setup with a simple Sinatra application](http://docs.cloudfoundry.com/tools/vmc/installing-vmc.html#creating-a-simple-sinatra-application).
-
-# Some resources
-
-Use these for some pointers for in depth understanding. The rest of this document tries to explain the basic concepts without getting into the really deep details.
-
-## Documentation
-
-* [Cloud Foundry documentation](http://cloudfoundry.github.com/)
-* [Cloud Foundry public cloud (commercial version) docs](http://docs.cloudfoundry.com/getting-started.html)
-* [Cloud Foundry community wiki](https://github.com/mrdavidlaing/cf-docs-contrib/wiki)
-
-## Community
-
-* [BOSH users Mailinglist](https://groups.google.com/a/cloudfoundry.org/forum/#!forum/bosh-users)
-* [Cloud Foundry Mailinglist](https://groups.google.com/a/cloudfoundry.org/forum/#!forum/vcap-dev)
-
-## Source code
-
-* [Cloud Foundry release](https://github.com/cloudfoundry/cf-release)
-* [Cloud Foundry jobs (within the release repo)](https://github.com/cloudfoundry/cf-release/tree/master/jobs)
-* [Cloud Foundry VCAP (core component)](https://github.com/cloudfoundry/vcap)
-
-# TODO/Questions
-
-* Is the BOSH controller a MicroBosh? Another name?
-* What is a microbosh?
-* Is the BOSH controller a part of BOSH, or something that just controls it?
-* Does the core Cloud Foundry instances scale out?
-* How do you enable HA for the core?
-* You run BOSH commands on the Inception VM, not BOSH Controller. Is it meant to be run on your computer? Is network the reason it isn't?
-* Add some nice to know commands
-* How can we lock this to a certain "release"?
-* Can we describe how the source is fetched and what happens to it?

data/lib/bosh-cloudfoundry.rb

@@ -1,33 +0,0 @@
-require "bosh-cloudfoundry/version"
-
-module Bosh; module CloudFoundry; end; end
-
-require "logger"
-require "common/common"
-require "common/thread_formatter"
-require "cli"
-
-# for generating password
-require 'openssl'
-
-# for the #sh helper
-require "rake"
-require "rake/file_utils"
-
-# for validating DNS -> IP setups
-require 'net/dns'
-
-# for:
-# * validating compute flavors
-# * provisioning IP addresses
-require "fog"
-require 'fog/aws/models/compute/flavors'
-
-# CLI mixins
-require "bosh-cloudfoundry/config_options"
-require "bosh-cloudfoundry/bosh_release_manager"
-
-require "bosh-cloudfoundry/config"
-require "bosh-cloudfoundry/providers"
-
-require "bosh-cloudfoundry/system_deployment_manifest_renderer"

data/lib/bosh-cloudfoundry/bosh_release_manager.rb

@@ -1,156 +0,0 @@
-# Copyright (c) 2012-2013 Stark & Wayne, LLC
-
-module Bosh; module CloudFoundry; end; end
-
-# There are two concepts of "latest".
-# * for upload: "latest" is the highest release in cf-release
-# * for manifest creation: "latest" is the highest release already uploaded to the BOSH
-module Bosh::CloudFoundry::BoshReleaseManager
-  
-  # @return [Array] BOSH releases available in target BOSH
-  # [{"name"=>"appcloud", "versions"=>["124", "126"], "in_use"=>[]}]
-  def bosh_releases
-    @bosh_releases ||= releases = director.list_releases
-  end
-
-  # @return [Array] BOSH release names available in target BOSH
-  def bosh_release_names
-    @bosh_release_names ||= bosh_releases.map { |rel| rel["name"] }
-  end
-
-  # @return [Array] BOSH release versions for specific release name in target BOSH
-  def bosh_release_versions(release_name)
-    if release = bosh_releases.find { |rel| rel["name"] == release_name }
-      release["versions"]
-    else
-      []
-    end
-  end
-
-  def release_name_version
-    "#{release_name}/#{release_version}"
-  end
-
-  # @return [Version String] BOSH version number; converts 'latest' into actual version
-  # TODO implement this; map "latest" to highest uploaded release in BOSH
-  # return "unknown" if BOSH has no releases of this name yet
-  def effective_release_version
-    release_version.to_s
-  end
-
-  # for upload, "latest" is the newest release in cf-release
-  def upload_final_release
-    release_number = use_latest_release? ? 
-      latest_uploadable_final_release_number :
-      release_version
-    chdir(cf_release_branch_dir) do
-      bosh_cmd "upload release releases/appcloud-#{release_number}.yml"
-    end
-    @bosh_releases = nil # reset cache
-  end
-
-  # Looks at the last line of releases/index.yml in cf-release 
-  # for the latest release number that could be uploaded
-  # @returns [String] a number such as "126"
-  def latest_uploadable_final_release_number
-    chdir(cf_release_branch_dir) do
-      return `tail -n 1 releases/index.yml | awk '{print $2}'`.strip
-    end
-  end
-
-  # Looks at the last line of releases/index.yml in cf-release 
-  # for the latest release number that could be uploaded
-  # @returns [String] a dev release code such as "126.8-dev"
-  def latest_uploadable_dev_release_number
-    chdir(cf_release_branch_dir) do
-      return `tail -n 1 dev_releases/index.yml | awk '{print $2}'`.strip
-    end
-  end
-
-  # @returns [String] absolute path to latest release to be uploaded
-  def latest_dev_release_filename
-    dev_release_number = latest_uploadable_dev_release_number
-    return nil unless dev_release_number.size > 0
-    File.join(cf_release_dir, "#{release_name}-#{dev_release_number}.yml")
-  end
-
-  def create_and_upload_dev_release
-    release_name = default_dev_release_name
-    chdir(cf_release_branch_dir) do
-      write_dev_config_file(release_name)
-      sh "bosh -n --color create release --with-tarball --force"
-      sh "bosh -n --color upload release"
-    end
-    @bosh_releases = nil # reset cache
-  end
-
-  def write_dev_config_file(release_name)
-    dev_config_file = "config/dev.yml"
-    if File.exist?(dev_config_file)
-      dev_config = YAML.load_file(dev_config_file)
-    else
-      dev_config = {}
-    end
-    dev_config["dev_name"] = release_name
-    File.open(dev_config_file, "w") { |file| file << dev_config.to_yaml }
-  end
-
-  # clones or updates cf-release for a specific branch
-  # For all defaults, clones into:
-  # * /var/vcap/store/releases/cf-release/master (cf_release_branch_dir)
-  #
-  # Uses:
-  # * releases_dir (e.g. '/var/vcap/store/releases')
-  # * cf_release_branch (e.g. 'staging')
-  # * cf_release_branch_dir (e.g. '/var/vcap/store/releases/cf-release/staging')
-  def clone_or_update_cf_release
-    raise "invoke #set_cf_release_branch(branch) first" unless cf_release_branch_dir
-    if File.directory?(cf_release_branch_dir)
-      chdir(cf_release_branch_dir) do
-        sh "git pull origin #{cf_release_branch}" # recursive is below
-      end
-    else
-      chdir(releases_dir) do
-        sh "git clone -b #{cf_release_branch} #{cf_release_git_repo} #{cf_release_branch_dir}"
-        chdir(cf_release_branch_dir) do
-          sh "git update-index --assume-unchanged config/final.yml 2>/dev/null"
-        end
-      end
-    end
-  end
-
-  # when creating a dev release, need to pull down submodules
-  def prepare_cf_release_for_dev_release
-    chdir(cf_release_branch_dir) do
-      say "Rewriting all git:// & git@ to https:// ..."
-      # Snippet written by Mike Reeves <swampfoxmr@gmail.com> on bosh-users mailing list
-      # Date 2012-12-06
-      sh "sed -i 's#git@github.com:#https://github.com/#g' .gitmodules"
-      sh "sed -i 's#git://github.com#https://github.com#g' .gitmodules"
-      sh "git submodule update --init --recursive"
-    end
-  end
-
-  def default_release_name
-    "appcloud"
-  end
-
-  def default_dev_release_name(branch_name=cf_release_branch)
-    suffix = "-#{branch_name || 'dev'}"
-    default_release_name + suffix
-  end
-
-  def switch_to_development_release
-    system_config.release_name = default_dev_release_name(cf_release_branch)
-    system_config.release_version = "latest"
-    system_config.release_type = "dev"
-    system_config.save
-  end
-
-  def switch_to_final_release
-    system_config.release_name = default_release_name
-    system_config.release_version = "latest"
-    system_config.release_type = "final"
-    system_config.save
-  end
-end

data/lib/bosh-cloudfoundry/config.rb

@@ -1,6 +0,0 @@
-require "bosh-cloudfoundry/config/common_config"
-require "bosh-cloudfoundry/config/system_config"
-require "bosh-cloudfoundry/config/microbosh_config"
-require "bosh-cloudfoundry/config/dea_config"
-require "bosh-cloudfoundry/config/postgresql_service_config"
-require "bosh-cloudfoundry/config/redis_service_config"

data/lib/bosh-cloudfoundry/config/common_config.rb

@@ -1,27 +0,0 @@
-# Copyright (c) 2012-2013 Stark & Wayne, LLC
-
-require "cli"
-
-module Bosh; module CloudFoundry; module Config; end; end; end
-
-module Bosh::CloudFoundry::Config
-  class CommonConfig < Bosh::Cli::Config
-
-    [
-      :base_systems_dir, # e.g. /var/vcap/store/systems
-      :target_system,        # e.g. /var/vcap/store/systems/production
-      :bosh_git_repo,    # e.g. "git://github.com/cloudfoundry/bosh.git"
-      :releases_dir,     # e.g. /var/vcap/store/releases
-      :stemcells_dir,    # e.g. /var/vcap/store/stemcells
-      :repos_dir,        # e.g. /var/vcap/store/repos
-    ].each do |attr|
-      define_method attr do
-        read(attr, false)
-      end
-
-      define_method "#{attr}=" do |value|
-        write_global(attr, value)
-      end
-    end
-  end
-end

data/lib/bosh-cloudfoundry/config/dea_config.rb

@@ -1,161 +0,0 @@
-# Copyright (c) 2012-2013 Stark & Wayne, LLC
-
-module Bosh; module CloudFoundry; module Config; end; end; end
-
-class Bosh::CloudFoundry::Config::DeaConfig
-  attr_reader :system_config
-
-  def initialize(system_config)
-    @system_config = system_config
-  end
-
-  def self.build_from_system_config(system_config)
-    system_config.dea ||= {}
-    new(system_config)
-  end
-
-  def bosh_provider_name
-    @system_config.bosh_provider
-  end
-
-  def update_count_and_flavor(server_count, server_flavor)
-    self.dea_server_count = server_count.to_i
-    self.dea_server_flavor = server_flavor.to_s
-    self.save
-  end
-
-  # Determine ow many DEA servers are required
-  # based on the system configuration
-  def dea_server_count
-    @system_config.dea["count"] || 0
-  end
-
-  def dea_server_count=(count)
-    @system_config.dea["count"] = count
-  end
-
-  def dea_server_flavor
-    @system_config.dea["flavor"]
-  end
-
-  def dea_server_flavor=(flavor)
-    @system_config.dea["flavor"] = flavor
-  end
-
-  def save
-    @system_config.save
-  end
-
-  # Adds additional cf-release jobs into the core server (the core job in the manifest)
-  def add_core_jobs_to_manifest(manifest)
-    if dea_server_count == 0
-      @core_job ||= manifest["jobs"].find { |job| job["name"] == "core" }
-      @core_job["template"] << "dea"
-    end
-  end
-
-  # Adds resource pools to the target manifest, if dea_server_count > 0
-  #
-  # - name: dea
-  #   network: default
-  #   size: 2
-  #   stemcell: 
-  #     name: bosh-stemcell
-  #     version: 0.7.0
-  #   cloud_properties: 
-  #     instance_type: m1.xlarge
-  def add_resource_pools_to_manifest(manifest)
-    if dea_server_count > 0
-      resource_pool = {
-        "name" => "dea",
-        "network" => "default",
-        "size" => dea_server_count,
-        "stemcell" => {
-          "name" => @system_config.stemcell_name,
-          "version" => @system_config.stemcell_version
-        },
-        # TODO how to create "cloud_properties" per-provider?
-        "cloud_properties" => {
-          "instance_type" => dea_server_flavor
-        }
-      }
-      manifest["resource_pools"] << resource_pool
-    end
-  end
-
-  # Jobs to add to the target manifest
-  #
-  # - name: dea
-  #   template: 
-  #   - dea
-  #   instances: 2
-  #   resource_pool: dea
-  #   networks: 
-  #   - name: default
-  #     default: 
-  #     - dns
-  #     - gateway
-  def add_jobs_to_manifest(manifest)
-    if dea_server_count > 0
-      job = {
-        "name" => "dea",
-        "template" => ["dea"],
-        "instances" => dea_server_count,
-        "resource_pool" => "dea",
-        # TODO are these AWS-specific networks?
-        "networks" => [{
-          "name" => "default",
-          "default" => ["dns", "gateway"]
-        }]
-      }
-      manifest["jobs"] << job
-    end
-  end
-
-  # Add extra configuration properties into the manifest
-  # to configure the allocation of RAM to the DEA
-  def merge_manifest_properties(manifest)
-    manifest["properties"]["dea"] ||= {}
-    manifest["properties"]["dea"] = {
-      "max_memory" => max_memory
-    }
-  end
-
-  # The RAM for a dedicated DEA node
-  # else the RAM of the core/0 VM
-  # minus the +preallocated_ram+.
-  def max_memory
-    if dea_server_count > 0
-      max_memory_for_dedicated_dea
-    else
-      dea_ram_for_core_vm_flavor
-    end
-  end
-
-  # @return [Integer] available ram for running CloudFoundry apps
-  def max_memory_for_dedicated_dea
-    ram_for_server_flavor - preallocated_ram
-  end
-
-  # @return [Integer] the ballpark ram for DEA, BOSH agent, etc
-  def preallocated_ram
-    300
-  end
-
-  def dea_ram_for_core_vm_flavor
-    ram_for_core_vm_flavor - preallocated_ram
-  end
-
-  def ram_for_core_vm_flavor
-    provider.ram_for_server_flavor(system_config.core_server_flavor)
-  end
-
-  def ram_for_server_flavor
-    provider.ram_for_server_flavor(dea_server_flavor)
-  end
-
-  # a helper object for the target BOSH provider
-  def provider
-    @provider ||= Bosh::CloudFoundry::Providers.for_bosh_provider_name(system_config)
-  end
-end