bosh-cloudfoundry 0.6.1 → 0.7.0.alpha.1

data/.gitignore

@@ -15,4 +15,7 @@ spec/reports
 test/tmp
 test/version_tmp
 tmp
-.DS_Store
+bosh_release/.final_builds/jobs/*/*.tgz
+bosh_release/.final_builds/packages/*/*.tgz
+bosh_release/releases/*.tgz
+deployments

data/.rspec

@@ -1,3 +1,3 @@
 --format progress
 --color
---profile
+--require spec_helper

data/.travis.yml

@@ -1,5 +1,11 @@
 language: ruby
+script: bundle exec rake
 rvm:
-  - 1.9.3
-  - rbx-19mode
-  # - ruby-head - generates "Cannot find Syck parser for YAML"
+  - ruby-1.9.3
+  - ruby-2.0.0
+notifications:
+  email:
+    recipients:
+      - drnicwilliams@gmail.com
+    on_success: change
+    on_failure: always

data/ChangeLog.md

@@ -1,55 +1,20 @@
 # Change Log
 
-Create and manage your Cloud Foundry deployments on AWS and OpenStack with a simple command line tool (a plugin for BOSH CLI in fact). Start small, grow big, and constantly upgrade your various Cloud Foundry deployments.
+## v0.7 - now supporting Cloud Foundry v2!
 
-## v0.6
+`bosh-cloudfoundry` is back, rewritten, and supporting Cloud Foundry v2.
 
-There is now a write up of [the concepts](https://github.com/StarkAndWayne/bosh-cloudfoundry/blob/master/docs/concept.md) and constructs being deployed from Anders Sveen!
+```
+$ bosh prepare cf
+$ bosh create cf --dns mycloud.com --public-ip 1.2.3.4
+```
 
-Tests are running on Travis CI. We have green builds!
+As a rewrite, v0.7 initially implements fewer features than were in v0.6. It takes advantage of the long-awaited first final release of [cf-release](https://github.com/cloudfoundry/cf-release) for Cloud Foundry v2 (v132).
 
-Automatically uploading base stemcell 0.7.0; rather than the very new 1.5.0-pre2 which only work with the also very new microbosh 1.5.0-pre2.
+The rewrite introduces some new implementation/feature concepts:
 
-Faster when using a final release because it no longer pulls down all the submodules; and some better code for handling the switch between a development and final release; but...
+* using `bosh diff` (aka biff) to generate the deployment file
+* bundles all final releases into the project & distributed rubygem/plugin (no runtime dependency on cf-release git repository; only the public blobstore)
+* templates are versioned for the final releases
 
-Still need to create development releases from cf-release's master branch. Hopefully all the required patches will be in cf-release's 130 release coming soon!
-
-### v0.6.1
-
-Constrain which fog version is installed to match bosh_cli. [thx @bonzofenix]
-
-Little bit of README clean up.
-
-## v0.5
-
-Gerrit is dead. Long live gerrit. This release is for everyone who is getting started or wants to upgrade. Gerrit is dead.
-
-A few days ago the core Cloud Foundry team shut down the gerrit hosting of cf-release and several patches that we needed to run Cloud Foundry on AWS and/or OpenStack. Fortunately, all the patches have been merged into the staging branch of the cf-release on github. This new release of bosh-cloudfoundry defaults to creating a development (non-final) release of cf-release from its staging branch.
-
-The `bosh cf upload release` command now has a `--branch BRANCH` for uploading a release based on a different branch. It currently defaults to `staging`.
-
-In future, when cf-release eventually merges all the required patches into master branch, we will switch to defaulting to `master`.
-
-And finally, the big requirement for going v1.0, is for all the required patches to be included in a final release of cf-release.
-
-### v0.5.1
-
-Fixes a bug in creating dev releases of cf-release based on branches that aren't "master". It now correctly clones each staging branch into its own repository, built solely off the targeted branch.
-
-## v0.4
-
-Defaults to patched dev release (since a cf-release final release doesn't work).
-
-### v0.4.1
-
-Workaround for cf-release HEAD change - providing `properties.uaa.scim` property now.
-
-## v0.3
-
-Additions:
-
-* OpenStack support! [thx @frodenas; tested by @danhighman] many hugs & kisses to you both!
-
-Fixes:
-
-* changed `health_manager` to `health_manager_next` job (legacy hm was removed in [cf-release patch](https://github.com/cloudfoundry/cf-release/commit/cba60f2e2dee13b7e09eb178eec72aa084a15b1a))
+The latter means that new versions of this rubygem can be published that are backwards compatible with aging deployments of Cloud Foundry. There should not be any forced coupling of old `bosh-cloudfoundry` to old `cf-release` final releases.

data/Gemfile

@@ -1,8 +1,14 @@
-source "https://rubygems.org"
+source 'https://rubygems.org'
+source 'https://s3.amazonaws.com/bosh-jenkins-gems/' # for prerelease bosh_cli
 
-# Specify your gem's dependencies in bosh-cloudfoundry.gemspec
+# Specify your gem's dependencies in redis-cf-plugin.gemspec
 gemspec
 
-gem "awesome_print"
-gem "rb-fsevent", "~> 0.9.1"
-gem "guard-rspec"
+bosh_cli_path = File.expand_path("~/gems/cloudfoundry/bosh/bosh_cli")
+if File.directory?(bosh_cli_path)
+  gem "bosh_cli", path: bosh_cli_path
+end
+
+group :development do
+  gem "guard-rspec"
+end

data/Guardfile

@@ -1,10 +1,7 @@
-# A sample Guardfile
-# More info at https://github.com/guard/guard#readme
-
-guard 'rspec' do
-  watch(%r{^spec/.+_spec\.rb$})
-  watch(%r{^lib/bosh/cli/commands/cf.rb$})    { |m| "spec/unit/cf_command_spec.rb" }
-  watch(%r{^lib/bosh-cloudfoundry/(.+)\.rb$}) { |m| "spec/unit/#{m[1]}_spec.rb" }
+guard 'rspec', spec_paths: ["spec"] do
+  watch(%r{^spec/(.+_spec)\.rb$})
+  watch(%r{^lib/bosh/cli/commands/(.+)\.rb$})    { |m| "spec/plugin_spec.rb" }
+  watch(%r{^lib/bosh/cloudfoundry/(.+)\.rb$})    { |m| "spec/#{m[1]}_spec.rb" }
   watch('spec/spec_helper.rb')  { "spec" }
 end
 

data/LICENSE.txt

@@ -1,4 +1,4 @@
-Copyright (c) 2012 Dr Nic Williams
+Copyright (c) 2013 Dr Nic Williams
 
 MIT License
 
@@ -19,4 +19,4 @@ MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
 NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
 LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
 OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
-WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -1,238 +1,122 @@
-# Bosh::CloudFoundry
+# Run Cloud Foundry on AWS or OpenStack
 
-You want Cloud Foundry? You can now create, scale, upgrade and patch one or more Cloud Foundry deployments using very simple, easy to learn and remember CLI commands.
+This is a simple `bosh` CLI plugin to boot up Cloud Foundry and then grow and upgrade and maintain it. Initially runs on AWS or OpenStack via bosh.
 
-Currently supports AWS and OpenStack only. vSphere and vCloud support will require someone to tell me that they really want it.
-
-[![Build Status](https://travis-ci.org/StarkAndWayne/bosh-cloudfoundry.png?branch=master)](https://travis-ci.org/StarkAndWayne/bosh-cloudfoundry) [![Code Climate](https://codeclimate.com/github/StarkAndWayne/bosh-cloudfoundry.png)](https://codeclimate.com/github/StarkAndWayne/bosh-cloudfoundry)
-
-## Zero to Cloud Foundry in a few hours
-
-If you have an AWS or OpenStack account then you can deploy a running Cloud Foundry VM by following this very simple tutorial. It first uses the very powerful [bosh-bootstrap](https://github.com/StarkAndWayne/bosh-bootstrap) tool to deploy a BOSH, and then using this project (a plugin for the BOSH CLI) it will do everything necessary to deploy Cloud Foundry via your shiny new, empty BOSH.
-
-**You should not need to know anything before running this tutorial.** If you do this tutorial and you feel it was too complex or you needed to know something additional, please [create a ticket](https://github.com/StarkAndWayne/bosh-cloudfoundry/issues).
+Example create/delete scenario:
 
 ```
-# on your laptop
-gem install bosh-bootstrap
-bosh-bootstrap deploy
-# prompts for AWS/OpenStack credentials
-# and some other fields that you can
-# just press Enter for.
-
-# SSH into inception VM (also see 'tmux' and 'mosh' commands)
-bosh-bootstrap ssh
-
-bosh cf prepare system production
-# prompts for a DNS host for your CloudFoundry, such as mycompany.com
-# will generate a new IP address
-# now setup your DNS for *.mycompany.com => new IP address
-# the re-run:
-bosh cf prepare system production
-
-# use special "cf deploy" to install VMC & create initial VMC user
-bosh cf deploy
+$ bosh prepare cf
+$ bosh create cf --dns mycloud.com --public-ip 1.2.3.4
 
-# now we can grow our single VM deployment
-
-bosh cf change deas 1
-bosh cf add service postgresql
-bosh deploy
+...
+$ bosh delete cf
 ```
 
-Overtime, as you add more DEAs and other service nodes, your set of VMs might look like:
+The deployed Cloud Foundry does not include any data or messaging services for the user applications. These are available as add-ons coming soon.
 
-```
-$ bosh vms
-+-----------------------------+---------+---------------------------+-----------------------------+
-| Job/index                   | State   | Resource Pool             | IPs                         |
-+-----------------------------+---------+---------------------------+-----------------------------+
-| core/0                      | running | core                      | 10.4.70.116, 54.235.200.165 |
-| dea/0                       | running | dea                       | 10.4.49.7                   |
-| dea/1                       | running | dea                       | 10.111.39.12                |
-| postgresql_m1_medium_free/0 | running | postgresql_m1_medium_free | 10.4.71.164                 |
-| postgresql_m1_small_free/0  | running | postgresql_m1_small_free  | 10.110.83.128               |
-| postgresql_m1_small_free/1  | running | postgresql_m1_small_free  | 10.189.103.26               |
-+-----------------------------+---------+---------------------------+-----------------------------+
-```
+[![Build Status](https://travis-ci.org/StarkAndWayne/bosh-cloudfoundry.png?branch=v0.7)](https://travis-ci.org/StarkAndWayne/bosh-cloudfoundry)
 
 ## Requirements
 
-* Ruby 1.9
-* BOSH running on AWS (other CPIs coming)
-* Unix-based local system (Windows support for bosh-bootstrap is a priority)
+You must use the same BOSH for deploying redis as you use to deploy Cloud Foundry (unless your DNS is configured to be shared across the two BOSH).
 
-## Usage
+It is also required that you have login access to the same BOSH being used to deploy your Cloud Foundry.
 
-The tool is very simple to use and to get CloudFoundry deployed on a small set of initial servers.
+Confirm this by running:
 
 ```
-gem install bosh-cloudfoundry
-bosh cf prepare system demo
-bosh cf deploy
+$ bosh status
+$ bosh deployments
 ```
 
-The above can be run the following from your local laptop or a server, such as an `bosh-bootstrap` inception server. 
+The former will confirm you are targeting a bosh. The latter will display the deployments. One of which should be your Cloud Foundry.
 
-The latter is preferred. As the Cloud Foundry BOSH release (`cf-release`) is 1.5 Gb, it may be preferable to manage your Cloud Foundry deployments from your inception server, as created/prepared via `bosh-bootstrap`.
+## Installation
 
-```
-bosh-bootstrap deploy --latest-stemcell
-bosh-bootstrap ssh
+Install via RubyGems:
 
-# now on the inception VM
-gem install bosh-cloudfoundry
-export TMPDIR=/var/vcap/store/tmp
-bosh cf prepare system production
-# prompts for a DNS host for your CloudFoundry, such as mycompany.com
-bosh cf change deas 1
-bosh cf add service postgresql 1
-bosh cf deploy
+```
+$ gem install bosh_cli -v "~> 1.5.0.pre" --source https://s3.amazonaws.com/bosh-jenkins-gems/ 
+$ gem install bosh-cloudfoundry
 ```
 
-During `bosh cf prepare system production`, it will automatically upload the latest release of CloudFoundry (the latest final [BOSH release](http://github.com/cloudfoundry/cf-release)) and the latest stable stemcell (becomes the base AMI for AWS, for example).
+The `bosh_cli` gem is currently only available from S3, rather than RubyGem itself. So it needs to be installed first.
 
-NOTE: `export TMPDIR=/var/vcap/store/tmp` tells the upload process to use the larger mounted volume at `/var/vcap/store`. 
+## Usage
 
-You can upload a more recent stemcell or create a new one from source, respectively:
+Each time you install the latest `bosh-cloudfoundry` gem you may want to re-upload the latest available Cloud Foundry bosh release to your bosh. If no newer release is available then nothing good nor bad will occur.
 
 ```
-bosh cf upload stemcell --latest
-bosh cf upload stemcell --custom
+$ bosh prepare cf
+Uploading new cf release to bosh...
 ```
 
-You can upload a more recent final BOSH release of CloudFoundry, or create a non-final version from the very latest commits to the CloudFoundry BOSH release, respectively:
+To create/provision a new Cloud Foundry you run the following command. By default, it will select the smallest possible deployment size.
 
 ```
-bosh cf upload release
-bosh cf upload release --dev
+$ bosh create cf --dns mycloud.com --public-ip 1.2.3.4
+$ bosh create cf --dns mycloud.com --public-ip 1.2.3.4 --size medium
+$ bosh create cf --dns mycloud.com --public-ip 1.2.3.4 --size large
+$ bosh create cf --dns mycloud.com --public-ip 1.2.3.4 --size xlarge
 ```
 
-### All available commands
+By default the core Cloud Foundry server is assigned a 4096 Mb persistent volume/disk. This can be changed later as your Cloud Foundry deployment grows.
 
-Prefix each with `bosh`:
+NOTE: By default, the `default` security group is used.
 
-```
-cf prepare system [<name>] [--core-ip ip] [--root-dns dns] 
-                  [--core-server-flavor flavor] [--release-name name] [--release-version 
-                  version] [--stemcell-name name] [--stemcell-version version] 
-                  [--admin-emails email1,email2] [--skip-validations] 
-    create CloudFoundry system 
-    --core-ip ip                 Static IP for CloudController/router, e.g. 1.2.3.4 
-    --root-dns dns               Base DNS for CloudFoundry applications, e.g. vcap.me 
-    --core-server-flavor flavor  Flavor of the CloudFoundry Core server. Default: 'm1.large' 
-    --release-name name          Name of BOSH release within target BOSH. Default: 'appcloud' 
-    --release-version version    Version of target BOSH release within target BOSH. Default: 'latest' 
-    --stemcell-name name         Name of BOSH stemcell within target BOSH. Default: 'bosh-stemcell' 
-    --stemcell-version version   Version of BOSH stemcell within target BOSH. Default: determines latest for stemcell 
-    --admin-emails email1,email2 Admin email accounts in created CloudFoundry 
-    --skip-validations           Skip all validations 
-
-cf change deas [<server_count>] [--flavor flavor] 
-    change the number/flavor of DEA servers (servers that run CF apps) 
-    --flavor flavor Change flavor of all DEA servers 
+You will be prompted to confirm that your chosen/default security group has ports `22`, `80`, `443` and `4222` open. To chose a different security group, use the `--security-group` option:
 
-cf add service <service_name> [<additional_count>] [--flavor flavor] 
-    add additional CloudFoundry service node 
-    --flavor flavor Server flavor for additional service nodes 
-
-cf upload stemcell [--latest] [--custom] 
-    download/create stemcell & upload to BOSH 
-    --latest Use latest stemcell; possibly not tagged stable 
-    --custom Create custom stemcell from BOSH git source 
+```
+$ bosh create redis --security-group cf-core
+```
 
-cf upload release [--dev] 
-    fetch & upload public cloudfoundry release to BOSH 
-    --dev Create development release from very latest cf-release commits 
+* TODO - how to show available instance sizes
+* TODO - how to update Cloud Foundry servers to a different instance size/flavor
+* TODO - how to scale from a small deployment to a large deployment
+* TODO - how to update the persistent disks of the deployment
 
-cf deploy 
-    deploy CloudFoundry system or apply any changes 
+## Releasing new plugin gem versions
 
-cf watch nats 
-    subscribe to all nats messages within CloudFoundry
-```
+There are two reasons to release new versions of this plugin.
 
-## Services
+1. Package the latest [cf-release](https://github.com/cloudfoundry/cf-release) bosh release (which describes how the core Cloud Foundry components are implemented)
+2. New features or bug fixes to the plugin
 
-By default your CloudFoundry deployment comes with no built-in services. Instead, you easily enable each one and allocate it resources using the `bosh cf add service NAME` command; then deploy again.
+To package the latest "final release" of the Cloud Foundry bosh release into this source repository, run the following command:
 
 ```
-$ bosh cf add service postgresql
-$ bosh cf add service redis
-$ bosh deploy
+$ cd /path/to/releases
+$ git clone https://github.com/cloudfoundry/cf-release.git
+$ cd -
+$ rake bosh:release:import[/path/to/releases/cf-release]
+# for zsh shell quotes are required around rake arguments:
+$ rake bosh:release:import'[/path/to/releases/cf-release]'
 ```
 
-Eventually the new service node servers will be up and running, and then the VMC client will be able to create/bind/delete these services with your CloudFoundry applications.
+Note: only the latest "final release" will be packaged.
 
-For example:
+To locally test the plugin (`bosh` cli loads plugins from its local path automatically):
 
-``` 
-============== System Services ==============
- 
-+------------+---------+---------------------------------------+
-| Service    | Version | Description                           |
-+------------+---------+---------------------------------------+
-| postgresql | 9.0     | PostgreSQL database service (vFabric) |
-| redis      | 2.2     | Redis key-value store service         |
-+------------+---------+---------------------------------------+
- 
-=========== Provisioned Services ============
- 
-+------------------+------------+
-| Name             | Service    |
-+------------------+------------+
-| postgresql-6d01e | postgresql |
-| postgresql-ff0c1 | postgresql |
-| redis-d0d3d      | redis      |
-+------------------+------------+g
 ```
-
-## Orders of easiness vs powerfulness
-
+$ cf /path/to/bosh-cloudfoundry
+$ bosh cf
 ```
 
-+--------------+     +-----------------+       +---------------------+
-|              |     |                 |       |                     |
-|  CLI         |     |                 |       |                     |
-|  commands    +---->|                 |       |                     |
-|              |     |  system         |       |  deployment         |
-|              |     |  config         |       |  manifest           |
-+--------------+     |  (yaml)         |       |  (yaml)             |
-                     |                 +------>|                     |
-                     |                 |       |                     |
-                     |                 |       |                     |
-                     |                 |       |                     |
-                     |                 |       |                     |
-                     +-----------------+       |                     |
-                                               |                     |
-                                               |                     |
-                                               |                     |
-                                               |                     |
-                                               |                     |
-                                               |                     |
-                                               |                     |
-                                               |                     |
-                                               |                     |
-                                               |                     |
-                                               +---------------------+
-```
-
-## Development
+To release a new version of the plugin as a RubyGem:
+
+1. Edit `bosh-cloudfoundry.gemspec` to update the major or minor or patch version.
+2. Run the release command:
 
 ```
-bundle
-bundle exec rake spec
-bundle exec rake install
+$ rake release
 ```
 
 ## Contributing
 
+For fixes or features to the `bosh_cli_plugin_redis` (`bosh redis`) plugin:
+
 1. Fork it
 2. Create your feature branch (`git checkout -b my-new-feature`)
 3. Commit your changes (`git commit -am 'Add some feature'`)
 4. Push to the branch (`git push origin my-new-feature`)
 5. Create new Pull Request
-
-
-[![Bitdeli Badge](https://d2weczhvl823v0.cloudfront.net/StarkAndWayne/bosh-cloudfoundry/trend.png)](https://bitdeli.com/free "Bitdeli Badge")
-

data/Rakefile

@@ -1,5 +1,3 @@
-# Copyright (c) 2012-2013 Stark & Wayne, LLC
-
 ENV["BUNDLE_GEMFILE"] ||= File.expand_path("../Gemfile", __FILE__)
 
 require "rubygems"
@@ -12,24 +10,50 @@ require "rake/dsl_definition"
 require "rake"
 require "rspec/core/rake_task"
 
-
-if defined?(RSpec)
-  namespace :spec do
-    desc "Run Unit Tests"
-    unit_rspec_task = RSpec::Core::RakeTask.new(:unit) do |t|
-      t.pattern = "spec/unit/**/*_spec.rb"
-      t.rspec_opts = %w(--format progress --color)
+bosh_release = "cf-release"
+target_bosh_release_path = File.expand_path("../bosh_release", __FILE__)
+
+namespace :bosh do
+  namespace :release do
+    desc "Import latest #{bosh_release}"
+    task :import, :path do |t, args|
+      source_bosh_release = File.expand_path(args[:path])
+      unless File.directory?(source_bosh_release)
+        $stderr.puts "Please pass path to source bosh release"
+        exit 1
+      end
+
+      sh "rm -rf #{target_bosh_release_path}"
+
+      # required directories to pass bosh_cli validations
+      %w[config jobs packages src].each do |dir|
+        sh "mkdir -p #{target_bosh_release_path}/#{dir}"
+        sh "touch #{target_bosh_release_path}/#{dir}/.gitkeep"
+      end
+
+      chdir(target_bosh_release_path) do
+        sh "cp -R #{source_bosh_release}/releases #{target_bosh_release_path}"
+        sh "cp -R #{source_bosh_release}/.final_builds #{target_bosh_release_path}"
+        sh "cp -R #{source_bosh_release}/config/final.yml #{target_bosh_release_path}/config"
+      end
     end
 
-    desc "Run Integration Tests"
-    functional_rspec_task = RSpec::Core::RakeTask.new(:functional) do |t|
-      t.pattern = "spec/functional/**/*_spec.rb"
-      t.rspec_opts = %w(--format progress --color)
+    desc "Remove any large temporary tgz from internal bosh_release"
+    task :clean do
+      chdir(target_bosh_release_path) do
+        sh "ls .final_builds/**/*.tgz | xargs rm; true"
+        sh "ls releases/**/*.tgz | xargs rm; true"
+      end
     end
   end
+end
+
+task :release => "bosh:release:clean"
 
-  desc "Install dependencies and run tests"
-  task :spec => %w(spec:unit spec:functional)
+desc "Run specs"
+unit_rspec_task = RSpec::Core::RakeTask.new(:spec) do |t|
+  t.pattern = "spec/*_spec.rb"
+  t.rspec_opts = %w(--format progress --color)
 end
 
-task :default => :spec
+task :default => :spec