lobot 0.9.7 → 0.10.0

data/.rspec

@@ -0,0 +1,2 @@
+--color
+--format=nested

data/.rvmrc

@@ -1 +1 @@
-rvm use ruby-1.8.7-p299@lobot
+rvm --create use ruby-1.9.3-p125@lobot

data/Gemfile

@@ -1,9 +1,6 @@
 source "http://rubygems.org"
 
-gem "rspec"
-gem "generator_spec"
-gem "rails"
-gem "cucumber"
+gem "rvm-capistrano", :group => :development, :require => false
 
 # Specify your gem's dependencies in lobot.gemspec
 gemspec

data/LICENSE.txt

@@ -18,4 +18,4 @@ FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
 AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
 LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
 OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
-THE SOFTWARE.
+THE SOFTWARE.

data/README.md

@@ -12,14 +12,14 @@ Lando Calrissian relies on Lobot to keep Cloud City afloat, and now you can rely
 * capistrano tasks for bootstrapping and deploying to an EC2 instance
 * chef recipes for configuring a Centos server to run Jenkins and build Rails projects.
 
-all you'll need to do is:
+all you'll need to do is run the following commands:
 
     rails g lobot:install
-    edit config/ci.yml
+    rails g lobot:config
     rake ci:server_start
     cap ci bootstrap
     cap ci chef
-	
+
 Read on for an explanation of what each one of these steps does.
 
 ## Install
@@ -34,52 +34,72 @@ Lobot is a Rails 3 generator.  Rails 2 can be made to work, but you will need to
     rails g lobot:install
 
 ## Setup
+You can use a generator to interactively generate the config/ci.yml that includes some reasonable defaults and explanations
+
+    rails g lobot:config
 
-Edit config/ci.yml
+Alternatively, manually edit config/ci.yml
 
-    --- 
+    ---
     app_name: # a short name for your application
     app_user: # the user created to run your CI process
     git_location: # The location of your remote git repository which Jenkins will poll and pull from on changes.
-    basic_auth: 
+    basic_auth:
     - username: # The username you will use to access the Jenkins web interface
       password: # The password you will use to access the Jenkins web interface
-    credentials: 
+    credentials:
       aws_access_key_id: # The Access Key for your Amazon AWS account
       aws_secret_access_key: The Secret Access Key for your Amazon AWS account
       provider: AWS # leave this one alone
-    server: 
+    server:
       name: run 'rake ci:server_start to populate'
       instance_id: run 'rake ci:server_start to populate'
     build_command: ./script/ci_build.sh
-    ec2_server_access: 
+    ec2_server_access:
       key_pair_name: myapp_ci
       id_rsa_path: ~/.ssh/id_rsa
-    id_rsa_for_github_access: |-
-      -----BEGIN RSA PRIVATE KEY-----
-      SSH KEY WITH ACCESS TO GITHUB GOES HERE
-      -----END RSA PRIVATE KEY-----
-      
-For security, you can add ci.yml to your .gitignore file and store a ci.yml.example without authentication credentials in your repository
+    github_private_ssh_key_path: ~/.ssh/id_rsa
+
+For security, the lobot:install task added config/ci.yml to the .gitignore file since it includes sensitive AWS credentials and your CI users password.
+Keep in mind that the default build script ci_build.sh uses the headless gem and jasmine. You'll want to add those to your Gemfile or change your build command.
+
+At this point you will need to create a commit of the files generated or modified and push those changes to your remote git repository so jenkins can execute your build command.
 
 ## Modify the soloistrc if necessary
 
 Switch postgres to mysql, or add your own recipes for your own dependencies.
 
-## Let 'er rip
+## Starting your lobot instance
 
-Launch an instance, allocate and associates an elastic IP and updates ci.yml:
+1. Launch an instance, allocate and associates an elastic IP and updates ci.yml:
 
     rake ci:server_start
 
-Bootstrap the instance using the boostrap_server.sh script generated by Lobot. The script instals ruby prerequisites, creates the app_user account, and installs RVM for that user:
+2. Bootstrap the instance using the boostrap_server.sh script generated by Lobot. The script instals ruby prerequisites, creates the app_user account, and installs RVM for that user:
 
     cap ci bootstrap
-	
-Upload the contents of your chef/cookbooks/ directory, upload the soloistrc, and run chef:
+
+3. Upload the contents of your chef/cookbooks/ directory, upload the soloistrc, and run chef:
 
     cap ci chef
 
+Your lobot instance should now be up and running. You will be able to access your CI server at: http://<your instance address>/ with the username and password you chose during configuration.
+For more information about Jenkins CI, see http://jenkins-ci.org/
+
+## Troubleshooting
+
+Shell access for your instance
+
+    rake ci:ssh
+
+Terminating your instance and deallocating the elastic IP
+
+    rake ci:terminate
+
+Suspending your instance
+
+    rake ci:stop
+
 ## Add your new CI instance to [cimonitor](http://github.com/pivotal/cimonitor) and CCMenu
 
 Lobot can generate the config for you, just run:
@@ -92,10 +112,14 @@ Lobot can generate the config for you, just run:
 * capistrano
 * capistrano-ext
 * rvm (the gem - it configures capistrano to use RVM on the server)
+* nokogiri
 
 # Tests
 
-Lobot is tested using rspec, generator_spec and cucumber.  Cucumber provides a full integration test which can generate a rails application, push it to github, start a server and bring up CI for the generated project.  You'll need a git repository(which should not have any code you care about) and an AWS account to run the spec.  It costs about $0.50 and takes about half an hour.  It does not clean up after itself, so be sure to terminate the server when you're done, or it will cost substantially more than $0.50.  Use the secrets.yml.example to create a secrets.yml file with your accounts.
+Lobot is tested using rspec, generator_spec and cucumber.  Cucumber provides a full integration test which can generate a rails application, push it to github, start a server and bring up CI for the generated project.
+You'll need a git repository (which should not have any code you care about) and an AWS account to run the whole test suite.  It costs about $0.50 and takes about half an hour.
+It will attempt to terminate the server however you should verify this so you do not get charged additional money.
+Use the secrets.yml.example to create a secrets.yml file with your account information.
 
 # Contributing
 

data/Rakefile

@@ -1,2 +1,13 @@
+require 'rspec/core/rake_task'
+require 'cucumber/rake/task'
 require 'bundler'
 Bundler::GemHelper.install_tasks
+
+task :default => [:spec, :features]
+
+desc "Run all specs in spec directory (excluding plugin specs)"
+RSpec::Core::RakeTask.new(:spec)
+
+desc "Run all features in the features directory"
+Cucumber::Rake::Task.new(:features) do |t|
+end

data/docs/ci_yml.md

@@ -15,7 +15,7 @@ The basic auth field is an array of hashes containing the usernames and password
 # credentials:
 Currently Lobot only supports managing EC2 servers. To start a new instance, the rake task needs to be able to connect to ec2 and ask it to launch a server.
 ##  aws_access_key_id:
-The Access Key for your Amazon AWS account.  You can obtain it by visiting 
+The Access Key for your Amazon AWS account.  You can obtain it by visiting
 https://aws-portal.amazon.com/gp/aws/developer/account/index.html?ie=UTF8&action=access-key
 ##  aws_secret_access_key: The Secret Access Key for your Amazon AWS account
 This will be available at the same URL.
@@ -32,16 +32,11 @@ The instance_id is also saved, in order to facilitate stopping and starting inst
 # build_command: ./script/ci_build.sh
 The build command is the shell script that Jenkins will execute to run your build.  While you could put this script is the Jenkins configuration, it's clearer to have it in a script directly checked into your project.
 
-# ec2_server_access: 
+# ec2_server_access:
 	key_pair_name: myapp_ci
 	id_rsa_path: ~/.ssh/id_rsa
 
 The EC2 server access section is perhaps the most confusing field in the ci.yml file. This section tells Lobot what ssh keys to use to access the servers it spins up.  When you run rake ci:sever_start, Lobot first looks in your ec2 account to see if they key pair name you specified exists - if it doesn't, it uploads the specified #{id_rsa_path}.pub to amazon, and names it #{key_pair_name}.  It then launches the server, telling EC2 to use the key pair specified.
 
-# id_rsa_for_github_access
-id_rsa_for_github_access: |-
-  -----BEGIN RSA PRIVATE KEY-----
-  SSH KEY WITH ACCESS TO GITHUB GOES HERE
-  -----END RSA PRIVATE KEY-----
-
-This is the rsa key that jenkins will use when connecting to github. Note that it is indented as it's a multiline markdown string.
+# github_private_ssh_key_path: ~/.ssh/id_github_current
+This is the path to the ssh private key that jenkins will use when connecting to github.

data/features/ci.feature

@@ -1,28 +1,29 @@
-Feature: CI 
+Feature: CI
 
   Background:
     Given the temp directory is clean
-    Given I am in the temp directory
-  
+    And I am in the temp directory
+
   Scenario: Installing Lobot on a rails3 project
-    When I create a new Rails project using a Rails template
+    When I create a new Rails project
     And I vendor Lobot
     And I put Lobot in the Gemfile
     And I run bundle install
     And I run the Lobot generator
     Then rake reports ci tasks as being available
-  
+
+  @aws
   Scenario: Install CI on Amazon AWS using new Rails template
-    When I create a new Rails project using a Rails template
+    When I create a new Rails project
     And I vendor Lobot
     And I put Lobot in the Gemfile
     And I add a gem with an https://github.com source
     And I run bundle install
     And I run the Lobot generator
     And I enter my info into the ci.yml file
-    And I change my ruby version
+    And I make changes to be committed
     And I push to git
-    And I run the server setup
+    And I start the server
     And I bootstrap
     And I deploy
     Then CI is green

data/features/config/secrets.yml.example

@@ -1,7 +1,5 @@
---- 
+---
 aws_access_key_id:
-aws_secret_access_key: 
-id_rsa_for_github_access: |-
-  -----BEGIN RSA PRIVATE KEY-----
-  -----END RSA PRIVATE KEY-----
-  
+aws_secret_access_key:
+github_private_ssh_key_path: ~/.ssh/id_github_current
+

data/features/step_definitions/ci_steps.rb

@@ -1,19 +1,16 @@
-
-def system!(str)
-  raise "Command Failed: #{str}" unless system(str)
-end
-
 Given /^the temp directory is clean$/ do
   system!("rm -rf /tmp/lobot-test")
   system!("mkdir -p /tmp/lobot-test")
 end
 
 Given /^I am in the temp directory$/ do
-  Dir.chdir('/tmp/lobot-test')
+  Dir.chdir(LOBOT_TEMP_DIRECTORY)
 end
 
-When /^I create a new Rails project using a Rails template$/ do
-  system!("echo -e '\nyes\nno\nno\nno\nno\nno\nno' | rails new testapp -m https://github.com/pivotal/guiderails/raw/master/main.rb")
+When /^I create a new Rails project$/ do
+  system!("rvm ruby-1.9.3-p125 do rvm gemset create testapp")
+  system!("rails new testapp")
+  system!("cd testapp && echo 'rvm use ruby-1.9.3-p125@testapp' > .rvmrc")
 end
 
 When /^I vendor Lobot$/ do
@@ -30,6 +27,7 @@ end
 
 When /^I add a gem with an https:\/\/github.com source$/ do
   system!(%{echo "gem 'greyhawkweather', :git => 'https://github.com/verdammelt/Greyhawk-Weather.git'" >> testapp/Gemfile})
+  system!("cd testapp && bundle install")
 end
 
 When /^I run bundle install$/ do
@@ -45,48 +43,61 @@ When /^I run the Lobot generator$/ do
 end
 
 When /^I enter my info into the ci\.yml file$/ do
-  secrets = YAML.load_file(File.expand_path('../config/secrets.yml', File.dirname(__FILE__)))
-  
+  hostname = `hostname`.strip
+  secrets_file = File.expand_path('../config/secrets.yml', File.dirname(__FILE__))
+  raise "Missing #{secrets_file}, needed for AWS test." unless File.exist?(secrets_file)
+  secrets = YAML.load_file(secrets_file)
+  raise "Missing AWS secret access key" unless secrets["aws_secret_access_key"].to_s != ""
+  raise "Missing AWS access key id" unless secrets["aws_access_key_id"].to_s != ""
+  raise "Missing github private key path" unless secrets["github_private_ssh_key_path"].to_s != ""
+
+  raise "Missing private SSH key for AWS!" unless File.exist?(File.expand_path("~/.ssh/id_github_current"))
   ci_conf_location = 'testapp/config/ci.yml'
   ci_yml = YAML.load_file(ci_conf_location)
   ci_yml.merge!(
-  'app_name' => 'testapp',
-  'app_user' => 'testapp-user',
-  'git_location' => 'git@github.com:pivotalprivate/ci-smoke.git',
-  'basic_auth' => [{ 'username' => 'testapp', 'password' => 'testpass' }],
-  'credentials' => { 'aws_access_key_id' => secrets['aws_access_key_id'], 'aws_secret_access_key' => secrets['aws_secret_access_key'], 'provider' => 'AWS' },
-  'ec2_server_access' => {'key_pair_name' => 'lobot_cucumber_key_pair', 'id_rsa_path' => '~/.ssh/id_rsa'},
-  'id_rsa_for_github_access' => secrets['id_rsa_for_github_access']
+    'app_name' => 'testapp',
+    'app_user' => 'testapp-user',
+    'git_location' => 'git@github.com:pivotalprivate/ci-smoke.git',
+    'basic_auth' => [{ 'username' => 'testapp', 'password' => 'testpass' }],
+    'credentials' => { 'aws_access_key_id' => secrets['aws_access_key_id'], 'aws_secret_access_key' => secrets['aws_secret_access_key'], 'provider' => 'AWS' },
+    'ec2_server_access' => {'key_pair_name' => "lobot_cucumber_key_pair_#{hostname}", 'id_rsa_path' => '~/.ssh/id_github_current'},
+    'github_private_ssh_key_path' => secrets["github_private_ssh_key_path"]
   )
-  # ci_yml['server']['name]  = '' # This can be used to merge in a server which is already running if you want to skip the setup steps while iterating on a test
+  # ci_yml['server']['name']  = '' # This can be used to merge in a server which is already running if you want to skip the setup steps while iterating on a test
   File.open(ci_conf_location, "w") do |f|
-    YAML.dump(ci_yml, f)
+    f << ci_yml.to_yaml
+    f << File.read(secrets_file)
   end
 end
 
-When /^I change my ruby version$/ do
-  system! "cd testapp && echo 'rvm use ruby-1.8.7-p299@lobot' > .rvmrc"
-end
-
-
-When /^I push to git$/ do
+When /^I make changes to be committed$/ do
   lobot_dir = File.expand_path('../../', File.dirname(__FILE__))
   system! "rm testapp/vendor/cache/*"
   system! "cp #{lobot_dir}/pkg/lobot-#{Lobot::VERSION}.gem testapp/vendor/cache/"
   system! "echo 'config/ci.yml' >> testapp/.gitignore"
+  ["headless", "rspec-rails", "jasmine"].each do |gem|
+    system!(%{echo "gem '#{gem}'" >> testapp/Gemfile})
+  end
+  system!("cd testapp && bundle install")
+  system!("cd testapp && bundle exec jasmine init .")
+end
+
+When /^I push to git$/ do
+  system! "cd testapp && git init"
   system! "cd testapp && git add ."
   system! "cd testapp && git commit -m'initial commit'"
-  system "cd testapp && git remote rm origin"
+  system "cd testapp && git remote rm origin" # Ignore failures
   system! "cd testapp && git remote add origin git@github.com:pivotalprivate/ci-smoke.git"
   system! "cd testapp && git push --force -u origin master"
 end
 
-When /^I run the server setup$/ do
-  system! "cd testapp && rake ci:server_start"
+When /^I start the server$/ do
+  system! "cd testapp && bundle exec rake ci:server_start"
 end
 
 When /^I bootstrap$/ do
-  system! "cd testapp && cap ci bootstrap"
+  system!("cd testapp && bundle install")
+  system! "cd testapp && bundle exec cap ci bootstrap"
 end
 
 When /^I deploy$/ do
@@ -95,12 +106,12 @@ end
 
 Then /^CI is green$/ do
   Timeout::timeout(300) do
-    until system("cd testapp && rake ci:status")
+    until system("cd testapp && bundle exec rake ci:status")
       sleep 5
     end
   end
 end
 
 Then /^rake reports ci tasks as being available$/ do
-  `cd testapp && rake -T`.should include("ci:start")
+  `cd testapp && bundle exec rake -T`.should include("ci:start")
 end

data/features/support/env.rb

@@ -1,3 +1,12 @@
 require "rubygems"
+require File.expand_path('../../lib/lobot', File.dirname(__FILE__))
+LOBOT_TEMP_DIRECTORY = "/tmp/lobot-test"
 
-require File.expand_path('../../lib/lobot', File.dirname(__FILE__))
+def system!(str)
+  system(str)
+  raise "Command Failed: #{str} with exit code #{$?.exitstatus}" unless $?.success?
+end
+
+After '@aws' do
+  system "cd #{LOBOT_TEMP_DIRECTORY}/testapp && rake ci:terminate" unless ENV["KEEP_INSTANCE"]
+end

data/lib/generators/lobot/config_generator.rb

@@ -0,0 +1,161 @@
+module Lobot
+  class ConfigGenerator < Rails::Generators::Base
+    source_root File.expand_path(File.join(File.dirname(__FILE__), 'templates'))
+
+    def generate_ci_config
+      default_git_location = nil
+      if File.exist?(Rails.root.join(".git/config"))
+        default_git_location = `cat .git/config | grep url`.strip.split.last
+      else
+        puts "nope"
+      end
+      default_app_name = File.basename(Rails.root)
+      config = {
+        'app_name' => default_app_name,
+        'app_user' => ENV['USER'],
+        'git_location' => default_git_location,
+        'basic_auth' => [
+          {
+            'username' => "#{default_app_name}_ci"
+          }
+        ],
+        'credentials' => {
+          'provider' => "AWS"
+        },
+        'server' => {
+          'flavor_id' => "m1.large",
+          'security_group' => "ci_servers",
+          'ssh_port' => "22"
+        },
+        'build_command' => "./script/ci_build.sh",
+        'ec2_server_access' => {
+          "key_pair_name" => "#{default_app_name}_ci",
+          "id_rsa_path" => File.expand_path("~/.ssh/id_rsa")
+        },
+        "github_private_ssh_key_path" => File.expand_path("~/.ssh/id_rsa")
+      }
+
+      say "* The name of your application as it will appear in CI"
+      app_name = ask "Application Name [#{config['app_name']}]: "
+      config['app_name'] = app_name if app_name != ""
+
+      say "* The user created to run your CI process"
+      app_user = ask "Application User [#{config['app_user']}]: "
+      config['app_user'] = app_user if app_user != ""
+
+      say "* The location of your remote git repository which CI will poll and pull from on changes"
+      git_location = ask "Git Repository Location [#{config['git_location']}]: "
+      config['git_location'] = git_location if git_location != ""
+
+      say "* The username you will use to access the CI web interface"
+      ci_username = ask "CI Username [#{config['basic_auth'][0]['username']}]: "
+      config['basic_auth'][0]['username'] = ci_username if ci_username != ""
+
+      say "* The password you will use to access the Jenkins web interface"
+      while true do
+        ci_password = ask "Choose a CI password: "
+        config['basic_auth'][0]['password'] = ci_password
+        if ci_password == ""
+          say "Password cannot be blank"
+        else
+          break
+        end
+      end
+
+      say <<-EOS
+* See https://aws-portal.amazon.com/gp/aws/developer/account/index.html?ie=UTF8&action=access-key
+* for access key id and secret access key
+      EOS
+
+      while true do
+        aws_access_key_id = ask "AWS Access Key ID: "
+        config['credentials']['aws_access_key_id'] = aws_access_key_id
+        if aws_access_key_id == ""
+          say "AWS Access Key ID cannot be blank"
+        else
+          break
+        end
+      end
+
+      while true do
+        aws_secret_access_key = ask "AWS Secret Access Key: "
+        config['credentials']['aws_secret_access_key'] = aws_secret_access_key
+        if aws_secret_access_key == ""
+          say "AWS Secret Access Key cannot be blank"
+        else
+          break
+        end
+      end
+
+      say <<-EOS
+* See http://aws.amazon.com/ec2/instance-types/ API name values (e.g. m1.large)
+      EOS
+      flavor_id = ask "Choose an EC2 instance type [#{config['server']['flavor_id']}]: "
+      config['server']['flavor_id'] = flavor_id if flavor_id != ""
+
+      security_group = ask "AWS EC2 Security Group Name [#{config['server']['security_group']}]: "
+      config['server']['security_group'] = security_group if security_group != ""
+
+      ssh_port = ask "Server SSH Port [#{config['server']['ssh_port']}]: "
+      config['server']['ssh_port'] = ssh_port if ssh_port != ""
+
+      build_command = ask "Build Command: [#{config['build_command']}]: "
+      config['build_command'] = build_command if build_command != ""
+
+      say <<-EOS
+* This should refer to an SSH key pair that you have already generated. You may wish to generate a new key
+* separate from what you may already be using for github or other systems.
+* For a tutorial on this see: http://open.bsdcow.org/histerical/tutorials/ssh_pubkey_auth#1.2
+      EOS
+      while true do
+        id_rsa_path = ask "Path to SSH Private Key for EC2 Access [#{config['ec2_server_access']['id_rsa_path']}]: "
+        config['ec2_server_access']['id_rsa_path'] = id_rsa_path if id_rsa_path != ""
+        if config['ec2_server_access']['id_rsa_path'] != File.expand_path(config['ec2_server_access']['id_rsa_path'])
+          config['ec2_server_access']['id_rsa_path'] = File.expand_path(File.join(ENV['HOME'], '.ssh', config['ec2_server_access']['id_rsa_path']))
+        end
+        if File.exist?(config['ec2_server_access']['id_rsa_path']) && File.exist?("#{config['ec2_server_access']['id_rsa_path']}.pub")
+          break
+        else
+          say "Unable to find both #{config['ec2_server_access']['id_rsa_path']} and #{config['ec2_server_access']['id_rsa_path']}.pub"
+        end
+      end
+
+      say <<-EOS
+* This is an arbitrary label corresponding to the SSH credentials that you just selected
+* You may name this anything you like. For example: your project name, hostname or name of the SSH key you just chose
+      EOS
+      key_pair_name = ask "AWS EC2 Key Pair Name [#{config['ec2_server_access']['key_pair_name']}]: "
+      config['ec2_server_access']['key_pair_name'] = key_pair_name if key_pair_name != ""
+
+      say <<-EOS
+* This needs to refer to an SSH Private Key that has been associated an account that has access to the git
+* repository you entered above. On github this will be listed here: https://github.com/settings/ssh
+      EOS
+      while true do
+        github_private_ssh_key_path = ask "Path to SSH Private Key for Github [#{config['github_private_ssh_key_path']}]: "
+        config['github_private_ssh_key_path'] = github_private_ssh_key_path if github_private_ssh_key_path != ""
+        if config['github_private_ssh_key_path'] != File.expand_path(config['github_private_ssh_key_path'])
+          config['github_private_ssh_key_path'] = File.expand_path(File.join(ENV['HOME'], '.ssh', config['github_private_ssh_key_path']))
+        end
+        if File.exist?(config['github_private_ssh_key_path'])
+          break
+        else
+          say "Unable to find #{config['github_private_ssh_key_path']}"
+        end
+      end
+
+
+
+      config_ci = YAML.load_file(Rails.root.join("config/ci.yml")) rescue {}
+      config_ci.merge!(config)
+
+      File.open(Rails.root.join("config/ci.yml"), "w") do |f|
+        f << config_ci.to_yaml
+      end
+
+      say "\n\nconfig/ci.yml configured. To launch your instance run rake ci:server_start."
+      say "Be sure to push uncommitted changes made by the lobot:install process first."
+    end
+  end
+end
+