relevance_rails 0.1.1.alpha → 0.1.2

data/CHANGELOG.md

@@ -1,3 +1,10 @@
+*v0.1.2*
+
+* add streaming output from chef convergence using a fog monkeypatch
+* retry apt-get command failures
+* add --help
+* add support for rvm > 1.12.0
+
 *v0.1.1.alpha*
 
 * Ruby manager agnostic - can use rvm, system ruby or rbenv

data/README.md

@@ -24,73 +24,18 @@ $ gem install relevance_rails
 $ relevance_rails new <your new project>
 ````
 
-For existing projects, first add to your Gemfile:
+For existing projects, please note we *only* support Rails 3.2 and higher!  To add to an existing project, first add to your Gemfile:
 
     group :development, :test do
       gem 'relevance_rails'
     end
 
-After a `bundle install`, pull in our chef recipes into provision/:
 
-```sh
-# defaults to mysql
-$ rails g provision_config
-# if using postgresql
-$ rails g provision_config postgresql
-```
+Using relevance_rails
+---------------------
 
-Configuring bundled ssh keys
-----------------------------
+[See USAGE](https://github.com/relevance/relevance_rails/blob/master/USAGE.md)
 
-By default, relevance_rails bundles all your keys from `ssh-add -L` with your instance. For bundling
-additional keys, you can create a ~/.relevance_rails/keys_git_url file and point it to a git repo that
-has additional keys. Keys in that git repo should exist as top level *.pub files. You *MUST* have at
-least one key to provision.
-
-Provisioning on EC2
--------------------
-
-First create an aws config in ~/.relevance\_rails/aws\_config.yml.
-An example config looks like this:
-
-```yaml
-aws_credentials:
-  :aws_access_key_id: <your aws access key id>
-  :aws_secret_access_key: <your aws secret access key>
-
-server:
-  creation_config:
-    :flavor_id: <instance type, e.g. 'm1.large'>
-    :image_id: <ami to bootstrap with. Must be some UBUNTU image. e.g. "ami-fd589594">
-    :groups: <security group to place the new deployment in, e.g. "default">
-    :key_name: <name of the public/private keypair to start instance with>
-  private_key: |
-    -----BEGIN RSA PRIVATE KEY-----
-    Include the RSA private key here. This should correspond to the keypair indicated
-    by :key_name above.
-    -----END RSA PRIVATE KEY-----
-```
-
-Now just provision your instance:
-
-```sh
-$ rake provision:ec2_and_generate NAME=qa
-$ cap qa deploy:setup deploy
-```
-
-Provisioning with Vagrant
--------------------------
-
-From your app's root directory:
-
-```sh
-$ cd provision
-# pull in vagrant and chef
-$ bundle install
-$ vagrant up
-```
-
-Vagrant instance should be up at 172.25.5.5.
 
 Supported Ruby Versions
 -----------------------

data/USAGE.md

@@ -0,0 +1,64 @@
+Setup Chef
+----------
+
+To setup chef recipes in provision/:
+
+```sh
+# defaults to mysql
+$ rails g provision_config
+# if using postgresql
+$ rails g provision_config postgresql
+```
+
+Configuring bundled ssh keys
+----------------------------
+
+By default, relevance_rails bundles all your keys from `ssh-add -L` with your instance. For bundling
+additional keys, you can create a ~/.relevance_rails/keys_git_url file and point it to a git repo that
+has additional keys. Keys in that git repo should exist as top level *.pub files. You *MUST* have at
+least one key to provision.
+
+Provisioning on EC2
+-------------------
+
+First create an aws config in ~/.relevance\_rails/aws\_config.yml.
+An example config looks like this:
+
+```yaml
+aws_credentials:
+  :aws_access_key_id: <your aws access key id>
+  :aws_secret_access_key: <your aws secret access key>
+
+server:
+  creation_config:
+    :flavor_id: <instance type, e.g. 'm1.large'>
+    :image_id: <ami to bootstrap with. Must be some UBUNTU image. e.g. "ami-fd589594">
+    :groups: <security group to place the new deployment in, e.g. "default">
+    :key_name: <name of the public/private keypair to start instance with>
+  private_key: |
+    -----BEGIN RSA PRIVATE KEY-----
+    Include the RSA private key here. This should correspond to the keypair indicated
+    by :key_name above.
+    -----END RSA PRIVATE KEY-----
+```
+
+Now just provision your instance:
+
+```sh
+$ rake provision:ec2_and_generate NAME=qa
+$ cap qa deploy:setup deploy
+```
+
+Provisioning with Vagrant
+-------------------------
+
+From your app's root directory:
+
+```sh
+$ cd provision
+# pull in vagrant and chef
+$ bundle install
+$ vagrant up
+```
+
+Vagrant instance should be up at 172.25.5.5.

data/lib/generators/provision_config/provision_config_generator.rb

@@ -19,6 +19,12 @@ class ProvisionConfigGenerator < Rails::Generators::Base
     Dir.chdir(destination_root)
   end
 
+  def check_git_status
+    unless `git status`[/working directory clean/]
+      abort "Your git repository is dirty. Clean up before reinvoking this command."
+    end
+  end
+
   def check_authorized_keys
     if (@authorized_keys = fetch_keys).empty?
       abort "No ssh keys were found! Check ssh-add -L and your keys_git_url config."

data/lib/generators/relevance_file/templates/Gemfile.erb

@@ -1,11 +1,11 @@
 source :rubygems
 
-gem 'rails', '3.2.2'
-gem 'haml', '3.1.4'
-gem 'configatron', '2.8.3'
+gem 'rails', '~> 3.2.2'
+gem 'haml', '~> 3.1.4'
+gem 'configatron', '~> 2.8.3'
 gem 'airbrake', '~> 3.0.4'
-gem 'factory_girl_rails', '1.2.0'
-gem 'jquery-rails', '1.0.14'
+gem 'factory_girl_rails', '~> 1.2.0'
+gem 'jquery-rails', '~> 1.0.14'
 gem 'therubyracer'
 <%- if database == 'mysql' -%>
 gem 'mysql2', '~> 0.3.11'

data/lib/generators/relevance_file/templates/README.markdown.erb

@@ -5,32 +5,7 @@
 gem install bundler
 [TODO other setup commands here]
 
-## Provisioning on EC2 (Optional)
+Using relevance_rails
+---------------------
 
-First create an aws config in ~/.relevance\_rails/aws\_config.yml.
-An example config looks like this:
-
-```yaml
-aws_credentials:
-  :aws_access_key_id: <your aws access key id>
-  :aws_secret_access_key: <your aws secret access key>
-
-server:
-  creation_config:
-    :flavor_id: <instance type, e.g. 'm1.large'>
-    :image_id: <ami to bootstrap with. Must be some UBUNTU image. e.g. "ami-fd589594">
-    :groups: <security group to place the new deployment in, e.g. "default">
-    :key_name: <name of the public/private keypair to start instance with>
-  private_key: |
-    -----BEGIN RSA PRIVATE KEY-----
-    Include the RSA private key here. This should correspond to the keypair indicated
-    by :key_name above.
-    -----END RSA PRIVATE KEY-----
-```
-
-Now just provision your instance:
-
-```sh
-$ rails g deployment qa
-$ cap qa deploy:setup deploy
-```
+[See USAGE](https://github.com/relevance/relevance_rails/blob/master/USAGE.md)

data/lib/relevance_rails/fog_ext/ssh.rb

@@ -0,0 +1,53 @@
+require 'fog'
+require 'fog/core/ssh'
+
+# Monkey-patch Fog 1.3.1 to stream SSH output 
+# (in real time) to stdout.
+class Fog::SSH::Real
+  def run(commands)
+    commands = [*commands]
+    results  = []
+    begin
+      Net::SSH.start(@address, @username, @options) do |ssh|
+        commands.each do |command|
+          result = Fog::SSH::Result.new(command)
+          ssh.open_channel do |ssh_channel|
+            ssh_channel.request_pty
+            ssh_channel.exec(command) do |channel, success|
+              unless success
+                raise "Could not execute command: #{command.inspect}"
+              end
+
+              channel.on_data do |ch, data|
+                result.stdout << data
+                puts data
+              end
+
+              channel.on_extended_data do |ch, type, data|
+                next unless type == 1
+                result.stderr << data
+                puts data
+              end
+
+              channel.on_request('exit-status') do |ch, data|
+                result.status = data.read_long
+              end
+
+              channel.on_request('exit-signal') do |ch, data|
+                result.status = 255
+              end
+            end
+          end
+          ssh.loop
+          results << result
+        end
+      end
+    rescue Net::SSH::HostKeyMismatch => exception
+      exception.remember_host!
+      sleep 0.2
+      retry
+    end
+    results
+  end
+end
+

data/lib/relevance_rails/provision.rb

@@ -1,9 +1,13 @@
 require 'fog'
 require 'thor'
 require 'timeout'
+require 'relevance_rails/fog_ext/ssh'
 
 module RelevanceRails
   module Provision
+    class AptInstallError < StandardError
+    end
+
     def self.create_ec2(name = nil)
       abort "Please provide a $NAME" unless name
       server = provision_ec2_instances(name)
@@ -54,10 +58,10 @@ module RelevanceRails
       puts "Provisioning an instance..."
       server = fog_connection.servers.create(config['server']['creation_config'])
       fog_connection.tags.create(:key => 'Name',
-                      :value => "#{Rails.application.class.parent_name} #{name}",
+                      :value => "#{name}",
                       :resource_id => server.id)
       server.private_key = config['server']['private_key']
-      
+
       File.open("config/ec2_instance.txt", "w") do |f|
         f.puts(server.id)
       end
@@ -65,17 +69,12 @@ module RelevanceRails
       puts "Provisioned!"
       server
     end
-    
+
     def self.run_commands(server)
-      puts "Updating apt cache..."
-      run_command(server, 'sudo apt-get update')
-      puts "Installing ruby..."
-      run_command(server, 'sudo apt-get -y install ruby')
-      puts "Installing rubygems..."
-      run_command(server, 'sudo apt-get -y install rubygems1.8')
+      apt_installs(server)
       puts "Installing chef..."
       run_command(server, 'sudo gem install chef --no-ri --no-rdoc --version 0.10.8')
-      puts "Copying chef resources from provision directory.."
+      puts "Copying chef resources from provision directory..."
       server.scp("#{Rails.root.join('provision')}/", '/tmp/chef-solo', :recursive => true)
       puts "Converging server, this may take a while (10-20 minutes)"
       run_command(server, 'cd /tmp/chef-solo && sudo /var/lib/gems/1.8/bin/chef-solo -c solo.rb -j dna.json')
@@ -85,6 +84,43 @@ module RelevanceRails
       server
     end
 
+    def self.retry_block(times, errors, failure)
+      succeeded = false
+      attempts = 0
+      last_error = nil
+      until succeeded || attempts > times-1
+        begin
+          retval = yield
+          succeeded = true
+        rescue *errors => e
+          puts failure
+          attempts +=1
+          last_error = e
+        end
+      end
+      if succeeded
+        return retval
+      else
+        exit 1
+      end
+    end
+
+    def self.apt_installs(server)
+      retry_block(3, [AptInstallError], "Apt-cache came from corrupt mirror, retrying update...") do
+        puts "Updating apt cache..."
+        run_apt_command(server, 'sudo apt-get update')
+        puts "Installing ruby..."
+        run_apt_command(server, 'sudo apt-get -y install ruby')
+        puts "Installing rubygems..."
+        run_apt_command(server, 'sudo apt-get -y install rubygems1.8')
+      end
+    end
+
+    def self.run_apt_command(server, command)
+      jobs = server.ssh(command)
+      raise AptInstallError unless jobs_succeeded?(jobs)
+    end
+
     def self.run_command(server, command)
       jobs = server.ssh(command)
       exit 1 unless jobs_succeeded?(jobs)
@@ -93,22 +129,10 @@ module RelevanceRails
     def self.wait_for_ssh(server)
       puts "Waiting for ssh connectivity..."
       server.wait_for { ready? }
-      succeeded = false
-      attempts = 0
-      last_error = nil
-      until succeeded || attempts > 4
+      retry_block(5, [Errno::ECONNREFUSED, Timeout::Error], "Connecting to Amazon refused. Retrying...") do
         sleep 10
-        begin
-          # Should be checked for compatability across rubies
-          Timeout.timeout(60) { server.ssh('ls') }
-          succeeded = true
-        rescue Errno::ECONNREFUSED, Timeout::Error => e
-          puts "Connecting to Amazon refused. Attempt #{attempts+1}..."
-          attempts += 1
-          last_error = e
-        end
+        Timeout.timeout(60) { server.ssh('ls') }
       end
-      raise last_error unless succeeded
       puts "Server up and listening for SSH!"
     end
 

data/lib/relevance_rails/runner.rb

@@ -3,7 +3,9 @@ require 'relevance_rails'
 module RelevanceRails
   class Runner
     def self.start(argv=ARGV)
-      if argv.delete '--version'
+      if argv.empty? || argv[0] == '--help' || argv[0] == '-h'
+        print_help
+      elsif argv.delete '--version'
         puts "RelevanceRails #{RelevanceRails::VERSION}"
       elsif argv[0] == 'new'
         add_default_options! argv
@@ -22,8 +24,37 @@ module RelevanceRails
 
     private
 
+    def self.print_help
+      puts <<-STR
+Usage: relevance_rails new APP [OPTIONS]
+
+Options:
+
+-d, --database <database>: Generate dependencies, configuration, and deployments
+                           for database, only supports mysql (default) and
+                           postgresql
+
+When invoked in an RVM shell, the new project and newly provisioned servers will
+inherit the current ruby. The new project will get its own gemset.
+
+When invoked in other contexts, newly provisioned servers will attempt to
+install a ruby which matches the ruby executing the relevance_rails process.
+STR
+    end
+
     def self.setup_rvm(app_name)
-      $LOAD_PATH.unshift "#{ENV['rvm_path']}/lib"
+      rvm_version = Gem::Version.new(`rvm --version`[/rvm (\d\.\d+\.\d+)/, 1].to_s)
+
+      if rvm_version < Gem::Version.new('1.10.2')
+        abort "Rvm version 1.10.2 or greater is required. Run 'rvm get stable'"
+      end
+
+      if rvm_version < Gem::Version.new('1.12.0')
+        # RVM's ruby drivers were factored out into a gem
+        # in 1.12.0, so you don't use this trick anymore.
+        $LOAD_PATH.unshift "#{ENV['rvm_path']}/lib"
+      end
+
       require 'rvm'
 
       env = RVM::Environment.current

data/lib/relevance_rails/version.rb

@@ -1,3 +1,3 @@
 module RelevanceRails
-  VERSION = "0.1.1.alpha"
+  VERSION = "0.1.2"
 end

data/lib/tasks/provision.rake

@@ -3,13 +3,15 @@ require 'relevance_rails/provision'
 namespace :provision do
   desc 'Provision a deployable EC2 instance and generate Capistrano config'
   task :ec2_and_generate do
-    server = RelevanceRails::Provision.create_ec2(ENV['NAME'])
+    server_name = "#{Rails.application.class.parent_name} #{ENV['NAME']}"
+    server = RelevanceRails::Provision.create_ec2(server_name)
     system('rails', 'generate', 'deployment', ENV['NAME'], server.public_ip_address)
   end
 
   desc 'Provision a deployable EC2 instance'
   task :ec2 do
-    RelevanceRails::Provision.create_ec2(ENV['NAME'])
+    server_name = "#{Rails.application.class.parent_name} #{ENV['NAME']}"
+    RelevanceRails::Provision.create_ec2(server_name)
   end
 
   desc 'Stop your previously-provisioned EC2 instance'

data/relevance_rails.gemspec

@@ -24,7 +24,12 @@ Gem::Specification.new do |s|
 
   # specify any dependencies here; for example:
   s.add_runtime_dependency 'rails', '~> 3.2'
-  s.add_runtime_dependency 'fog', '~> 1.3.0'
+
+  # be aware that we're monkey-patching fog in fog_ext/ssh.rb;
+  # if we update fog, that monkey-patch might need to be
+  # revisited
+  s.add_runtime_dependency 'fog', '1.3.1'
+
   s.add_runtime_dependency 'thor', '~> 0.14.6'
   s.add_development_dependency 'pry'
   s.add_development_dependency 'capybara'

data/script/ci

@@ -40,35 +40,25 @@ bundle exec rake spec
 
 echo "Building new relevance_rails gem..."
 rm -Rf relevance_rails-*.gem
-gem build --verbose relevance_rails.gemspec
+rake install --trace
 echo "Done"
 
-# NOTE: Manual install of just-built gem to workaround bundler wanting
-# it to live on rubygems.org.
-# NOTE: RVM 1.x doesn't work with "set -e", so you have to unset/set it.
+echo "Ensure we remove any old app directory or gemset..."
+rm -Rf ./$RAILS_APP
 set +e
 rvm --force gemset delete $RAILS_APP
-rvm gemset create $RAILS_APP
-rvm gemset use $RAILS_APP
-echo "RVM is now using:` rvm current`"
 set -e
+echo "Done"
 
-if $(gem list relevance_rails --installed); then
-  echo "Removing old relevance_rails gem..."
-  gem uninstall --verbose --all --executables relevance_rails
-  echo "Done"
-fi
-
-echo "Installing relevance_rails gem into $RAILS_APP gemset..."
-gem install relevance_rails-*.gem
-echo "Done."
-
-rm -Rf ./$RAILS_APP
-relevance_rails new $RAILS_APP --database=postgresql
+relevance_rails new $RAILS_APP --database=postgresql --relevance-dev
 
-# NOTE: this does NOT change the RVM gemset; we set it manually above
+# NOTE: this does NOT change the RVM gemset; we set it manually below
 builtin cd $RAILS_APP
+
+set +e
+rvm gemset use $RAILS_APP
 echo "RVM is now using:` rvm current`"
+set -e
 
 files_must_NOT_exist public/index.html
 files_must_exist config/database.yml provision/dna.json config/deploy.rb config/deploy/recipes/deploy.rb

data/spec/lib/provision_spec.rb

@@ -2,10 +2,14 @@ require 'spec_helper'
 require 'relevance_rails/provision'
 
 describe RelevanceRails::Provision do
+  def job(options={})
+    options = {:stdout => '', :stderr => '', :status => 0}.update options
+    mock(options)
+  end
+
   describe '.run_commands' do
     it "fails fast if a command fails" do
-      job = mock(:status => 1, :stdout => '', :stderr => 'FAIL WHALE', :command => 'exit 1')
-      server = mock(:ssh => [job])
+      server = mock(:ssh => [job(:status => 1, :command => 'exit 1', :stderr => 'FAIL WHALE')])
       capture_stdout do
         expect do
           RelevanceRails::Provision.run_commands server
@@ -13,4 +17,52 @@ describe RelevanceRails::Provision do
       end.should =~ /STDERR: FAIL WHALE/
     end
   end
+
+  describe '.wait_for_ssh' do
+
+    let(:server) { mock("server") }
+
+    before do
+      server.should_receive(:wait_for).ordered.and_return(true)
+    end
+
+    it 'retries if the first attempt fails' do
+      server.should_receive(:ssh).ordered.and_raise(Errno::ECONNREFUSED)
+      server.should_receive(:ssh).ordered.and_return([job(:command => 'echo')])
+      RelevanceRails::Provision.should_receive(:sleep).twice.with(10).and_return(10)
+      expect do
+        capture_stdout { RelevanceRails::Provision.wait_for_ssh(server) }
+      end.to_not raise_error
+    end
+
+    it 'retries up to five times, then fails' do
+      server.should_receive(:ssh).ordered.exactly(5).times.and_raise(Errno::ECONNREFUSED)
+      RelevanceRails::Provision.should_receive(:sleep).exactly(5).times.with(10).and_return(10)
+      expect do
+        capture_stdout { RelevanceRails::Provision.wait_for_ssh(server) }
+      end.to raise_error SystemExit
+    end
+  end
+
+  describe '.apt_installs' do
+    it 'retries if the first attempt fails' do
+      server = mock("server")
+      server.should_receive(:ssh).ordered.with('sudo apt-get update').and_return([job(:command => 'sudo apt-get update')])
+      server.should_receive(:ssh).ordered.with('sudo apt-get -y install ruby').and_return([job(:status => 1, :command => 'sudo apt-get -y install ruby')])
+      server.should_receive(:ssh).ordered.with('sudo apt-get update').and_return([job(:command => 'sudo apt-get update')])
+      server.should_receive(:ssh).ordered.with('sudo apt-get -y install ruby').and_return([job(:command => 'sudo apt-get -y install ruby')])
+      server.should_receive(:ssh).ordered.with('sudo apt-get -y install rubygems1.8').and_return([job(:command => 'sudo apt-get -y install rubygems1.8')])
+      expect do
+        capture_stdout { RelevanceRails::Provision.apt_installs(server) }
+      end.to_not raise_error
+    end
+
+    it 'tries twice, then fails' do
+      server = mock("server")
+      server.should_receive(:ssh).exactly(3).with('sudo apt-get update').and_return([job(:status => 1, :command => 'sudo apt-get update')])
+      expect do
+        capture_stdout { RelevanceRails::Provision.apt_installs(server) }
+      end.to raise_error SystemExit
+    end
+  end
 end

metadata

@@ -1,15 +1,15 @@
 --- !ruby/object:Gem::Specification
 name: relevance_rails
 version: !ruby/object:Gem::Version
-  version: 0.1.1.alpha
-  prerelease: 6
+  version: 0.1.2
+  prerelease: 
 platform: ruby
 authors:
 - Alex Redington
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2012-04-12 00:00:00.000000000 Z
+date: 2012-04-19 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rails
@@ -32,17 +32,17 @@ dependencies:
   requirement: !ruby/object:Gem::Requirement
     none: false
     requirements:
-    - - ~>
+    - - '='
       - !ruby/object:Gem::Version
-        version: 1.3.0
+        version: 1.3.1
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     none: false
     requirements:
-    - - ~>
+    - - '='
       - !ruby/object:Gem::Version
-        version: 1.3.0
+        version: 1.3.1
 - !ruby/object:Gem::Dependency
   name: thor
   requirement: !ruby/object:Gem::Requirement
@@ -139,6 +139,7 @@ files:
 - Gemfile
 - README.md
 - Rakefile
+- USAGE.md
 - aws_config.example.yml
 - bin/relevance_rails
 - lib/generators/deployment/deployment_generator.rb
@@ -158,6 +159,7 @@ files:
 - lib/generators/relevance_file/templates/spec_helper.rb
 - lib/relevance_rails.rb
 - lib/relevance_rails/chef_dna.rb
+- lib/relevance_rails/fog_ext/ssh.rb
 - lib/relevance_rails/generator_overrides.rb
 - lib/relevance_rails/provision.rb
 - lib/relevance_rails/public_key_fetcher.rb
@@ -190,9 +192,9 @@ required_ruby_version: !ruby/object:Gem::Requirement
 required_rubygems_version: !ruby/object:Gem::Requirement
   none: false
   requirements:
-  - - ! '>'
+  - - ! '>='
     - !ruby/object:Gem::Version
-      version: 1.3.1
+      version: '0'
 requirements: []
 rubyforge_project: relevance_rails
 rubygems_version: 1.8.21