pawnee 0.0.5

data/.gitignore

@@ -0,0 +1,18 @@
+*.gem
+*.rbc
+.bundle
+.config
+.yardoc
+Gemfile.lock
+InstalledFiles
+_yardoc
+coverage
+doc/
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp
+coverage

data/.rspec

@@ -0,0 +1,2 @@
+--colour
+--format nested

data/.travis.yml

@@ -0,0 +1,7 @@
+language: ruby
+rvm:
+  - 1.9.2
+  - 1.9.3
+  - jruby-19mode # JRuby in 1.9 mode
+# uncomment this line if your project needs to run something other than `rake`:
+# script: bundle exec rspec spec

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in pawnee.gemspec
+gemspec

data/LICENSE

@@ -0,0 +1,22 @@
+Copyright (c) 2012 Ryan Stout
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,158 @@
+# Pawnee - making server provisioning easier
+
+[NOTE: This is still a work in progress and not ready for use yet]
+
+## Why
+
+You may wonder why do we need another provisioning system, [here's](https://github.com/ryanstout/pawnee/blob/master/docs/FAQ.md) my answer.
+
+## Goals
+
+This system will:
+
+1. **Setup recipes as gems**
+  - Gems get prefixed so you can search for them
+  - Gems inherit from main gem
+2. **Gems recipes can be overridden in a rails folder**
+	- either by a load path solution or monkeypatching
+	  - they have a classname that can be predicted from the gem name
+			- so anything that overrides that class can monkeypatch or replace parts of it
+3. **It should use bundler**
+  - so you just list the things you need on your server in a certain group (that only gets run on when installing)
+  - you could use :path => '...'
+  - the gems don't install the things in a bundle, just provide a task you can run to set them up
+4. **It should run gems locally or remote**
+	- based on the --servers option passed in
+5. **It should integrate with capistrano**
+	- we need a place to store server info....
+		- redis somewhere?
+		- Serverfile/Serverfile.lock
+  - the servers and roles should be maintained by this system
+  - but deploying should be a separate thing (not like rubber)
+6. **It should use thor for template stuff**
+	- and just overwrite it so files get copied to remote destinations if needed
+
+
+
+## Installation
+
+Add any pawnee gem's you want to use to bundler:
+
+    gem 'pawnee-nginx'
+
+You can see currently available recipe gems [here](https://rubygems.org/search?utf8=%E2%9C%93&query=pawnee)
+
+[Or you can create your own recipy gems](https://github.com/ryanstout/pawnee/blob/master/docs/GUIDE.md)
+
+## Usage
+
+### Config File
+
+Setup a config/pawnee.yml file.  In this file you can specify any options you want to pass along with the servers:
+
+Here's an example config with server's:
+
+		servers:
+		  - domain: server1.com
+		    roles: [nginx, unicorn]
+		  - domain: server2.com
+		    roles: [unicorn]
+
+Then run:
+
+		bundle exec pawnee setup
+
+## Contributing
+
+1. Fork it
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Added some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create new Pull Request
+
+
+
+
+## Pawnee Provides Helpers For:
+
+- files
+- directories
+- packages
+- service calls (via invoking methods in other recipes)
+- compile (untar/make/make install)
+- users
+- cron? [not complete]  (maybe leverage whenever gem)
+- env [not complete]  (manage adding things to .base_profile somehow - maybe leverage insert_into_file stuff?)
+
+## Standard Tasks
+
+Pawnee defined some convention for task names so all gem's can provide standard ways to call things
+
+	- #setup
+	- #teardown
+	- #restart
+	- #stop
+	- #start
+
+
+## Options
+
+In a task, recipes can access and change the self.options hash.  These values will be
+passed from one task to another.  Gem tasks will be executed in the order they are defined
+in bundler (at the moment, this may change)
+
+## Configuration
+
+standard config options:
+- servers
+- git_repo_url
+- web_root
+- aws...
+- s3...
+
+## exposed by unicorn for example
+app_server_locations ['localhost:3000', 'localhost:3001'] - gets picked up on by nginx maybe?
+
+
+
+### ThorSsh Extensions to Thor
+We make a few modifications to allow it to be used as the base to pawnee
+1) We make it so all actions can be run either locally or remotely via ssh through the thor-ssh gem
+2) We make options mutable and use that to allow the passing around of options
+
+
+
+
+server connection system
+  - fog
+  - looks at fog api and tags to figure out what servers are running and their roles
+
+core gem
+  - provides dsl features for setting stuff up
+    - packages ['...', '...']  # uses apt
+    - file copy in with erb
+  - all DSL commands can be run either locally or remote
+    - so file copy uploads if needed
+  - provides setup for defaults
+    - ec2/s3 credentials
+    - deploy_path(/mnt/[name])
+  - provides generator for building new gems
+
+deployment:
+  - include in deploy.rb to setup servers and roles
+  - standard cap deploy
+
+
+
+
+
+### Roles
+- gems define a roles they provide
+	- [logically, this can only be one role right?]
+	- multiple gems can provide the same role, but one gem can not provide multiple
+- you say which roles you want to run (as --roles)
+- servers say which roles they provide
+
+1) list of gems that run the listed roles
+2) run on each server that provides
+

data/Rakefile

@@ -0,0 +1,11 @@
+#!/usr/bin/env rake
+require "bundler/gem_tasks"
+require 'rake/testtask'
+
+Rake::TestTask.new do |t|
+  t.libs.push "lib"
+  t.test_files = FileList['test/*_test.rb']
+  t.verbose = true
+end
+
+task :default => :test

data/bin/pawnee

@@ -0,0 +1,10 @@
+#!/usr/bin/env ruby
+require "rubygems"
+require "bundler/setup"
+require 'pawnee/cli'
+
+Bundler.require(:default)
+
+Pawnee::CLI.start(ARGV)
+
+

data/docs/FAQ.md

@@ -0,0 +1,33 @@
+# Why is this better than Chef/Puppet?
+
+Thats a tough question.  I try to avoid reinventing the wheel, but it seems like the current wheel has some problems.
+
+The main reasons is that Chef and Puppet are both huge solutions to a problem I think could be simplified with a few constraints and better choices.  (chef is over 110k lines of ruby, it doesn't feel like this is a 110k lines of ruby problem)
+
+Also, both seem to have business models that actually harm the product.  For example with Chef, they charge you for a product to help set everything up.  That means that they have incentives to make it difficult to setup.
+
+Here's a few reason why Pawnee is better:
+
+1) Simpler, well written, documented code
+2) Leverages existing technology people know (rubygems, thor)
+3) Code reuse is built in (all recipes are gems, any part of the gem can be overridden without changing the gem - like rails engines)
+4) No need to bootstrap ruby - All actions can be run either locally (with a local ruby) or from a remote server over ssh
+5) Clean logging (yea, this is important)
+6) Tests - everything is tested using Vagrant
+
+
+Also, just for good measure, here's a few reasons why its worse:
+
+1) Only supports ubuntu (currently - the plan is to get the kinks worked out on ubuntu first)
+2) Fewer recipes (again, currently)
+
+
+## Problems with Chef/Puppet
+
+[Note: this is my opinion]
+
+1. Little to no testing of recipes (at least for Chef)
+2. Complicated recipe upgrading/code reuse patterns
+3. You need to run the code locally (should have option to run over ssh)
+
+[chef's providers as an example: http://wiki.opscode.com/display/chef/Resources]

data/docs/GUIDE.md

@@ -0,0 +1,102 @@
+# Recipe Gem Creation Guide
+
+## Overview
+
+Pawnee tries to make it very easy to create recipes for setting up things on remote or local servers.  Pawnee provides a gem generator to get started, simply run:
+
+		pawnee gem [gemname]
+
+This will generate files for a gem in the current directory.  The word Recipe is used to refer to these pawnee gems (since it is a standard term in the provisioning world.)
+
+## base.rb
+
+The main code for your gem will go in lib/pawnee/[gemname]/base.rb  This file is a recipe class that inherits from Pawnee::Base.  
+
+Pawnee uses the [thor gem](https://github.com/wycats/thor) to provide things like tasks and actions.  Pawnee::Base extends thor and adds in things to do the following:
+
+1. allow actions to run remotely over ssh or locally.
+2. allow tasks to run multiple times (once per server).
+3. allow defaults in a config file for options
+4. allow the options hash to be changed and passed between tasks
+
+By adding these to thor, Pawnee::Base provides a great starting point for creating recipes.  It also tracks any class that inherits from it and adds it to a list of recipes so you can run:
+
+		bundle exec pawnee setup
+		
+Running the above will automatically call setup on all pawnee recipes in the current Gemfile
+
+## Tasks
+
+Recipes have two default tasks, #setup and #teardown.  For most situations, think of #setup as "install" and #teardown as "uninstall".  To run setup now, in your gem directory you can do:
+
+		bundle
+		bundle exec pawnee [gemname] setup
+
+Also, you could place the gem in another projects Gemfile (using the :path option for now) and then run something like:
+
+		bundle exec pawnee setup
+
+Again, this will invoke the setup task on all pawnee gems in the Gemfile.
+
+Tasks can call other tasks using the #invoke method (see thor)
+
+## Servers Option
+
+We'll come back to options in a minute, but quickly we have to talk about one special option, 'servers'.  'servers' should be setup as a default option on your setup and teardown tasks.
+
+Servers can be a few things:
+
+1. an array of domains (with an assumed default user of ubuntu)
+2. an array of hashs like the following:
+		[{'domain' => 'something.com', 'roles' => ['your gemname']}, ...]
+3. a mix of either
+
+These options can either be passed in on the command line like so:
+
+		bundle exec pawnee yourgemname setup --servers something.com
+
+Or they can be setup in the [config.yml file](https://github.com/ryanstout/pawnee#config-file).
+
+## Actions
+
+### Thor Actions
+
+Since Pawnee is build on thor, we can use any of the [thor actions](https://github.com/wycats/thor/wiki/Actions) inside our class.  Pawnee includes a gem called thor-ssh (which was written for pawnee) that extends thor and allows a .destination_connection to be set on any thor class.  .destination_connection is automatically setup for any task if the 'servers' option is set (again, either from the command line or in the config.yml file)  When destination_connection is set, any of the actions will use the local machine for the source, and the destination machine for the destination.
+
+So running something like the following:
+
+		copy_file('templates/test.txt', '/home/ubuntu/test.txt')
+
+Would copy the file from templates/test.txt (in the gem) to /home/ubuntu/test.txt on the remote server (or servers if multiple servers were passed in)
+
+**note**: in the current system, if multiple servers are passed in, the task is simply run once for each server.
+
+thor-ssh also provides two methods for running any command:
+
+ #run and #exec
+
+ #run will log the command being run, #exec will not
+
+### Pawnee Actions
+
+Pawnee also provides its own set of actions for common server tasks.
+
+install_package, remove_package, compile, create_user, delete_user
+
+## Options
+
+Pawnee also uses thor's option system.  This means your tasks should use thor's #desc and #method_option.  ([Read more here](https://github.com/wycats/thor/wiki/Getting-Started) and [here](https://github.com/wycats/thor/wiki/Method-Options))  This allows for an easily defined set of options for any task.  This also allows anyone to see what options the tasks take:
+
+		bundle exec pawnee [yourgemname] help setup
+		
+^ would return a list of the options along with descriptions
+
+By default pawnee makes it so any method options are scoped to the current gem.  However pawnee makes it easy for gems to share options.  Any gem can change the self.options hash and it will be passed (in its changed form) to the next task.  This means you could easily have one gem provide things like path to another gem.  (Just make sure the gem providing the paths is before the 2nd gem in the Gemfile).  When sharing options, some options make since to not be scoped to the gem.  In this case, you can run the #method_options inside of a global_options do block.
+
+		global_options do
+			method_option :deploy_path, :required => true
+		end
+
+This makes it easy to have the same option be used between gems.
+
+	

data/docs/TODO.md

@@ -0,0 +1,10 @@
+TODO: Provide testing stubs for gems
+TODO: Add a as_user('user') do .. end option
+				- needs to look to options for how to get to root
+				- have it run all commands as that user (for sftp actions we'll set the own after)
+				- we'll need to change exec and run to work from within a shell session
+					- maybe have an option to run from within a shell, or we could get them into the right place every time
+				- maybe we should add a system to "get you to root, then get you to another user"
+TODO: Need to make a clear way for pawnee gems (and recipes) to provide actions (example, git gem provides git actions)
+TODO: Run actions in threads (across multiple servers)
+TODO: Test to make sure arguments work directly as well (they probably don't right now)

data/lib/pawnee/pawnee/actions/base_model.rb

@@ -0,0 +1,35 @@
+require 'active_model'
+
+module Pawnee
+  module Actions
+    
+    class BaseModel
+      include ActiveModel::Dirty
+      attr_accessor :new_record
+      
+      def new_record?
+        !!@new_record
+      end
+
+      def update_attributes(attributes)
+        attributes.each_pair do |key,value|
+          self.send(:"#{key}=", value) if self.respond_to?(:"#{key}=")
+        end
+      end
+      
+      def self.change_attr_accessor(method_names)
+        [method_names].flatten.each do |method_name|
+          self.send(:define_method, :"#{method_name}") do
+            return instance_variable_get("@#{method_name}")
+          end
+          
+          self.send(:define_method, :"#{method_name}=") do |val|
+            self.send(:"#{method_name}_will_change!") unless val == instance_variable_get("@#{method_name}")
+            instance_variable_set("@#{method_name}", val)
+          end
+        end
+      end
+    end
+    
+  end
+end

data/lib/pawnee/pawnee/actions/compile.rb

@@ -0,0 +1,140 @@
+module Pawnee
+  module Actions
+    
+    # Takes a tar.gz or zip file and unzips it, and runs the 
+    # standard ./configure, make, sudo make install
+    #
+    # It attempts to raise an exception at any compilation 
+    # failure.
+    #
+    #  compile 'http://nginx.org/download/nginx-1.2.0.tar.gz'
+    #
+    # === Parameters
+    # url<String>:: The url to download
+    # temp_dir<String>:: Where the compilation should take place
+    # options<Hash>:: Hash of options, see below
+    #
+    # === Options
+    # #compile requires that you either specify a :bin_file option
+    # that the method can check for on the remote system, or that 
+    # you pass a block that returns true if the app has already been
+    # installed
+    #
+    # :bin_file   - the name of an executable that the method can check for in the path
+    # :configure  - a string of options to pass to the ./configure command.
+    #
+    # === Block
+    # You can also pass a block that if it returns true, it will not 
+    # recompile.  So the general idea is return true if the exe is already
+    # installed.
+    def compile(url, temp_dir, options={})
+      # TODO: Add invoke/revoke support using action(...), maybe 
+      # make things run via Thor::Group
+      
+      installed = false
+      if options[:bin_file]
+        # Check if the bin file is installed
+        installed = exec(options[:bin_file]).strip != ''
+      else
+        raise "You must pass :bin_file or a block to compile" unless block_given?
+        installed = yield()
+      end
+      
+      if installed
+        say_status :already_compiled, url, :blue
+        return true
+      else
+        # Compile and install
+        Compile.new(self, url, temp_dir, options)
+      end
+    end
+    
+    class Compile
+      attr_accessor :base
+      attr_accessor :options
+      
+      # Sets up a compile object
+      #
+      # === Parameters
+      # base<Thor>:: The main class
+      # url<String>:: The url to download from
+      # temp_dir<String>:: A path to a temporary directory where the compilation 
+      # can take place
+      def initialize(base, url, temp_dir, options)
+        @base = base
+        @temp_dir = temp_dir
+        self.options = options
+        @file = File.basename(url)
+        @zip_file = @file[/[.]zip$/]
+        @file_path = File.join(temp_dir, @file)
+        
+        # TODO: GET THE FIRST DIRECTORY INSTEAD
+        dir_name = @file.gsub(/[.]tar[.]gz$/, '').gsub(/[.]zip$/, '')
+        @extracted_path = File.join(temp_dir, dir_name)
+        
+        @base.say_status 'download and compile', url
+        
+        download(url)
+        extract
+        configure
+        make
+        make_install
+      end
+      
+      # Uses get to download the file and place it in the temp path
+      def download(url)
+        base.get(url, @file_path)
+      end
+      
+      # Extract the compressed file
+      def extract
+        base.say_status 'extract', @file
+        if @zip_file
+          base.exec("cd #{@temp_dir} ; unzip #{@file}")
+        else
+          base.exec("cd #{@temp_dir} ; tar xvfpz #{@file}")
+        end
+      
+        # Remove the file
+        base.destination_files.rm_rf(@file_path)
+      end
+      
+      # Takes a command and an action_name.  It says its running the action_name, then
+      # runs the command and if there is non-zero exit status, it prints out an error,
+      # stdout, and stderr, then raises an exception
+      #
+      # === Parameters
+      # command<String>:: The command to be run
+      # action_name<String>:: An action name (used to explain what is happening)
+      def run_with_failure_handler(command, action_name)
+        base.say_status action_name.downcase, ''
+        stdout, stderr, exit_code, exit_status = base.exec(command, true)
+        
+        if exit_code != 0
+          base.say_status :error, "Unable to #{action_name}, see output below", :red
+          puts stdout
+          puts stderr
+          
+          raise "Unable to configure #{action_name}"
+        end
+      end
+
+      # Runs ./configure on the files
+      def configure
+        run_with_failure_handler("cd #{@extracted_path} ; ./configure #{options[:configure] || ''}", 'configure')
+      end
+      
+      # Runs make
+      def make
+        run_with_failure_handler("cd #{@extracted_path} ; make", 'make')
+      end
+      
+      # Runs sudo make install
+      def make_install
+        run_with_failure_handler("cd #{@extracted_path} ; sudo make install", 'make install')
+      end
+      
+    end
+  end
+
+end

data/lib/pawnee/pawnee/actions/package.rb

@@ -0,0 +1,111 @@
+module Pawnee
+  module Actions
+    
+    # Returns true if the package is installed.  If a version is 
+    # listed, it will make sure it matches the version.
+    #
+    #  package_installed? 'memcached'
+    #  -> true
+    #
+    #  package_installed? 'memcached', '1.4.7-0.1ubuntu1'
+    #  -> false
+    #
+    # === Parameters
+    # package_name<String>:: The name of package
+    def package_installed?(package_name, version=nil)
+      installed_version = installed_package_version(package_name)
+      if version
+        return (installed_version == version)
+      else
+        return !!installed_version
+      end
+    end
+    
+    # Returns the version of a package that is installed
+    #
+    #  installed_package_version 'memcached'
+    #  -> 1.4.7-0.1ubuntu1
+    #
+    # === Parameters
+    # package_name<String>:: The name of package
+    def installed_package_version(package_name)
+      packages = nil
+      as_root do
+        packages = exec("dpkg -l")
+      end
+      
+      packages.split(/\n/).grep(/^ii /).each do |package|
+        _, name, version = package.split(/\s+/)
+        
+        if name == package_name
+          return version
+        end
+      end
+      
+      return nil
+    end
+    
+    # Installs a package using the operating system's
+    # package management system (currentl apt-get only)
+    #
+    #  install_package 'mysql-server5'
+    #  install_package 'gcc', '4.5'
+    #
+    # === Parameters
+    # package_name<String>:: The name of package to be installed
+    # version<String>:: The version of the package
+    def install_package(package_name, version=nil)
+      if package_installed?(package_name)
+        say_status "package already installed", package_name
+      else
+        package_name = "#{package_name}=#{version}" if version
+        as_root do
+          exec("apt-get -y install #{package_name}")
+        end
+        say_status "installed package", package_name.gsub('=', ' ')
+      end
+    end
+    
+    # Installs a set of packages, use install_package if you 
+    # wish to specify a version
+    # 
+    #  install_packages 'build-essential', 'zlib1g-dev'
+    #
+    # === Parameters
+    # packages<Array>:: An array of strings of package names
+    def install_packages(*package_names)
+      package_names.flatten.each do |package_name|
+        install_package(package_name)
+      end
+    end
+    
+    # Removes package_name
+    #
+    # === Parameters
+    # package_name<String>:: The name of package
+    def remove_package(package_name)
+      if package_installed?(package_name)
+        say_status "removed package", package_name
+        as_root do
+          exec("apt-get -y remove #{package_name}")
+        end
+      else
+        say_status "package not removed", package_name
+      end
+    end
+    
+    # Removes a set of packages, use remove_package if you 
+    # wish to specify a version
+    #
+    #  remove_packages 'build-essential', 'zlib1g-dev'
+    #
+    # === Parameters
+    # packages<Array>:: An array of strings of package names
+    def remove_packages(*package_names)
+      package_names.flatten.each do |package_name|
+        remove_package(package_name)
+      end
+    end
+    
+  end
+end