octopusci 0.2.3 → 0.3.0

data/README.markdown

@@ -0,0 +1,132 @@
+Octopusci
+=========
+
+Octopusci is fresh new take on a continuous integration server centralized
+around the concept of getting the same great CI benefits when using a
+multi-branch workflow.
+
+How's it Different?
+-------------------
+
+The impetus that brought Octopusci into being was simply the lack of CI servers
+that cleanly supported a software development workflow based on multiple
+branches. Secondarily, it was the excessive amount of effort necessary to get a
+basic CI server up and running.
+
+Octopsuci fills this gap by providing intelligent multi-branch queueing
+and multi-server job distribution. Beyond that it provides a
+solid continous integration server that is trivial to get setup and running. A number
+of the concepts used in Octopusci are pulled from
+[Continuous Delivery](http://continuousdelivery.com/),
+[Continuous Integration](http://martinfowler.com/articles/continuousIntegration.html)
+as well as Scott Chacon's post on the 
+[GitHub Flow](http://scottchacon.com/2011/08/31/github-flow.html).
+
+The following is a listing of a number of some of its more significant features.
+
+### Dynamic Multi-Branch Triggering
+
+Octopusci detects branch creation/modification and dynamically generates a build for
+that branch based on the project the pushed branch belongs to. Most existing CI servers
+that I have used force you to manually define jobs for each branch you would like it to
+manage.
+
+### Multi-Server Job Distribution
+
+Octopusci allows you to configure it to run "remote jobs" on numerous servers and it
+keeps track of which servers are currently busy as well as handing new jobs to the
+correct servers as they become available. This is extremely valuable if you are
+interested in running automated acceptance tests that take a long time to run such
+as Selenium/Cucumber & Capybara Tests.
+
+### Intelligent Multi-Branch Queueing
+
+Octopusci intelligently manages its job queue by by simply updating any pending jobs with
+the newly pushed branch data. This means that at any given point in time there is only
+ever one pending job for each branch. When, a code push comes into Octopusci it
+first looks to see if there is already a pending job for the branch that was pushed. If
+there is, it simply updates the jobs associated branch data. If there is not already a
+pending job then it queues a new job for that branch.
+
+### GitHub Integration ###
+
+Octopusci was designed specifically to integrate cleanly with GitHub's push notifications
+system. At some point in the future Octopusci may support more than just GitHub but for
+the time being GitHub is our primary focus.
+
+Install Guide
+-------------
+
+### Install Dependencies ###
+
+Octopusci has one major dependency at the moment, [Redis](http://redis.io/).
+[Redis](http://redis.io/) needs to be installed and configured to startup appropriately
+on the box you plan to run Octopusci on.
+
+On Debian/Ubuntu machines this is to my knowledge as easy as `apt-get install redis-server`.
+
+On Mac OS X machines this can easly be installed via [brew](http://mxcl.github.com/homebrew/)
+using `brew install redis`. Follow the on screen instructions to configure it to auto
+startup when you boot up as well as simply how to run the server manually.
+
+### Gem & Init Skel ###
+
+    $ gem install octopusci
+    $ sudo octopusci-skel
+
+The `octopusci-skel` command will make sure the `/etc/octopusci` path exists and its
+underlying structure. It will also create a default example `/etc/ocotpusci/config.yml`
+if one is not found.
+
+### Update Example Config ###
+
+Now that the `/etc/octopusci/config.yml` example config has been created for you it is
+time to go check it out and update some of the values in it.
+
+TODO: Fill this out with details on the config, required fields, optional fields, etc.
+
+### Jobs ###
+
+Add any jobs you would like to the `/etc/octopusci/jobs` directory as .rb files
+and Octopusci will load them appropriately when started.
+
+### Web Interface ###
+
+Figure out what directory the gem is installed in by running the following
+command and stripping off the `lib/octopusci.rb` at the end.
+
+    gem which octopusci
+
+Once you have the path we can use that path to setup Passenger with Apache
+or something else like nginx.
+
+Apache virtual host example
+
+    <VirtualHost *:80>
+      ServerName octopusci.example.com
+      PassengerAppRoot /path/of/octpusci/we/got/before
+      DocumentRoot /path/of/octpusci/we/got/before/lib/octopusci/server/public
+      <Directory /path/of/octpusci/we/got/before/lib/octopusci/server/public>
+        Order allow,deny
+        Allow from all
+        AllowOverride all
+        Options -MultiViews
+      </Directory>
+    </VirtualHost>
+
+The above will give us the web Octopusci web interface.
+
+If you are developing you can simply start this up by running
+`rackup -p whatever_port` while inside the octopusci directory where the
+`config.ru` file exists.
+
+I recommend you setup the second half of Octopusci (`octopusci-tentacles`) with
+God or some other monitoring system. However, for development you can simply
+run `octopusci-tentacles` directoly as follows:
+
+    otopusci-tentacles
+
+Screenshots
+-----------
+
+![Octopusci - Dashboard](https://img.skitch.com/20111005-tfxgw59mec5msnfu3pd6is3btf.jpg)

data/bin/octopusci-reset-redis

@@ -0,0 +1,26 @@
+#!/usr/bin/env ruby
+
+$LOAD_PATH.unshift File.expand_path(File.dirname(__FILE__) + '/../lib')
+
+require 'rubygems'
+require 'octopusci'
+
+num_jobs = Octopusci::JobStore.size
+jobs = Octopusci::JobStore.list(0, num_jobs)
+
+jobs.each do |j|
+  # delete the actual job record
+  Octopusci::JobStore.redis.del("octopusci:jobs:#{j['id']}")
+
+  # delete the repo_name, branch_name job references
+  Octopusci::JobStore.redis.del("octopusci:#{j['repo_name']}:#{j['branch_name']}:jobs")
+
+  # delete the github payload for the repo_name, branch_name
+  Octopusci::JobStore.redis.del("octopusci:github_payload:#{j['repo_name']}:#{j['branch_name']}")
+
+end
+
+Octopusci::JobStore.redis.del('octopusci:job_count')
+Octopusci::JobStore.redis.del('octopusci:jobs')
+Octopusci::JobStore.redis.del('octopusci:stagelocker')
+Octopusci::JobStore.redis.del('queue:octopusci:commit')

data/bin/octopusci-skel

@@ -19,6 +19,8 @@ if !File.exists?(CONFIG_PATH)
   File.open(CONFIG_PATH, 'w') do |f|
     f << "general:
   jobs_path: \"/etc/octopusci/jobs\"
+  base_url: \"http://localhost:9998\"
+  workspace_base_path: \"/Users/octopusci/.octopusci\"
 
 http_basic:
   username: admin
@@ -34,13 +36,6 @@ smtp:
   password: somepassword
   raise_delivery_errors: true
 
-db:
-  adapter: mysql
-  host: localhost
-  database: octopusci
-  username: someusername
-  password: somepassword
-
 projects:
   - { name: octopusci, owner: cyphactor, job_klass: SomeJobClass, repo_uri: 'git@github.com:cyphactor/octopusci.git', default_email: devs@example.com }
 

data/bin/octopusci-tentacles

@@ -26,8 +26,8 @@ require 'octopusci'
 # valid initial state. But, also in the case that octopusci-tentacles was
 # killed it nukes the potentially screwed up state the redis data is in and
 # initializes it the proper state based on the config.
-if Octopusci::CONFIG.has_key?('stages')
-  Octopusci::StageLocker.load(Octopusci::CONFIG['stages'])
+if Octopusci::Config.has_key?('stages')
+  Octopusci::StageLocker.load(Octopusci::Config['stages'])
 end
 
 case ARGV[0]

data/config.ru

@@ -4,7 +4,7 @@ require 'octopusci/server'
 
 # Set the AUTH env variable to your basic auth password to protect Resque.
 Resque::Server.use Rack::Auth::Basic do |username, password|
-  (username == Octopusci::CONFIG['http_basic']['username']) && (password == Octopusci::CONFIG['http_basic']['password'])
+  (username == Octopusci::Config['http_basic']['username']) && (password == Octopusci::Config['http_basic']['password'])
 end
 
 run Rack::URLMap.new("/" => Octopusci::Server.new, "/resque" => Resque::Server.new)

data/lib/octopusci.rb

@@ -1,17 +1,13 @@
 require 'time-ago-in-words'
 
 require 'octopusci/version'
+require 'octopusci/errors'
 require 'octopusci/helpers'
-require 'octopusci/schema'
+require 'octopusci/io'
+require 'octopusci/job_store'
 require 'octopusci/notifier'
 require 'octopusci/queue'
 require 'octopusci/stage_locker'
 require 'octopusci/job'
 require 'octopusci/config'
 require 'octopusci/worker_launcher'
-
-module Octopusci
-  def self.greet
-    return "Hello RSpec!"
-  end
-end

data/lib/octopusci/config.rb

@@ -1,25 +1,34 @@
 require 'octopusci/notifier'
-require 'active_record'
 
 module Octopusci
-  class Config
+  class ConfigStore
     class MissingConfigField < RuntimeError; end
-    class ConfigNotInitialized < RuntimeError; end
     
-    def initialize()
+    def initialize
+      reset()
+    end
+
+    def reset
       @options = {}
     end
-    
-    # read the configuration values into the object from a YML file.
-    def load_yaml(yaml_file)
-      @options = YAML.load_file(yaml_file)
+
+    def options
+      @options
+    end
+
+    def load(yaml_file = nil, &block)
+      load_yaml(yaml_file) if !yaml_file.nil?
+      yield self if block
+      after_load()
+    end
+
+    def reload(yaml_file = nil, &block)
+      reset()
+      load(yaml_file, &block)
     end
     
     # allow options to be accessed as if this object is a Hash.
     def [](key_name)
-      if @options.nil?
-        raise ConfigNotInitialized, "Can't access the '#{key_name}' field because the config hasn't been initialized."
-      end
       if !@options.has_key?(key_name.to_s())
         raise MissingConfigField, "'#{key_name}' is NOT defined as a config field."
       end
@@ -46,50 +55,55 @@ module Octopusci
         return self[key_name_str]
       end
     end
-  end
-  
-  # On evaluation of the module it defines a new singleton of Config.
-  if (!defined?(CONFIG))
-    CONFIG = Config.new()
-  end
 
-  def self.configure(yaml_file = nil, &block)    
-    CONFIG.load_yaml(yaml_file) if !yaml_file.nil?
-    yield CONFIG if block
+    def after_load(&block)
+      if block
+        @after_load = block
+      elsif @after_load
+        @after_load.call
+      end
+    end
+
+    private
 
-    Notifier.default :from => Octopusci::CONFIG['smtp']['notification_from_email']
-    Notifier.delivery_method = :smtp
-    Notifier.smtp_settings = {
-      :address => Octopusci::CONFIG['smtp']['address'],
-      :port => Octopusci::CONFIG['smtp']['port'].to_s,
-      :authentication => Octopusci::CONFIG['smtp']['authentication'],
-      :enable_starttls_auto => Octopusci::CONFIG['smtp']['enable_starttls_auto'],
-      :user_name => Octopusci::CONFIG['smtp']['user_name'],
-      :password => Octopusci::CONFIG['smtp']['password'],
-      :raise_delivery_errors => Octopusci::CONFIG['smtp']['raise_delivery_errors']
-    }
-    Notifier.logger = Logger.new(STDOUT)
+    # read the configuration values into the object from a YML file.
+    def load_yaml(yaml_file)
+      @options.merge!(YAML.load_file(yaml_file))
+    end
   end
-end
-
-# Load the actual config file
-Octopusci.configure("/etc/octopusci/config.yml")
 
-if Octopusci::CONFIG['stages'] == nil
-  raise "You have defined stages as an option but have no items in it."
+  # On evaluation of the module it defines a new singleton of Config.
+  if (!defined?(::Octopusci::Config))
+    ::Octopusci::Config = ConfigStore.new()
+  end
 end
 
-ActiveRecord::Base.establish_connection(
-  :adapter => Octopusci::CONFIG['db']['adapter'],
-  :host => Octopusci::CONFIG['db']['host'],
-  :database => Octopusci::CONFIG['db']['database'],
-  :username => Octopusci::CONFIG['db']['username'],
-  :password => Octopusci::CONFIG['db']['password']
-)
+# Setup the config after load callback
+Octopusci::Config.after_load do
+  if Octopusci::Config['stages'] == nil
+    raise "You have defined stages as an option but have no items in it."
+  end
+
+  Octopusci::Notifier.default :from => Octopusci::Config['smtp']['notification_from_email']
+  Octopusci::Notifier.delivery_method = :smtp
+  Octopusci::Notifier.smtp_settings = {
+    :address => Octopusci::Config['smtp']['address'],
+    :port => Octopusci::Config['smtp']['port'].to_s,
+    :authentication => Octopusci::Config['smtp']['authentication'],
+    :enable_starttls_auto => Octopusci::Config['smtp']['enable_starttls_auto'],
+    :user_name => Octopusci::Config['smtp']['user_name'],
+    :password => Octopusci::Config['smtp']['password'],
+    :raise_delivery_errors => Octopusci::Config['smtp']['raise_delivery_errors']
+  }
+  Octopusci::Notifier.logger = Logger.new(STDOUT)
 
-Dir.open(Octopusci::CONFIG['general']['jobs_path']) do |d|
-  job_file_names = d.entries.reject { |e| e == '..' || e == '.' }
-  job_file_names.each do |f_name|
-    require Octopusci::CONFIG['general']['jobs_path'] + "/#{f_name}"
+  Dir.open(Octopusci::Config['general']['jobs_path']) do |d|
+    job_file_names = d.entries.reject { |e| e == '..' || e == '.' }
+    job_file_names.each do |f_name|
+      load Octopusci::Config['general']['jobs_path'] + "/#{f_name}"
+    end
   end
 end
+
+# Load the actual config file
+Octopusci::Config.load("/etc/octopusci/config.yml")

data/lib/octopusci/errors.rb

@@ -1,4 +1,6 @@
 module Octopusci
   # Raised when a method that is meant to be pure virtual that is not overloaded is called
   class PureVirtualMethod < RuntimeError; end
+  class JobRunFailed < RuntimeError; end
+  class JobHalted < RuntimeError; end
 end

data/lib/octopusci/helpers.rb

@@ -7,31 +7,32 @@ module Octopusci
       attrs = {}
             
       # ref
-      attrs[:ref] = gh_pl["ref"]
+      attrs['ref'] = gh_pl["ref"]
+      attrs['branch_name'] = gh_pl["ref"].split('/').last
       # compare
-      attrs[:compare] = gh_pl["compare"]
+      attrs['compare'] = gh_pl["compare"]
       # repo_name
-      attrs[:repo_name] = gh_pl["repository"]["name"]
+      attrs['repo_name'] = gh_pl["repository"]["name"]
       # repo_owner_name
-      attrs[:repo_owner_name] = gh_pl["repository"]["owner"]["name"]
+      attrs['repo_owner_name'] = gh_pl["repository"]["owner"]["name"]
       # repo_owner_email
-      attrs[:repo_owner_email] = gh_pl["repository"]["owner"]["email"]
+      attrs['repo_owner_email'] = gh_pl["repository"]["owner"]["email"]
       # repo_pushed_at
-      attrs[:repo_pushed_at] = gh_pl["repository"]["pushed_at"]
+      attrs['repo_pushed_at'] = gh_pl["repository"]["pushed_at"]
       # repo_created_at
-      attrs[:repo_created_at] = gh_pl["repository"]["created_at"]
+      attrs['repo_created_at'] = gh_pl["repository"]["created_at"]
       # repo_desc
-      attrs[:repo_desc] = gh_pl["repository"]["description"]
+      attrs['repo_desc'] = gh_pl["repository"]["description"]
       # repo_url
-      attrs[:repo_url] = gh_pl["repository"]["url"]
+      attrs['repo_url'] = gh_pl["repository"]["url"]
       # before_commit
-      attrs[:before_commit] = gh_pl["before"]
+      attrs['before_commit'] = gh_pl["before"]
       # forced
-      attrs[:forced] = gh_pl["forced"]
+      attrs['forced'] = gh_pl["forced"]
       # after_commit
-      attrs[:after_commit] = gh_pl["after"]
+      attrs['after_commit'] = gh_pl["after"]
       
-      attrs[:payload] = gh_pl
+      attrs['payload'] = gh_pl
       
       return attrs
     end
@@ -41,7 +42,7 @@ module Octopusci
     # this method returns nil. Otherwise, this project returns a hash of the
     # project info that it found in the config.
     def self.get_project_info(project_name, project_owner)
-      Octopusci::CONFIG["projects"].each do |proj|
+      Octopusci::Config["projects"].each do |proj|
         if (proj['name'] == project_name) && (proj['owner'] == project_owner)
           return proj
         end
@@ -58,7 +59,7 @@ module Octopusci
     end
     
     def self.workspace_path(stage)
-      return Octopusci::CONFIG['general']['workspace_base_path'] + "/#{stage}"
+      return Octopusci::Config['general']['workspace_base_path'] + "/#{stage}"
     end
   end
 end

data/lib/octopusci/io.rb

@@ -0,0 +1,70 @@
+module Octopusci
+  class IO
+    attr_accessor :job
+
+    def initialize(job)
+      @job = job
+    end
+
+    def read_all_out
+      if File.exists?(abs_output_file_path)
+        cont = ""
+        f = File.open(abs_output_file_path, 'r')
+        cont = f.read()
+        f.close
+        return cont
+      else
+        return ""
+      end
+    end
+
+    def read_all_log
+      if File.exists?(abs_log_file_path)
+        return File.open(abs_log_file_path, 'r').read()
+      else
+        return ""
+      end
+    end
+
+    def write_out(msg="", &block)
+      write_raw_output(false, msg, &block)
+    end
+
+    def write_log(msg="", &block)
+      write_raw_output(true, msg, &block)
+    end
+
+    def write_raw_output(silently=false, msg="")
+      # Make sure that the directory structure is in place for the job output.
+      if !File.directory?(abs_output_base_path)
+        FileUtils.mkdir_p(abs_output_base_path)
+      end
+      
+      # Run the command and output the output to the job file
+      out_f = if silently
+        File.open(abs_log_file_path, 'a')
+      else
+        File.open(abs_output_file_path, 'a')
+      end
+
+      yield(out_f) if block_given?
+      out_f << msg unless msg.nil? || msg.empty?
+
+      out_f.close
+    end
+
+    private
+
+    def abs_output_file_path
+      return "#{abs_output_base_path}/output.txt"
+    end
+
+    def abs_log_file_path
+      return "#{abs_output_base_path}/silent_output.txt"
+    end
+
+    def abs_output_base_path
+      return "#{Octopusci::Config['general']['workspace_base_path']}/jobs/#{@job['id']}"
+    end
+  end
+end