octopusci 0.0.1

Sign up to get free protection for your applications and to get access to all the features.
data/LICENSE ADDED
@@ -0,0 +1 @@
1
+ TODO: add license
data/README ADDED
@@ -0,0 +1,63 @@
1
+ The purpose of this project is provide a simple CI server that will work with
2
+ GitHub Post-Receive hook. It is also specifically designed to handle multiple
3
+ build/job queues for each branch.
4
+
5
+ This would basically allow you to every time code is pushed to the central
6
+ repository enqueue a build/job for the specific branch. That way if you
7
+ have topic branches that are being pushed to the central repository along
8
+ side the mainline branch they will get queue properly as well.
9
+
10
+ My idea for implementation at this point is that if I can detect the branch
11
+ from the GitHub Post-Receive hook then I can identify branch. If I can
12
+ identify branch then I can maintain individual queues for each branch.
13
+
14
+ Then the execution would look at config values for predefined branches and
15
+ priorities for order of running them so that the mainline branch running its
16
+ jobs could take precedence over non specified topic branches.
17
+
18
+ Install redis and get it starting up appropriately
19
+
20
+ gem install octopusci
21
+ sudo octopusci-skel
22
+ octopusci-db-migrate
23
+
24
+ Then update the /etc/octopusci/config.yml appropriately.
25
+
26
+ Add any jobs you would like to the /etc/octopusci/jobs directory as rb files
27
+ and octopusci will load them appropriately when started.
28
+
29
+ Figure out what directory the gem is installed in by running the following
30
+ command and stripping off the lib/octopusci.rb at the end.
31
+
32
+ gem which octopusci
33
+
34
+ Once you have the path we can use that path to setup Passenger with Apache
35
+ or something else like nginx as well as setup the database. Note: You will
36
+ need to setup a database user and a database for octopusci. The settings for
37
+ these should be stored in /etc/octopusci/config.yml.
38
+
39
+ rake -f /path/of/octpusci/we/got/before/Rakefile db:migrate
40
+
41
+ <VirtualHost *:80>
42
+ ServerName octopusci.example.com
43
+ PassengerAppRoot /path/of/octpusci/we/got/before
44
+ DocumentRoot /path/of/octpusci/we/got/before/lib/octopusci/server/public
45
+ <Directory /path/of/octpusci/we/got/before/lib/octopusci/server/public>
46
+ Order allow,deny
47
+ Allow from all
48
+ AllowOverride all
49
+ Options -MultiViews
50
+ </Directory>
51
+ </VirtualHost>
52
+
53
+ The above will give us the web Octopusci web interface.
54
+
55
+ If you are developing you can simply start this up by running
56
+ rackup -p whatever_port while inside the octopusci directory where the
57
+ config.ru file exists.
58
+
59
+ I recommend you setup the second half of Octopusci (octopusci-tentacles) with
60
+ God or some other monitoring system. However, for development you can simply
61
+ run octopusci-tentacles directoly as follows:
62
+
63
+ otopusci-tentacles
data/bin/octopusci ADDED
@@ -0,0 +1,56 @@
1
+ #!/usr/bin/env ruby
2
+
3
+ $LOAD_PATH.unshift File.expand_path(File.dirname(__FILE__) + '/../lib')
4
+
5
+ require 'rubygems'
6
+ require 'optparse'
7
+
8
+ # This hash will hold all of the options
9
+ # parsed from the command-line by
10
+ # OptionParser.
11
+ options = {}
12
+
13
+ optparse = OptionParser.new do|opts|
14
+ # Set a banner, displayed at the top
15
+ # of the help screen.
16
+ opts.banner = "Usage: optparse1.rb [options] file1 file2 ..."
17
+
18
+ # Define the options, and what they do
19
+ options[:verbose] = false
20
+ opts.on( '-v', '--verbose', 'Output more information' ) do
21
+ options[:verbose] = true
22
+ end
23
+
24
+ options[:quick] = false
25
+ opts.on( '-q', '--quick', 'Perform the task quickly' ) do
26
+ options[:quick] = true
27
+ end
28
+
29
+ options[:logfile] = nil
30
+ opts.on( '-l', '--logfile FILE', 'Write log to FILE' ) do |file|
31
+ options[:logfile] = file
32
+ end
33
+
34
+ # This displays the help screen, all programs are
35
+ # assumed to have this option.
36
+ opts.on( '-h', '--help', 'Display this screen' ) do
37
+ puts opts
38
+ exit
39
+ end
40
+ end
41
+
42
+ # Parse the command-line. Remember there are two forms
43
+ # of the parse method. The 'parse' method simply parses
44
+ # ARGV, while the 'parse!' method parses ARGV and removes
45
+ # any options found there, as well as any parameters for
46
+ # the options. What's left is the list of files to resize.
47
+ optparse.parse!
48
+
49
+ puts "Being verbose" if options[:verbose]
50
+ puts "Being quick" if options[:quick]
51
+ puts "Logging to file #{options[:logfile]}" if options[:logfile]
52
+
53
+ ARGV.each do|f|
54
+ puts "Resizing image #{f}..."
55
+ sleep 0.5
56
+ end
@@ -0,0 +1,10 @@
1
+ #!/usr/bin/env ruby
2
+
3
+ $LOAD_PATH.unshift File.expand_path(File.dirname(__FILE__) + '/../lib')
4
+
5
+ require 'rubygems'
6
+ require 'octopusci'
7
+
8
+ ActiveRecord::Base.logger = Logger.new(STDOUT)
9
+ ActiveRecord::Migration.verbose = true
10
+ ActiveRecord::Migrator.migrate(File.expand_path(File.dirname(__FILE__) + "/../db/migrate"))
@@ -0,0 +1,52 @@
1
+ #!/usr/bin/env ruby
2
+
3
+ $LOAD_PATH.unshift File.expand_path(File.dirname(__FILE__) + '/../lib')
4
+
5
+ require 'rubygems'
6
+ require 'fileutils'
7
+
8
+ if Process.uid != 0
9
+ puts "Must run as root"
10
+ exit 1
11
+ end
12
+
13
+ JOBS_PATH = '/etc/octopusci/jobs'
14
+ CONFIG_PATH = '/etc/octopusci/config.yml'
15
+
16
+ FileUtils.mkdir_p(JOBS_PATH)
17
+
18
+ if !File.exists?(CONFIG_PATH)
19
+ File.open(CONFIG_PATH, 'w') do |f|
20
+ f << "general:
21
+ jobs_path: \"/etc/octopusci/jobs\"
22
+
23
+ smtp:
24
+ notification_from_email: somefrom@example.com
25
+ address: smtp.gmail.com
26
+ port: 587
27
+ authentication: plain
28
+ enable_starttls_auto: true
29
+ user_name: someuser@example.com
30
+ password: somepassword
31
+ raise_delivery_errors: true
32
+
33
+ db:
34
+ adapter: mysql
35
+ host: localhost
36
+ database: octopusci
37
+ username: someusername
38
+ password: somepassword
39
+
40
+ projects:
41
+ - { name: octopusci, owner: cyphactor, job_klass: SomeJobClass, repo_uri: 'git@github.com:cyphactor/octopusci.git', default_email: devs@example.com }
42
+
43
+ stages:
44
+ - test_b
45
+ "
46
+ end
47
+ puts "Created example #{CONFIG_PATH}, please modify appropriately"
48
+ else
49
+ puts "#{CONFIG_PATH} already exists, exiting to avoid modification."
50
+ puts "If you would like to generated the example config again please rename the existing #{CONFIG_PATH}."
51
+ end
52
+
@@ -0,0 +1,8 @@
1
+ #!/usr/bin/env ruby
2
+
3
+ $LOAD_PATH.unshift File.expand_path(File.dirname(__FILE__) + '/../lib')
4
+
5
+ require 'rubygems'
6
+ require 'octopusci'
7
+
8
+ Octopusci::WorkerLauncher.launch
data/bin/pusci-stage ADDED
@@ -0,0 +1,67 @@
1
+ #!/usr/bin/env ruby
2
+
3
+ $LOAD_PATH.unshift File.expand_path(File.dirname(__FILE__) + '/../lib')
4
+
5
+ require 'rubygems'
6
+ require 'optparse'
7
+ require 'octopusci'
8
+
9
+ # This hash will hold all of the options
10
+ # parsed from the command-line by
11
+ # OptionParser.
12
+ options = {}
13
+
14
+ optparse = OptionParser.new do|opts|
15
+ # Set a banner, displayed at the top
16
+ # of the help screen.
17
+ opts.banner = "Usage: pusci-stage [options] stage_name"
18
+
19
+ # Define the options, and what they do
20
+ options[:add] = false
21
+ opts.on('-a', '--add', 'Add a stage back into the pool') do
22
+ options[:add] = true
23
+ end
24
+
25
+ options[:rem] = false
26
+ opts.on( '-r', '--rem', 'Rem a stage from the pool' ) do
27
+ options[:rem] = true
28
+ end
29
+
30
+ options[:list] = false
31
+ opts.on( '-l', '--list', 'List all stages' ) do
32
+ options[:list] = true
33
+ end
34
+
35
+ options[:pool] = false
36
+ opts.on( '-p', '--pool', 'List all stages currently in the pool' ) do
37
+ options[:pool] = true
38
+ end
39
+
40
+ # This displays the help screen, all programs are
41
+ # assumed to have this option.
42
+ opts.on( '-h', '--help', 'Display the help screen' ) do
43
+ puts opts
44
+ exit
45
+ end
46
+ end
47
+
48
+ # Parse the command-line. Remember there are two forms
49
+ # of the parse method. The 'parse' method simply parses
50
+ # ARGV, while the 'parse!' method parses ARGV and removes
51
+ # any options found there, as well as any parameters for
52
+ # the options. What's left is the list of files to resize.
53
+ optparse.parse!
54
+
55
+ if options[:list] == true
56
+ Octopusci::StageLocker.stages.each do |s|
57
+ puts s
58
+ end
59
+ elsif options[:pool] == true
60
+ puts Octopusci::StageLocker.pool
61
+ elsif options[:add] == true
62
+ Octopusci::StageLocker.push(ARGV[0])
63
+ elsif options[:rem] == true
64
+ Octopusci::StageLocker.rem(ARGV[0])
65
+ end
66
+
67
+ exit
data/config.ru ADDED
@@ -0,0 +1,5 @@
1
+ $LOAD_PATH.unshift File.expand_path(File.dirname(__FILE__) + '/lib')
2
+ require 'resque/server'
3
+ require 'octopusci/server'
4
+
5
+ run Rack::URLMap.new("/" => Octopusci::Server.new, "/resque" => Resque::Server.new)
@@ -0,0 +1,29 @@
1
+ class Init < ActiveRecord::Migration
2
+ def self.up
3
+ create_table :jobs do |t|
4
+ t.string :ref
5
+ t.string :compare
6
+ t.string :repo_name
7
+ t.string :repo_owner_name
8
+ t.string :repo_owner_email
9
+ t.timestamp :repo_pushed_at
10
+ t.timestamp :repo_created_at
11
+ t.text :repo_desc
12
+ t.string :repo_url
13
+ t.string :before_commit
14
+ t.boolean :forced
15
+ t.string :after_commit
16
+ t.boolean :running
17
+ t.boolean :successful
18
+ t.text :output
19
+ t.timestamps
20
+ t.timestamp :started_at
21
+ t.timestamp :ended_at
22
+ t.text :payload
23
+ end
24
+ end
25
+
26
+ def self.down
27
+ drop_table :jobs
28
+ end
29
+ end
@@ -0,0 +1,95 @@
1
+ require 'octopusci/notifier'
2
+ require 'active_record'
3
+
4
+ module Octopusci
5
+ class Config
6
+ class MissingConfigField < RuntimeError; end
7
+ class ConfigNotInitialized < RuntimeError; end
8
+
9
+ def initialize()
10
+ @options = {}
11
+ end
12
+
13
+ # read the configuration values into the object from a YML file.
14
+ def load_yaml(yaml_file)
15
+ @options = YAML.load_file(yaml_file)
16
+ end
17
+
18
+ # allow options to be accessed as if this object is a Hash.
19
+ def [](key_name)
20
+ if @options.nil?
21
+ raise ConfigNotInitialized, "Can't access the '#{key_name}' field because the config hasn't been initialized."
22
+ end
23
+ if !@options.has_key?(key_name.to_s())
24
+ raise MissingConfigField, "'#{key_name}' is NOT defined as a config field."
25
+ end
26
+ return @options[key_name.to_s()]
27
+ end
28
+
29
+ # allow options to be set as if this object is a Hash.
30
+ def []=(key_name, value)
31
+ @options[key_name.to_s()] = value
32
+ end
33
+
34
+ def has_key?(key_name)
35
+ return @options.has_key?(key_name)
36
+ end
37
+
38
+ # allow options to be read and set using method calls. This capability is primarily for
39
+ # allowing the configuration to be defined through a block passed to the configure() function
40
+ # from an initializer or similar file.
41
+ def method_missing(key_name, *args)
42
+ key_name_str = key_name.to_s()
43
+ if key_name_str =~ /=$/ then
44
+ self[key_name_str.chop()] = args[0]
45
+ else
46
+ return self[key_name_str]
47
+ end
48
+ end
49
+ end
50
+
51
+ # On evaluation of the module it defines a new singleton of Config.
52
+ if (!defined?(CONFIG))
53
+ CONFIG = Config.new()
54
+ end
55
+
56
+ def self.configure(yaml_file = nil, &block)
57
+ CONFIG.load_yaml(yaml_file) if !yaml_file.nil?
58
+ yield CONFIG if block
59
+
60
+ Notifier.default :from => Octopusci::CONFIG['smtp']['notification_from_email']
61
+ Notifier.delivery_method = :smtp
62
+ Notifier.smtp_settings = {
63
+ :address => Octopusci::CONFIG['smtp']['address'],
64
+ :port => Octopusci::CONFIG['smtp']['port'].to_s,
65
+ :authentication => Octopusci::CONFIG['smtp']['authentication'],
66
+ :enable_starttls_auto => Octopusci::CONFIG['smtp']['enable_starttls_auto'],
67
+ :user_name => Octopusci::CONFIG['smtp']['user_name'],
68
+ :password => Octopusci::CONFIG['smtp']['password'],
69
+ :raise_delivery_errors => Octopusci::CONFIG['smtp']['raise_delivery_errors']
70
+ }
71
+ Notifier.logger = Logger.new(STDOUT)
72
+ end
73
+ end
74
+
75
+ # Load the actual config file
76
+ Octopusci.configure("/etc/octopusci/config.yml")
77
+
78
+ if Octopusci::CONFIG['stages'] == nil
79
+ raise "You have defined stages as an option but have no items in it."
80
+ end
81
+
82
+ ActiveRecord::Base.establish_connection(
83
+ :adapter => Octopusci::CONFIG['db']['adapter'],
84
+ :host => Octopusci::CONFIG['db']['host'],
85
+ :database => Octopusci::CONFIG['db']['database'],
86
+ :username => Octopusci::CONFIG['db']['username'],
87
+ :password => Octopusci::CONFIG['db']['password']
88
+ )
89
+
90
+ Dir.open(Octopusci::CONFIG['general']['jobs_path']) do |d|
91
+ job_file_names = d.entries.reject { |e| e == '..' || e == '.' }
92
+ job_file_names.each do |f_name|
93
+ require Octopusci::CONFIG['general']['jobs_path'] + "/#{f_name}"
94
+ end
95
+ end
@@ -0,0 +1,4 @@
1
+ module Octopusci
2
+ # Raised when a method that is meant to be pure virtual that is not overloaded is called
3
+ class PureVirtualMethod < RuntimeError; end
4
+ end
@@ -0,0 +1,60 @@
1
+ module Octopusci
2
+ module Helpers
3
+ # Take the github payload hash and translate it to the Job model's attrs
4
+ # so that we can easily use the github payload hash to update_attributes
5
+ # on the Job mode.l
6
+ def self.gh_payload_to_job_attrs(gh_pl)
7
+ attrs = {}
8
+
9
+ # ref
10
+ attrs[:ref] = gh_pl["ref"]
11
+ # compare
12
+ attrs[:compare] = gh_pl["compare"]
13
+ # repo_name
14
+ attrs[:repo_name] = gh_pl["repository"]["name"]
15
+ # repo_owner_name
16
+ attrs[:repo_owner_name] = gh_pl["repository"]["owner"]["name"]
17
+ # repo_owner_email
18
+ attrs[:repo_owner_email] = gh_pl["repository"]["owner"]["email"]
19
+ # repo_pushed_at
20
+ attrs[:repo_pushed_at] = gh_pl["repository"]["pushed_at"]
21
+ # repo_created_at
22
+ attrs[:repo_created_at] = gh_pl["repository"]["created_at"]
23
+ # repo_desc
24
+ attrs[:repo_desc] = gh_pl["repository"]["description"]
25
+ # repo_url
26
+ attrs[:repo_url] = gh_pl["repository"]["url"]
27
+ # before_commit
28
+ attrs[:before_commit] = gh_pl["before"]
29
+ # forced
30
+ attrs[:forced] = gh_pl["forced"]
31
+ # after_commit
32
+ attrs[:after_commit] = gh_pl["after"]
33
+
34
+ attrs[:payload] = gh_pl
35
+
36
+ return attrs
37
+ end
38
+
39
+ # Get the information specified in the config about this project. If
40
+ # project info can't be found for the given project_name and project_owner
41
+ # this method returns nil. Otherwise, this project returns a hash of the
42
+ # project info that it found in the config.
43
+ def self.get_project_info(project_name, project_owner)
44
+ Octopusci::CONFIG["projects"].each do |proj|
45
+ if (proj['name'] == project_name) && (proj['owner'] == project_owner)
46
+ return proj
47
+ end
48
+ end
49
+ return nil
50
+ end
51
+
52
+ def self.decode(str)
53
+ ::MultiJson.decode(str)
54
+ end
55
+
56
+ def self.encode(str)
57
+ ::MultiJson.encode(str)
58
+ end
59
+ end
60
+ end
@@ -0,0 +1,43 @@
1
+ module Octopusci
2
+ class Job
3
+ def self.run(github_payload, stage, job_id, job_conf)
4
+ raise PureVirtualMethod, "The self.commit_run method needs to be defined on your Octopusci::Job."
5
+ end
6
+
7
+ def self.perform(project_name, branch_name, job_id, job_conf)
8
+ if Octopusci::CONFIG.has_key?('stages')
9
+ # Get the next available stage from redis which locks it by removing it
10
+ # from the list of available
11
+ stage = Octopusci::StageLocker.pop
12
+ end
13
+
14
+ begin
15
+ # Using redis to get the associated github_payload
16
+ github_payload = Octopusci::Queue.github_payload(project_name, branch_name)
17
+
18
+ job = ::Job.where("jobs.repo_name = ? && jobs.ref = ?", github_payload['repository']['name'], github_payload['ref']).order('jobs.created_at DESC').first
19
+ if job
20
+ job.started_at = Time.new
21
+ job.running = true
22
+ job.save
23
+ end
24
+
25
+ # Run the commit run and report about status and output
26
+ Bundler.with_clean_env {
27
+ self.run(github_payload, stage, job_id, job_conf)
28
+ }
29
+
30
+ if job
31
+ job.ended_at = Time.new
32
+ job.running = false
33
+ job.save
34
+ end
35
+ ensure
36
+ if Octopusci::CONFIG.has_key?('stages')
37
+ # Unlock the stage by adding it back to the list of available stages
38
+ Octopusci::StageLocker.push(stage)
39
+ end
40
+ end
41
+ end
42
+ end
43
+ end
@@ -0,0 +1,16 @@
1
+ <!DOCTYPE html>
2
+ <html>
3
+ <head>
4
+ <meta content="text/html; charset=UTF-8" http-equiv="Content-Type" />
5
+ </head>
6
+ <body>
7
+ <h1>Octopusci Build (<%= @status_str %>) - <%= @job.repo_name %> / <%= @job.ref.gsub(/refs\/heads\//, '') %></h1>
8
+ <h2>Status: <%= @status_str %></h2>
9
+ <h2>Command Output:</h2>
10
+ <pre>
11
+ <code>
12
+ <%= @cmd_output %>
13
+ </code>
14
+ </pre>
15
+ </body>
16
+ </html>
@@ -0,0 +1,9 @@
1
+ # Octopusci Build
2
+
3
+ ## Command Status
4
+
5
+ <%= @cmd_status %>
6
+
7
+ ## Command Output
8
+
9
+ <%= @cmd_output %>
@@ -0,0 +1,37 @@
1
+ require 'action_mailer'
2
+
3
+ # Set the ActionMailer view_path to lib where this library is so that when
4
+ # it searches for class_name/method for the templates it can find them when
5
+ # we don't use the standard Rails action mailer view locations.
6
+ ActionMailer::Base.view_paths = File.dirname(__FILE__) + '/../'
7
+
8
+ module Octopusci
9
+ class Notifier < ActionMailer::Base
10
+ def job_complete(recipient, cmd_output, cmd_status, github_payload, job_id)
11
+ @job = ::Job.find(job_id)
12
+ @job.output = cmd_output
13
+ @job.running = false
14
+ if cmd_status == 0
15
+ @job.successful = true
16
+ else
17
+ @job.successful = false
18
+ end
19
+ @job.save
20
+
21
+ if recipient
22
+ @cmd_output = cmd_output
23
+ @cmd_status = cmd_status
24
+ @github_payload = github_payload
25
+ if @cmd_status == 0
26
+ @status_str = 'success'
27
+ else
28
+ @status_str = 'failed'
29
+ end
30
+ mail(:to => recipient, :subject => "Octopusci Build (#{@status_str}) - #{@job.repo_name} / #{@job.ref.gsub(/refs\/heads\//, '')}") do |format|
31
+ format.text
32
+ format.html
33
+ end
34
+ end
35
+ end
36
+ end
37
+ end
@@ -0,0 +1,41 @@
1
+ require 'resque'
2
+
3
+ module Octopusci
4
+ module Queue
5
+ def self.enqueue(job_klass, proj_name, branch_name, github_payload, job_conf)
6
+ resque_opts = { "class" => job_klass, "args" => [proj_name, branch_name] }
7
+ gh_pl_key = github_payload_key(proj_name, branch_name)
8
+
9
+ if lismember('commit', resque_opts)
10
+ Resque.redis.set(gh_pl_key, Resque::encode(github_payload))
11
+ # Get the most recent job for this project and update it with the data
12
+ job = ::Job.where("jobs.repo_name = ? && jobs.ref = ?", proj_name, '/refs/heads/' + branch_name).order('jobs.created_at DESC').first
13
+ if job
14
+ job.update_attributes(Octopusci::Helpers.gh_payload_to_job_attrs(github_payload))
15
+ end
16
+ else
17
+ # Create a new job for this project with the appropriate data
18
+ job = ::Job.create(Octopusci::Helpers.gh_payload_to_job_attrs(github_payload).merge({ :running => false }))
19
+ resque_opts["args"] << job.id
20
+ resque_opts["args"] << job_conf
21
+ Resque.redis.set(gh_pl_key, Resque::encode(github_payload))
22
+ Resque.push('commit', resque_opts)
23
+ end
24
+ end
25
+
26
+ def self.lismember(queue, item)
27
+ size = Resque.size(queue)
28
+ [Resque.peek(queue, 0, size)].flatten.any? { |v|
29
+ v == item
30
+ }
31
+ end
32
+
33
+ def self.github_payload(project_name, branch_name)
34
+ Resque::decode(Resque.redis.get(github_payload_key(project_name, branch_name)))
35
+ end
36
+
37
+ def self.github_payload_key(proj_name, branch_name)
38
+ "octpusci:github_payload:#{proj_name}:#{branch_name}"
39
+ end
40
+ end
41
+ end
@@ -0,0 +1,9 @@
1
+ require 'active_record'
2
+
3
+ class Job < ActiveRecord::Base
4
+ serialize :payload
5
+
6
+ def branch_name
7
+ self.ref.gsub(/refs\/heads\//, '')
8
+ end
9
+ end