octopusci 0.0.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
data/LICENSE ADDED
@@ -0,0 +1 @@
1
+ TODO: add license
data/README ADDED
@@ -0,0 +1,63 @@
1
+ The purpose of this project is provide a simple CI server that will work with
2
+ GitHub Post-Receive hook. It is also specifically designed to handle multiple
3
+ build/job queues for each branch.
4
+
5
+ This would basically allow you to every time code is pushed to the central
6
+ repository enqueue a build/job for the specific branch. That way if you
7
+ have topic branches that are being pushed to the central repository along
8
+ side the mainline branch they will get queue properly as well.
9
+
10
+ My idea for implementation at this point is that if I can detect the branch
11
+ from the GitHub Post-Receive hook then I can identify branch. If I can
12
+ identify branch then I can maintain individual queues for each branch.
13
+
14
+ Then the execution would look at config values for predefined branches and
15
+ priorities for order of running them so that the mainline branch running its
16
+ jobs could take precedence over non specified topic branches.
17
+
18
+ Install redis and get it starting up appropriately
19
+
20
+ gem install octopusci
21
+ sudo octopusci-skel
22
+ octopusci-db-migrate
23
+
24
+ Then update the /etc/octopusci/config.yml appropriately.
25
+
26
+ Add any jobs you would like to the /etc/octopusci/jobs directory as rb files
27
+ and octopusci will load them appropriately when started.
28
+
29
+ Figure out what directory the gem is installed in by running the following
30
+ command and stripping off the lib/octopusci.rb at the end.
31
+
32
+ gem which octopusci
33
+
34
+ Once you have the path we can use that path to setup Passenger with Apache
35
+ or something else like nginx as well as setup the database. Note: You will
36
+ need to setup a database user and a database for octopusci. The settings for
37
+ these should be stored in /etc/octopusci/config.yml.
38
+
39
+ rake -f /path/of/octpusci/we/got/before/Rakefile db:migrate
40
+
41
+ <VirtualHost *:80>
42
+ ServerName octopusci.example.com
43
+ PassengerAppRoot /path/of/octpusci/we/got/before
44
+ DocumentRoot /path/of/octpusci/we/got/before/lib/octopusci/server/public
45
+ <Directory /path/of/octpusci/we/got/before/lib/octopusci/server/public>
46
+ Order allow,deny
47
+ Allow from all
48
+ AllowOverride all
49
+ Options -MultiViews
50
+ </Directory>
51
+ </VirtualHost>
52
+
53
+ The above will give us the web Octopusci web interface.
54
+
55
+ If you are developing you can simply start this up by running
56
+ rackup -p whatever_port while inside the octopusci directory where the
57
+ config.ru file exists.
58
+
59
+ I recommend you setup the second half of Octopusci (octopusci-tentacles) with
60
+ God or some other monitoring system. However, for development you can simply
61
+ run octopusci-tentacles directoly as follows:
62
+
63
+ otopusci-tentacles
data/bin/octopusci ADDED
@@ -0,0 +1,56 @@
1
+ #!/usr/bin/env ruby
2
+
3
+ $LOAD_PATH.unshift File.expand_path(File.dirname(__FILE__) + '/../lib')
4
+
5
+ require 'rubygems'
6
+ require 'optparse'
7
+
8
+ # This hash will hold all of the options
9
+ # parsed from the command-line by
10
+ # OptionParser.
11
+ options = {}
12
+
13
+ optparse = OptionParser.new do|opts|
14
+ # Set a banner, displayed at the top
15
+ # of the help screen.
16
+ opts.banner = "Usage: optparse1.rb [options] file1 file2 ..."
17
+
18
+ # Define the options, and what they do
19
+ options[:verbose] = false
20
+ opts.on( '-v', '--verbose', 'Output more information' ) do
21
+ options[:verbose] = true
22
+ end
23
+
24
+ options[:quick] = false
25
+ opts.on( '-q', '--quick', 'Perform the task quickly' ) do
26
+ options[:quick] = true
27
+ end
28
+
29
+ options[:logfile] = nil
30
+ opts.on( '-l', '--logfile FILE', 'Write log to FILE' ) do |file|
31
+ options[:logfile] = file
32
+ end
33
+
34
+ # This displays the help screen, all programs are
35
+ # assumed to have this option.
36
+ opts.on( '-h', '--help', 'Display this screen' ) do
37
+ puts opts
38
+ exit
39
+ end
40
+ end
41
+
42
+ # Parse the command-line. Remember there are two forms
43
+ # of the parse method. The 'parse' method simply parses
44
+ # ARGV, while the 'parse!' method parses ARGV and removes
45
+ # any options found there, as well as any parameters for
46
+ # the options. What's left is the list of files to resize.
47
+ optparse.parse!
48
+
49
+ puts "Being verbose" if options[:verbose]
50
+ puts "Being quick" if options[:quick]
51
+ puts "Logging to file #{options[:logfile]}" if options[:logfile]
52
+
53
+ ARGV.each do|f|
54
+ puts "Resizing image #{f}..."
55
+ sleep 0.5
56
+ end
@@ -0,0 +1,10 @@
1
+ #!/usr/bin/env ruby
2
+
3
+ $LOAD_PATH.unshift File.expand_path(File.dirname(__FILE__) + '/../lib')
4
+
5
+ require 'rubygems'
6
+ require 'octopusci'
7
+
8
+ ActiveRecord::Base.logger = Logger.new(STDOUT)
9
+ ActiveRecord::Migration.verbose = true
10
+ ActiveRecord::Migrator.migrate(File.expand_path(File.dirname(__FILE__) + "/../db/migrate"))
@@ -0,0 +1,52 @@
1
+ #!/usr/bin/env ruby
2
+
3
+ $LOAD_PATH.unshift File.expand_path(File.dirname(__FILE__) + '/../lib')
4
+
5
+ require 'rubygems'
6
+ require 'fileutils'
7
+
8
+ if Process.uid != 0
9
+ puts "Must run as root"
10
+ exit 1
11
+ end
12
+
13
+ JOBS_PATH = '/etc/octopusci/jobs'
14
+ CONFIG_PATH = '/etc/octopusci/config.yml'
15
+
16
+ FileUtils.mkdir_p(JOBS_PATH)
17
+
18
+ if !File.exists?(CONFIG_PATH)
19
+ File.open(CONFIG_PATH, 'w') do |f|
20
+ f << "general:
21
+ jobs_path: \"/etc/octopusci/jobs\"
22
+
23
+ smtp:
24
+ notification_from_email: somefrom@example.com
25
+ address: smtp.gmail.com
26
+ port: 587
27
+ authentication: plain
28
+ enable_starttls_auto: true
29
+ user_name: someuser@example.com
30
+ password: somepassword
31
+ raise_delivery_errors: true
32
+
33
+ db:
34
+ adapter: mysql
35
+ host: localhost
36
+ database: octopusci
37
+ username: someusername
38
+ password: somepassword
39
+
40
+ projects:
41
+ - { name: octopusci, owner: cyphactor, job_klass: SomeJobClass, repo_uri: 'git@github.com:cyphactor/octopusci.git', default_email: devs@example.com }
42
+
43
+ stages:
44
+ - test_b
45
+ "
46
+ end
47
+ puts "Created example #{CONFIG_PATH}, please modify appropriately"
48
+ else
49
+ puts "#{CONFIG_PATH} already exists, exiting to avoid modification."
50
+ puts "If you would like to generated the example config again please rename the existing #{CONFIG_PATH}."
51
+ end
52
+
@@ -0,0 +1,8 @@
1
+ #!/usr/bin/env ruby
2
+
3
+ $LOAD_PATH.unshift File.expand_path(File.dirname(__FILE__) + '/../lib')
4
+
5
+ require 'rubygems'
6
+ require 'octopusci'
7
+
8
+ Octopusci::WorkerLauncher.launch
data/bin/pusci-stage ADDED
@@ -0,0 +1,67 @@
1
+ #!/usr/bin/env ruby
2
+
3
+ $LOAD_PATH.unshift File.expand_path(File.dirname(__FILE__) + '/../lib')
4
+
5
+ require 'rubygems'
6
+ require 'optparse'
7
+ require 'octopusci'
8
+
9
+ # This hash will hold all of the options
10
+ # parsed from the command-line by
11
+ # OptionParser.
12
+ options = {}
13
+
14
+ optparse = OptionParser.new do|opts|
15
+ # Set a banner, displayed at the top
16
+ # of the help screen.
17
+ opts.banner = "Usage: pusci-stage [options] stage_name"
18
+
19
+ # Define the options, and what they do
20
+ options[:add] = false
21
+ opts.on('-a', '--add', 'Add a stage back into the pool') do
22
+ options[:add] = true
23
+ end
24
+
25
+ options[:rem] = false
26
+ opts.on( '-r', '--rem', 'Rem a stage from the pool' ) do
27
+ options[:rem] = true
28
+ end
29
+
30
+ options[:list] = false
31
+ opts.on( '-l', '--list', 'List all stages' ) do
32
+ options[:list] = true
33
+ end
34
+
35
+ options[:pool] = false
36
+ opts.on( '-p', '--pool', 'List all stages currently in the pool' ) do
37
+ options[:pool] = true
38
+ end
39
+
40
+ # This displays the help screen, all programs are
41
+ # assumed to have this option.
42
+ opts.on( '-h', '--help', 'Display the help screen' ) do
43
+ puts opts
44
+ exit
45
+ end
46
+ end
47
+
48
+ # Parse the command-line. Remember there are two forms
49
+ # of the parse method. The 'parse' method simply parses
50
+ # ARGV, while the 'parse!' method parses ARGV and removes
51
+ # any options found there, as well as any parameters for
52
+ # the options. What's left is the list of files to resize.
53
+ optparse.parse!
54
+
55
+ if options[:list] == true
56
+ Octopusci::StageLocker.stages.each do |s|
57
+ puts s
58
+ end
59
+ elsif options[:pool] == true
60
+ puts Octopusci::StageLocker.pool
61
+ elsif options[:add] == true
62
+ Octopusci::StageLocker.push(ARGV[0])
63
+ elsif options[:rem] == true
64
+ Octopusci::StageLocker.rem(ARGV[0])
65
+ end
66
+
67
+ exit
data/config.ru ADDED
@@ -0,0 +1,5 @@
1
+ $LOAD_PATH.unshift File.expand_path(File.dirname(__FILE__) + '/lib')
2
+ require 'resque/server'
3
+ require 'octopusci/server'
4
+
5
+ run Rack::URLMap.new("/" => Octopusci::Server.new, "/resque" => Resque::Server.new)
@@ -0,0 +1,29 @@
1
+ class Init < ActiveRecord::Migration
2
+ def self.up
3
+ create_table :jobs do |t|
4
+ t.string :ref
5
+ t.string :compare
6
+ t.string :repo_name
7
+ t.string :repo_owner_name
8
+ t.string :repo_owner_email
9
+ t.timestamp :repo_pushed_at
10
+ t.timestamp :repo_created_at
11
+ t.text :repo_desc
12
+ t.string :repo_url
13
+ t.string :before_commit
14
+ t.boolean :forced
15
+ t.string :after_commit
16
+ t.boolean :running
17
+ t.boolean :successful
18
+ t.text :output
19
+ t.timestamps
20
+ t.timestamp :started_at
21
+ t.timestamp :ended_at
22
+ t.text :payload
23
+ end
24
+ end
25
+
26
+ def self.down
27
+ drop_table :jobs
28
+ end
29
+ end
@@ -0,0 +1,95 @@
1
+ require 'octopusci/notifier'
2
+ require 'active_record'
3
+
4
+ module Octopusci
5
+ class Config
6
+ class MissingConfigField < RuntimeError; end
7
+ class ConfigNotInitialized < RuntimeError; end
8
+
9
+ def initialize()
10
+ @options = {}
11
+ end
12
+
13
+ # read the configuration values into the object from a YML file.
14
+ def load_yaml(yaml_file)
15
+ @options = YAML.load_file(yaml_file)
16
+ end
17
+
18
+ # allow options to be accessed as if this object is a Hash.
19
+ def [](key_name)
20
+ if @options.nil?
21
+ raise ConfigNotInitialized, "Can't access the '#{key_name}' field because the config hasn't been initialized."
22
+ end
23
+ if !@options.has_key?(key_name.to_s())
24
+ raise MissingConfigField, "'#{key_name}' is NOT defined as a config field."
25
+ end
26
+ return @options[key_name.to_s()]
27
+ end
28
+
29
+ # allow options to be set as if this object is a Hash.
30
+ def []=(key_name, value)
31
+ @options[key_name.to_s()] = value
32
+ end
33
+
34
+ def has_key?(key_name)
35
+ return @options.has_key?(key_name)
36
+ end
37
+
38
+ # allow options to be read and set using method calls. This capability is primarily for
39
+ # allowing the configuration to be defined through a block passed to the configure() function
40
+ # from an initializer or similar file.
41
+ def method_missing(key_name, *args)
42
+ key_name_str = key_name.to_s()
43
+ if key_name_str =~ /=$/ then
44
+ self[key_name_str.chop()] = args[0]
45
+ else
46
+ return self[key_name_str]
47
+ end
48
+ end
49
+ end
50
+
51
+ # On evaluation of the module it defines a new singleton of Config.
52
+ if (!defined?(CONFIG))
53
+ CONFIG = Config.new()
54
+ end
55
+
56
+ def self.configure(yaml_file = nil, &block)
57
+ CONFIG.load_yaml(yaml_file) if !yaml_file.nil?
58
+ yield CONFIG if block
59
+
60
+ Notifier.default :from => Octopusci::CONFIG['smtp']['notification_from_email']
61
+ Notifier.delivery_method = :smtp
62
+ Notifier.smtp_settings = {
63
+ :address => Octopusci::CONFIG['smtp']['address'],
64
+ :port => Octopusci::CONFIG['smtp']['port'].to_s,
65
+ :authentication => Octopusci::CONFIG['smtp']['authentication'],
66
+ :enable_starttls_auto => Octopusci::CONFIG['smtp']['enable_starttls_auto'],
67
+ :user_name => Octopusci::CONFIG['smtp']['user_name'],
68
+ :password => Octopusci::CONFIG['smtp']['password'],
69
+ :raise_delivery_errors => Octopusci::CONFIG['smtp']['raise_delivery_errors']
70
+ }
71
+ Notifier.logger = Logger.new(STDOUT)
72
+ end
73
+ end
74
+
75
+ # Load the actual config file
76
+ Octopusci.configure("/etc/octopusci/config.yml")
77
+
78
+ if Octopusci::CONFIG['stages'] == nil
79
+ raise "You have defined stages as an option but have no items in it."
80
+ end
81
+
82
+ ActiveRecord::Base.establish_connection(
83
+ :adapter => Octopusci::CONFIG['db']['adapter'],
84
+ :host => Octopusci::CONFIG['db']['host'],
85
+ :database => Octopusci::CONFIG['db']['database'],
86
+ :username => Octopusci::CONFIG['db']['username'],
87
+ :password => Octopusci::CONFIG['db']['password']
88
+ )
89
+
90
+ Dir.open(Octopusci::CONFIG['general']['jobs_path']) do |d|
91
+ job_file_names = d.entries.reject { |e| e == '..' || e == '.' }
92
+ job_file_names.each do |f_name|
93
+ require Octopusci::CONFIG['general']['jobs_path'] + "/#{f_name}"
94
+ end
95
+ end
@@ -0,0 +1,4 @@
1
+ module Octopusci
2
+ # Raised when a method that is meant to be pure virtual that is not overloaded is called
3
+ class PureVirtualMethod < RuntimeError; end
4
+ end
@@ -0,0 +1,60 @@
1
+ module Octopusci
2
+ module Helpers
3
+ # Take the github payload hash and translate it to the Job model's attrs
4
+ # so that we can easily use the github payload hash to update_attributes
5
+ # on the Job mode.l
6
+ def self.gh_payload_to_job_attrs(gh_pl)
7
+ attrs = {}
8
+
9
+ # ref
10
+ attrs[:ref] = gh_pl["ref"]
11
+ # compare
12
+ attrs[:compare] = gh_pl["compare"]
13
+ # repo_name
14
+ attrs[:repo_name] = gh_pl["repository"]["name"]
15
+ # repo_owner_name
16
+ attrs[:repo_owner_name] = gh_pl["repository"]["owner"]["name"]
17
+ # repo_owner_email
18
+ attrs[:repo_owner_email] = gh_pl["repository"]["owner"]["email"]
19
+ # repo_pushed_at
20
+ attrs[:repo_pushed_at] = gh_pl["repository"]["pushed_at"]
21
+ # repo_created_at
22
+ attrs[:repo_created_at] = gh_pl["repository"]["created_at"]
23
+ # repo_desc
24
+ attrs[:repo_desc] = gh_pl["repository"]["description"]
25
+ # repo_url
26
+ attrs[:repo_url] = gh_pl["repository"]["url"]
27
+ # before_commit
28
+ attrs[:before_commit] = gh_pl["before"]
29
+ # forced
30
+ attrs[:forced] = gh_pl["forced"]
31
+ # after_commit
32
+ attrs[:after_commit] = gh_pl["after"]
33
+
34
+ attrs[:payload] = gh_pl
35
+
36
+ return attrs
37
+ end
38
+
39
+ # Get the information specified in the config about this project. If
40
+ # project info can't be found for the given project_name and project_owner
41
+ # this method returns nil. Otherwise, this project returns a hash of the
42
+ # project info that it found in the config.
43
+ def self.get_project_info(project_name, project_owner)
44
+ Octopusci::CONFIG["projects"].each do |proj|
45
+ if (proj['name'] == project_name) && (proj['owner'] == project_owner)
46
+ return proj
47
+ end
48
+ end
49
+ return nil
50
+ end
51
+
52
+ def self.decode(str)
53
+ ::MultiJson.decode(str)
54
+ end
55
+
56
+ def self.encode(str)
57
+ ::MultiJson.encode(str)
58
+ end
59
+ end
60
+ end
@@ -0,0 +1,43 @@
1
+ module Octopusci
2
+ class Job
3
+ def self.run(github_payload, stage, job_id, job_conf)
4
+ raise PureVirtualMethod, "The self.commit_run method needs to be defined on your Octopusci::Job."
5
+ end
6
+
7
+ def self.perform(project_name, branch_name, job_id, job_conf)
8
+ if Octopusci::CONFIG.has_key?('stages')
9
+ # Get the next available stage from redis which locks it by removing it
10
+ # from the list of available
11
+ stage = Octopusci::StageLocker.pop
12
+ end
13
+
14
+ begin
15
+ # Using redis to get the associated github_payload
16
+ github_payload = Octopusci::Queue.github_payload(project_name, branch_name)
17
+
18
+ job = ::Job.where("jobs.repo_name = ? && jobs.ref = ?", github_payload['repository']['name'], github_payload['ref']).order('jobs.created_at DESC').first
19
+ if job
20
+ job.started_at = Time.new
21
+ job.running = true
22
+ job.save
23
+ end
24
+
25
+ # Run the commit run and report about status and output
26
+ Bundler.with_clean_env {
27
+ self.run(github_payload, stage, job_id, job_conf)
28
+ }
29
+
30
+ if job
31
+ job.ended_at = Time.new
32
+ job.running = false
33
+ job.save
34
+ end
35
+ ensure
36
+ if Octopusci::CONFIG.has_key?('stages')
37
+ # Unlock the stage by adding it back to the list of available stages
38
+ Octopusci::StageLocker.push(stage)
39
+ end
40
+ end
41
+ end
42
+ end
43
+ end
@@ -0,0 +1,16 @@
1
+ <!DOCTYPE html>
2
+ <html>
3
+ <head>
4
+ <meta content="text/html; charset=UTF-8" http-equiv="Content-Type" />
5
+ </head>
6
+ <body>
7
+ <h1>Octopusci Build (<%= @status_str %>) - <%= @job.repo_name %> / <%= @job.ref.gsub(/refs\/heads\//, '') %></h1>
8
+ <h2>Status: <%= @status_str %></h2>
9
+ <h2>Command Output:</h2>
10
+ <pre>
11
+ <code>
12
+ <%= @cmd_output %>
13
+ </code>
14
+ </pre>
15
+ </body>
16
+ </html>
@@ -0,0 +1,9 @@
1
+ # Octopusci Build
2
+
3
+ ## Command Status
4
+
5
+ <%= @cmd_status %>
6
+
7
+ ## Command Output
8
+
9
+ <%= @cmd_output %>
@@ -0,0 +1,37 @@
1
+ require 'action_mailer'
2
+
3
+ # Set the ActionMailer view_path to lib where this library is so that when
4
+ # it searches for class_name/method for the templates it can find them when
5
+ # we don't use the standard Rails action mailer view locations.
6
+ ActionMailer::Base.view_paths = File.dirname(__FILE__) + '/../'
7
+
8
+ module Octopusci
9
+ class Notifier < ActionMailer::Base
10
+ def job_complete(recipient, cmd_output, cmd_status, github_payload, job_id)
11
+ @job = ::Job.find(job_id)
12
+ @job.output = cmd_output
13
+ @job.running = false
14
+ if cmd_status == 0
15
+ @job.successful = true
16
+ else
17
+ @job.successful = false
18
+ end
19
+ @job.save
20
+
21
+ if recipient
22
+ @cmd_output = cmd_output
23
+ @cmd_status = cmd_status
24
+ @github_payload = github_payload
25
+ if @cmd_status == 0
26
+ @status_str = 'success'
27
+ else
28
+ @status_str = 'failed'
29
+ end
30
+ mail(:to => recipient, :subject => "Octopusci Build (#{@status_str}) - #{@job.repo_name} / #{@job.ref.gsub(/refs\/heads\//, '')}") do |format|
31
+ format.text
32
+ format.html
33
+ end
34
+ end
35
+ end
36
+ end
37
+ end
@@ -0,0 +1,41 @@
1
+ require 'resque'
2
+
3
+ module Octopusci
4
+ module Queue
5
+ def self.enqueue(job_klass, proj_name, branch_name, github_payload, job_conf)
6
+ resque_opts = { "class" => job_klass, "args" => [proj_name, branch_name] }
7
+ gh_pl_key = github_payload_key(proj_name, branch_name)
8
+
9
+ if lismember('commit', resque_opts)
10
+ Resque.redis.set(gh_pl_key, Resque::encode(github_payload))
11
+ # Get the most recent job for this project and update it with the data
12
+ job = ::Job.where("jobs.repo_name = ? && jobs.ref = ?", proj_name, '/refs/heads/' + branch_name).order('jobs.created_at DESC').first
13
+ if job
14
+ job.update_attributes(Octopusci::Helpers.gh_payload_to_job_attrs(github_payload))
15
+ end
16
+ else
17
+ # Create a new job for this project with the appropriate data
18
+ job = ::Job.create(Octopusci::Helpers.gh_payload_to_job_attrs(github_payload).merge({ :running => false }))
19
+ resque_opts["args"] << job.id
20
+ resque_opts["args"] << job_conf
21
+ Resque.redis.set(gh_pl_key, Resque::encode(github_payload))
22
+ Resque.push('commit', resque_opts)
23
+ end
24
+ end
25
+
26
+ def self.lismember(queue, item)
27
+ size = Resque.size(queue)
28
+ [Resque.peek(queue, 0, size)].flatten.any? { |v|
29
+ v == item
30
+ }
31
+ end
32
+
33
+ def self.github_payload(project_name, branch_name)
34
+ Resque::decode(Resque.redis.get(github_payload_key(project_name, branch_name)))
35
+ end
36
+
37
+ def self.github_payload_key(proj_name, branch_name)
38
+ "octpusci:github_payload:#{proj_name}:#{branch_name}"
39
+ end
40
+ end
41
+ end
@@ -0,0 +1,9 @@
1
+ require 'active_record'
2
+
3
+ class Job < ActiveRecord::Base
4
+ serialize :payload
5
+
6
+ def branch_name
7
+ self.ref.gsub(/refs\/heads\//, '')
8
+ end
9
+ end