promotion 1.0.7

data/.yardopts

@@ -0,0 +1 @@
+--title "Promotion" -m rdoc --main README lib/**/*.rb - README

data/README

@@ -0,0 +1,256 @@
+= Promotion
+Promotion makes it possible to repeatedly deploy an application in a fast and
+reliable way.
+
+== Staging
+A key concept is the *staging* area. It is a folder (+/var/staging+ by default)
+where the files for each application are loaded into a named sub-folder (eg. +/var/staging/myapp+).
+For example, a staging area may look like this:
+
+   /var
+    |__ staging
+      |__ webapp1        -- a web application
+      |__ monitor        -- sysadmin tool to monitor availability
+      |__ analytics      -- analytics app for admins
+      |__ snort          -- a custom config for snort
+
+In this example, note that we also have an area for an installed application like snort.
+Although the binaries are installed through normal package installation,
+you may have a lot of time and work invested in the
+configuration. These files can of course be backed up and restored manually, but the purpose of
+Promotion is to make deployment fast and reliable. One way to achieve that is to store your
+valuable configuration files in source control, so you can reliably checkout the latest version
+for deployment each and every time. So the +snort+ staging area need only hold a few configuration
+files.
+
+Aside: why not just use version control?::
+ The important files are spread all over the file system, so the only way to restore them all
+ in a single command is to make the root directory a subversion working folder, add selected
+ files to version control and then have <code>.svn</code> folders spread all over the place.
+ Moreover, you'd need to run svn as root.
+
+== Deployment Descriptor
+Each application has an XML <b>deployment descriptor</b> (eg. +/var/staging/myapp/deploy.xml+)
+that describes the conditions necessary for the successful operation of the application.
+
+The deployment descriptor describes how to move the files from staging to their correct
+locations in the file system. This simple requirement unrolls into a chain of others:
+- we need the folder to be there to put the file in
+- the permissions need to be correct on the file and folder
+- the ownerships must be set
+- that means we need the necessary user accounts created
+- which means we first need the right groups set up.
+
+In addition to installing the application's files, there are system-wide configuration
+files that need to be adjusted to enable an application to startup or run properly:
+- /etc/profile
+- /etc/rc.conf.local
+- /etc/sudoers
+- /var/cron/tabs/*
+
+Promotion generates these files based on the set of all of the deployment descriptors it finds
+in the staging area.
+
+An important aspect is that a deployment is *idempotent* - you can do it as many
+times as you like and the result will be the same. This means its safe to make a
+small change to an application and re-deploy it in a few seconds, ready for testing.
+
+Promotion is useful in a testing environment when changes and redeployments are frequent,
+but it shines in a production environment by reducing maintenance downtime and risk.
+
+Promotion was originally designed for use on an OpenBSD system, but is configurable enough to work
+on any *nix system. Just modify the <code>promotion/config.rb</code> configuration file
+to suit your system's paths.
+
+== Installation
+*WARNING*::
+ Promotion will makes changes to your operating system, including important system-wide files.
+
+ DO NOT USE ON A PRODUCTION SYSTEM without rigorous testing in a virtual machine first.
+
+Create the staging area (and set permissions appropriately):
+
+  $ sudo mkdir /var/staging
+
+Check out the promotion app into staging, and bootstrap promotion
+  $ sudo wget http://rubyforge.org/frs/download.php/76463/promotion-x.y.z.zip
+  $ sudo unzip promotion-x.y.z.zip -d promotion     # creates the promotion sub-directory
+  $ cd promotion
+  $ sudo ruby bin/bootstrap
+  Promoting promotion...OK.
+  $ cd /var/staging
+Now you are ready to use Promotion to deploy other applications.
+
+== Usage
+For each application you want to deploy, create its staging folder:
+  $ cd /var/staging
+  $ sudo svn checkout https://hosted/path/to/project/dist myapp
+The project (or perhaps a +dist+ folder designed for deployment) is checked out into
+a folder with the given name.
+
+Now you can promote the application:
+  $ sudo promote myapp
+  Promoting myapp...OK.
+
+If a database is involved, you can also migrate the database schema:
+  $ sudo evolve myapp
+  Evolving the database to version 1023
+which will execute any new schema migration scripts. If you execute it again,
+you will see:
+  $ sudo evolve myapp
+  Already at version 1023.
+
+If you want to revert to an earlier version of the database schema:
+  $ sudo devolve myapp 1017
+  Devolving the database to version 1017
+which will execute the schema migration scripts in the +devolve+ folder,
+in reverse order from 1023 down to 1017.
+
+== Deployment descriptor syntax
+The deployment descriptor is typically named <code>deploy.xml</code> and placed at the top
+of the application's staging folder (eg. <code>/var/staging/myapp</code>).
+
+It's structure is described by the XML schema which can be found at
+at +/var/staging/promotion/promotion.xsd+ or at
+http://finalstep.com.au/schema/promotion.v100.xsd
+
+=== +Specification+
+The top level element is a Specification
+  <?xml version="1.0" encoding="UTF-8" ?>
+  <Specification Name="myapp" Title="My Test App"
+    xmlns="http://finalstep.com.au/promotion/v100"
+    xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+    xsi:schemaLocation="http://finalstep.com.au/schema/promotion.v100.xsd ../promotion/promotion.xsd">
+which has a +Name+ matching the staging folder it resides in (+myapp+), and a fuller +Title+.
+
+=== +Description+
+A description element is the first child:
+  <Description>
+    The promotion manager ensures that the environment for an application
+    is set up correctly, including folder and file permissions and ownership,
+    startup scripts, sudoers privileges and environment variables.
+  </Description>
+
+=== +Groups+
+The sequence is then driven by dependencies: you cannot set ownerships until you have
+users, and you cannot create a user without a group, so Groups come first. Each of these
+elements is optional however.
+  <Groups>
+    <Group Gid="502" Name="_mysql" />
+  </Groups>
+This creates a group with group ID <code>502</code> and the name <code>_mysql</code>.
+
+=== +Users+
+We can also create an admin and a user to run the MySQL daemon:
+  <Users>
+    <User Name="richard" Class="staff" Gecos="Richard Kernahan" Uid="1002" Gid="1002"
+      Home="/home/richard" Shell="/bin/ksh" Groups="wheel webadmin"/>
+    <User Name="_mysql" Class="daemon" Gecos="MySQL Account" Uid="502"Gid="502"
+      Home="/nonexistent" Shell="/sbin/nologin"  />
+  </Users>
+
+- +Name+, +Uid+, and +Gid+ are mandatory
+- +Gecos+ defaults to the Name
+- +Home+ for +richard+ defaults to +/home/richard+
+- +Shell+ defaults to +/sbin/nologin+
+- +Groups+ is an optional space-separated list of groups the user should be added to
+
+=== +Folders+
+Before we can move files into place, we need the Folders set up:
+  <Folders Mode="0750" Owner="root" Group="wheel">
+    <Folder Owner="_mysql" Group="_mysql">/var/mysql</Folder>
+    <Folder>/home/myapp</Folder>
+    <Folder>/var/myapp</Folder>
+  </Folders>
+You can have several +Folders+ elements, if desired. The defaults for all +Folder+
+child elements may be set as attributes of the parent +Folders+ element. In this example,
+all the folders will have permissions <code>rwxr-x---</code>, and ownership of <code>root:wheel</code>.
+Note that the folder <code>/var/mysql</code> has a different owner and group. Any Folder can
+override the default Mode, Owner, and Group.
+
+=== +Files+
+Finally we can specify how to move the files into place:
+  <Files Owner="root" Group="wheel" Mode="0644" Source="conf">
+    <File>/etc/my.cnf</File>
+    <File>/etc/myapp.conf</File>
+  </Files>
+  <Files Owner="root" Group="wheel" Mode="0750" Source="sbin">
+    <File>/usr/local/sbin/up!</File>
+    <File>/usr/local/sbin/down!</File>
+    <File>/usr/local/sbin/push!</File>
+  </Files>
+
+In this example, we have two sets of files. The first set is expected to be found under the +conf+
+folder. Looking at the first file, we see the destination path +/etc/my.cnf+.
+The source file should have the same filename +my.cnf+ and be located in the +conf+ folder.
+This results in Promotion executing a command such as:
+
+  cp -f /var/staging/myapp/conf/my.cnf /etc/my.cnf
+
+This allows a flatter, more convenient project folder structure, since the deployment descriptor maps
+the files into their proper operating system locations.
+
+As for Folders, the Files element can define the default Owner, Group and Mode for all File children.
+
+=== +Allfiles+
+A convenient shorthand for copying all the files in a folder to another folder is the
++Allfiles+ element. In this example, we will copy everything from the conf folder to
+the destination folder specified.
+  <Allfiles Group="bin" Mode="0644" Owner="root" Source="conf">/var/axonsec/conf</Allfiles>
+
+=== +Daemon+
+To enable an application to startup automatically, a +Daemon+ element is needed. The name of the
+startup script is expected to be <code>/etc/rc.d/myapp</code>.
+  <Daemon Flags="" Priority="10" User="" />
+
+The +Flags+ attribute contains the command line options provided to the executable by
+<code>/etc/rc.conf.local</code>.
+
+The lower the +Priority+ value, the earlier that script is run (high values start later).
+
+The +User+ is the user running the process, typically an unprivileged user named <code>_myapp</code>.
+Leave blank to run the startup script as root (eg. as when dropping privileges).
+
+=== +Crontab+
+Cron jobs are often needed for an application to run smoothly. Instead of creating them by hand,
+you can specify each job as part of the deployment descriptor and let promotion set up the crontab
+automatically.
+  <Crontab>
+    <Schedule User="root" Hour="2" Minute="7"
+      Comment="Backup the entire database at 2:07am each morning">
+      <Command><![CDATA[/usr/local/sbin/myappbak]]></Command>
+    </Schedule>
+  </Crontab>
+In this example, we specify a job to be added to root's crontab, to backup the database
+at 2:07am each morning. The time specification is as defined in <code>crontab(5)</code>:
+- Minute
+- Hour
+- DayOfMonth
+- Month
+- DayOfWeek
+The +Command+ is best wrapped safely in a CDATA section in case of XML-unsafe characters
+like $.
+
+The +Comment+ will be inserted into the final crontab file just before the job specification.
+
+=== +Database+
+If the application has a database, we need to specify the path to the DBMS client command
+line tool.
+  <Database>/usr/local/bin/mysql</Database>
+
+If using SQLite3, a +Database+ attribute is also needed to specify the file to operate on:
+  <Database database="/var/myapp/data/orders.db">/usr/bin/sqlite3</Database>
+
+== Database schema migration
+Four components are required to automate database schema migration:
+1. Database migration scripts, stored in the +evolve+ and +devolve+ sub-folders
+   of the application's staging folder. These are normal migration scripts you might apply
+   manually.
+2. A <code><Database></code> element in the deployment descriptor, containing the path to
+   the database client command line tool. In the case of SQLite3, it also needs a <code>database</code>
+   attribute containing the path of the database file to operate on.
+3. Privileges to allow the user to apply the migration scripts to the database, or else
+   it will of course fail.
+4. Credentials for the user stored privately in <code>~/.my.cnf</code> or <code>~/.pgpass</code>
+   (unless using SQLite3). This allows the scripts to be executed in batch mode, without
+   prompting for a password.

data/VERSION

@@ -0,0 +1 @@
+1.0.7

data/bin/devolve

@@ -0,0 +1,59 @@
+#!/usr/local/bin/ruby -I/usr/local/
+# This script devolves the database
+
+require 'getoptlong'
+#______________________
+# Set usage
+#
+usage=<<-EOT
+	devolve - devolve the database schema using migration scripts
+	usage:  sudo /usr/local/bin/devolve [--config configFilepath] appname target
+    where appname is the name of a folder in /var/staging containing a file deploy.xml
+	      target  integer version to devolve to
+EOT
+#______________________
+# parse options
+#
+opts = GetoptLong.new(
+  [ "--config",  "-c",            GetoptLong::REQUIRED_ARGUMENT ],
+  [ "--help",    "-h",            GetoptLong::NO_ARGUMENT ]
+)
+config = nil
+configFilename = nil
+opts.each do |opt, arg|
+	case opt
+		when "--config"
+			configFilename = arg
+		when "--help"
+			puts usage
+			exit 1
+	end
+end
+#______________________
+# check arguments
+#
+if ARGV.length != 2
+	puts(usage)
+	exit 1
+end
+appname = ARGV[0]
+targetVersion = ARGV[1].to_i()
+unless targetVersion >= 1000
+	puts("target version must be 1000 or greater")
+	puts(usage)
+	exit 1
+end
+configFilename ||= "promotion/config.rb"
+require(configFilename)
+unless File.directory?(File.expand_path(appname, Folders::Staging))
+	puts("The application #{appname} was not found in the staging area #{Folders::Staging}")
+	puts("Aborting")
+	exit 1
+end
+#______________________
+# start application
+#
+$stdout.flush()
+require 'promotion/application'
+app = Promotion::Application.new(appname)
+app.devolve(targetVersion)

data/bin/evolve

@@ -0,0 +1,60 @@
+#!/usr/local/bin/ruby -I/usr/local/
+# This script evolves the database
+
+require 'getoptlong'
+#______________________
+# Set usage
+#
+usage=<<-EOT
+	evolve - evolve the database schema using migration scripts
+	usage:  sudo /usr/local/bin/evolve [--config configFilepath] appname [target]
+		where appname is the name of a folder in /var/staging containing a file deploy.xml
+			target is an integer version to evolve to (default is to evolve as far as possible)
+EOT
+#______________________
+# parse options
+#
+opts = GetoptLong.new(
+  [ "--config",  "-c",            GetoptLong::REQUIRED_ARGUMENT ],
+  [ "--help",    "-h",            GetoptLong::NO_ARGUMENT ]
+)
+config = nil
+configFilename = nil
+opts.each do |opt, arg|
+  case opt
+    when "--config"
+        configFilename = arg
+    when "--help"
+      puts usage
+      exit 1
+  end
+end
+#______________________
+# check arguments
+#
+if ARGV.length < 1
+	puts(usage)
+	exit 1
+end
+appname = ARGV[0]
+if ARGV.length > 1
+	targetVersion = ARGV[1].to_i()
+	unless targetVersion > 1000
+		puts(usage)
+		exit 1
+	end
+end
+configFilename ||= "promotion/config.rb"
+require(configFilename)
+unless File.directory?(File.expand_path(appname, Folders::Staging))
+	puts("The application #{appname} was not found in the staging area #{Folders::Staging}")
+	puts("Aborting")
+	exit 1
+end
+#______________________
+# start application
+#
+$stdout.flush()
+require 'promotion/application'
+app = Promotion::Application.new(appname)
+app.evolve(targetVersion)

data/bin/promote

@@ -0,0 +1,67 @@
+#!/usr/local/bin/ruby -I/usr/local/
+# This script starts the promotion application
+
+#______________________
+if Process.euid != 0
+  puts("This script must be run as root")
+  exit 1
+end
+
+require 'getoptlong'
+#______________________
+# Set usage
+#
+usage=<<-EOT
+  promotion - installer and configuration manager
+  usage:  sudo /usr/local/bin/promotion [--config configFilepath] appname
+    where appname is the name of a folder under /var/staging
+EOT
+#______________________
+# parse options
+#
+opts = GetoptLong.new(
+  [ "--config",  "-c",            GetoptLong::REQUIRED_ARGUMENT ],
+  [ "--verbose", "-v",            GetoptLong::NO_ARGUMENT ],
+  [ "--help",    "-h",            GetoptLong::NO_ARGUMENT ]
+)
+config = nil
+configFilename = nil
+opts.each do |opt, arg|
+  case opt
+    when "--config"
+        configFilename = arg
+    when "--help"
+      puts usage
+      exit 1
+  end
+end
+#______________________
+# check arguments
+#
+if ARGV.length() == 0
+  puts("No application to install. We will just regenerate the environment files.")
+elsif ARGV.length() > 1
+  puts("Please specify just one application at a time.")
+  puts(usage)
+  exit 1
+end
+appname = ARGV[0]
+unless File.directory?(File.expand_path(appname, "/var/staging"))
+  puts("The application #{appname} was not found in the staging area /var/staging")
+  puts("Aborting")
+  exit 1
+end
+#______________________
+# Configure
+#
+configFilename ||= "promotion/config.rb"
+require(configFilename)
+#______________________
+# start application
+#
+print("Promoting #{appname}...")
+$stdout.flush()
+require 'promotion/application'
+app = Promotion::Application.new(appname)
+app.promote()
+puts("OK.")

data/deploy.xml

@@ -0,0 +1,35 @@
+<?xml version="1.0" encoding="UTF-8" ?>
+<Specification Name="promotion" Title="Promotion Manager"
+	xmlns="http://finalstep.com.au/promotion/v100"
+	xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
+	xsi:schemaLocation="http://finalstep.com.au/schema/promotion.v100.xsd ../promotion/promotion.xsd">
+	<Description>
+		The promotion manager ensures that the environment for an application
+		is set up correctly, including folder and file permissions and ownership,
+		startup scripts, sudoers privileges and environment variables.
+	</Description>
+	<Folders>
+		<Folder Mode="0770">/var/staging</Folder>
+		<Folder Mode="0770">/var/staging/promotion</Folder>
+		<Folder Clear="true">/usr/local/promotion</Folder>
+		<Folder>/usr/local/promotion/erb</Folder>
+	</Folders>
+	<Files Source="bin" Mode="0755">
+		<File>/usr/local/sbin/promote</File>
+		<File>/usr/local/sbin/evolve</File>
+		<File>/usr/local/sbin/devolve</File>
+	</Files>
+	<Files Source="lib/promotion">
+		<File>/usr/local/promotion/application.rb</File>
+		<File>/usr/local/promotion/config.rb</File>
+		<File>/usr/local/promotion/enforcer.rb</File>
+		<File>/usr/local/promotion/generator.rb</File>
+		<File>/usr/local/promotion/evolver.rb</File>
+	</Files>
+	<Files Source="lib/promotion/erb">
+		<File>/usr/local/promotion/erb/profile.erb</File>
+		<File>/usr/local/promotion/erb/rc.conf.local.erb</File>
+		<File>/usr/local/promotion/erb/sudoers.erb</File>
+		<File>/usr/local/promotion/erb/crontab.erb</File>
+	</Files>
+</Specification>

data/ext/promotion/extconf.rb

@@ -0,0 +1,19 @@
+#!/usr/local/bin/ruby
+# Working directory is the ext/promotion/ folder
+
+# generate Makefile to install binaries
+source = "../../bin"
+target = "/usr/local"
+makeContents=<<-EOT
+# Makefile for Promotion
+.PHONY : all
+all :
+.PHONY : install
+install :
+	install -m 0750 -o root -g wheel #{source}/promote #{target}/sbin
+	install -m 0750 -o root -g wheel #{source}/evolve  #{target}/sbin
+	install -m 0750 -o root -g wheel #{source}/devolve #{target}/sbin
+EOT
+makefile = File.new("Makefile", "w")
+makefile.puts(makeContents)
+makefile.close()

data/lib/promotion/application.rb

@@ -0,0 +1,49 @@
+require 'rubygems'
+require 'promotion/enforcer'
+require 'promotion/generator'
+require 'promotion/evolver'
+require 'log4r'
+include Log4r
+
+module Promotion	# :nodoc:
+
+# The Promotion::Application coordinates the activity of the components:
+# - Enforcer
+# - Generator
+# - Evolver
+class Application
+
+	# Creates a new Promotion::Application instance for a specific app to be promoted.
+	# A global +$log+ logger is also created.
+	def initialize(appname)
+		@appname = appname
+		$log = Logger.new(@appname)
+		FileOutputter[@appname] = FileOutputter.new(@appname, :filename => Files::Log)
+		FileOutputter[@appname].formatter = PatternFormatter.new(:pattern => "%d %m")
+		$log.outputters = FileOutputter[@appname]
+	end
+
+	# Promotes an application into production.
+	def promote()
+		enforcer = Promotion::Enforcer.new(@appname)
+		generator = Promotion::Generator.new()
+		enforcer.start()
+		generator.start()
+		$log.info("Application #{@appname} successfully promoted.")
+	end
+
+	# Evolves the database to a target version, or as far as possible, by
+	# executing database migration scripts in the +evolve+ folder
+	def evolve(target=nil)
+		evolver = Promotion::Evolver.new(@appname, true, target)
+		evolver.start()
+	end
+
+	# Devolve a database back to an earlier version
+	def devolve(target=nil)
+		evolver = Promotion::Evolver.new(@appname, false, target)
+		evolver.start()
+	end
+
+end
+end

data/lib/promotion/config.rb

@@ -0,0 +1,55 @@
+# application configuration for promotion
+# PRODUCTION
+
+# A section of the configuration file that contains folder paths
+module Folders
+	# The folder where applications are staged (eg. from Subversion)
+	# ready for deployment
+	Staging = "/var/staging"
+	# The folder containing the ERB templates
+	Templates = "/usr/local/promotion/erb"
+	# The system crontab folder
+	Crontabs = "/var/cron/tabs"
+end
+
+# A section of the configuration file that contains file paths
+module Files
+	# The standard name of the deployment descriptor for all apps
+	Spec = "deploy.xml"
+	# The log file for promotion
+	Log = "/var/staging/Promotion.log"
+	# system path to the shared profile
+	Profile = "/etc/profile"
+	# system path to the sudoers file
+	Sudoers = "/etc/sudoers"
+	# system path to rc.conf.local
+	Rc_conf = "/etc/rc.conf.local"
+	# path to useradd executable
+	Useradd = "/usr/sbin/useradd"
+	# path to groupadd executable
+	Groupadd = "/usr/sbin/groupadd"
+	# path to visudo executable
+	Visudo = "/usr/sbin/visudo"
+	# path to crontab executable
+	Crontab = "/usr/bin/crontab"
+	# path to zip executable
+	Zip = "/usr/local/bin/zip -q "
+end
+
+# A section of the configuration file that contains ERB template filenames
+module Templates
+	# filename for Profile template
+	Profile = "profile.erb"
+	# filename for Rc_conf template
+	Rc_conf = "rc.conf.local.erb"
+	# filename for Sudoers template
+	Sudoers = "sudoers.erb"
+	# filename for Crontab template
+	Crontab = "crontab.erb"
+end
+
+raise "staging area not found" unless File.directory?(Folders::Staging)
+raise "template folder not found" unless File.directory?(Folders::Templates)
+Templates.constants { |template|
+	raise "template #{template} not found" unless File.exist?(File.expand_path(Templates.const_get(template), Folders::Templates))
+}