flintlock 0.1.0

data/CHANGES.md

@@ -0,0 +1,4 @@
+
+# Version 0.1.0
+
+- Initial release.

data/LICENSE

@@ -0,0 +1,27 @@
+Copyright (c) 2013, Jon McKenzie
+All rights reserved.
+
+Redistribution and use in source and binary forms, with or without modification,
+are permitted provided that the following conditions are met:
+
+    * Redistributions of source code must retain the above copyright notice, this
+      list of conditions and the following disclaimer.
+
+    * Redistributions in binary form must reproduce the above copyright notice,
+      this list of conditions and the following disclaimer in the documentation
+      and/or other materials provided with the distribution.
+
+    * The names of the contributors to this project may not be used to endorse
+      or promote products derived from this software without specific prior
+      written permission.
+
+THIS SOFTWARE IS PROVIDED BY THE COPYRIGHT HOLDERS AND CONTRIBUTORS "AS IS" AND
+ANY EXPRESS OR IMPLIED WARRANTIES, INCLUDING, BUT NOT LIMITED TO, THE IMPLIED
+WARRANTIES OF MERCHANTABILITY AND FITNESS FOR A PARTICULAR PURPOSE ARE DISCLAIMED.
+IN NO EVENT SHALL THE COPYRIGHT HOLDER OR CONTRIBUTORS BE LIABLE FOR ANY DIRECT,
+INDIRECT, INCIDENTAL, SPECIAL, EXEMPLARY, OR CONSEQUENTIAL DAMAGES (INCLUDING, BUT
+NOT LIMITED TO, PROCUREMENT OF SUBSTITUTE GOODS OR SERVICES; LOSS OF USE, DATA,
+OR PROFITS; OR BUSINESS INTERRUPTION) HOWEVER CAUSED AND ON ANY THEORY OF LIABILITY,
+WHETHER IN CONTRACT, STRICT LIABILITY, OR TORT (INCLUDING NEGLIGENCE OR OTHERWISE)
+ARISING IN ANY WAY OUT OF THE USE OF THIS SOFTWARE, EVEN IF ADVISED OF THE
+POSSIBILITY OF SUCH DAMAGE.

data/README.md

@@ -0,0 +1,208 @@
+# flintlock
+
+``flintlock`` is a simple application deployer inspired by Heroku's buildpacks. 
+
+At its core, it's a simple scripting API which allows developers/ops the ability
+to create re-usable application deployments. In ``flintlock``, these deployments 
+are called "modules".
+
+## Installation
+
+The latest release of ``flintlock`` will be published to ``rubygems.org``. To install,
+just run:
+
+```console
+$ gem install flintlock
+```
+
+If you're running on a RHEL/CentOS 6 machine (or derivative), you might be able to
+make use of the ``flintlock`` RPM spec file located in the git repository 
+(``flintlock.spec``) to build RPM packages. Assuming all of the dependencies are
+installed, this should just be a matter of running:
+
+```console
+$ rpmbuild -ba flintlock.spec --define "scl ruby193"
+```
+
+## Tutorial
+
+Let's deploy a sample ``redis`` module I've written. This tutorial assumes you 
+are running on a CentOS 6 machine with access to the EPEL package repository.
+You'll also need ``git``.
+
+After installing ``flintlock``, run the following:
+
+```console
+flintlock deploy git://github.com/jcmcken/flintlock-redis.git /some/empty/directory
+```
+
+In this case, the ``deploy`` command will recognize that you want to deploy from ``git``.
+It will clone the remote repository, stage it, and then begin deploying the necessary
+files and directories to ``/some/empty/directory``. Let's see what happens:
+
+
+```console
+$ flintlock deploy git://github.com/jcmcken/flintlock-redis.git /some/empty/directory
+         run  fetching module
+        info  deploying jcmcken/redis (0.0.1) to '/some/empty/directory'
+      create  creating deploy directory
+         run  installing and configuring dependencies
+      create  staging application files
+         run  launching the application
+         run  altering application runtime environment
+        info  complete!
+$
+```
+
+Assuming the module was written well enough, these messages should indicate that our
+``redis`` server is running. Let's verify:
+
+```console
+$ ps -ef | grep redis
+jcmcken  24846     1  0 17:41 ?        00:00:00 /usr/sbin/redis-server /some/empty/directory/etc/redis.conf
+jcmcken  24865 19343  0 17:41 pts/1    00:00:00 grep redis
+```
+
+Let's take a look at the deploy directory, ``/some/empty/directory``:
+
+```console
+$ tree /some/empty/directory
+/some/empty/directory
+|-- bin
+|   `-- redis
+|-- data
+|-- etc
+|   `-- redis.conf
+|-- log
+|   |-- redis.log
+|   |-- stderr.log
+|   `-- stdout.log
+`-- run
+    `-- redis.pid
+```
+
+You'll notice that everything for this ``redis`` server is self-contained within our deploy
+directory. This is a central tenet of ``flintlock``:
+
+**An application deployment is always self-contained within a single directory**
+
+How well an application adheres to this philosophy depends on the application. For instance,
+some applications may not have configurable ``/tmp`` directories. For transient data, this
+is usually acceptable. But all of the important files should really be located together.
+
+## Supported Formats
+
+Currently ``flintlock`` can install modules from a number of sources. In addition to
+local directories, ``flintlock`` supports the following protocols/formats:
+
+* ``git``
+* ``tar`` or ``tar.gz`` over ``http``/``https``
+
+Attempting to install any other way will throw an error message similar to the following:
+
+```console
+         run  fetching module
+       error  don't know how to download 'https://github.com'!
+```
+
+## Writing a Module
+
+### Introduction
+
+``flintlock`` modules are simply a bunch of scripts which follow a certain convention.
+
+At its heart, a module has the following minimal layout:
+
+```text
+sample-app-1
+|-- bin
+|   |-- defaults
+|   |-- modify
+|   |-- prepare
+|   |-- stage
+|   |-- start
+|   `-- stop
+`-- metadata.json
+```
+
+Running ``flintlock new`` in an empty directory of your choosing will automatically
+generate this structure.
+
+The top-level directory (in this case, ``sample-app-1``) can be called anything. 
+
+The files under ``bin`` are executable scripts (using any language you care to use). All
+of these scripts must exist, but they need not do anything. More on these later.
+
+The ``metadata.json`` file contains metadata about the module. This metadata looks as follows:
+
+```json
+{
+  "author": "jcmcken",
+  "name": "sample-app-1",
+  "version": "0.0.1"
+}
+```
+
+All three keys (``author``, ``name``, ``version``) are required, but can be any value. These
+metadata are merely used to namespace the module.
+
+``flintlock`` developers can choose to include more files in their modules if needed.
+
+### Stages
+
+``flintlock`` has different "stages" of execution that occur in a specific order every time
+you run a deployment.
+
+These stages correspond directly to the scripts under ``bin/``.
+
+The most important stages, and their purpose, are as follows. (The stages occur in the order listed below)
+
+* ``prepare``: Install or compile any required dependencies. This script takes no arguments.
+* ``stage``: Stage the application directories and files. This script takes a single argument,
+  which is the directory where your app will be deployed. This directory need not exist, but if
+  it does, it must be empty.
+* ``start``: Start the application. This script takes the same argument passed to ``stage``.
+* ``modify``: Once the application is started, perform some runtime modifications. For instance,
+  if you've just started a MySQL server, you may want to remove the default tables or add a 
+  password to the database superuser. This script takes the same argument passed to ``stage``
+  and ``start``.
+
+The API between these scripts and ``flintlock`` is as follows:
+
+* If the script exits with a return code of ``0``, ``flintlock`` will think that the script
+  succeeded.
+* If the script exits with a return code of ``1``, ``flintlock`` will think that the script
+  has failed.
+* Any other return code, and ``flintlock`` will think that some sort of internal error has
+  occurred. In other words, something outside of the script's control failed.
+
+When ``flintlock`` encounters a non-zero exit code, it will halt execution and display an
+error.
+
+### Configuration Defaults
+
+A ``flintlock`` module may also choose to utilize the ``bin/defaults`` script to set
+configuration defaults.
+
+By default, ``flintlock`` will source this script prior to executing any of the other 
+stages.
+
+A user can choose to override these defaults at the command line. For example, if your
+``defaults`` script looks like:
+
+```bash
+PORT=80
+```
+
+The user can override this at the command line by running:
+
+```console
+PORT=8080 flintlock deploy <module> <deploy_dir>
+```
+
+``flintlock`` will transparently override the default ``PORT`` with the env var passed at the
+command line.
+
+### Examples
+
+An example ``flintlock`` module can be found @ http://github.com/jcmcken/flintlock-redis.git.

data/bin/flintlock

@@ -0,0 +1,5 @@
+#!/usr/bin/env ruby
+
+require 'flintlock/cli'
+
+Flintlock::Cli.start

data/lib/flintlock/cli.rb

@@ -0,0 +1,71 @@
+require 'thor'
+require 'flintlock/module'
+
+module Flintlock
+  class Cli < Thor
+    include Thor::Actions
+
+    desc "deploy MODULE DIRECTORY", "Deploy a flintlock module MODULE to DIRECTORY"
+    method_option :debug, :type => :boolean, :description => "enable debug output", :default => false
+    def deploy(uri, app_dir)
+      say_status "run", "fetching module", :magenta
+      mod = get_module(uri, options)
+      say_status "info", "deploying #{mod.full_name} to '#{app_dir}'", :blue
+      say_status "create", "creating deploy directory"
+      mod.create_app_dir(app_dir) rescue abort("deploy directory is not empty")
+
+      begin
+        say_status "run", "installing and configuring dependencies", :magenta
+        mod.prepare
+        say_status "create", "staging application files"
+        mod.stage(app_dir)
+        say_status "run", "launching the application", :magenta
+        mod.start(app_dir)
+        say_status "run", "altering application runtime environment", :magenta
+        mod.modify(app_dir)
+        say_status "info", "complete!", :blue
+      rescue Errno::EACCES => e
+        abort("#{e.message.gsub(/Permission denied/, 'permission denied')}")
+      rescue RunFailure
+        abort('stage failed!')
+      end
+    end
+
+    desc "new [DIRECTORY]", "Generate a new, minimal flintlock module"
+    def new(directory = Dir.pwd)
+      abort("directory isn't empty!") if ! Util.empty_directory?(directory)
+      inside(directory) do
+        empty_directory "bin"
+        inside("bin") do
+          Module.script_names.each do |script|
+            create_file script
+          end
+        end 
+        create_file(Metadata.filename, Metadata.empty)
+      end
+    end
+
+    private
+
+    def get_module(uri, options={})
+      begin
+        Flintlock::Module.new(uri, options)
+      rescue InvalidModule => e
+        abort("invalid flintlock module '#{e}'")
+      rescue UnsupportedModuleURI => e
+        abort("don't know how to download '#{e}'!")
+      rescue ModuleDownloadError => e
+        abort("failed to download '#{e}'")
+      end
+    end
+
+    def error(message)
+      say_status "error", message, :red
+    end
+
+    def abort(message)
+      error(message)
+      exit(1)
+    end
+  end
+end

data/lib/flintlock/logger.rb

@@ -0,0 +1,13 @@
+require 'logger'
+
+module Flintlock
+  class Logger < ::Logger
+    def silence!
+      @saved_logdev, @logdev = @logdev, nil
+    end
+
+    def unsilence!
+      @logdev, @saved_logdev = @saved_logdev, nil if @saved_logdev
+    end
+  end
+end

data/lib/flintlock/metadata.rb

@@ -0,0 +1,53 @@
+require 'json'
+
+module Flintlock
+  class Metadata
+    attr_reader :filename
+
+    def initialize(filename = nil)
+      @filename = filename || default_metadata_file
+      @data = Metadata.load(@filename)
+    end
+
+    def self.filename
+      'metadata.json'
+    end
+
+    def valid?
+      begin
+        result = ! [author, version, name].map(&:empty?).any?
+      rescue
+        result = false
+      end 
+      return result
+    end
+
+    def default_metadata_file
+      File.join(Dir.pwd, Metadata.filename)
+    end
+
+    def self.load(filename)
+      JSON.load(File.read(filename))
+    end
+
+    def author
+      @data.fetch('author')
+    end
+
+    def version
+      @data.fetch('version')
+    end
+
+    def name
+      @data.fetch('name')
+    end
+
+    def full_name
+      "#{author}/#{name} (#{version})"
+    end
+
+    def self.empty
+      {"author" => "", "version" => "", "name" => ""}.to_json
+    end
+  end
+end

data/lib/flintlock/module.rb

@@ -0,0 +1,208 @@
+require 'flintlock/metadata'
+require 'flintlock/logger'
+require 'flintlock/util'
+require 'open3'
+require 'fileutils'
+require 'logger'
+require 'shellwords'
+require 'uri'
+require 'tmpdir'
+require 'tempfile'
+require 'open-uri'
+
+module Flintlock
+  class InvalidModule < RuntimeError; end
+  class UnsupportedModuleURI < RuntimeError; end
+  class ModuleDownloadError < RuntimeError; end
+  class RunFailure < RuntimeError; end
+
+  class Module
+    attr_reader :uri, :metadata
+
+    def initialize(uri = nil, options={})
+      # track temporary files and directories for deletion
+      @tmpfiles = []
+     
+      # destroy tmp files on exit 
+      at_exit { handle_exit }
+
+      @debug = !!options[:debug]
+      @uri = uri || Dir.pwd
+      @log = load_logger
+      @root_dir = download_from_uri(@uri)
+      @metadata = load_metadata
+
+      load_scripts!
+      validate
+
+      @env = load_env(@defaults_script)
+
+    end
+
+    def download_from_uri(uri)
+      case URI.parse(uri).scheme
+      when nil # no scheme == local file
+        uri
+      when 'git'
+        handle_git_uri(uri)
+      when 'http', 'https'
+        raise UnsupportedModuleURI.new(uri) if ! Util.supported_archive?(uri)
+        # over these protocols, we're getting an archive
+        handle_archive(handle_http_uri(uri))
+      else
+        raise UnsupportedModuleURI, uri
+      end
+    end
+
+    def handle_exit
+      @tmpfiles.each { |x| FileUtils.rm_rf(x, :secure => true) }
+    end
+
+    def handle_git_uri(uri)
+      root_dir = Dir.mktmpdir
+      @tmpfiles << root_dir
+      command = Shellwords.join(['git', 'clone', uri, root_dir])
+      stdout, stderr, status = Open3.capture3(command)
+      raise ModuleDownloadError, uri if status.exitstatus != 0 
+      root_dir
+    end
+
+    def handle_http_uri(uri, buffer=8192)
+      tmpfile = Tempfile.new(['flintlock', Util.full_extname(uri)]).path
+      @tmpfiles << tmpfile
+      open(uri) do |input|
+        open(tmpfile, 'wb') do |output|
+          while ( buf = input.read(buffer))
+            output.write buf
+          end
+        end
+      end
+      tmpfile
+    rescue OpenURI::HTTPError
+      raise ModuleDownloadError, uri
+    end
+
+    def handle_archive(filename)
+      tmpdir = Dir.mktmpdir
+      @tmpfiles << tmpdir
+      case filename
+      when /\.tar\.gz$/
+        command = ['tar', 'xfz', filename, '-C', tmpdir]
+      when /\.tar$/
+        command = ['tar', 'xf', filename, '-C', tmpdir]
+      else
+        raise UnsupportedModuleURI, filename
+      end
+      _, _, status = Open3.capture3(Shellwords.join(command))
+      raise ModuleDownloadError if status.exitstatus != 0
+      tmpdir
+    end
+
+    def full_name
+      @metadata.full_name
+    end
+
+    def self.script_names
+      ['defaults', 'modify', 'prepare', 'stage', 'start', 'stop']
+    end
+
+    def scripts
+      [@modify_script, @prepare_script, @stage_script, @start_script, @stop_script, @defaults_script]
+    end
+
+    def scripts_exist?
+      scripts.map { |x| File.file?(x) }.all?
+    end
+
+    def valid?
+      @metadata.valid? && scripts_exist?
+    end
+
+    def prepare
+      @log.info("running prepare stage: #{@prepare_script}")
+      run_script(@prepare_script)
+    end
+
+    def stage(app_dir)
+      @log.info("running stage stage: #{@stage_script}")
+      run_script(@stage_script, app_dir)
+    end
+    
+    def modify(app_dir)
+      @log.info("running modify stage: #{@modify_script}")
+      run_script(@modify_script, app_dir)
+    end
+  
+    def start(app_dir)
+      @log.info("running start stage: #{@start_script}")
+      run_script(@start_script, app_dir)
+    end
+    
+    def stop(app_dir)
+      @log.info("running stop stage: #{@stop_script}")
+      run_script(@stop_script, app_dir)
+    end
+
+    def current_env
+      Hash[ENV.to_a] # get rid of ENV obj
+    end
+
+    def load_env(defaults_script)
+      # hokey, but seems to work
+      env_data = %x{set -a && source #{defaults_script} && env}.split.map{ |x| x.split('=', 2) }
+      env = Hash[env_data]
+      @log.debug("defaults script is #{defaults_script}")
+      @log.debug("defaults env is #{env.inspect}")
+      env = env.merge(current_env)
+      @log.debug("merged env is #{env.inspect}")
+      env
+    end
+
+    def create_app_dir(app_dir)
+      FileUtils.mkdir_p(app_dir)
+      raise if ! Util.empty_directory?(app_dir)
+    end
+
+    private
+
+    def load_scripts!
+      Module.script_names.map do |x|
+        instance_variable_set("@#{x}_script".to_sym, File.join(@root_dir, 'bin', x))
+      end
+    end
+
+    def validate
+      raise InvalidModule.new(@uri) if ! valid?
+    end
+
+    def load_logger
+      log = Logger.new(STDOUT)
+      log.silence! if ! @debug
+      log
+    end
+
+    def load_metadata
+      begin
+        Metadata.new(File.join(@root_dir, Metadata.filename)) 
+      rescue Errno::ENOENT
+        raise InvalidModule, uri
+      end
+    end
+
+    def run(command)
+      handle_run(*Open3.capture3(@env, command))
+    end
+
+    def run_script(script, *args)
+      run(Shellwords.join([script, *args]))
+    end
+
+    def handle_run(stdout, stderr, status)
+      stdout.lines.each { |x| @log.info(x) }
+      if status.exitstatus != 0
+        stderr.lines.each { |x| @log.error(x) }
+        raise RunFailure
+      end
+    end
+  end
+end

data/lib/flintlock/util.rb

@@ -0,0 +1,27 @@
+module Flintlock
+  class Util
+    def self.empty_directory?(directory)
+      Dir[File.join(directory, '*')].empty?
+    end
+  
+    def self.supported_archives
+      ['.tar.gz', '.tar']
+    end
+  
+    def self.supported_archive?(filename)
+      Util.supported_archives.include?(full_extname(filename))
+    end
+  
+    def self.full_extname(filename)
+      data = []
+      current_filename = filename.dup
+      while true
+        ext = File.extname(current_filename)
+        break if ext.empty?
+        current_filename = current_filename.gsub(/#{ext}$/, '')
+        data << ext
+      end
+      data.reverse.join
+    end
+  end
+end

data/lib/flintlock/version.rb

@@ -0,0 +1,3 @@
+module Flintlock
+  VERSION = '0.1.0'
+end

data/lib/flintlock.rb

@@ -0,0 +1,3 @@
+require 'rubygems'
+require 'flintlock/metadata'
+require 'flintlock/module'

metadata

@@ -0,0 +1,89 @@
+--- !ruby/object:Gem::Specification
+name: flintlock
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+  prerelease: 
+platform: ruby
+authors:
+- Jon McKenzie
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2014-05-26 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: thor
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: json
+  requirement: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    none: false
+    requirements:
+    - - ! '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+description: A simple application deployer inspired by Heroku's buildpacks
+email: jcmcken@gmail.com
+executables:
+- flintlock
+extensions: []
+extra_rdoc_files: []
+files:
+- lib/flintlock.rb
+- lib/flintlock/module.rb
+- lib/flintlock/metadata.rb
+- lib/flintlock/util.rb
+- lib/flintlock/cli.rb
+- lib/flintlock/logger.rb
+- lib/flintlock/version.rb
+- LICENSE
+- README.md
+- CHANGES.md
+- bin/flintlock
+homepage: https://github.com/jcmcken/flintlock
+licenses:
+- MIT
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: 1.9.3
+required_rubygems_version: !ruby/object:Gem::Requirement
+  none: false
+  requirements:
+  - - ! '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 1.8.23
+signing_key: 
+specification_version: 3
+summary: A simple application deployer
+test_files: []