conjure 0.2.10 → 0.3.0

data/History.md

@@ -1,3 +1,11 @@
+### Version 0.3.0
+2016-01-04
+
+* Remove command-line app in favor of the provisioning API
+* Revise API with support for updating existing Instances
+* Monitor and restart background workers for deployed apps using DelayedJob
+* Upgrade to Passenger base images v0.9.18
+
 ### Version 0.2.10
 2015-06-24
 

data/README.md

@@ -4,36 +4,27 @@
 [![Code Climate](https://codeclimate.com/github/brianauton/conjure.png)](https://codeclimate.com/github/brianauton/conjure)
 [![Dependency Status](https://gemnasium.com/brianauton/conjure.png)](https://gemnasium.com/brianauton/conjure)
 
+Conjure is a Ruby library for creating and updating cloud server instances as part of a Rails deployment workflow.
 
-Magically powerful deployment for Rails applications.
+Conjure creates a cloud server instance, and uses Docker to start separate containers for the database and the web server. It sets up data volume containers for both of these service containers, so all data will be preserved if any individual containers or the server itself are restarted.
 
-### Requirements
-
-Deploying a Rails application with Conjure currently requires the
-following:
-
-  * A DigitalOcean account
+Conjure sets up an environment suitable for running the Rails application into which you've installed the Conjure gem, but it does not install your application. See the [capistrano-conjure](https://github.com/brianauton/capistrano-conjure) gem for an easy way to combine Conjure with deployment of your application.
 
-  * A Public/private SSH keypair to use for bootstrapping new cloud
-    servers. Generating a new keypair for this purpose is recommended.
+WARNING: Conjure creates server instances and other resources using service provider accounts that you specify. In most cases this incurs service charges that will recur until you explicitly cancel them. You are responsible for all charges incurred through your use of Conjure.
 
-Also, your Rails application requires all of the following:
+WARNING: Conjure attempts to have a good security model and to treat your code and data responsibly, but this is not guaranteed. You are responsible for the security and confidentiality of the code of applications deployed with Conjure, the data handled by these applications, and your service credentials and public keys used for deployment.
 
-  * It must have a `.ruby-version` file indicating which version of
-    Ruby to run
+### Requirements
 
-  * It must be able to run in production mode with a single Postgres
-    or MySQL database (any existing database.yml will be ignored)
+* A DigitalOcean account (other cloud services may be supported in the future).
 
-  * It must be checked out locally into a git repository with a valid
-    `origin` remote
+* A public SSH key located in `~/.ssh/id_rsa.pub`. This public key will be uploaded to the instance to allow subsequent access to the server. (other methods of granting server access to developers may be supported in the future).
 
-  * The public SSH key you're using must have permission to check out
-    the project from `origin`
+* A Rails app that uses Postgres as its database (other databases may be supported in the future).
 
 ### Getting Started
 
-First, install the Conjure gem by either adding it to your Gemfile
+First, install the Conjure gem either by adding it to your Gemfile
 
     group :development do
       gem "conjure"
@@ -43,84 +34,52 @@ and then running `bundle`, OR by installing it directly:
 
     gem install conjure
 
-Then add a file to your Rails project called
-`config/conjure.yml`. This should be a YAML file with the following
-fields:
-
-  * `digitalocean_client_id` and `digitalocean_api_key`: These
-    credentials are available after logging in to your Digital Ocean
-    account.
-
-  * `digitalocean_region`: The geographic region for deploying new
-    cloud servers. If unsure, use "New York 1".
-
-  * `private_key_file` and `public_key_file` (optional): Pathnames to
-    local files (absolute paths, or relative to your project's
-    `config` directory) that contain the private and public SSH keys
-    to use for deployment. If these aren't specified, Conjure will try
-    to find identity files in `~/.ssh`.
-
-Here's an example conjure.yml file:
-
-    digitalocean_client_id: XXXXXXXX
-    digitalocean_api_key: XXXXXXXX
-    digitalocean_region: New York 1
+Then set the DIGITALOCEAN_API_TOKEN environment variable to your DigitalOcean API token. Note that you need a single token that was generated for DigitalOcean's v2 API, not a key and secret pair as were used with their older v1 API.
 
-Finally, tell Conjure to deploy your app:
+    export DIGITALOCEAN_API_TOKEN=xxxxxxxxxxxxxxxx
 
-    conjure deploy
+Now you're ready to provision a new instance of the Rails app. Here's an example of Ruby code you could run from the Rails console or from a Rake task in your Rails app:
 
-The last line of the output will tell you the IP address of the
-deployed server. Repeating the command will reuse the existing server
-rather than deploying a new one. Specify a branch to deploy with
-`--branch` or `-b` (default is `master`):
+    Conjure::Instance.create app_name: "widget_store", rails_env: "demo", ruby_version: "2.0"
 
-    conjure deploy --branch=mybranch
+### Creating a new instance
 
-### Additional Commands
+To create a new instance on a new DigitalOcean droplet, call `Conjure::Instance.create` with a list of options. The following options are required:
 
-Additional commands are available after you've deployed with `conjure
-deploy`.
+* app_name: The application name; included in the name of the DigitalOcean droplet for easy identification.
 
-#### Export
+Conjure will create the instance, and then give you a summary of information about the instance (including its IP address) that you'll need in order to access the instance and deploy your application.
 
-Produce a native-format (Postgres or MySQL) dump of the
-currently-deployed server's production database, and save it to the
-local file `FILE`.
+### Updating an existing instance
 
-    conjure export FILE
+To update an existing instance, call `Conjure::Instance.update` with a list of options. The following options are required:
 
-#### Import
+* ip_address: The IP address of the DigitalOcean droplet to update.
 
-Overwrite the production database on the currently-deployed server
-with a dump from the local file `FILE`. The dump should be in the same
-format as that produced by the `export` command (either a Postgres or
-MySQL dump according to the database type).
+Updating an instance simply involves checking that all the necessary Docker containers are running, and starting any that aren't running (after building them according to the supplied options). This means that if you want to change the configuration of your Rails application's web server, you can SSH to the droplet and manually stop and remove the `passenger` container, then run an update to have it rebuilt. This will preserve all your application's data, since that is stored in volume containers. An easier method for changing the options on existing instances may be added in the future.
 
-    conjure import FILE
+### Options
 
-#### Log
+The following additional options are supported for both `create` and `update`:
 
-Show logs from the deployed application. Optionally specify the number
-of lines with `-n`, and use --tail to continue streaming new lines as
-they are added to the log.
+* max_upload_mb: The maximum size in megabytes of uploaded files that the web server should allow. Default is 20.
 
-    conjure log [-n=NUM] [--tail|-t]
+* rails_env: The Rails environment, e.g. "staging" or "production". Default is "staging".
 
-#### Console
+* ruby_version: Valid values are "1.9", "2.0", "2.1", and "2.2" (the default).
 
-Open a console on the deployed application.
+* rubygems_version: Use this only if your application requires a specific version of RubyGems, otherwise a reasonably recent version will be used.
 
-    conjure console
+* ssl_hostname: Optionally configure the web server to respond to SSL connections at the given hostname. Currently this requires you to upload certificate and key files to the server after the instance is created.
 
-#### Rake
+* system_packages: An array of the names of any additional `apt` packages that should be installed in the Ubuntu container that runs your Rails app.
 
-Run a rake task on the deployed application and show the output.
+### Development
 
-    conjure rake [ARGUMENTS...]
+After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment. Run `bundle exec conjure` to use the gem in this directory, ignoring other installed copies of this gem.
 
-#### Show
+To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
 
-List the current status of all deployed instances of your command.
+### Contributing
 
-    conjure show
+Bug reports and pull requests are welcome on GitHub at [https://github.com/brianauton/conjure](https://github.com/brianauton/conjure).

data/lib/conjure.rb

@@ -1,12 +1 @@
-module Conjure
-  require File.join(File.dirname(__FILE__), "conjure/provider")
-  Dir[File.join(File.dirname(__FILE__), "conjure/**/*.rb")].each { |f| require f }
-
-  def self.config
-    @config ||= Config.load Dir.pwd
-  end
-
-  def self.identity
-    @identity ||= Identity.new(config)
-  end
-end
+require "conjure/instance"

data/lib/conjure/delayed_job.rb

@@ -0,0 +1,39 @@
+module Conjure
+  class DelayedJob
+    def initialize(options)
+      @rails_env = options[:rails_env]
+    end
+
+    def apply(template)
+      template.add_file_data monit_rc, "/etc/monit/monitrc"
+      template.run "chmod 0600 /etc/monit/monitrc"
+      template.add_file_data monit_run, "/etc/service/monit/run"
+      template.run "chmod 0700 /etc/service/monit/run"
+    end
+
+    def system_packages
+      ["monit"]
+    end
+
+    private
+
+    def monit_rc
+      command = "/usr/bin/env RAILS_ENV=#{@rails_env} /home/app/application/current/bin/delayed_job"
+      'set daemon 10
+set logfile /var/log/monit.log
+set idfile /var/lib/monit/id
+set statefile /var/lib/monit/state
+check process delayed_job
+  with pidfile /home/app/application/shared/tmp/pids/delayed_job.pid
+  start program = "' + command + ' start" as uid "app" and gid "app"
+  stop program = "' + command + ' stop" as uid "app" and gid "app"
+'
+    end
+
+    def monit_run
+      '#!/bin/sh
+exec /usr/bin/monit -I
+'
+    end
+  end
+end

data/lib/conjure/digital_ocean/droplet.rb

@@ -1,5 +1,6 @@
 require "conjure/digital_ocean/account"
 require "conjure/digital_ocean/key_set"
+require "securerandom"
 
 module Conjure
   module DigitalOcean
@@ -20,9 +21,10 @@ module Conjure
       private
 
       def create
+        puts "Creating DigitalOcean droplet..."
         response = account.post("droplets", {
           image: @options[:image],
-          name: @options[:name],
+          name: "#{@options[:name_prefix]}-#{SecureRandom.hex 4}",
           region: @options[:region],
           size: @options[:size],
           ssh_keys: [key.id],
@@ -39,7 +41,8 @@ module Conjure
       end
 
       def account
-        @account ||= Account.new(:token => @options[:token])
+        raise "Error: DIGITALOCEAN_API_TOKEN must be set." unless ENV["DIGITALOCEAN_API_TOKEN"]
+        @account ||= Account.new(:token => ENV["DIGITALOCEAN_API_TOKEN"])
       end
 
       def key

data/lib/conjure/docker/host.rb

@@ -0,0 +1,75 @@
+module Conjure
+  module Docker
+    class Host
+      def initialize(server)
+        @server = server
+      end
+
+      def start(image_name, daemon_command, options = {})
+        container_name = options[:name]
+        all_options = "#{start_options options} #{image_name} #{daemon_command}"
+        if running? container_name
+          puts "Detected #{container_name} container running."
+        else
+          puts "Starting #{container_name} container..."
+          @server.run("docker run #{all_options}").strip
+          sleep 2
+          raise "Container failed to start" unless running? container_name
+        end
+      end
+
+      def build(image_source_files)
+        Dir.mktmpdir do |dir|
+          image_source_files.each { |filename, data| File.write "#{dir}/#{filename}", data }
+          result = with_directory(dir) { |remote_dir| @server.run "docker build #{remote_dir}" }
+          match = result.match(/Successfully built ([0-9a-z]+)/)
+          raise "Failed to build Docker image, output was #{result}" unless match
+          match[1]
+        end
+      end
+
+      def running?(container_name)
+        running_container_names.include? container_name
+      end
+
+      private
+
+      def with_directory(local_path, &block)
+        local_archive = remote_archive = "/tmp/archive.tar.gz"
+        remote_path = "/tmp/unpacked_archive"
+        `cd #{local_path}; tar czf #{local_archive} *`
+        @server.send_file local_archive, remote_archive
+        @server.run "mkdir #{remote_path}; cd #{remote_path}; tar mxzf #{remote_archive}"
+        yield remote_path
+      ensure
+        `rm #{local_archive}`
+        @server.run "rm -Rf #{remote_path} #{remote_archive}"
+      end
+
+      def start_options(options)
+        [
+          "-d",
+          "--restart=always",
+          mapped_options("--link", options[:linked_containers]),
+          ("--name #{options[:name]}" if options[:name]),
+          mapped_options("-p", options[:ports]),
+          listed_options("--volumes-from", options[:volume_containers]),
+        ].flatten.compact.join(" ")
+      end
+
+      def listed_options(command, values)
+        values ||= []
+        values.map { |v| "#{command} #{v}" }
+      end
+
+      def mapped_options(command, values)
+        values ||= {}
+        values.map { |from, to| "#{command} #{from}:#{to}" }
+      end
+
+      def running_container_names
+        @server.run("docker ps --format='{{.Names}}'").split("\n").compact
+      end
+    end
+  end
+end

data/lib/conjure/docker/template.rb

@@ -0,0 +1,71 @@
+require "conjure/docker/host"
+require "tmpdir"
+
+module Conjure
+  module Docker
+    class Template
+      def initialize(base_image_name)
+        @commands = ["FROM #{base_image_name}"]
+        @file_data = {}
+      end
+
+      def add_file(filename, remote_name)
+        add_file_data File.read(filename), remote_name
+      end
+
+      def add_file_data(data, remote_name)
+        local_name = "file#{@file_data.length+1}"
+        @file_data[local_name] = data
+        @commands << "ADD #{local_name} #{remote_name}"
+      end
+
+      def run(command)
+        @commands << "RUN #{command}"
+      end
+
+      def volume(name)
+        @commands << "VOLUME #{name}"
+      end
+
+      def source
+        @commands.join "\n"
+      end
+
+      def environment(hash)
+        hash.each { |key, value| @commands << "ENV #{key} #{value}" }
+      end
+
+      def start(container_host, command, options = {})
+        if container_names(options).all? { |name| container_host.running? name }
+          puts "Detected all #{options[:name]} containers running."
+        else
+          puts "Building #{options[:name]} base image..."
+          image_name = container_host.build(image_source_files)
+          options = options.merge(volume_options(container_host, image_name, options)) if options[:volumes]
+          container_host.start(image_name, command, options)
+        end
+      end
+
+      private
+
+      def container_names(options)
+        [options[:name]] + options[:volumes].to_h.keys
+      end
+
+      def volume_options(container_host, image_name, options)
+        {
+          volume_containers: options[:volumes].map do |name, path|
+            volume_template = Docker::Template.new(image_name)
+            volume_template.volume path
+            volume_template.start(container_host, "/bin/true", name: name)
+            name
+          end
+        }
+      end
+
+      def image_source_files
+        @file_data.merge "Dockerfile" => @commands.join("\n")
+      end
+    end
+  end
+end

data/lib/conjure/instance.rb

@@ -1,102 +1,57 @@
+require "conjure/docker/host"
+require "conjure/rails_application"
+require "conjure/server"
+require "conjure/swap"
+require "yaml"
+
 module Conjure
   class Instance
-    def initialize(options)
-      @origin = options[:origin]
-      @branch = options[:branch]
-      @rails_environment = options[:rails_environment]
-      @server = options[:server]
+    def initialize(ip_address, options)
+      @server = Server.new ip_address
+      @options = options
     end
 
-    def self.where(options = {})
-      Collection.new(options)
+    def self.create(options)
+      @server = Server.create server_name_prefix(options), options
+      new(@server.ip_address, options).tap(&:update)
     end
 
-    def origin
-      @origin ||= @server.name.split("-")[0]
+    def self.update(options)
+      ip_address = options.delete(:ip_address)
+      new(ip_address, options).tap(&:update)
     end
 
-    def rails_environment
-      @rails_environment ||= @server.name.split("-")[1]
+    def update
+      components.each(&:install)
     end
 
     def ip_address
       @server.ip_address
     end
 
-    def shell
-      rails_server.base_image
-    end
-
-    def branch
-      @branch ||= codebase.branch
+    def port
+      2222
     end
 
-    def database
-      codebase.database
+    def user
+      "app"
     end
 
-    def create
-      @server_name = Service::CloudServer.ensure_unique_name(server_name)
-      deploy
+    def pending_files
+      components.flat_map(&:pending_files)
     end
 
-    def deploy
-      Log.info "[deploy] Deploying #{branch} to #{rails_environment}"
-      codebase.install
-      rails_server.run
-      Log.info "[deploy] Application deployed to #{ip_address}"
-    end
-
-    def codebase
-      @codebase ||= Service::RailsCodebase.new target, origin, @branch, rails_environment
-    end
+    private
 
-    def rails_server
-      @rails_server ||= Service::RailsServer.new target, rails_environment
+    def self.server_name_prefix(options)
+      "#{options[:app_name]}-#{options[:rails_env]}"
     end
 
-    def server_name
-      @server_name ||= "#{application_name}-#{rails_environment}"
-    end
-
-    def server
-      @server ||= Service::CloudServer.new(server_name)
-    end
-
-    def target
-      @target ||= Target.new(:machine_name => server.name)
-    end
-
-    def application_name
-      Application.new(:origin => @origin).name
-    end
-
-    def status
-      "running"
-    end
-
-    def name
-      server.name
-    end
-
-    class Collection
-      include Enumerable
-
-      def initialize(options)
-        @origin = options[:origin]
-      end
-
-      def application_name
-        Application.new(:origin => @origin).name
-      end
-
-      def each(&block)
-        return unless @origin
-        Service::CloudServer.each_with_name_prefix("#{application_name}-") do |server|
-          match = server.name.match(/^#{application_name}-([^-]+)(-[0-9]+)?$/)
-          yield Instance.new(:server => server) if match
-        end
-      end
+    def components
+      @components ||= [
+        Swap.new(@server),
+        RailsApplication.new(Docker::Host.new(@server), @options),
+      ]
     end
   end
 end