centurion 1.4.2 → 1.5.0

data/README.md

@@ -25,7 +25,7 @@ Commercial Docker Registry Providers:
 Open-source:
 - The [Docker registry](https://github.com/dotcloud/docker-registry) project,
   built and maintained by Docker. You host this yourself.
-- (*NEW!*) [Dogestry](https://github.com/newrelic-forks/dogestry) is an
+- (*NEW!*) [Dogestry](https://github.com/dogestry/dogestry) is an
   s3-backed Docker registry alternative that removes the requirement to set up
   a centralized registry service or host anything yourself.
 
@@ -184,6 +184,23 @@ This is currently pretty inflexible in that the hostname will now be the same
 on all of your hosts. A possible future addition is the ability to pass
 a block to be evaluated on each host.
 
+### CGroup Resource Constraints
+
+Limits on memory and CPU can be specified with the `memory` and `cpu_shares`
+settings. Both of these expect a 64-bit integer describing the number of
+bytes, and the number of CPU shares, respectively.
+
+For example, to limit the memory to 1G, and the cpu time slice to half the
+normal length, include the following:
+
+```ruby
+memory 1024000000
+cpu_shares 512
+```
+
+For more information on Docker's CGroup limits see [the Docker
+docs](https://docs.docker.com/reference/run/#runtime-constraints-on-cpu-and-memory).
+
 ### Interpolation
 
 Currently there is one special string for interpolation that can be added to
@@ -317,7 +334,7 @@ $ bundle exec centurion -p radio-radio -e staging -a list
 
 Centurion needs to have access to some registry in order to pull images to
 remote Docker servers. This needs to be either a hosted registry (public or
-private), or [Dogestry](https://github.com/newrelic-forks/dogestry).
+private), or [Dogestry](https://github.com/dogestry/dogestry).
 
 #### Access to the registry 
 
@@ -340,7 +357,7 @@ These correspond to the following settings:
 Centurion normally uses the built-in registry support in the Docker daemon to
 handle pushing and pulling images. But Centurion also has the ability to use
 external tooling to support hosting your registry on Amazon S3. That tooling is
-from a project called [Dogestry](https://github.com/newrelic-forks/dogestry).
+from a project called [Dogestry](https://github.com/dogestry/dogestry).
 We have recently improved that tooling substantially in coordination with the
 Centurion support.
 
@@ -349,7 +366,7 @@ with Amazon S3 to provide reliable hosting of images.  Setting Centurion up to
 use Dogestry is pretty trivial:
 
  1. Install Dogestry binaries on the client from which Dogestry is run.
-    Binaries are provided in the [GitHub release](https://github.com/newrelic-forks/dogestry).
+    Binaries are provided in the [GitHub release](https://github.com/dogestry/dogestry).
  1. Add the settings necessary to get Centurion to pull from Dogestry. A config
     example is provided below:
 
@@ -365,6 +382,24 @@ namespace :environment do
 end
 ```
 
+Development
+-----------
+
+Sometimes when you're doing development you want to try out some configuration
+settings in environment variables that aren't in the config yet. Or perhaps you
+want to override existing settings to test with. You can provide the
+`--override-env` command line flag with some overrides or new variables to set.
+Here's how to use it:
+
+```bash
+$ centurion -e development -a deploy -p radio-radio --override-env=SERVICE_PORT=8080,NAME=radio
+```
+
+Centurion is aimed at repeatable deployments so we don't recommend that you use
+this functionality for production deployments. It will work, but it means that
+the config is not the whole source of truth for your container configuration.
+Caveat emptor.
+
 Future Additions
 ----------------
 
@@ -388,6 +423,7 @@ Pull requests should:
  * Have a description that explains the need for the changes
  * Include tests!
  * Not break the public API
+ * Add yourself to the CONTRIBUTORS file. I might forget.
 
 If you are simply looking to contribute to the project, taking on one of the
 items in the "Future Additions" section above would be a great place to start.
@@ -400,4 +436,4 @@ patents, and ideas in that code in our products if we so choose. You also agree
 the code is provided as-is and you provide no warranties as to its fitness or
 correctness for any purpose
 
-Copyright (c) 2014 New Relic, Inc. All rights reserved.
+Copyright (c) 2014-2015 New Relic, Inc. All rights reserved.

data/bin/centurion

@@ -22,16 +22,17 @@ Dir.glob(File.join(task_dir, '*.rake')).each { |file| load file }
 require 'trollop'
 
 opts = Trollop::options do
-  opt :project,          'project (blog, forums...)',                   type: String, required: true,  short: '-p'
-  opt :environment,      "environment (production, staging...)",        type: String, required: true,  short: '-e'
-  opt :action,           'action (deploy, list...)',                    type: String, default: 'list', short: '-a'
-  opt :image,            'image (yourco/project...)',                   type: String, required: false, short: '-i'
-  opt :tag,              'tag (latest...)',                             type: String, required: false, short: '-t'
-  opt :hosts,            'hosts, comma separated',                      type: String, required: false, short: '-h'
-  opt :docker_path,      'path to docker executable (default: docker)', type: String, default: 'docker', short: '-d'
-  opt :no_pull,          'Skip the pull_image step',                    type: :flag, default: false, short: '-n'
-  opt :registry_user,    'user for registry auth (default: nil)',       type: String, default: nil, short: :none
-  opt :registry_password,'password for registry auth (default: nil)',   type: String, default: nil, short: :none
+  opt :project,          'project (blog, forums...)',            type: String, required: true,  short: '-p'
+  opt :environment,      "environment (production, staging...)", type: String, required: true,  short: '-e'
+  opt :action,           'action (deploy, list...)',             type: String, default: 'list', short: '-a'
+  opt :image,            'image (yourco/project...)',            type: String, short: '-i'
+  opt :tag,              'tag (latest...)',                      type: String, short: '-t'
+  opt :hosts,            'hosts, comma separated',               type: String, short: '-h'
+  opt :docker_path,      'path to docker executable',            type: String, default: 'docker', short: '-d'
+  opt :no_pull,          'Skip the pull_image step',             type: :flag,  default: false, short: '-n'
+  opt :registry_user,    'user for registry auth',               type: String, short: :none
+  opt :registry_password,'password for registry auth',           type: String, short: :none
+  opt :override_env,     'override environment variables, comma separated', type: String
 end
 
 set_current_environment(opts[:environment].to_sym)
@@ -55,6 +56,14 @@ set :image, opts[:image] if opts[:image]
 set :tag,   opts[:tag] if opts[:tag]
 set :hosts, opts[:hosts].split(",") if opts[:hosts]
 
+# Override environment variables when specified
+if opts[:override_env]
+  opts[:override_env].split(',').each do |envvar|
+    key, value = envvar.split('=')
+    env_vars(key => value)
+  end
+end
+
 # Default tag should be "latest"
 set :tag, 'latest' unless any?(:tag)
 set :docker_registry, Centurion::DockerRegistry::OFFICIAL_URL unless any?(:docker_registry)

data/lib/centurion/deploy.rb

@@ -4,6 +4,8 @@ module Centurion; end
 
 module Centurion::Deploy
   FAILED_CONTAINER_VALIDATION = 100
+  INVALID_CGROUP_CPUSHARES_VALUE = 101
+  INVALID_CGROUP_MEMORY_VALUE = 102
 
   def stop_containers(target_server, port_bindings, timeout = 30)
     public_port    = public_port_for(port_bindings)
@@ -71,6 +73,17 @@ module Centurion::Deploy
     false
   end
 
+  def is_a_uint64?(value)
+    result = false
+    if !value.is_a? Integer
+      return result
+    end
+    if value < 0 || value > 0xFFFFFFFFFFFFFFFF
+      return result
+    end
+    return true
+  end
+
   def wait_for_load_balancer_check_interval
     sleep(fetch(:rolling_deploy_check_interval, 5))
   end
@@ -87,13 +100,26 @@ module Centurion::Deploy
     end
   end
 
-  def container_config_for(target_server, image_id, port_bindings=nil, env_vars=nil, volumes=nil, command=nil, hostname=nil)
+  def container_config_for(target_server, image_id, port_bindings=nil, env_vars=nil, volumes=nil, command=nil, memory=nil, cpu_shares=nil)
+
+    if memory && ! is_a_uint64?(memory)
+      error "Invalid value for CGroup memory constraint: #{memory}, value must be a between 0 and 18446744073709551615"
+      exit(INVALID_CGROUP_MEMORY_VALUE)
+    end
+
+    if cpu_shares && ! is_a_uint64?(cpu_shares)
+      error "Invalid value for CGroup CPU constraint: #{cpu_shares}, value must be between 0 and 18446744073709551615"
+      exit(INVALID_CGROUP_CPUSHARES_VALUE)
+    end
+
     container_config = {
       'Image'        => image_id,
       'Hostname'     => fetch(:container_hostname, target_server.hostname),
     }
 
     container_config.merge!('Cmd' => command) if command
+    container_config.merge!('Memory' => memory) if memory
+    container_config.merge!('CpuShares' => cpu_shares) if cpu_shares
 
     if port_bindings
       container_config['ExposedPorts'] ||= {}
@@ -119,8 +145,8 @@ module Centurion::Deploy
     container_config
   end
 
-  def start_new_container(target_server, image_id, port_bindings, volumes, env_vars=nil, command=nil)
-    container_config = container_config_for(target_server, image_id, port_bindings, env_vars, volumes, command)
+  def start_new_container(target_server, image_id, port_bindings, volumes, env_vars=nil, command=nil, memory=nil, cpu_shares=nil)
+    container_config = container_config_for(target_server, image_id, port_bindings, env_vars, volumes, command, memory, cpu_shares)
     start_container_with_config(target_server, volumes, port_bindings, container_config)
   end
 

data/lib/centurion/deploy_dsl.rb

@@ -20,6 +20,14 @@ module Centurion::DeployDSL
     set(:hosts, current)
   end
 
+  def memory(memory)
+    set(:memory, memory)
+  end
+
+  def cpu_shares(cpu_shares)
+    set(:cpu_shares, cpu_shares)
+  end
+
   def command(command)
     set(:command, command)
   end

data/lib/centurion/docker_server.rb

@@ -34,10 +34,18 @@ class Centurion::DockerServer
 
   def find_containers_by_public_port(public_port, type='tcp')
     ps.select do |container|
-      if container['Ports']
-        container['Ports'].find do |port|
-          port['PublicPort'] == public_port.to_i && port['Type'] == type
-        end
+      next unless container && container['Ports']
+      container['Ports'].find do |port|
+        port['PublicPort'] == public_port.to_i && port['Type'] == type
+      end
+    end
+  end
+
+  def find_containers_by_name(wanted_name)
+    ps.select do |container|
+      next unless container && container['Names']
+      container['Names'].find do |name|
+        name =~ /\A\/#{wanted_name}(-[a-f0-9]{7})?\Z/
       end
     end
   end

data/lib/centurion/version.rb

@@ -1,3 +1,3 @@
 module Centurion
-  VERSION = '1.4.2'
+  VERSION = '1.5.0'
 end

data/lib/tasks/deploy.rake

@@ -91,7 +91,9 @@ namespace :deploy do
         fetch(:port_bindings),
         fetch(:binds),
         fetch(:env_vars),
-        fetch(:command)
+        fetch(:command),
+        fetch(:memory),
+        fetch(:cpu_shares)
       )
     end
   end
@@ -118,7 +120,9 @@ namespace :deploy do
         fetch(:port_bindings),
         fetch(:binds),
         fetch(:env_vars),
-        fetch(:command)
+        fetch(:command),
+        fetch(:memory),
+        fetch(:cpu_shares)
       )
 
       skip_ports = Array(fetch(:rolling_deploy_skip_ports, [])).map(&:to_s)

data/spec/deploy_spec.rb

@@ -1,3 +1,4 @@
+require 'spec_helper'
 require 'centurion/deploy'
 require 'centurion/deploy_dsl'
 require 'centurion/logging'
@@ -217,6 +218,36 @@ describe Centurion::Deploy do
         expect(config['Cmd']).to eq(command)
       end
     end
+
+    context 'when cgroup limits are specified' do
+      let(:memory) { 10000000 }
+      let(:cpu_shares) { 1234 }
+
+      before do
+        allow(test_deploy).to receive(:error) # Make sure we don't have red output in tests
+      end
+
+      it 'sets cgroup limits in the config' do
+        config = test_deploy.container_config_for(server, image_id, port_bindings, env, volumes, command, memory, cpu_shares)
+
+        expect(config).to be_a(Hash)
+        expect(config.keys).to match_array(%w{ Hostname Image Memory CpuShares })
+        expect(config['Memory']).to eq(10000000)
+        expect(config['CpuShares']).to eq(1234)
+      end
+
+      it 'throws a fatal error value for Cgroup Memory limit is invalid' do
+        expect { config = test_deploy.container_config_for(server, image_id, port_bindings, env, volumes, command, "I like pie", cpu_shares) }.to terminate.with_code(102)
+        expect { test_deploy.container_config_for(server, image_id, port_bindings, env, volumes, command, -100, cpu_shares) }.to terminate.with_code(102)
+        expect { test_deploy.container_config_for(server, image_id, port_bindings, env, volumes, command, 0xFFFFFFFFFFFFFFFFFF, cpu_shares) }.to terminate.with_code(102)
+      end
+
+      it 'throws a fatal error value for Cgroup CPU limit is invalid' do
+        expect { test_deploy.container_config_for(server, image_id, port_bindings, env, volumes, command, memory, "I like pie") }.to terminate.with_code(101)
+        expect { test_deploy.container_config_for(server, image_id, port_bindings, env, volumes, command, memory, -100) }.to terminate.with_code(101)
+        expect { test_deploy.container_config_for(server, image_id, port_bindings, env, volumes, command, memory, 0xFFFFFFFFFFFFFFFFFF) }.to terminate.with_code(101)
+      end
+    end
   end
 
   describe '#start_container_with_config' do
@@ -359,7 +390,7 @@ describe Centurion::Deploy do
     let(:command)  { ['/bin/echo', 'hi'] }
 
     it 'configures the container' do
-      expect(test_deploy).to receive(:container_config_for).with(server, 'image_id', bindings, nil, {}, nil).once
+      expect(test_deploy).to receive(:container_config_for).with(server, 'image_id', bindings, nil, {}, nil, nil, nil).once
 
       allow(test_deploy).to receive(:start_container_with_config)
 
@@ -417,16 +448,18 @@ describe Centurion::Deploy do
   end
 
   describe '#launch_console' do
-    let(:bindings) { {'80/tcp'=>[{'HostIp'=>'0.0.0.0', 'HostPort'=>'80'}]} }
-    let(:volumes)  { nil }
-    let(:env)      { nil }
-    let(:command)  { nil }
+    let(:bindings)   { {'80/tcp'=>[{'HostIp'=>'0.0.0.0', 'HostPort'=>'80'}]} }
+    let(:volumes)    { nil }
+    let(:env)        { nil }
+    let(:command)    { nil }
+    let(:memory)     { nil }
+    let(:cpu_shares) { nil }
 
     it 'configures the container' do
-      expect(test_deploy).to receive(:container_config_for).with(server, 'image_id', bindings, env, volumes, command).once
+      expect(test_deploy).to receive(:container_config_for).with(server, 'image_id', bindings, env, volumes, command, memory, cpu_shares).once
       allow(test_deploy).to receive(:start_container_with_config)
 
-      test_deploy.start_new_container(server, 'image_id', bindings, volumes, env, command)
+      test_deploy.start_new_container(server, 'image_id', bindings, volumes, env, command, memory, cpu_shares)
     end
 
     it 'augments the container_config' do

data/spec/docker_server_spec.rb

@@ -2,9 +2,21 @@ require 'spec_helper'
 require 'centurion/docker_server'
 
 describe Centurion::DockerServer do
-  let(:host) { 'host1' }
+  let(:host)   { 'host1' }
   let(:docker_path) { 'docker' }
   let(:server) { Centurion::DockerServer.new(host, docker_path) }
+  let(:container) {
+     {
+       'Command' => '/bin/bash',
+       'Created' => 1414797234,
+       'Id'      => '28970c706db0f69716af43527ed926acbd82581e1cef5e4e6ff152fce1b79972',
+       'Image'   => 'centurion-test:latest',
+       'Names'   => ['/centurion-783aac4'],
+       'Ports'   => [{'PrivatePort'=>80, 'Type'=>'tcp', 'IP'=>'0.0.0.0', 'PublicPort'=>23235}],
+       'Status'  => 'Up 3 days'
+     }
+  }
+  let(:ps)     { [ container, {}, nil ] }
 
   it 'knows its hostname' do
     expect(server.hostname).to eq('host1')
@@ -40,4 +52,26 @@ describe Centurion::DockerServer do
     allow(server).to receive(:ps).and_return(image_names.map {|name| { 'Image' => name } })
     expect(server.current_tags_for('target')).to eq(%w[latest production])
   end
+
+  context 'finding containers' do
+    before do
+      allow(server).to receive(:ps).and_return(ps)
+    end
+
+    it 'finds containers by port' do
+      expect(server.find_containers_by_public_port(23235, 'tcp')).to eq([container])
+    end
+
+    it 'only returns correct matches by port' do
+      expect(server.find_containers_by_public_port(1234, 'tcp')).to be_empty
+    end
+
+    it 'finds containers by name' do
+      expect(server.find_containers_by_name('centurion')).to eq([container])
+    end
+
+    it 'only returns correct matches by name' do
+      expect(server.find_containers_by_name('fbomb')).to be_empty
+    end
+  end
 end

data/spec/support/matchers/exit_code_matches.rb

@@ -0,0 +1,38 @@
+# https://gist.github.com/mmasashi/58bd7e2668836a387856
+RSpec::Matchers.define :terminate do |code|
+  actual = nil
+
+  def supports_block_expectations?
+    true
+  end
+
+  match do |block|
+    begin
+      block.call
+    rescue SystemExit => e
+      actual = e.status
+    end
+    actual and actual == status_code
+  end
+
+  chain :with_code do |status_code|
+    @status_code = status_code
+  end
+
+  failure_message do |block|
+    "expected block to call exit(#{status_code}) but exit" +
+      (actual.nil? ? " not called" : "(#{actual}) was called")
+  end
+
+  failure_message_when_negated do |block|
+    "expected block not to call exit(#{status_code})"
+  end
+
+  description do
+    "expect block to call exit(#{status_code})"
+  end
+
+  def status_code
+    @status_code ||= 0
+  end
+end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: centurion
 version: !ruby/object:Gem::Version
-  version: 1.4.2
+  version: 1.5.0
   prerelease: 
 platform: ruby
 authors:
@@ -21,7 +21,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2015-01-26 00:00:00.000000000 Z
+date: 2015-02-24 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: trollop
@@ -208,6 +208,7 @@ files:
 - spec/logging_spec.rb
 - spec/spec_helper.rb
 - spec/support/matchers/capistrano_dsl_matchers.rb
+- spec/support/matchers/exit_code_matches.rb
 homepage: https://github.com/newrelic/centurion
 licenses:
 - MIT
@@ -249,3 +250,4 @@ test_files:
 - spec/logging_spec.rb
 - spec/spec_helper.rb
 - spec/support/matchers/capistrano_dsl_matchers.rb
+- spec/support/matchers/exit_code_matches.rb