broadside 2.0.0 → 3.0.0.pre.prerelease

data/CHANGELOG.md

@@ -1,9 +1,47 @@
+# 3.0.0
+### Breaking Changes
+- `ssh`, `bash`, `logtail`, `status`, and `run` are now top level commands, not subcommands of `deploy`
+- No more `RAKE_DB_MIGRATE` constant
+- Configuration changes:
+  - `config.git_repo=` and `config.type=` were removed.
+  - `config.base` and `config.deploy` are no longer backwards compatible - any options configured at `config.base.something` or `config.deploy.something` must now be configured at `config.something`
+  - `config.ecs.cluster` and `config.ecs.poll_frequency` are now configured at `config.aws.ecs_default_cluster` and `config.aws.ecs_poll_frequency`
+  - `config.docker_image` is now `config.default_docker_image`
+  - `instance` can no longer be configured on a per `Target` basis
+
+#### Added Features
+- Allow configuration of separate `:docker_image` per target
+- Put back ability to configure a default `:tag` per target
+- Add `broadside targets` command to display all the targets' deployed images and CPU/memory allocations
+- `broadside status` has an added `--verbose` switch that displays service and task information
+- [#11](https://github.com/lumoslabs/broadside/issues/11): Add option for ssh proxy user and proxy keyfile
+- [#2](https://github.com/lumoslabs/broadside/issues/2): Add flag for changing loglevel, and add `--debug` switch that enables GLI debug output
+- Failed deploys will rollback the service to the last successfully running scale
+- Allow setting an environment variable `BROADSIDE_SYSTEM_CONFIG_FILE` to be used instead of `~/.broadside/config.rb`
+
+#### General Improvements
+- Only load `env_files` for the selected target (rather than preloading from unrelated targets)
+- Make `env_files` configuration optional
+- `Utils` has been replaced in favor of `LoggingUtils`
+- Exceptions will be raised if a target is configured with an invalid hash key
+- Tasks run have a more relevant `started_by` tag
+- Default log level changed to `INFO`
+- [#21](https://github.com/lumoslabs/broadside/issues/21) Print more useful messages when tasks die without exit codes.
+- `Command` class to encapsulate the running of various commands
+
 # 2.0.0
-- **BREAKING CHANGE** `rake db:migrate` is no longer the default `predeploy_command`
+#### Breaking Changes
+- [#27](https://github.com/lumoslabs/broadside/issues/27) `rake db:migrate` is no longer the default `predeploy_command`
+- Remove ability to configure a default tag for each target
+
+#### Added Features
+- [#38](https://github.com/lumoslabs/broadside/issues/38) ECS cluster can be configured for each target by setting `config.ecs.cluster`
+
+#### General Improvements
+- `base` configuration has been removed - the main `Configuration` object holds all the `base` config. `Broadside.config.base` may be called but will display a deprecation warning.
+- `deploy` configuration has been removed - primarily handled through the main `Configuration` object and in `targets=`. `Broadside.config.deploy` may be called but will display a deprecation warning.
 - `Target` is a first class object
 - `Deploy` is composed of a `Target` plus command line options
-- There is no more `base` configuration - the main `Configuration` object holds all the `base` config.  You can still call `Broadside.config.base` though you will get a deprecation warning.
-- There is no more `deploy` configuration - most of that is handled in the main `Configuration` object and in `targets=`. You can still call `Broadside.config.deploy` though you will get a deprecation warning.
 
 # 1.4.0
 - [#42](https://github.com/lumoslabs/broadside/pull/42/files): Update the task definition when running bootstrap

data/README.md

@@ -1,93 +1,75 @@
 # Broadside [![Build Status](https://travis-ci.org/lumoslabs/broadside.svg?branch=master)](https://travis-ci.org/lumoslabs/broadside)
-A command-line tool for deploying applications on AWS EC2 Container Service (ECS)
-
-This tool is primarily intended for use with ruby applications.
 
+A [GLI](https://github.com/davetron5000/gli) based command-line tool for deploying applications on [AWS EC2 Container Service (ECS)](https://aws.amazon.com/ecs/)
 
 ## Overview
-Amazon ECS presents a low barrier to entry for production-level docker applications. Combined with ECS's built-in blue-green deployment, Elastic Load Balancers, Autoscale Groups, and CloudWatch, one can set up a robust cluster that can scale to serve any number of applications in a short amount of time. Broadside seeks to leverage these benefits and improve the deployment process for developers.
+Amazon ECS presents a low barrier to entry for production-level docker applications. Combined with ECS's built-in blue-green deployment, Elastic Load Balancers, Autoscale Groups, and CloudWatch, one can theoretically set up a robust cluster that can scale to serve any number of applications in a short amount of time. The ECS GUI, CLI, overall architecture are not the easiest to work with, however, so Broadside seeks to leverage the [ECS ruby API](http://docs.aws.amazon.com/sdkforruby/api/Aws/ECS.html) to dramatically simplify and improve the configuration and deployment process for developers, offering a simple command line interface and configuration format that should meet most needs.
+
+Broadside does _not_ attempt to handle operational tasks like infrastructure setup and configuration, which are better suited to tools like [terraform](https://www.terraform.io/).
+
+### Capabilities
 
-Broadside offers a simple command-line interface to perform deployments on ECS. It does not attempt to handle operational tasks like infrastructure setup and configuration, which are better suited for tools like [terraform](https://www.terraform.io/). This allows applications using broadside to employ a clean configuration file that looks something like:
+- **Trigger** ECS deployments
+- **Launch** a bash shell directly onto a running container in the cluster
+- **SSH** directly onto a host running a container
+- **Launch** an ECS task running an arbitrary command
+- **Tail** logs of a running container
+- **Scale** an existing deployment on the fly
+
+### Example Config for Quickstarters
+Applications using broadside employ a configuration file that looks something like:
 
 ```ruby
 Broadside.configure do |config|
   config.application = 'hello_world'
-  config.docker_image = 'lumoslabs/hello_world'
-  config.type = 'ecs'
-  config.ecs.cluster = 'micro-cluster'
-  config.deploy.targets = {
+  config.default_docker_image = 'lumoslabs/hello_world'
+  config.aws.ecs_default_cluster = 'production-cluster'
+  config.aws.region = 'us-east-1'                  # 'us-east-1 is the default
+  config.targets = {
     production_web: {
       scale: 7,
       command: ['bundle', 'exec', 'unicorn', '-c', 'config/unicorn.conf.rb'],
-      env_file: '../.env.production'
+      env_file: '.env.production'
       predeploy_commands: [
-        Broadside::Predeploy::RAKE_DB_MIGRATE,     # RAKE_DB_MIGRATE is just a constant for your convenience
+        ['bundle', 'exec', 'rake', 'db:migrate'],
         ['bundle', 'exec', 'rake', 'data:migrate']
       ]
     },
-    production_worker: {
-      scale: 15,
-      command: ['bundle', 'exec', 'rake', 'resque:work'],
-      env_file: '../.env.production'
-    },
+    # If you have multiple images or clusters, you can configure them per target
     staging_web: {
-      cluster: 'staging-cluster', # Overrides config.ecs.cluster
       scale: 1,
       command: ['bundle', 'exec', 'puma'],
-      env_file: '../.env.staging'
+      env_file: '.env.staging',
+      tag: 'latest',                                # Set a default tag for this target
+      cluster: 'staging-cluster',                   # Overrides config.ecs.cluster
+      docker_image: 'lumoslabs/staging_hello_world' # Overrides config.docker_image
     },
-    staging_worker: {
-      scale: 1,
-      command: ['bundle', 'exec', 'rake', 'resque:work'],
-      env_file: '../.env.staging'
-    },
-    # Example with a task_definition and service configuration which you use to bootstrap a service and
-    # initial task definition.  Accepts all the options AWS does - read their documentation for details:
-    #
-    # Service config: https://docs.aws.amazon.com/sdkforruby/api/Aws/ECS/Client.html#create_service-instance_method
-    # Task Definition Config: https://docs.aws.amazon.com/sdkforruby/api/Aws/ECS/Client.html#register_task_definition-instance_method
-    game_save_as_json_blob_stream: {
+    json_stream: {
       scale: 1,
       command: ['java', '-cp', '*:.', 'path.to.MyClass'],
-      env_file: '.env.production',
-      service_config: {
-        deployment_configuration: {
-          minimum_healthy_percent: 0.5,
-        }
-      },
-      task_definition_config: {
-        container_definitions: [
-          {
-            cpu: 1,
-            memory: 2000,
-          }
-        ]
-      }
+      # This target has a task_definition and service config which you use to bootstrap a new AWS Service
+      service_config: { deployment_configuration: { minimum_healthy_percent: 0.5 } },
+      task_definition_config: { container_definitions: [ { cpu: 1, memory: 2000, } ] }
     }
   }
 end
 ```
 
-From here, developers can use broadside's command-line interface to initiate a basic deployment:
+From here, developers can use broadside's command-line interface to initiate a basic deployment and launch the
+configured `command` as an ECS Service:
 
 ```bash
-broadside deploy short --target production_web --tag $GIT_SHA
+bundle exec broadside deploy full --target production_web --tag v.1.1.example.tag
 ```
-or run
-
-```bash
-broadside deploy full --target production_web --tag $GIT_SHA
-```
-
-which will run the listed `predeploy_commands` listed in the config above prior to the deployment.
 
 In the case of an error or timeout during a deploy, broadside will automatically rollback to the latest stable version. You can perform manual rollbacks as well through the command-line.
 
-See the complete command-line reference in the wiki.
+[For more information on broadside commands, see the complete command-line reference in the wiki](https://github.com/lumoslabs/broadside/wiki/CLI-reference).
 
 
-## Setup
-First, install broadside by adding it to your application gemfile:
+## Installation
+### Via Gemfile
+First, install broadside by adding it to your application `Gemfile`:
 ```ruby
 gem 'broadside'
 ```
@@ -95,23 +77,29 @@ gem 'broadside'
 Then run
 ```bash
 bundle install
-bundle binstubs broadside
 ```
 
-It's recommended that you specify broadside as a development gem so it doesn't inflate your production image.
-
 You can now run the executable in your app directory:
 ```bash
-bin/broadside --help
+bundle exec broadside --help
+```
+
+You may also generate binstubs using
+```bash
+bundle binstubs broadside
 ```
 
-For a full application setup, see the detailed instructions in the wiki.
+### System Wide
+Alternatively, you can install broadside using:
+```
+gem install broadside
+```
 
+## Configuration
+For full application setup, see the [detailed instructions in the wiki](https://github.com/lumoslabs/broadside/wiki).
 
-## Contributing
-Pull requests, bug reports, and feature suggestions are welcome! Before starting on a contribution, I recommend opening an issue or replying to an existing one to give others some initial context on the work needing to be done.
+## Debugging
+Use the `--debug` switch to enable stacktraces and debug output.
 
-1. Create your feature branch (`git checkout -b my-new-feature`)
-2. Commit your changes (`git commit -am 'Add some feature'`)
-3. Push to the branch (`git push origin my-new-feature`)
-4. Create a new Pull Request
+## Contributing
+Pull requests, bug reports, and feature suggestions are welcome! Before starting on a contribution, we recommend opening an issue or replying to an existing one to give others some initial context on the work needing to be done.

data/bin/broadside

@@ -4,187 +4,15 @@ require 'gli'
 require 'broadside'
 
 include GLI::App
-include Broadside::Utils
+include Broadside::LoggingUtils
 
 program_desc 'A command-line tool for deployment and development of docker applications.'
-
 version Broadside::VERSION
 
 subcommand_option_handling :normal
 arguments :strict
 synopsis_format :full
 
-desc 'Configuration file to use.'
-default_value 'config/broadside.conf.rb'
-arg_name 'FILE'
-flag [:c, :config]
-
-def add_shared_deploy_configs(subcmd)
-  # deployment always requires target
-  subcmd.desc 'Deployment target to use, e.g. production_web'
-  subcmd.arg_name 'TARGET'
-  subcmd.flag [:t, :target], type: Symbol
-
-  subcmd.action do |global_options, options, args|
-    _DeployObj = Kernel.const_get("Broadside::#{Broadside.config.type.capitalize}Deploy")
-    _target = Broadside.config.targets.select { |t| t.name == options[:target] }.first
-    raise "Bad target: #{options[:target]}" unless _target
-    _DeployObj.new(_target, options).public_send(subcmd.name)
-  end
-end
-
-# GLI type coercions
-accept Symbol do |val|
-  val.to_sym
-end
-accept Array do |val|
-  val.split(' ')
-end
-accept Fixnum do |val|
-  val.to_i
-end
-
-desc 'Bootstrap your service and task definition from the configured definition.'
-command :bootstrap do |b|
-  b.desc 'Docker tag for application container'
-  b.arg_name 'TAG'
-  b.flag [:tag]
-
-  add_shared_deploy_configs(b)
-end
-
-desc 'Deploy your application.'
-command :deploy do |d|
-  d.desc 'Deploys without running migrations'
-  d.command :short do |subcmd|
-    subcmd.desc 'Docker tag for application container'
-    subcmd.arg_name 'TAG'
-    subcmd.flag [:tag]
-
-    add_shared_deploy_configs(subcmd)
-  end
-
-  d.desc 'Performs a full deployment (with migration)'
-  d.command :full do |subcmd|
-    subcmd.desc 'Docker tag for application container'
-    subcmd.arg_name 'TAG'
-    subcmd.flag [:tag]
-
-    add_shared_deploy_configs(subcmd)
-  end
-
-  d.desc 'Scales application to a given count'
-  d.command :scale do |subcmd|
-    subcmd.desc 'Specify a new scale for application'
-    subcmd.arg_name 'NUM'
-    subcmd.flag [:s, :scale], type: Fixnum
-
-    add_shared_deploy_configs(subcmd)
-  end
-
-  d.desc 'Creates a single instance of the application to run a command.'
-  d.command :run do |subcmd|
-    subcmd.desc 'Docker tag for application container'
-    subcmd.arg_name 'TAG'
-    subcmd.flag [:tag]
-
-    subcmd.desc 'Command to run (wrap argument in quotes)'
-    subcmd.arg_name 'COMMAND'
-    subcmd.flag [:command], type: Array
-
-    add_shared_deploy_configs(subcmd)
-  end
-
-  d.desc 'Rolls back n releases and deploys'
-  d.command :rollback do |subcmd|
-    subcmd.desc 'Number of releases to rollback'
-    subcmd.arg_name 'COUNT'
-    subcmd.flag [:r, :rollback], type: Fixnum
-
-    add_shared_deploy_configs(subcmd)
-  end
-
-  d.desc 'Gets information about what is currently deployed.'
-  d.command :status do |subcmd|
-    add_shared_deploy_configs(subcmd)
-  end
-
-  d.desc 'Tail the logs inside the running container.'
-  d.command :logtail do |subcmd|
-    subcmd.desc '0-based instance index'
-    subcmd.default_value 0
-    subcmd.arg_name 'INSTANCE'
-    subcmd.flag [:n, :instance], type: Fixnum
-
-    subcmd.desc 'Number of lines to tail'
-    subcmd.default_value 10
-    subcmd.arg_name 'TAIL_LINES'
-    subcmd.flag [:l, :lines], type: Fixnum
-
-    add_shared_deploy_configs(subcmd)
-  end
-
-  d.desc 'Establish a secure shell on the instance running the container.'
-  d.command :ssh do |subcmd|
-    subcmd.desc '0-based instance index'
-    subcmd.default_value 0
-    subcmd.arg_name 'INSTANCE'
-    subcmd.flag [:n, :instance], type: Fixnum
-
-    add_shared_deploy_configs(subcmd)
-  end
-
-  d.desc 'Establish a shell inside the running container.'
-  d.command :bash do |subcmd|
-    subcmd.desc '0-based instance index'
-    subcmd.default_value 0
-    subcmd.arg_name 'INSTANCE'
-    subcmd.flag [:n, :instance], type: Fixnum
-
-    add_shared_deploy_configs(subcmd)
-  end
-end
-
-def call_hook(type, command)
-  hook = Broadside.config.send(type)
-
-  if hook.is_a?(Proc)
-    hook_args =
-      if command.parent.is_a?(GLI::Command)
-        {
-          command: command.parent.name,
-          subcommand: command.name
-        }
-      else
-        { command: command.name }
-      end
-    debug "Calling", type, "with args", hook_args
-    hook.call(hook_args)
-  end
-end
-
-pre do |global, command, options, args|
-  Broadside.load_config(global[:config])
-
-  call_hook(:prehook, command)
-  true
-end
-
-post do |global, command, options, args|
-  call_hook(:posthook, command)
-  true
-end
-
-
-on_error do |exception|
-  # false skips default error handling
-  case exception
-  when Broadside::MissingVariableError
-    error exception.message, "Run your last command with --help for more information."
-    false
-  else
-    true
-  end
-end
+commands_from File.expand_path(File.join(File.dirname(__FILE__), '/../lib/broadside/gli'))
 
 exit run(ARGV)

data/broadside.gemspec

@@ -12,18 +12,18 @@ Gem::Specification.new do |spec|
   spec.summary       = 'A command-line tool for EC2 Container Service deployment.'
   spec.homepage      = 'https://github.com/lumoslabs/broadside'
   spec.license       = 'MIT'
-  spec.files         = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec|features)/}) }
-  spec.bindir        = "bin"
+  spec.files         = `git ls-files -z`.split("\x0").reject { |f| f.match(%r{^(test|spec)/}) }
+  spec.bindir        = 'bin'
   spec.executables   = ['broadside']
-  spec.require_paths = ["lib"]
+  spec.require_paths = ['lib']
 
   spec.add_dependency 'activesupport', '>= 3', '< 5'
+  spec.add_dependency 'activemodel', '>= 3', '< 5'
   spec.add_dependency 'aws-sdk', '~> 2.3'
-  spec.add_dependency 'dotenv', '>= 0.9.0'
+  spec.add_dependency 'dotenv', '>= 0.9.0', '< 3.0'
   spec.add_dependency 'gli', '~> 2.13'
-  spec.add_dependency 'rainbow', '~> 2.1'
+  spec.add_dependency 'tty', '~> 0.5'
 
   spec.add_development_dependency 'rspec', '~> 3.4.0'
   spec.add_development_dependency 'bundler', '~> 1.9'
-  spec.add_development_dependency 'fakefs'
 end

data/lib/broadside.rb

@@ -1,42 +1,50 @@
+require 'active_model'
+require 'active_support/core_ext'
+require 'aws-sdk'
+
 require 'broadside/error'
-require 'broadside/utils'
-require 'broadside/configuration/verify_instance_variables'
-require 'broadside/configuration'
-require 'broadside/configuration/aws_config'
-require 'broadside/configuration/ecs_config'
+require 'broadside/logging_utils'
+require 'broadside/configuration/invalid_configuration'
+require 'broadside/configuration/configuration'
+require 'broadside/configuration/aws_configuration'
+require 'broadside/command'
 require 'broadside/target'
 require 'broadside/deploy'
-require 'broadside/predeploy_commands'
 require 'broadside/ecs/ecs_deploy'
 require 'broadside/ecs/ecs_manager'
 require 'broadside/version'
 
 module Broadside
-  extend Utils
+  extend LoggingUtils
 
-  SYSTEM_CONFIG_FILE = "#{Dir.home}/.broadside/config.rb"
+  USER_CONFIG_FILE = (ENV['BROADSIDE_SYSTEM_CONFIG_FILE'] || File.join(Dir.home, '.broadside', 'config.rb')).freeze
 
   def self.configure
     yield config
+    raise ConfigurationError, config.errors.full_messages unless config.valid?
   end
 
-  def self.load_config(config_file)
-    begin
-      load SYSTEM_CONFIG_FILE if File.exists?(SYSTEM_CONFIG_FILE)
-    rescue LoadError => e
-      error "Encountered an error loading system configuration file '#{SYSTEM_CONFIG_FILE}' !"
-      raise e
-    end
+  def self.load_config_file(config_file)
+    raise ArgumentError, "#{config_file} does not exist" unless File.exist?(config_file)
+    config.config_file = config_file
 
     begin
-      config.file = config_file
-      load config_file
-    rescue LoadError => e
-      error "Encountered an error loading required configuration file '#{config_file}' !"
-      raise e
+      if File.exist?(USER_CONFIG_FILE)
+        debug "Loading user configuration from #{USER_CONFIG_FILE}"
+
+        begin
+          load(USER_CONFIG_FILE)
+        rescue ConfigurationError
+          # Suppress the exception because the system config file can be incomplete and validation failure is expected
+        end
+      end
+
+      debug "Loading application configuration from #{config_file}"
+      load(config_file)
+    rescue LoadError
+      error 'Encountered an error loading broadside configuration'
+      raise
     end
-
-    config.verify
   end
 
   def self.config
@@ -45,5 +53,7 @@ module Broadside
 
   def self.reset!
     @config = nil
+    EcsManager.instance_variable_set(:@ecs_client, nil)
+    EcsManager.instance_variable_set(:@ec2_client, nil)
   end
 end