ufo 1.2.0 → 1.5.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: ff5b6f6bdcbb6495ea2cf2531698fe558246696e
-  data.tar.gz: 0a206451dca821129ae71f77e14375c61e386c66
+  metadata.gz: 88ffa32343a3f2d384a43df599aab3521141bd03
+  data.tar.gz: d305e9b86ea4dbb552014b1c1ea4ee67d79dd4f6
 SHA512:
-  metadata.gz: cea5a0c53eaf714b1185b529550aa24e51e2f0e6b0320d1879a4c3c5342362db60c7b0b26fdcd5bd61b17516764b12af2cf5acee96ed7ee8d787336b59169bcb
-  data.tar.gz: 2041c3b1723d6fd113094a258f5adac6d5b5f8128a7ba1733cf7b26109b0e8130f59a5e9b0a0775990df26d144d44bd9f1839a015226cc494983ff1cd0959279
+  metadata.gz: 200363f4ff73473eb9ca774ded09805f459af00d01130a9a4e7f9dee41238b5d0522be360908ba63f56b480f33f7e171f6fd7f3267e49af00c529fc43916911f
+  data.tar.gz: 0c1c498e29e3669907137abc93dbb9aa5b914a764ef506ec310e395124950171d8c78b39dbe7dfdaae74e93df743d99dcc0e5e063b4925b5fb5d5a4675134fcc

data/CHANGELOG.md

@@ -3,6 +3,12 @@
 All notable changes to this project will be documented in this file.
 This project *tries* to adhere to [Semantic Versioning](http://semver.org/), even before v1.0.
 
+## [1.5.0]
+* add ufo ships command
+* refactor code into modules: ecr, docker, tasks
+* improve error message when task_definitions.rb evaluation errors
+* rename --force option to --sure
+
 ## [1.2.0]
 
 * allow -h, --help, help options at the end of the command

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    ufo (1.1.0)
+    ufo (1.2.0)
       aws-sdk
       colorize
       deep_merge
@@ -20,13 +20,13 @@ GEM
       minitest (~> 5.1)
       thread_safe (~> 0.3, >= 0.3.4)
       tzinfo (~> 1.1)
-    aws-sdk (2.9.25)
-      aws-sdk-resources (= 2.9.25)
-    aws-sdk-core (2.9.25)
+    aws-sdk (2.9.28)
+      aws-sdk-resources (= 2.9.28)
+    aws-sdk-core (2.9.28)
       aws-sigv4 (~> 1.0)
       jmespath (~> 1.0)
-    aws-sdk-resources (2.9.25)
-      aws-sdk-core (= 2.9.25)
+    aws-sdk-resources (2.9.28)
+      aws-sdk-core (= 2.9.28)
     aws-sigv4 (1.0.0)
     builder (3.2.3)
     byebug (9.0.6)
@@ -37,7 +37,7 @@ GEM
     diff-lcs (1.3)
     docile (1.1.5)
     hashie (3.5.5)
-    i18n (0.8.3)
+    i18n (0.8.4)
     jmespath (1.3.1)
     json (2.1.0)
     minitest (5.10.2)

data/README.md

@@ -158,7 +158,7 @@ task_definition "hi-web-prod" do
     family: task_definition_name,
     # image: tongueroo/hi:ufo-[timestamp]=[sha]
     image: helper.full_image_name,
-    environment: env_file('.env.prod')
+    environment: helper.env_file('.env.prod')
     name: "web",
     container_port: helper.dockerfile_port,
     command: ["bin/web"]
@@ -170,8 +170,8 @@ The task\_definitions.rb file has some special variables and helper methods avai
 
 * **helper.full\_image\_name** — The full docker image name that ufo builds. The “base” portion of the docker image name is defined in ufo/settings.yml. For example, the base portion is `tongueroo/hi` and the full image name is `tongueroo/hi:ufo-[timestamp]-[sha]`. The base name does not include the generated Docker tag, which contains a timestamp and git sha of the Dockerfile that is used.
 * **helper.dockerfile\_port** — Exposed port extracted from the Dockerfile of the project. 
-* **env_vars(text)** — This method takes a block of text that contains the env values in key=value format and converts that block of text to the proper task definition json format.
-* **env_file(path)** — This method takes an `.env` file which contains a simple key value list of environment variables and converts the list to the proper task definition json format.
+* **helper.env_vars(text)** — This method takes a block of text that contains the env values in key=value format and converts that block of text to the proper task definition json format.
+* **helper.env_file(path)** — This method takes an `.env` file which contains a simple key value list of environment variables and converts the list to the proper task definition json format.
 
 The 2 classes which provide these special helper methods are in [ufo/dsl.rb](https://github.com/tongueroo/ufo/blob/master/lib/ufo/dsl.rb) and [ufo/dsl/helper.rb](https://github.com/tongueroo/ufo/blob/master/lib/ufo/dsl/helper.rb). Refer to these classes for the full list of the special variables and methods.
 
@@ -184,9 +184,9 @@ This is easily accomplished with the `bin/deploy` wrapper script that the `ufo i
 ```bash
 #!/bin/bash -xe
 
-ufo ship hi-worker-prod --cluster stag --no-wait
-ufo ship hi-clock-prod --cluster stag --no-wait --no-docker
-ufo ship hi-web-prod --cluster stag --no-docker
+ufo ship hi-worker-prod --cluster prod --no-wait
+ufo ship hi-clock-prod --cluster prod --no-wait --no-docker
+ufo ship hi-web-prod --cluster prod --no-docker
 ```
 
 The first `ufo ship hi-worker-prod` command build and ships docker image to ECS, but the following two `ufo ship` commands use the `--no-docker` flag to skip the `docker build` step.  `ufo ship` will use the last built docker image as the image to be shipped.  For those curious, this is stored in `ufo/docker_image_name_ufo.txt`.
@@ -196,7 +196,7 @@ The first `ufo ship hi-worker-prod` command build and ships docker image to ECS,
 Ufo assumes a convention that service\_name and the task\_name are the same.  If you would like to override this convention then you can specify the task name.
 
 ```
-ufo ship hi-web-prod--task my-task
+ufo ship hi-web-prod --task my-task
 ```
 
 This means that in the task_definition.rb you will also defined it with `my-task`.  For example:
@@ -233,7 +233,7 @@ ufo tasks register # will register all genreated task definitinos in the ufo/out
 Skips all the build docker phases of a deploy sequence and only update the service with the task definitions.
 
 ```bash
-ufo ship hi-web-prod--no-docker
+ufo ship hi-web-prod --no-docker
 ```
 Note if you use the `--no-docker` option you should ensure that you have already push a docker image to your docker register.  Or else the task will not be able to spin up because the docker image does not exist.  I recommend that you normally use `ufo ship` most of the time.
 

data/bin/ufo

@@ -1,5 +1,12 @@
 #!/usr/bin/env ruby
 
+# Trap ^C
+Signal.trap("INT") {
+  puts "\nCtrl-C detected. Exiting..."
+  sleep 1
+  exit
+}
+
 require 'rubygems'
 
 $:.unshift(File.expand_path('../../lib', __FILE__))

data/lib/ufo.rb

@@ -1,28 +1,24 @@
-$:.unshift(File.expand_path("../", __FILE__))
-require "ufo/version"
-require "pp"
+$:.unshift(File.expand_path('../', __FILE__))
+require 'ufo/version'
 require 'deep_merge'
-require "colorize"
+require 'colorize'
+require 'fileutils'
 
 module Ufo
+  autoload :Defaults, 'ufo/defaults'
+  autoload :AwsServices, 'ufo/aws_services'
   autoload :Command, 'ufo/command'
   autoload :Settings, 'ufo/settings'
-  autoload :PrettyTime, 'ufo/pretty_time'
-  autoload :Execute, 'ufo/execute'
+  autoload :Util, 'ufo/util'
   autoload :Init, 'ufo/init'
-  autoload :EcrAuth, 'ufo/ecr_auth'
   autoload :CLI, 'ufo/cli'
-  autoload :DockerBuilder, 'ufo/docker_builder'
-  autoload :DockerfileUpdater, 'ufo/dockerfile_updater'
-  autoload :DockerCleaner, 'ufo/docker_cleaner'
-  autoload :TasksBuilder, 'ufo/tasks_builder'
-  autoload :TasksRegister, 'ufo/tasks_register'
   autoload :Ship, 'ufo/ship'
   autoload :Task, 'ufo/task'
-  autoload :EcrCleaner, 'ufo/ecr_cleaner'
   autoload :Destroy, 'ufo/destroy'
+  autoload :DSL, 'ufo/dsl'
   autoload :Scale, 'ufo/scale'
-  # modules
-  autoload :Defaults, 'ufo/defaults'
-  autoload :AwsServices, 'ufo/aws_services'
+
+  autoload :Docker, 'ufo/docker'
+  autoload :Ecr, 'ufo/ecr'
+  autoload :Tasks, 'ufo/tasks'
 end

data/lib/ufo/cli.rb

@@ -3,63 +3,6 @@ require 'ufo/command'
 require 'ufo/cli/help'
 
 module Ufo
-  class Docker < Command
-    desc "build", "builds docker image"
-    long_desc CLI::Help.docker_build
-    option :push, type: :boolean, default: false
-    def build
-      builder = DockerBuilder.new(options)
-      builder.build
-      builder.push if options[:push]
-    end
-
-    desc "base", "builds docker image from Dockerfile.base and update current Dockerfile"
-    long_desc CLI::Help.docker_base
-    option :push, type: :boolean, default: true
-    def base
-      builder = DockerBuilder.new(options.dup.merge(
-        image_namespace: "base",
-        dockerfile: "Dockerfile.base"
-      ))
-      builder.build
-      builder.push if options[:push]
-      builder.update_dockerfile
-      DockerCleaner.new(builder.image_name, options.merge(tag_prefix: "base")).cleanup
-      EcrCleaner.new(builder.image_name, options.merge(tag_prefix: "base")).cleanup
-    end
-
-    desc "image_name", "displays the full docker image with tag that will be generated"
-    option :generate, type: :boolean, default: false, desc: "Generate a name without storing it"
-    long_desc CLI::Help.docker_full_image_name
-    def image_name
-      full_image_name = DockerBuilder.new(options).full_image_name
-      puts full_image_name
-    end
-
-    desc "cleanup IMAGE_NAME", "Cleans up old images.  Keeps a specified amount."
-    option :keep, type: :numeric, default: 3
-    option :tag_prefix, default: "ufo"
-    long_desc CLI::Help.docker_cleanup
-    def cleanup(image_name)
-      DockerCleaner.new(image_name, options).cleanup
-    end
-  end
-
-  class Tasks < Command
-    desc "build", "builds task definitions"
-    long_desc CLI::Help.tasks_build
-    option :pretty, type: :boolean, default: true, desc: "Pretty format the json for the task definitions"
-    def build
-      TasksBuilder.new(options).build
-    end
-
-    desc "register", "register all built task definitions in ufo/output"
-    long_desc CLI::Help.tasks_register
-    def register
-      TasksRegister.register(:all, options)
-    end
-  end
-
   class CLI < Command
     class_option :verbose, type: :boolean
     class_option :mute, type: :boolean
@@ -67,11 +10,11 @@ module Ufo
     class_option :project_root, type: :string, default: '.'
     class_option :cluster, desc: "Cluster.  Overrides ufo/settings.yml."
 
-    desc "docker ACTION", "docker related tasks"
+    desc "docker [ACTION]", "docker related tasks"
     long_desc Help.docker
     subcommand "docker", Docker
 
-    desc "tasks ACTION", "task definition related tasks"
+    desc "tasks [ACTION]", "task definition related tasks"
     long_desc Help.tasks
     subcommand "tasks", Tasks
 
@@ -84,30 +27,48 @@ module Ufo
       Init.new(options).setup
     end
 
-    desc "ship [SERVICE]", "ships container to the ECS service"
-    option :task, desc: "ECS task name, to override the task name convention."
-    option :target_group, desc: "ELB Target Group ARN."
-    option :elb, desc: "ELB Name associated with the target_group.  Assumes first "
-    option :elb_prompt, type: :boolean, desc: "Enable ELB prompt", default: true
-    option :docker, type: :boolean, desc: "Enable docker build and push", default: true
-    option :wait, type: :boolean, desc: "Wait for deployment to complete", default: false
-    option :pretty, type: :boolean, default: true, desc: "Pretty format the json for the task definitions"
-    option :stop_old_tasks, type: :boolean, default: false, desc: "Stop old tasks after waiting for deploying to complete"
-    option :ecr_keep, type: :numeric, desc: "ECR specific cleanup of old images.  Specifies how many images to keep.  Only runs if the images are ECR images. Defaults to keeping all the images."
+    # common options to ship and ships command
+    ship_options = Proc.new do
+      option :task, desc: "ECS task name, to override the task name convention."
+      option :target_group, desc: "ELB Target Group ARN."
+      option :elb, desc: "ELB Name associated with the target_group.  Assumes first "
+      option :elb_prompt, type: :boolean, desc: "Enable ELB prompt", default: true
+      option :docker, type: :boolean, desc: "Enable docker build and push", default: true
+      option :wait, type: :boolean, desc: "Wait for deployment to complete", default: false
+      option :pretty, type: :boolean, default: true, desc: "Pretty format the json for the task definitions"
+      option :stop_old_tasks, type: :boolean, default: false, desc: "Stop old tasks after waiting for deploying to complete"
+      option :ecr_keep, type: :numeric, desc: "ECR specific cleanup of old images.  Specifies how many images to keep.  Only runs if the images are ECR images. Defaults to keeping all the images."
+    end
+
+    desc "ship [SERVICE]", "builds and ships container image to the ECS service"
     long_desc Help.ship
+    ship_options.call
     def ship(service)
-      builder = build_docker(options)
-      task_definition = options[:task] || service # convention
-      register_task(task_definition, options)
-      return if ENV['TEST'] # allows quick testing of the ship CLI portion only
+      builder = build_docker
 
+      task_definition = options[:task] || service # convention
+      Tasks::Builder.register(task_definition, options)
       ship = Ship.new(service, task_definition, options)
       ship.deploy
-      if options[:docker]
-        DockerCleaner.new(builder.image_name, options).cleanup
-        EcrCleaner.new(builder.image_name, options).cleanup
+
+      cleanup(builder.image_name)
+    end
+
+    desc "ships [LIST-OF-SERVICES]", "builds and ships same container image to multiple ECS services"
+    long_desc Help.ships
+    ship_options.call
+    def ships(*services)
+      builder = build_docker
+
+      services.each_with_index do |service|
+        service_name, task_defintion_name = service.split(':')
+        task_definition = task_defintion_name || service_name # convention
+        Tasks::Builder.register(task_definition, options)
+        ship = Ship.new(service, task_definition, options)
+        ship.deploy
       end
-      puts "Docker image shipped: #{builder.full_image_name.green}"
+
+      cleanup(builder.image_name)
     end
 
     desc "task [TASK_DEFINITION]", "runs a one time task"
@@ -115,14 +76,14 @@ module Ufo
     option :docker, type: :boolean, desc: "Enable docker build and push", default: true
     option :command, type: :array, desc: "Override the command used for the container"
     def task(task_definition)
-      build_docker(options)
-      register_task(task_definition, options)
+      Docker::Builder.build(options)
+      Tasks::Builder.register(task_definition, options)
       Task.new(task_definition, options).run
     end
 
     desc "destroy [SERVICE]", "destroys the ECS service"
     long_desc Help.destroy
-    option :force, type: :boolean, desc: "By pass are you sure prompt."
+    option :sure, type: :boolean, desc: "By pass are you sure prompt."
     def destroy(service)
       task_definition = options[:task] || service # convention
       Destroy.new(service, options).bye
@@ -140,8 +101,8 @@ module Ufo
     end
 
     no_tasks do
-      def build_docker(options)
-        builder = DockerBuilder.new(options) # outside if because it need docker.full_image_name
+      def build_docker
+        builder = Docker::Builder.new(options)
         if options[:docker]
           builder.build
           builder.push
@@ -149,14 +110,11 @@ module Ufo
         builder
       end
 
-      def register_task(task_definition, options)
-        # task definition and deploy logic are coupled in the Ship class.
-        # Example: We need to know if the task defintion is a web service to see if we need to
-        # add the elb target group.  The web service information is in the TasksBuilder
-        # and the elb target group gets set in the Ship class.
-        # So we always call these together.
-        TasksBuilder.new(options).build
-        TasksRegister.register(task_definition, options)
+      def cleanup(image_name)
+        return unless options[:docker]
+
+        Docker::Cleaner.new(image_name, options).cleanup
+        Ecr::Cleaner.new(image_name, options).cleanup
       end
     end
   end

data/lib/ufo/cli/help.rb

@@ -22,66 +22,6 @@ $ ufo docker tag
 EOL
         end
 
-        def docker_base
-<<-EOL
-
-The docker cache task builds a docker image using the Dockerfile.base file and
-updates the FROM Dockerfile image with the generated image from Dockerfile.base.
-
-Examples:
-
-$ ufo docker base
-
-$ ufo docker base --no-push # do not push the image to the registry
-
-Docker image tongueroo/hi:base-2016-10-21T15-50-57-88071f5 built.
-EOL
-        end
-
-        def docker_build
-<<-EOL
-Examples:
-
-$ ufo docker build
-
-$ ufo docker build --push # also pushes the image to the docker registry
-
-Docker image tongueroo/hi:ufo-2016-10-21T15-50-57-88071f5 built.
-EOL
-        end
-
-        def docker_full_image_name
-<<-EOL
-Examples:
-
-$ ufo docker full_image_name
-
-Docker image name that will be used: tongueroo/hi:ufo-2016-10-15T19-29-06-88071f5
-EOL
-        end
-
-        def docker_cleanup
-<<-EOL
-Examples:
-
-Say you currently have these images:
-
-* tongueroo/hi:ufo-2016-10-15T19-29-06-88071f5
-
-* tongueroo/hi:ufo-2016-10-16T19-29-06-88071f5
-
-* tongueroo/hi:ufo-2016-10-17T19-29-06-88071f5
-
-* tongueroo/hi:ufo-2016-10-18T19-29-06-88071f5
-
-To clean them up and keep the 3 more recent:
-
-$ ufo docker cleanup tongueroo/hi
-
-This will remove tongueroo/hi:ufo-2016-10-15T19-29-06-88071f5.
-EOL
-        end
-
         def tasks
 <<-EOL
 Examples:
@@ -94,46 +34,49 @@ Note all the existing ufo/output generated task defintions are wiped out.
 EOL
         end
 
-        def tasks_build
+        def ship
 <<-EOL
 Examples:
 
-$ ufo tasks build
+To build the docker image, generate the task definition and ship it, run:
 
-Builds all the task defintiions.
+$ ufo ship hi-web-prod
 
-Note all the existing ufo/output generated task defintions are wiped out.
-EOL
-        end
+By convention the task and service names match. If you need override to this convention then you can specific the task.  For example if you want to ship to the `hi-web-prod-1` service and use the `hi-web-prod` task, run:
 
-        def tasks_register
-<<-EOL
-Examples:
+$ ufo ship hi-web-prod-1 --task hi-web-prod
 
-$ ufo tasks register
-All the task defintiions in ufo/output registered.
+The deploy will also created the ECS service if the service does not yet exist on the cluster.  The deploy will prompt you for the ELB `--target-group` if you are shipping a web container that does not yet exist.  If it is not a web container the `--target-group` option gets ignored.
+
+The prommpt can be bypassed by specifying a valid `--target-group` option or specifying the `---no-elb-prompt` option.
+
+$ ufo ship hi-web-prod --target-group arn:aws:elasticloadbalancing:us-east-1:123456789:targetgroup/hi-web-prod/jsdlfjsdkd
+
+$ ufo ship hi-web-prod --no-elb-prompt
 EOL
         end
 
-        def ship
+        def ships
 <<-EOL
+Builds docker image, registers it and ships it to multiple services.  This deploys the same docker image to multiple ECS services.
+
 Examples:
 
-To build the docker image, generate the task definition and ship it, run:
+$ ufo ships hi-web-prod hi-clock-prod hi-worker-prod
 
-$ ufo ship hi-web-prod
+By convention the task definition and service names match for each of the services you ship. If you need to override to this convention then you can specify the task definition for each service with a special syntax.  In the special syntax the service and task definition is separated by a colon.  Example:
 
-By convention the task and service names match. If you need override to this convention then you can specific the task.  For example if you want to ship to the `hi-web-prod-1` service and use the `hi-web-prod` task, run:
+$ ufo ships hi-web-prod-1:hi-web-prod hi-clock-prod-1 hi-worker-prod-1
 
-$ ufo ship hi-web-prod-1 --task hi-web-prod
+Here ufo will deploy to the hi-web-prod-1 ECS Service using the hi-web-prod task definition, but use the convention for the rest of the service.
 
-The deploy will also created the ECS service if the service does not yet exist on the cluster.  The deploy will prompt you for the ELB `--target-group` if you are shipping a web container that does not yet exist.  If it is not a web container the `--target-group` option gets ignored.
+For each service being deployed to, ufo will create the ECS service if the service does not yet exist on the cluster.  The deploy process will prompt you for the ELB `--target-group` if you are deploying to a 'web' service that does not yet exist.  Ufo determines that it is a web service by the name of the service. If the service has 'web' in the name then it is considered a web service. If it is not a web service then the `--target-group` option gets ignored.
 
-The prommpt can be bypassed by specifying a valid `--target-group` option or specifying the `---no-elb-prompt` option.
+The prommpt can be bypassed by specifying a valid `--target-group` option or specifying the `---no-elb-prompt` option.  Examples:
 
-$ ufo ship hi-web-prod --target-group arn:aws:elasticloadbalancing:us-east-1:123456789:targetgroup/hi-web-prod/jsdlfjsdkd
+$ ufo ships hi-web-prod hi-clock-prod hi-worker-prod --target-group arn:aws:elasticloadbalancing:us-east-1:123456789:targetgroup/hi-web-prod/jsdlfjsdkd
 
-$ ufo ship hi-web-prod --no-elb-prompt
+$ ufo ships hi-web-prod hi-clock-prod hi-worker-prod --no-elb-prompt
 EOL
         end
 

data/lib/ufo/destroy.rb

@@ -50,7 +50,7 @@ module Ufo
     end
 
     def are_you_sure?
-      return true if @options[:force]
+      return true if @options[:sure]
       puts "You are about to destroy #{@service} service on #{@cluster} cluster."
       print "Are you sure you want to do this? (y/n) "
       answer = $stdin.gets.strip

data/lib/ufo/docker.rb

@@ -0,0 +1,48 @@
+module Ufo
+  class Docker < Command
+    autoload :Help, 'ufo/docker/help'
+    autoload :Builder, 'ufo/docker/builder'
+    autoload :Dockerfile, 'ufo/docker/dockerfile'
+    autoload :Cleaner, 'ufo/docker/cleaner'
+
+    desc "build", "builds docker image"
+    long_desc Help.build
+    option :push, type: :boolean, default: false
+    def build
+      builder = Docker::Builder.new(options)
+      builder.build
+      builder.push if options[:push]
+    end
+
+    desc "base", "builds docker image from Dockerfile.base and update current Dockerfile"
+    long_desc Help.base
+    option :push, type: :boolean, default: true
+    def base
+      builder = Docker::Builder.new(options.dup.merge(
+        image_namespace: "base",
+        dockerfile: "Dockerfile.base"
+      ))
+      builder.build
+      builder.push if options[:push]
+      builder.update_dockerfile
+      Docker::Cleaner.new(builder.image_name, options.merge(tag_prefix: "base")).cleanup
+      Ecr::Cleaner.new(builder.image_name, options.merge(tag_prefix: "base")).cleanup
+    end
+
+    desc "image_name", "displays the full docker image with tag that will be generated"
+    option :generate, type: :boolean, default: false, desc: "Generate a name without storing it"
+    long_desc Help.full_image_name
+    def image_name
+      full_image_name = Docker::Builder.new(options).full_image_name
+      puts full_image_name
+    end
+
+    desc "cleanup IMAGE_NAME", "Cleans up old images.  Keeps a specified amount."
+    option :keep, type: :numeric, default: 3
+    option :tag_prefix, default: "ufo"
+    long_desc Help.cleanup
+    def cleanup(image_name)
+      Docker::Cleaner.new(image_name, options).cleanup
+    end
+  end
+end

data/lib/ufo/{docker_builder.rb → docker/builder.rb}

@@ -1,7 +1,15 @@
 module Ufo
-  class DockerBuilder
-    include PrettyTime
-    include Execute
+  class Docker::Builder
+    include Util
+
+    def self.build(options)
+      builder = Docker::Builder.new(options) # outside if because it need builder.full_image_name
+      if options[:docker]
+        builder.build
+        builder.push
+      end
+      builder
+    end
 
     def initialize(options={})
       @options = options
@@ -54,7 +62,7 @@ module Ufo
     def update_auth_token
       return unless ecr_image?
       repo_domain = "https://#{image_name.split('/').first}"
-      auth = EcrAuth.new(repo_domain)
+      auth = Ecr::Auth.new(repo_domain)
       auth.update
     end
 
@@ -117,8 +125,8 @@ module Ufo
     end
 
     def update_dockerfile
-      updater = DockerfileUpdater.new(full_image_name, @options)
-      updater.update
+      dockerfile = Docker::Dockerfile.new(full_image_name, @options)
+      dockerfile.update
     end
 
     def say(msg)

data/lib/ufo/{docker_cleaner.rb → docker/cleaner.rb}

@@ -1,6 +1,6 @@
 module Ufo
-  class DockerCleaner
-    include Execute
+  class Docker::Cleaner
+    include Util
 
     def initialize(docker_image_name, options)
       # docker_image_name does not containg the tag

data/lib/ufo/{dockerfile_updater.rb → docker/dockerfile.rb}

@@ -1,5 +1,5 @@
 module Ufo
-  class DockerfileUpdater
+  class Docker::Dockerfile
     def initialize(full_image_name, options={})
       @full_image_name = full_image_name
       @options = options

data/lib/ufo/docker/help.rb

@@ -0,0 +1,65 @@
+module Ufo
+  module Docker::Help
+    def base
+<<-EOL
+
+The docker cache task builds a docker image using the Dockerfile.base file and
+updates the FROM Dockerfile image with the generated image from Dockerfile.base.
+
+Examples:
+
+$ ufo docker base
+
+$ ufo docker base --no-push # do not push the image to the registry
+
+Docker image tongueroo/hi:base-2016-10-21T15-50-57-88071f5 built.
+EOL
+    end
+
+    def build
+<<-EOL
+Examples:
+
+$ ufo docker build
+
+$ ufo docker build --push # also pushes the image to the docker registry
+
+Docker image tongueroo/hi:ufo-2016-10-21T15-50-57-88071f5 built.
+EOL
+    end
+
+    def full_image_name
+<<-EOL
+Examples:
+
+$ ufo docker full_image_name
+
+Docker image name that will be used: tongueroo/hi:ufo-2016-10-15T19-29-06-88071f5
+EOL
+    end
+
+    def cleanup
+<<-EOL
+Examples:
+
+Say you currently have these images:
+
+* tongueroo/hi:ufo-2016-10-15T19-29-06-88071f5
+
+* tongueroo/hi:ufo-2016-10-16T19-29-06-88071f5
+
+* tongueroo/hi:ufo-2016-10-17T19-29-06-88071f5
+
+* tongueroo/hi:ufo-2016-10-18T19-29-06-88071f5
+
+To clean them up and keep the 3 more recent:
+
+$ ufo docker cleanup tongueroo/hi
+
+This will remove tongueroo/hi:ufo-2016-10-15T19-29-06-88071f5.
+EOL
+    end
+
+    extend self
+  end
+end

data/lib/ufo/dsl.rb

@@ -25,7 +25,37 @@ module Ufo
     # of this class.
     def evaluate_template_definitions
       source_code = IO.read(@template_definitions_path)
-      instance_eval(source_code, @template_definitions_path)
+      begin
+        instance_eval(source_code, @template_definitions_path)
+      rescue Exception => e
+        task_definition_error(e)
+        puts "\nFull error:"
+        raise
+      end
+    end
+
+    # Prints out a user friendly task_definition error message
+    def task_definition_error(e)
+      error_info = e.backtrace.first
+      path, line_no, _ = error_info.split(':')
+      line_no = line_no.to_i
+      puts "Error evaluating #{path}:".colorize(:red)
+      puts e.message
+      puts "Here's the line in #{path} with the error:\n\n"
+
+      contents = IO.read(path)
+      content_lines = contents.split("\n")
+      context = 5 # lines of context
+      top, bottom = [line_no-context-1, 0].max, line_no+context-1
+      spacing = content_lines.size.to_s.size
+      content_lines[top..bottom].each_with_index do |line_content, index|
+        line_number = top+index+1
+        if line_number == line_no
+          printf("%#{spacing}d %s\n".colorize(:red), line_number, line_content)
+        else
+          printf("%#{spacing}d %s\n", line_number, line_content)
+        end
+      end
     end
 
     def build_task_definitions

data/lib/ufo/dsl/helper.rb

@@ -23,7 +23,7 @@ module Ufo
       end
 
       def full_image_name
-        DockerBuilder.new(@options).full_image_name
+        Docker::Builder.new(@options).full_image_name
       end
 
       #############

data/lib/ufo/ecr.rb

@@ -0,0 +1,6 @@
+module Ufo
+  module Ecr
+    autoload :Auth, 'ufo/ecr/auth'
+    autoload :Cleaner, 'ufo/ecr/cleaner'
+  end
+end

data/lib/ufo/{ecr_auth.rb → ecr/auth.rb}

@@ -1,5 +1,5 @@
 module Ufo
-  class EcrAuth
+  class Ecr::Auth
     include AwsServices
 
     def initialize(repo_domain)

data/lib/ufo/{ecr_cleaner.rb → ecr/cleaner.rb}

@@ -4,7 +4,7 @@ require "json"
 #
 #   ufo ship app-web --cluster prod --noop
 module Ufo
-  class EcrCleaner
+  class Ecr::Cleaner
     include AwsServices
     include Defaults
 
@@ -50,7 +50,7 @@ module Ufo
 
     def update_auth_token
       repo_domain = "https://#{@docker_image_name.split('/').first}"
-      auth = EcrAuth.new(repo_domain)
+      auth = Ecr::Auth.new(repo_domain)
       auth.update
     end
 

data/lib/ufo/init.rb

@@ -21,7 +21,7 @@ module Ufo
         dest = src.gsub(%r{.*starter_project/},'')
         dest = "#{@project_root}/#{dest}"
 
-        if File.exist?(dest) and !@options[:force]
+        if File.exist?(dest) and !@options[:sure]
           puts "exists: #{dest}".yellow unless @options[:quiet]
         else
           dirname = File.dirname(dest)

data/lib/ufo/ship.rb

@@ -24,9 +24,8 @@ module Ufo
   class Ship
     include Defaults
     include AwsServices
-    include PrettyTime
+    include Util
 
-    # service can be a pattern
     def initialize(service, task_definition, options={})
       @service = service
       @task_definition = task_definition
@@ -51,7 +50,15 @@ module Ufo
     # Example:
     #   No way to map: hi-.*-prod -> hi-web-prod hi-worker-prod hi-clock-prod
     def deploy
-      puts "Shipping #{@service}...".green unless @options[:mute]
+      message = "Shipping #{@service}..."
+      unless @options[:mute]
+        if @options[:noop]
+          puts "NOOP: #{message}"
+          return
+        else
+          puts message.green
+        end
+      end
 
       ensure_cluster_exist
       process_single_service
@@ -333,8 +340,8 @@ module Ufo
       task_definition_path = "ufo/output/#{task_definition}.json"
       task_definition_full_path = "#{@project_root}/#{task_definition_path}"
       unless File.exist?(task_definition_full_path)
-        puts "ERROR: Unable to find the task definition at #{task_definition_path}."
-        puts "Are you sure you have defined it in ufo/template_definitions.rb?"
+        puts "ERROR: Unable to find the task definition at #{task_definition_path}.".colorize(:red)
+        puts "Are you sure you have defined it in ufo/template_definitions.rb?".colorize(:red)
         exit
       end
       task_definition = JSON.load(IO.read(task_definition_full_path))

data/lib/ufo/tasks.rb

@@ -0,0 +1,20 @@
+module Ufo
+  class Tasks < Command
+    autoload :Help, 'ufo/tasks/help'
+    autoload :Builder, 'ufo/tasks/builder'
+    autoload :Register, 'ufo/tasks/register'
+
+    desc "build", "builds task definitions"
+    long_desc Help.build
+    option :pretty, type: :boolean, default: true, desc: "Pretty format the json for the task definitions"
+    def build
+      Tasks::Builder.new(options).build
+    end
+
+    desc "register", "register all built task definitions in ufo/output"
+    long_desc Help.register
+    def register
+      Tasks::Register.register(:all, options)
+    end
+  end
+end

data/lib/ufo/{tasks_builder.rb → tasks/builder.rb}

@@ -1,7 +1,16 @@
 module Ufo
-  autoload :DSL, 'ufo/dsl'
+  class Tasks::Builder
+    # build and registers together
+    def self.register(task_definition, options)
+      # task definition and deploy logic are coupled in the Ship class.
+      # Example: We need to know if the task defintion is a web service to see if we need to
+      # add the elb target group.  The web service information is in the Tasks::Builder
+      # and the elb target group gets set in the Ship class.
+      # So we always call these together.
+      Tasks::Builder.new(options).build
+      Tasks::Register.register(task_definition, options)
+    end
 
-  class TasksBuilder
     def initialize(options={})
       @options = options
       @project_root = options[:project_root] || '.'

data/lib/ufo/tasks/help.rb

@@ -0,0 +1,26 @@
+module Ufo
+  module Tasks::Help
+    def build
+<<-EOL
+Examples:
+
+$ ufo tasks build
+
+Builds all the task defintiions.
+
+Note all the existing ufo/output generated task defintions are wiped out.
+EOL
+    end
+
+    def register
+<<-EOL
+Examples:
+
+$ ufo tasks register
+All the task defintiions in ufo/output registered.
+EOL
+    end
+
+    extend self
+  end
+end

data/lib/ufo/{tasks_register.rb → tasks/register.rb}

@@ -2,14 +2,14 @@ require 'plissken' # Hash#to_snake_keys
 require 'json'
 
 module Ufo
-  class TasksRegister
+  class Tasks::Register
     include AwsServices
 
     def self.register(task_name, options={})
       project_root = options[:project_root] || '.'
       Dir.glob("#{project_root}/ufo/output/*").each do |path|
         if task_name == :all or path.include?(task_name)
-          task_register = TasksRegister.new(path, options)
+          task_register = Tasks::Register.new(path, options)
           task_register.register
         end
       end

data/lib/ufo/{execute.rb → util.rb}

@@ -1,5 +1,5 @@
 module Ufo
-  module Execute
+  module Util
     def execute(command, local_options={})
       command = "cd #{@project_root} && #{command}"
       # local_options[:live] overrides the global @options[:noop]
@@ -15,5 +15,16 @@ module Ufo
       end
       result
     end
+
+    # http://stackoverflow.com/questions/4175733/convert-duration-to-hoursminutesseconds-or-similar-in-rails-3-or-ruby
+    def pretty_time(total_seconds)
+      minutes = (total_seconds / 60) % 60
+      seconds = total_seconds % 60
+      if total_seconds < 60
+        "#{seconds.to_i}s"
+      else
+        "#{minutes.to_i}m #{seconds.to_i}s"
+      end
+    end
   end
 end

data/lib/ufo/version.rb

@@ -1,3 +1,3 @@
 module Ufo
-  VERSION = "1.2.0"
+  VERSION = "1.5.0"
 end

data/spec/fixtures/home_existing/.aws/config

@@ -0,0 +1,3 @@
+[default]
+region = us-west-2
+

data/spec/lib/cli_spec.rb

@@ -47,6 +47,16 @@ describe Ufo::CLI do
       end
     end
 
+    context "ships" do
+      it "deploys software to multiple services" do
+        out = execute("bin/ufo ships hi-web-prod hi-worker-prod #{@args} --no-wait")
+        # cannot look for Software shipped! because
+        #   ship.deploy unless ENV['TEST'] # to allow me to quickly test CLI portion only
+        # just testing the CLI portion.  The ship class itself is tested via ship_spec.rb
+        expect(out).to include("Task Definitions built")
+      end
+    end
+
     context "task" do
       it "runs one time task" do
         out = execute("bin/ufo task hi-migrate-prod #{@args}")

data/spec/lib/ecr_auth_spec.rb

@@ -1,8 +1,8 @@
 require 'spec_helper'
 
-describe Ufo::EcrAuth do
+describe Ufo::Ecr::Auth do
   let(:repo_domain) { "https://123456789.dkr.ecr.us-east-1.amazonaws.com" }
-  let(:auth) { Ufo::EcrAuth.new(repo_domain) }
+  let(:auth) { Ufo::Ecr::Auth.new(repo_domain) }
   before(:each) do
     allow(auth).to receive(:fetch_auth_token).and_return("opensesame")
   end
@@ -10,7 +10,6 @@ describe Ufo::EcrAuth do
   context("update") do
     before(:each) do
       clean_home
-      ENV['HOME'] = "spec/fixtures/home"
     end
 
     context("missing ~/.docker/config.json") do
@@ -24,7 +23,6 @@ describe Ufo::EcrAuth do
 
     context("existing ~/.docker/config.json") do
       it "should update the auth token" do
-        FileUtils.cp_r("spec/fixtures/home_existing", "spec/fixtures/home")
         auth.update
         data = JSON.load(IO.read("spec/fixtures/home/.docker/config.json"))
         auth_token = data["auths"][repo_domain]["auth"]
@@ -35,5 +33,6 @@ describe Ufo::EcrAuth do
 
   def clean_home
     FileUtils.rm_rf("spec/fixtures/home")
+    FileUtils.cp_r("spec/fixtures/home_existing", "spec/fixtures/home")
   end
 end

data/spec/lib/ecr_cleaner_spec.rb

@@ -1,10 +1,10 @@
 require 'spec_helper'
 
-describe Ufo::EcrCleaner do
+describe Ufo::Ecr::Cleaner do
   let(:docker_image_name) { "123456789.dkr.ecr.us-east-1.amazonaws.com/my-name" }
   let(:repo_domain) { "https://123456789.dkr.ecr.us-east-1.amazonaws.com" }
   let(:cleaner) do
-    Ufo::EcrCleaner.new(docker_image_name,
+    Ufo::Ecr::Cleaner.new(docker_image_name,
       project_root: "spec/fixtures/hi",
       ecr_keep: 3, # using 3 to test, default is 30
       mute: true

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: ufo
 version: !ruby/object:Gem::Version
-  version: 1.2.0
+  version: 1.5.0
 platform: ruby
 authors:
 - Tung Nguyen
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2017-06-01 00:00:00.000000000 Z
+date: 2017-06-11 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: thor
@@ -181,26 +181,31 @@ files:
 - lib/ufo/command.rb
 - lib/ufo/defaults.rb
 - lib/ufo/destroy.rb
-- lib/ufo/docker_builder.rb
-- lib/ufo/docker_cleaner.rb
-- lib/ufo/dockerfile_updater.rb
+- lib/ufo/docker.rb
+- lib/ufo/docker/builder.rb
+- lib/ufo/docker/cleaner.rb
+- lib/ufo/docker/dockerfile.rb
+- lib/ufo/docker/help.rb
 - lib/ufo/dsl.rb
 - lib/ufo/dsl/helper.rb
 - lib/ufo/dsl/outputter.rb
 - lib/ufo/dsl/task_definition.rb
-- lib/ufo/ecr_auth.rb
-- lib/ufo/ecr_cleaner.rb
-- lib/ufo/execute.rb
+- lib/ufo/ecr.rb
+- lib/ufo/ecr/auth.rb
+- lib/ufo/ecr/cleaner.rb
 - lib/ufo/init.rb
-- lib/ufo/pretty_time.rb
 - lib/ufo/scale.rb
 - lib/ufo/settings.rb
 - lib/ufo/ship.rb
 - lib/ufo/task.rb
-- lib/ufo/tasks_builder.rb
-- lib/ufo/tasks_register.rb
+- lib/ufo/tasks.rb
+- lib/ufo/tasks/builder.rb
+- lib/ufo/tasks/help.rb
+- lib/ufo/tasks/register.rb
 - lib/ufo/templates/default.json.erb
+- lib/ufo/util.rb
 - lib/ufo/version.rb
+- spec/fixtures/home_existing/.aws/config
 - spec/fixtures/home_existing/.docker/config.json
 - spec/lib/cli_spec.rb
 - spec/lib/ecr_auth_spec.rb
@@ -234,6 +239,7 @@ signing_key:
 specification_version: 4
 summary: Build Docker Containers and Ship Them to AWS ECS
 test_files:
+- spec/fixtures/home_existing/.aws/config
 - spec/fixtures/home_existing/.docker/config.json
 - spec/lib/cli_spec.rb
 - spec/lib/ecr_auth_spec.rb

data/lib/ufo/pretty_time.rb

@@ -1,14 +0,0 @@
-module Ufo
-  module PrettyTime
-    # http://stackoverflow.com/questions/4175733/convert-duration-to-hoursminutesseconds-or-similar-in-rails-3-or-ruby
-    def pretty_time(total_seconds)
-      minutes = (total_seconds / 60) % 60
-      seconds = total_seconds % 60
-      if total_seconds < 60
-        "#{seconds.to_i}s"
-      else
-        "#{minutes.to_i}m #{seconds.to_i}s"
-      end
-    end
-  end
-end