ufo 3.1.2 → 3.2.0

data/lib/ufo/docker.rb

@@ -5,7 +5,7 @@ module Ufo
     autoload :Dockerfile, 'ufo/docker/dockerfile'
     autoload :Cleaner, 'ufo/docker/cleaner'
 
-    desc "build", "builds docker image"
+    desc "build", "Build docker image."
     long_desc Help.text("docker:build")
     option :push, type: :boolean, default: false
     def build
@@ -14,7 +14,7 @@ module Ufo
       push if options[:push]
     end
 
-    desc "push IMAGE", "push the docker image"
+    desc "push IMAGE", "Push the docker image."
     long_desc Help.text("docker:push")
     option :push, type: :boolean, default: false
     def push(full_image_name=nil)
@@ -23,7 +23,7 @@ module Ufo
       pusher.push
     end
 
-    desc "base", "builds docker image from Dockerfile.base and update current Dockerfile"
+    desc "base", "Build docker image from `Dockerfile.base` and update current `Dockerfile`."
     long_desc Help.text("docker:base")
     option :push, type: :boolean, default: true
     def base
@@ -38,7 +38,7 @@ module Ufo
       Ecr::Cleaner.new(builder.image_name, options.merge(tag_prefix: "base")).cleanup
     end
 
-    desc "name", "displays the full docker image with tag that was last generated."
+    desc "name", "Display the full docker image with tag that was last generated."
     option :generate, type: :boolean, default: false, desc: "Generate a name without storing it"
     long_desc Help.text("docker:name")
     def name
@@ -46,7 +46,7 @@ module Ufo
       puts full_image_name
     end
 
-    desc "clean IMAGE_NAME", "Cleans up old images.  Keeps a specified amount."
+    desc "clean IMAGE_NAME", "Clean up old images.  Keeps a specified amount."
     option :keep, type: :numeric, default: 3
     option :tag_prefix, default: "ufo"
     long_desc Help.text("docker:clean")

data/lib/ufo/ecr/auth.rb

@@ -28,8 +28,13 @@ module Ufo
         data = JSON.load(IO.read(docker_config))
         data["auths"][@repo_domain] = {auth: auth_token}
       else
-        data = {auths: {@repo_domain => {auth: auth_token}}}
+        data = {"auths" => {@repo_domain => {auth: auth_token}}}
       end
+
+      # Handle legacy docker clients that still have old format with https://
+      legacy_entry = "https://#{@repo_domain}"
+      data["auths"][legacy_entry] = {auth: auth_token}
+
       ensure_dotdocker_exists
       IO.write(docker_config, JSON.pretty_generate(data))
     end

data/lib/ufo/ecr/cleaner.rb

@@ -49,7 +49,7 @@ module Ufo
     end
 
     def update_auth_token
-      repo_domain = "https://#{@docker_image_name.split('/').first}"
+      repo_domain = "#{@docker_image_name.split('/').first}"
       auth = Ecr::Auth.new(repo_domain)
       auth.update
     end

data/lib/ufo/help/completion.md

@@ -4,7 +4,7 @@ Example:
 
 Prints words for TAB auto-completion.
 
-Examples:
+## Examples
 
   ufo completion
   ufo completion hello

data/lib/ufo/help/completions.md

@@ -1,6 +1,6 @@
 Provides completions words.
 
-Examples:
+## Examples
 
   ufo completions ship service --mute
 

data/lib/ufo/help/deploy.md

@@ -1,6 +1,6 @@
 It is useful to sometimes deploy only the task definition without re-building it.  Say for example, you are debugging the task definition and just want to directly edit the `.ufo/output/hi-web.json` definition. You can accomplish this with the `ufo deploy` command.  The `ufo deploy` command will deploy the task definition in `.ufo/output` unmodified.  Example:
 
-  ufo deploy hi-web
+    ufo deploy hi-web
 
 The above command does the following:
 
@@ -12,3 +12,7 @@ The `ufo deploy` command does less than the `ufo ship` command.  Typically, peop
 1. build the Docker image
 2. register the ECS task definition
 3. update the ECS service
+
+The `ufo ships`, `ufo ship`, `ufo deploy` command support the same options. The options are presented here again for convenience:
+
+{% include ufo-ship-options.md %}

data/lib/ufo/help/destroy.md

@@ -1,5 +1,9 @@
-Examples:
+## Examples
 
-Destroys the service.  It will automatcally set the desired task size to 0 and stop all task so the destory happens in one command.
+Ufo provides a quick way to destroy an ECS service. To destroy an ECS service, you must make sure that the desired number of tasks is first set to 0. It is easy to forget to do this and waste time. So as part of destroying the service ufo will scale the ECS service down to 0 automatically first and then destroys the service.  Ufo also prompts you before destroying the service.
 
-  ufo destroy hi-web
+    ufo destroy hi-web
+
+If you would like to bypass the prompt, you can use the `--sure` option.
+
+    ufo destroy hi-web --sure

data/lib/ufo/help/docker.md

@@ -1,4 +1,4 @@
-Examples:
+## Examples
 
   ufo docker build
   ufo docker tag

data/lib/ufo/help/docker/base.md

@@ -1,9 +1,39 @@
-The docker cache task builds a docker image using the Dockerfile.base file and
-updates the FROM Dockerfile image with the generated image from Dockerfile.base.
+The docker cache task builds a docker image using the `Dockerfile.base` file and
+updates the FROM `Dockerfile` image with the generated image from `Dockerfile.base`.
 
-Examples:
+## Summarized Example
 
   ufo docker base
   ufo docker base --no-push # do not push the image to the registry
 
-Docker image tongueroo/hi:base-2016-10-21T15-50-57-88071f5 built.
+Docker image `tongueroo/hi:base-2016-10-21T15-50-57-88071f5` built.
+
+## Concept
+
+Docker is fantastic and has given developers more power and control over the OS their application runs on.  Sometimes building Docker images can be slow though.  Docker layer caching technology helps immensely to speed up the process as Docker will only rebuild layers that require rebuilding. But sometimes one little dependency changes it results in having to rebuild many layers.
+
+The `ufo docker base` commands allows you to build an Docker image from `Dockerfile.base` to be used as a cache and in the FROM instruction in the main `Dockerfile`.  This base Docker cache image technique speeds up development substantially if you have dependencies like gems in your project.  Let's say you have 20 gems in your project and are actively developing your project. You are experimenting with 3 different gems, adding and removing them while you are investigating the best gem to use.  Without the Docker image cache that already has all of your other gems installed each time you adjust `Gemfile` it would have to wait until all the gems from scratch again installed again.
+
+There are pros and cons of using this approach.  Remember there are 2 hard problems in computer science: 1. Naming and 2. Caching.  The main con about this approach is if you forget to update the base Docker image you will have cached artifacts that will not disappear unless you rebuild the base Docker image.  While some folks are completely against introducing this cache layer, some have found it being a huge win in speeding up their Docker development workflow.  If you are using this technique it is recommended that you set up some automation that rebuilds the base Docker image at least nightly.
+
+## Demo
+
+To demonstrate this command, there's a `docker-cache` branch in the [tongueroo/hi](https://github.com/tongueroo/hi/tree/docker-cache) repo.
+
+ Let's see the command in action:
+
+    ufo docker base
+    Building docker image with:
+      docker build -t tongueroo/hi:base-2017-06-12T14-36-44-2af505e -f Dockerfile.base .
+      ...
+    Pushed tongueroo/hi:base-2017-06-12T14-36-44-2af505e docker image. Took 28s.
+    The Dockerfile FROM statement has been updated with the latest base image:
+      tongueroo/hi:base-2017-06-12T14-36-44-2af505e
+
+Some of the output has been excluded so we can focus on the important parts to point out. First notice that the commmand simply shells out to the docker command and calls:
+
+    docker build -t tongueroo/hi:base-2017-06-12T14-36-44-2af505e -f Dockerfile.base .
+
+It is using the docker `-f Dockerfile.base` option to build the base image.  It names the image with `tongueroo/hi:base-2017-06-12T14-36-44-2af505e`.  The image tag contains useful information: the timestamp when the image was built and the exact git sha of the code.  The image gets push to a registry immediately.
+
+Notice at the very end, the *current* `Dockerfile`'s FROM statement has been updated with the newly built base Docker image automatically.  This saves you from forgetting to copying and pasting it the `Dockerfile` yourself.

data/lib/ufo/help/docker/build.md

@@ -1,6 +1,61 @@
-Examples:
+The `ufo docker build` builds a Docker image using the Dockerfile in the current project folder.  This simply is a wrapper command that shells out and calls the `docker` command.  We're use the [tongueroo/hi](https://github.com/tongueroo/hi) project and it's Dockerfile for demonstration.  Example:
 
-  ufo docker build
-  ufo docker build --push # also pushes the image to the docker registry
+    ufo docker build
+
+You'll see that it calls:
+
+    docker build -t tongueroo/hi:ufo-2017-06-11T22-18-03-a18aa30 -f Dockerfile .
+
+You should see similar output (some of the output has been truncated for conciseness):
+
+    $ ufo docker build
+    Building docker image with:
+      docker build -t tongueroo/hi:ufo-2017-06-11T22-18-03-a18aa30 -f Dockerfile .
+    Sending build context to Docker daemon 734.2 kB
+    Step 1 : FROM ruby:2.3.3
+     ---> 0e1db669d557
+    Step 2 : RUN apt-get update &&   apt-get install -y     build-essential     nodejs &&   rm -rf /var/lib/apt/lists/* && apt-get clean && apt-get purge
+     ---> Using cache
+     ---> 931ace833716
+    ...
+    Step 7 : ADD . /app
+     ---> fae2452e6c35
+    Removing intermediate container 4c93f92a7fd8
+    Step 8 : RUN bundle install --system
+     ---> Running in f851b9cb7d27
+    Using rake 12.0.0
+    Using i18n 0.8.1
+    ...
+    Using web-console 2.3.0
+    Bundle complete! 12 Gemfile dependencies, 56 gems now installed.
+    Bundled gems are installed into /usr/local/bundle.
+     ---> 194830c5c1a8
+    ...
+    Removing intermediate container 67545cd4cd09
+    Step 11 : CMD bin/web
+     ---> Running in b1b26e68d957
+     ---> 8547bb48b21f
+    Removing intermediate container b1b26e68d957
+    Successfully built 8547bb48b21f
+    Docker image tongueroo/hi:ufo-2017-06-11T22-18-03-a18aa30 built.  Took 33s.
+    $
+
+The docker image tag that is generated contains a useful timestamp and the current HEAD git sha of the project that you are on.
+
+By default when you are running `ufo docker build` directly it does not push the docker image to the registry.  If you would like it to automaticaly push the built image to a registry at the end of the build use the `--push` flag.
+
+    ufo docker build --push
+
+You should see it being pushed at the end:
+
+    Docker image tongueroo/hi:ufo-2017-06-11T22-22-32-a18aa30 built.  Took 34s.
+    The push refers to a repository [docker.io/tongueroo/hi]
+    ef375857f165: Pushed
+    4d791d7cde66: Pushed
+    277ff31e79b4: Layer already exists
+    a361a4de05df: Layer already exists
+    ufo-2017-06-11T22-22-32-a18aa30: digest: sha256:c5385a5084e87643bd943eb120e110321c59e8acd30736ba7b5223eb1143baa8 size: 3464
+    Pushed tongueroo/hi:ufo-2017-06-11T22-22-32-a18aa30 docker image. Took 9s.
+
+Note in order to push the image to a registry you will need to login into the registry.  If you are using DockerHub use the `docker login` command.  If you are using AWS ECR then you can use the `aws ecr get-login` command.
 
-Docker image tongueroo/hi:ufo-2016-10-21T15-50-57-88071f5 built.

data/lib/ufo/help/docker/clean.md

@@ -1,14 +1,20 @@
-Examples:
+Ufo comes with a handy command to clean up old images that ufo generates. Ufo only deletes images from the docker daemon and does not remove any images from any registry.  To use it you pass the base portion of the image name to the command.
 
 Say you currently have these images:
 
-  * tongueroo/hi:ufo-2016-10-15T19-29-06-88071f5
-  * tongueroo/hi:ufo-2016-10-16T19-29-06-88071f5
-  * tongueroo/hi:ufo-2016-10-17T19-29-06-88071f5
-  * tongueroo/hi:ufo-2016-10-18T19-29-06-88071f5
+    tongueroo/hi:ufo-2016-10-15T19-29-06-88071f5
+    tongueroo/hi:ufo-2016-10-16T19-29-06-88071f5
+    tongueroo/hi:ufo-2016-10-17T19-29-06-88071f5
+    tongueroo/hi:ufo-2016-10-18T19-29-06-88071f5
 
 To clean them up and keep the 3 more recent:
 
-  ufo docker clean tongueroo/hi
+    $ ufo docker clean tongueroo/hi
+    Cleaning up docker images...
+    Running: docker rmi tongueroo/hi:ufo-2016-10-15T19-29-06-88071f5
 
 This will remove tongueroo/hi:ufo-2016-10-15T19-29-06-88071f5.
+
+By default the clean command keeps the most 3 recent docker images. If you would like to override this setting you can use the `--keep` option. Example:
+
+    ufo docker clean tongueroo/hi --keep 5

data/lib/ufo/help/docker/name.md

@@ -1,16 +1,16 @@
-This command fetches the last name that was generated when `docker build` was ran internally by `ufo docker build`.  You can use it after you have built a docker image with `ufo docker build`.
+It is sometimes handy to grab the name of the Docker image that was just generated.  Let's say ou needed the image name so you can feed it into another script that you have. The command `ufo docker name` returns the image of the most recently built Docker image.
 
-Examples:
+## Examples
 
-  ufo docker build # stores the docker image name in the .ufo/data folder
-  ufo docker name  # fetches image name from .ufo/data folder
+    ufo docker build # stores the docker image name in the .ufo/data folder
+    ufo docker name  # fetches image name from .ufo/data folder
 
-An example image name would look something like this: tongueroo/hi:ufo-2018-02-15T19-29-06-88071f5
+An example image name would look something like this: `tongueroo/hi:ufo-2018-02-15T19-29-06-88071f5`
 
-Note, the .ufo/data folder is an internal data folder and it's structure can change in future releases.
+Note, the `.ufo/data` folder is an internal data folder and it's structure can change in future releases.
 
-If you want to generate a brand new name for whatever purpose, you can use the `--generate` flag.  The generate does not write to the `.ufo/data` folder.  It only generates a fresh name to stdout.  If you run it multiple times, it will generate new names.  You can notice this by seeing that the timestamp will always update. Examples:
+If you want to generate a brand new name for whatever purpose, you can use the `--generate` flag.  The generate does not write to the `.ufo/data` folder.  It only generates a fresh name to stdout.  If you run it multiple times, it will generate new names.  You can notice this by seeing that the timestamp will always update:
 
-  ufo docker name --generate  # example: tongueroo/hi:ufo-2018-02-15T10-00-00-88071f5
-  ufo docker name --generate  # example: tongueroo/hi:ufo-2018-02-15T10-00-08-88071f5
-  ufo docker name --generate  # example: tongueroo/hi:ufo-2018-02-15T10-00-16-88071f5
+    ufo docker name --generate  # example: tongueroo/hi:ufo-2018-02-15T10-00-00-88071f5
+    ufo docker name --generate  # example: tongueroo/hi:ufo-2018-02-15T10-00-08-88071f5
+    ufo docker name --generate  # example: tongueroo/hi:ufo-2018-02-15T10-00-16-88071f5

data/lib/ufo/help/docker/push.md

@@ -1,11 +1,28 @@
-This command pushes a docker image up to the registry.  By default it pushes the last image that was built with `ufo docker build`.  To see what the image name is you can run `ufo docker name`. Example:
 
-  ufo docker build # to build the image
-  ufo docker name  # to see the image name
-  ufo docker push  # push up the registry
+The `ufo docker push` command pushes the most recent Docker image built by `ufo docker build` to a registry.  This command pushes a docker image up to the registry.  By default it pushes the last image that was built with `ufo docker build`.  To see what the image name is you can run `ufo docker name`. Example:
+
+    ufo docker build # to build the image
+    ufo docker name  # to see the image name
+    ufo docker push  # push up the registry
+
+You'll see that `ufo docker push` simply shells out and calls `docker push`:
+
+    $ ufo docker push
+    => docker push 123456789.dkr.ecr.us-east-1.amazonaws.com/hi:ufo-2018-02-13T10-51-44-e0cc7be
+    The push refers to a repository [123456789.dkr.ecr.us-east-1.amazonaws.com/hi]
+    399c739c257d: Layer already exists
+    ...
+    Pushed 123456789.dkr.ecr.us-east-1.amazonaws.com/hi:ufo-2018-02-13T10-51-44-e0cc7be docker image. Took 1s.
+    $
 
 You can also push up a custom image by specifying the image name as the first parameter.
 
-  ufo docker push my/image:tag
+    ufo docker push my/image:tag
+
+You could also use the `--push` flag as part of the `ufo docker build` command to achieve the same thing as `ufo docker push`. The `ufo docker push` command might be more intutitive.
+
+    ufo docker build --push # same as above
+
+## Docker Authorization
 
-The command also updates your ECR auth token in `~/.docker/config.json` in case it has expired.
+Note in order to push the image to a registry you will need to login into the registry.  If you are using DockerHub use the `docker login` command.  If you are using AWS ECR then, ufo will automatically try to authorize you and configure your `~/.docker/config.json`.  If can also use `aws ecr get-login` command.

data/lib/ufo/help/hello.md

@@ -1,4 +1,4 @@
-Examples:
+## Examples
 
   ufo hello
   ufo hello NAME

data/lib/ufo/help/init.md

@@ -1,11 +1,49 @@
-Examples:
+The `ufo init` command provides a way to quickly setup a project to be ufo ready. It creates a ufo folder with all the starter supporting files in order to use ufo.  This page demonstrates how to use `ufo init`.  The command requires these options: `--app` and `--image`.
 
-  ufo init --app=hi --image=tongueroo/hi
+## Examples
 
-The app is that application name that you want to show up on the ECS dashboard.  It is strongly encourage to have the app name be a single word.
+For this example we will use [tongueroo/hi](https://github.com/tongueroo/hi) which is a small test sinatra app.  Let's run the command in our newly clone project.
 
-The image is the base portion of image name that will be pushed to the docker registry, ie: DockerHub or AWS ECR.  The image should not include the tag since the tag is generated upon a `ufo ship`.  For example:
+    $ git clone https://github.com/tongueroo/hi.git
+    $ cd hi
+    $ ufo init --app=hi --image=tongueroo/hi
+    Setting up ufo project...
+          create  .ufo/settings.yml
+          create  .ufo/task_definitions.rb
+          create  .ufo/templates/main.json.erb
+          create  .ufo/variables/base.rb
+          create  .ufo/variables/development.rb
+          create  .ufo/variables/production.rb
+          create  Dockerfile
+          create  bin/deploy
+          append  .gitignore
+    Starter ufo files created.
 
-  tongueroo/hi => tongueroo/hi:ufo-2018-02-08T21-04-02-3c86158
+## Options: app and image
+
+The `app` is that application name that you want to show up on the ECS dashboard.  It is encouraged to have the app name be a single word.
+
+The `image` is the base portion of image name that will be pushed to the docker registry, ie: DockerHub or AWS ECR.  The image should **not** include the tag since the tag is generated upon a `ufo ship`.  For example:
+
+    tongueroo/hi => tongueroo/hi:ufo-2018-02-08T21-04-02-3c86158
 
 The generated `tongueroo/hi:ufo-2018-02-08T21-04-02-3c86158` image name gets pushed to the docker registry.
+
+## Directory Structure
+
+The standard directory structure of the `.ufo` folder that was created looks like this:
+
+    ufo
+    ├── output
+    ├── settings.yml
+    ├── task_definitions.rb
+    ├── templates
+    ├   └── main.json.erb
+    └── variables
+        ├── base.rb
+        ├── production.rb
+        └── development.rb
+
+For a explanation of the folders and files refer to [Structure]({% link _docs/structure.md %}).
+
+

data/lib/ufo/help/scale.md

@@ -1,5 +1,6 @@
-Examples:
+Ufo provides a command to scale up and down an ECS service quickly. It is a simple wrapper for `aws ecs update-service --service xxx ----desired-count xxx`.  Here's an example of how you use it:
 
-Scales the service.  Simple wrapper for `aws ecs update-service --service xxx ----desired-count xxx`
+    $ ufo scale hi-web 3
+    Scale hi-web service in stag cluster to 3
 
-  ufo scale hi-web 5
+While scaling via this method is quick and convenient the [ECS Service AutoScaling](http://docs.aws.amazon.com/AmazonECS/latest/developerguide/service-auto-scaling.html) that is built into ECS is a much more powerful way to manage scaling your ECS service.

data/lib/ufo/help/ship.md

@@ -1,17 +1,68 @@
-Examples:
+The main command you use when using ufo is: `ufo ship`.  This command:
 
-To build the docker image, generate the task definition and ship it, run:
+1. builds the docker image
+2. registers the generated ECS task definition
+3. deploys the definition to AWS ECS
 
-  ufo ship hi-web
+Basic usage is:
+
+    ufo ship hi-web
+
+The ECS service gets created if the service does not yet exist on the cluster.
+
+### Conventions
 
 By convention the task and service names match. If you need override to this convention then you can specific the task.  For example if you want to ship to the `hi-web-1` service and use the `hi-web` task, run:
 
-  ufo ship hi-web-1 --task hi-web
+    ufo ship hi-web-1 --task hi-web
+
+## Options in Detail
+
+The command has a decent amount of options, you can see the options available with `ufo ship -h`.  The table below covers some of the options in detail:
+
+{% include ufo-ship-options.md %}
+
+As you can see there are plenty of options for `ufo ship`.  Let's demonstrate usage of them in a few examples.
+
+### Load Balancer Target Group
+
+When you are deploying to a service with the word 'web' in it, ufo assumes that this is a web service that uses a load balancer in front of it.  This is also covered a in the [Conventions]({% link _docs/conventions.md %}) page.  The deploy will prompt you for the ELB `--target-group`  if the ECS does not yet exist.  For non-web container the `--target-group` option gets ignored.  The prommpt can be bypassed by specifying a valid `--target-group` option or specifying the `---no-target-group-prompt` option.
+
+    ufo ship hi-web --no-target-group-prompt
+
+Or if you would like to specify the target-group upfront and not be bother with the prompted later you can use the `--target-group` option.
+
+    ufo ship hi-web --target-group=arn:aws:elasticloadbalancing:us-east-1:12345689:targetgroup/hi-web/12345
+
+### Deploying Task Definition without Docker Build
+
+Let's you want skip the docker build phase and only want use ufo to deploy a task definition. You can do this with the `ufo deploy` command.  Refer to [ufo deploy](http://ufoships.com/reference/ufo-deploy/) for more info.
+
+### Waiting for Deployments to Complete
+
+By default when ufo updates the ECS service with the new task definition it does so asynchronuously. You then normally visit the ECS service console and then refresh until you see that the deployment is completed.  You can also have ufo poll and wait for the deployment to be done with the `--wait` option
+
+    ufo ship hi-web --wait
+
+You should see output similar to this:
+
+    Shipping hi-web...
+    hi-web service updated on cluster with task hi-web
+    Waiting for deployment of task definition hi-web:8 to complete
+    ......
+    Time waiting for ECS deployment: 31s.
+    Software shipped!
+
+### Cleaning up Docker Images Automatically
+
+Since ufo builds the Docker image every time there's a deployment you will end up with a long list of docker images.  Ufo automatically cleans up older docker images at the end of the deploy process if you are using AWS ECR.  By default ufo keeps the most recent 30 Docker images. This can be adjust with the `--ecr-keep` option.
+
+    docker ship hi-web --ecr-keep 2
 
-The deploy will also created the ECS service if the service does not yet exist on the cluster.  The deploy will prompt you for the ELB `--target-group` if you are shipping a web container that does not yet exist.  If it is not a web container the `--target-group` option gets ignored.
+You should see something like this:
 
-The prommpt can be bypassed by specifying a valid `--target-group` option or specifying the `---no-target-group-prompt` option.
+    Cleaning up docker images...
+    Running: docker rmi tongueroo/hi:ufo-2017-06-12T06-46-12-a18aa30
 
-  ufo ship hi-web --target-group arn:aws:elasticloadbalancing:us-east-1:123456789:targetgroup/hi-web/jsdlfjsdkd
+If you are using DockerHub or another registry, ufo does not automatically clean up images.
 
-  ufo ship hi-web --no-target-group-prompt