ufo 4.6.1 → 5.0.2

data/docs/_docs/quick-start-ec2.md

@@ -26,6 +26,7 @@ This quickstart assumes:
 
 * You have push access to the repo. Refer to the Notes "Repo Push Access" section below for more info.
 * You are using ECS EC2 and have an ECS cluster with EC2 Container instances running. Refer to the Notes "ECS EC2 vs ECS Fargate" section below for more info.
+* The ECS Cluster is in the default VPC. If it is not you need to use the `--vpc-id`, `--ecs-subnets`, and `--elb-subnets` options in the [ufo init]({% link _reference/ufo-init.md %}) command.
 
 ## What Happened
 

data/docs/_docs/secrets.md

@@ -0,0 +1,135 @@
+---
+title: Secrets
+nav_order: 20
+---
+
+## What are Secrets?
+
+[ECS supports injecting secrets or sensitive data](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/specifying-sensitive-data.html) into the the environment as variables.  ECS decrypts the secrets straight from AWS to the ECS task environment. It never passes through the machine calling `ufo ship` IE: your laptop, a deploy server, or CodeBuild, etc.
+
+ECS supports 2 storage backends for secrets:
+
+1. [Secrets Manager](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/specifying-sensitive-data-secrets.html#secrets-envvar)
+2. [Systems Manager Parameter Store](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/specifying-sensitive-data-parameters.html#secrets-envvar-parameters)
+
+Here are both of the formats:
+
+Secrets manager format:
+
+```json
+{
+  "containerDefinitions": [{
+    "secrets": [{
+      "name": "environment_variable_name",
+      "valueFrom": "arn:aws:secretsmanager:region:aws_account_id:secret:secret_name-AbCdEf"
+    }]
+  }]
+}
+```
+
+Parameter store format:
+
+```json
+{
+  "containerDefinitions": [{
+    "secrets": [{
+      "name": "environment_variable_name",
+      "valueFrom": "arn:aws:ssm:region:aws_account_id:parameter/parameter_name"
+    }]
+  }]
+}
+```
+
+## UFO Support
+
+Ufo supports both forms of secrets. You create a `.secrets` file and can reference it in the `.ufo/templates/main.json.erb`
+
+```json
+{
+  "family": "<%= @family %>",
+  # ...
+  <% if @secrets %>
+  "secrets": <%= helper.secrets_file(".secrets") %>,
+  <% end %>
+}
+```
+
+The `.secrets` file is like an env file that will understand a secrets-smart format.  Example:
+
+    NAME1=SSM:my/parameter_name
+    NAME2=SECRETSMANAGER:/my/secret_name-AbCdEf
+
+The `SSM:` and `SECRETSMANAGER:` prefix will be expanded to the full ARN. You can also just specify the full ARN.
+
+    NAME1=arn:aws:ssm:region:aws_account_id:parameter/my/parameter_name
+    NAME2=arn:aws:secretsmanager:region:aws_account_id:secret:my/secret_name-AbCdEf
+
+In turn, this generates:
+
+```json
+{
+  "containerDefinitions": [{
+    "secrets": [
+      {
+        "name": "NAME1",
+        "valueFrom": "arn:aws:ssm:us-west-2:111111111111:parameter/demo/development/foo"
+      },
+      {
+        "name": "NAME2",
+        "valueFrom": "arn:aws:secretsmanager:us-west-2:111111111111:secret:/demo/development/my-secret-test-qRoJel"
+      }
+    ]
+  }]
+}
+```
+
+## SSM Parameter Names with Leading Slash
+
+If your SSM parameter has a leading slash then do **not** include when using it in the .secrets file. Example:
+
+    aws ssm get-parameter --name /demo/development/foo
+
+So use:
+
+    FOO=SSM:demo/development/foo
+
+The extra slash seems to confuse ECS. For secretsmanager names, you do include the leading slash.
+
+## Substitution
+
+Ufo also does a simple substition on the value. For example, the `:UFO_ENV` is replaced with the actual value of `UFO_ENV=development`. Example:
+
+    NAME1=SSM:demo/:UFO_ENV/parameter_name
+    NAME2=SECRETSMANAGER:demo/:UFO_ENV/secret_name-AbCdEf
+
+Expands to:
+
+    NAME1=arn:aws:ssm:region:aws_account_id:parameter/demo/development/parameter_name
+    NAME2=arn:aws:secretsmanager:region:aws_account_id:secret:/demo/development/secret_name-AbCdEf
+
+## IAM Permission
+
+If you're using secrets, you'll need to provide an IAM execution role so the EC2 instance has permission to read the secrets. Here's a starter example:
+
+.ufo/iam_roles/execution_role.rb
+
+```ruby
+managed_iam_policy("AmazonEC2ContainerRegistryReadOnly")
+managed_iam_policy("AmazonSSMReadOnlyAccess")
+managed_iam_policy("CloudWatchLogsFullAccess")
+managed_iam_policy("SecretsManagerReadWrite")
+```
+
+More info [ECS IAM Roles]({% link _docs/iam-roles.md %})
+
+## Debugging Tip
+
+Be sure that the secrets exist. If they do not you will see an error like this in the ecs-agent.log:
+
+/var/log/ecs/ecs-agent.log
+
+    level=info time=2020-06-26T00:59:46Z msg="Managed task [arn:aws:ecs:us-west-2:111111111111:task/development/91828be6a02b48f982cd9122db5e39b2]: error transitioning resource [ssmsecret] to [CREATED]: fetching secret data from SSM Parameter Store in us-west-2: invalid parameters: /my-parameter-name" module=task_manager.go
+
+Sometimes there is even no error message in the ecs-agent.log. As a debugging step, try removing all secrets and seeing if that the container will start up.
+
+{% include prev_next.md %}

data/docs/_docs/settings.md

@@ -16,8 +16,8 @@ base:
   image: tongueroo/demo-ufo
   # clean_keep: 30 # cleans up docker images on your docker server.
   # ecr_keep: 30 # cleans up images on ECR and keeps this remaining amount. Defaults to keep all.
-  network_profile: default # .ufo/settings/network/default.yml file
-  cfn_profile: default # .ufo/settings/cfn/default.yml file
+  # network_profile: default # .ufo/settings/network/default.yml file
+  # cfn_profile: default # .ufo/settings/cfn/default.yml file
 
 development:
   # cluster: dev # uncomment if you want the cluster name be other than the default
@@ -35,13 +35,14 @@ The table below covers each setting:
 
 Setting  | Description
 ------------- | -------------
-`aws_profile`  | If you have the `AWS_PROFILE` environment variable set, this will ensure that you are deploying the right `UFO_ENV` to the right AWS environment. It is explained below.
-`cfn_profile` | The name of the cfn profile settings file to use. Maps to .ufo/settings/cfn/NAME.yml file
-`clean_keep`  | Docker images generated from ufo are cleaned up automatically for you at the end of `ufo ship`. This controls how many docker images to keep around. The default is 3.
-`cluster`  | By convention, the ECS cluster that ufo deploys to matches the `UFO_ENV`. If `UFO=development`, then `ufo ship` deploys to the `development` ECS cluster. This is option overrides this convention.
-`ecr_keep`  | If you are using AWS ECR, then the ECR images can also be automatically cleaned up at the end of `ufo ship`. By default this is set to `nil` and all AWS ECR are kept.
-`image`  | The `image` value is the name that ufo will use for the Docker image name to be built.  Only provide the basename part of the image name without the tag because ufo automatically generates the tag for you. For example, `tongueroo/demo-ufo` is correct and `tongueroo/demo-ufo:my-tag` is incorrect.
-`network_profile` | The name of the network profile settings file to use. Maps to .ufo/settings/network/NAME.yml file
+aws_profile  | If you have the `AWS_PROFILE` environment variable set, this will ensure that you are deploying the right `UFO_ENV` to the right AWS environment. It is explained below.
+cfn_profile | The name of the cfn profile settings file to use. Maps to .ufo/settings/cfn/NAME.yml file. Will match an `UFO_ENV` file if it exists. IE: .ufo/settings/cfn/development.yml. Otherwise it defaults to .ufo/settings/cfn/default.yml.
+clean_keep  | Docker images generated from ufo are cleaned up automatically for you at the end of `ufo ship`. This controls how many docker images to keep around. The default is 3.
+cluster  | By convention, the ECS cluster that ufo deploys to matches the `UFO_ENV`. If `UFO=development`, then `ufo ship` deploys to the `development` ECS cluster. This is option overrides this convention.
+ecr_keep  | If you are using AWS ECR, then the ECR images can also be automatically cleaned up at the end of `ufo ship`. By default this is set to `nil` and all AWS ECR are kept.
+image  | The `image` value is the name that ufo will use for the Docker image name to be built.  Only provide the basename part of the image name without the tag because ufo automatically generates the tag for you. For example, `tongueroo/demo-ufo` is correct and `tongueroo/demo-ufo:my-tag` is incorrect.
+managed\_security\_groups | Create managed security groups for application ELBs. Defaults to true. If you disable it with false then no managed security groups will be created by UFO.
+network_profile | The name of the network profile settings file to use. Maps to .ufo/settings/network/NAME.yml file. Will match an `UFO_ENV` file if it exists. IE: .ufo/settings/network/development.yml. Otherwise it defaults to .ufo/settings/network/default.yml.
 
 ## AWS_PROFILE support
 

data/docs/_docs/settings/cluster.md

@@ -11,25 +11,19 @@ Normally, the ECS cluster defaults to whatever UFO_ENV is set to by [convention]
 
 By default, these are all the same:
 
-```sh
-ufo ship demo-web
-UFO_ENV=development ufo ship demo-web # same
-UFO_ENV=development ufo ship demo-web --cluster development # same
-```
+    ufo ship demo-web
+    UFO_ENV=development ufo ship demo-web # same
+    UFO_ENV=development ufo ship demo-web --cluster development # same
 
 If you use a specific `UFO_ENV=production`, these are the same
 
-```
-UFO_ENV=production ufo ship demo-web
-UFO_ENV=production ufo ship demo-web --cluster production # same
-```
+    UFO_ENV=production ufo ship demo-web
+    UFO_ENV=production ufo ship demo-web --cluster production # same
 
 Override the convention by explicitly specifying the `--cluster` option in the CLI.
 
-```sh
-ufo ship demo-web --cluster custom-cluster # override the cluster
-UFO_ENV=production ufo ship demo-web --cluster production-cluster # override the cluster
-```
+    ufo ship demo-web --cluster custom-cluster # override the cluster
+    UFO_ENV=production ufo ship demo-web --cluster production-cluster # override the cluster
 
 The cavaet is that you must remember to specify `--cluster`.  A wrapper `bin/deploy` script could be useful here.
 

data/docs/_docs/settings/manage-security-groups.md

@@ -0,0 +1,24 @@
+---
+title: Managed Security Groups
+short_title: Security Groups
+categories: settings
+nav_order: 16
+---
+
+Ufo creates and manages two security groups. One for the ELB and one for the ECS tasks. Details here: [UFO Security Groups]({% link _docs/extras/security-groups.md %}).
+
+You can disable the creation of managed security groups with: `managed_security_groups: false`. Example:
+
+```yaml
+base:
+  image: tongueroo/demo-ufo
+  managed_security_groups: false
+```
+
+## Why?
+
+Security Groups managed by UFO are transient. If you delete the UFO app and recreate it entirely. Any manual changes to the security groups will be lost.
+
+You can precreate security groups and add them generated UFO CloudFormation template, see [Settings Network]({% link _docs/settings/network.md %}). So then you won't lose any manual changes. If you're taking this approach, it's nice to have UFO not create any managed security groups at all. This removes security group clutter.
+
+{% include prev_next.md %}

data/docs/_docs/settings/network.md

@@ -2,7 +2,7 @@
 title: Settings Network
 short_title: Network
 categories: settings
-nav_order: 16
+nav_order: 17
 ---
 
 The settings.yml file references a network settings file with the `network_profile` option. This file has configurations that are related to the network.  The source code for the starter template file is at [network/default.yml.tt](https://github.com/tongueroo/ufo/blob/master/lib/template/.ufo/settings/network/default.yml.tt)  Here's an example network settings file.
@@ -22,6 +22,16 @@ elb_subnets: # defaults to same subnets as ecs_subnets when not set
 #   - sg-aaa
 # ecs_security_groups:
 #   - sg-bbb
+
+# Also supports extra security groups specific to each ECS service.
+# So you can target security groups on a per-role basis.
+# ecs_security_groups:
+#   demo-web:
+#   - sg-bbb
+#   - sg-ccc
+#   demo-worker:
+#   - sg-bbb
+#   - sg-ccc
 ```
 
 Option | Description

data/docs/_docs/structure.md

@@ -25,15 +25,16 @@ The table below covers the purpose of each folder and file.
 
 File / Directory  | Description
 ------------- | -------------
-<code>output/</code>  | The folder where the generated task definitions are written to.  The way the task definitions are generated is covered in [ufo tasks build]({% link _docs/tutorial-ufo-tasks-build.md %}).
-<code>params</code>  | This is where you can adjust the params that get send to the aws-sdk api calls for the [ufo task](https://ufoships.com/reference/ufo-task/) command. More info at [Params]({% link _docs/ufo-task-params.md %}).
-<code>settings.yml</code>  | Ufo's general settings file, where you adjust the default [settings]({% link _docs/settings.md %}).
-<code>settings/cfn/default.yml</code>  | Ufo's cfn settings. You can customize the CloudFormation resource properties here.
-<code>settings/network/default.yml</code>  | Ufo's network settings. You can customize the vpc and subnets to used here.
-<code>task_definitions.rb</code>  | This is where you define the task definitions and specify the variables to be used by the ERB templates.
-<code>templates/</code>  | The ERB templates with the task definition json code.  The templates are covered in more detail in [ufo tasks build]({% link _docs/tutorial-ufo-tasks-build.md %}).
-<code>templates/main.json.erb</code>  | This is the main and starter template task definition json file that ufo initially generates.
-<code>variables</code>  | This is where you can define shared variables that are made available to the `template_definitions.rb` and your templates. More info at [Variables]({% link _docs/variables.md %}).
+iam_roles/  | Where ufo managed iam roles associated with the task definition can be defined. For more details see: [IAM Roles]({% link _docs/iam-roles.md %}).
+output/  | The folder where the generated task definitions are written to.  The way the task definitions are generated is covered in [ufo tasks build]({% link _docs/tutorial-ufo-tasks-build.md %}).
+params  | This is where you can adjust the params that get send to the aws-sdk api calls for the [ufo task](https://ufoships.com/reference/ufo-task/) command. More info at [Params]({% link _docs/ufo-task-params.md %}).
+settings.yml  | Ufo's general settings file, where you adjust the default [settings]({% link _docs/settings.md %}).
+settings/cfn/default.yml  | Ufo's cfn settings. You can customize the CloudFormation resource properties here.
+settings/network/default.yml  | Ufo's network settings. You can customize the vpc and subnets to used here.
+task_definitions.rb  | This is where you define the task definitions and specify the variables to be used by the ERB templates.
+templates/  | The ERB templates with the task definition json code.  The templates are covered in more detail in [ufo tasks build]({% link _docs/tutorial-ufo-tasks-build.md %}).
+templates/main.json.erb  | This is the main and starter template task definition json file that ufo initially generates.
+variables  | This is where you can define shared variables that are made available to the `template_definitions.rb` and your templates. More info at [Variables]({% link _docs/variables.md %}).
 
 Now that you know where the ufo configurations are located and what they look like, let’s use ufo!
 

data/docs/_docs/tutorial-ufo-init.md

@@ -65,8 +65,6 @@ base:
   # clean_keep: 30 # cleans up docker images on your docker server.
   # ecr_keep: 30 # cleans up images on ECR and keeps this remaining amount. Defaults to keep all.
   # defaults when an new ECS service is created by ufo ship
-  network_profile: default # .ufo/settings/network/default.yml file
-  cfn_profile: default # .ufo/settings/cfn/default.yml file
 
 development:
   # cluster: dev # uncomment if you want the cluster name be other than the default
@@ -75,14 +73,10 @@ development:
   # When you have AWS_PROFILE set to one of these values, ufo will switch to the desired
   # environment. This prevents you from switching AWS_PROFILE, forgetting to
   # also switch UFO_ENV, and accidentally deploying to production vs development.
-  # aws_profiles:
-  #   - dev_profile1
-  #   - dev_profile2
+  # aws_profile: dev_profile1
 
 production:
   # cluster: prod
-  # aws_profiles:
-  #   - prod_profile
 ```
 
 The `image` value is the name that ufo will use as a base portion of the name to generate a Docker image name, it should not include the tag portion.

data/docs/_docs/ufo-current.md

@@ -1,6 +1,6 @@
 ---
 title: Ufo Current
-nav_order: 23
+nav_order: 26
 ---
 
 ## service

data/docs/_docs/ufo-env-extra.md

@@ -1,6 +1,6 @@
 ---
 title: UFO_ENV_EXTRA
-nav_order: 22
+nav_order: 25
 ---
 
 <div class="video-box"><div class="video-container"><iframe src="https://www.youtube.com/embed/UVQuwQGToYE" frameborder="0" allowfullscreen=""></iframe></div></div>

data/docs/_docs/ufo-env.md

@@ -1,6 +1,6 @@
 ---
 title: UFO_ENV
-nav_order: 21
+nav_order: 24
 ---
 
 Ufo's behavior is controlled by the `UFO_ENV` environment variable.  For example, the `UFO_ENV` variable is used to layer different ufo variable files together to make it easy to specify settings for different environments like production and development.  This is covered thoroughly in the [Variables]({% link _docs/variables.md %}) section.  `UFO_ENV` defaults to `development` when not set.
@@ -27,12 +27,10 @@ The most interesting way to set `UFO_ENV` is with the `aws_profiles` setting in
 
 ```yaml
 development:
-  aws_profiles:
-    - my-dev-profile
+  aws_profile: my-dev-profile
 
 production:
-  aws_profiles:
-    - my-prod-profile
+  aws_profile: my-prod-profile
 ```
 
 In this case, when you set `AWS_PROFILE` to switch AWS profiles, ufo picks this up and maps the `AWS_PROFILE` value to the specified `UFO_ENV` using the `aws_profiles` lookup.  Example:

data/docs/_docs/ufo-logs.md

@@ -1,6 +1,6 @@
 ---
 title: ufo logs command
-nav_order: 20
+nav_order: 23
 ---
 
 The ufo logs command will tail the logs of the ecs service if you are using the awslogs driver.
@@ -46,5 +46,4 @@ The generated .ufo task definition defaults to the awslogs driver. If you need i
 }
 ```
 
-
 {% include prev_next.md %}

data/docs/_docs/ufo-task-params.md

@@ -1,6 +1,6 @@
 ---
 title: Ufo Task Params
-nav_order: 24
+nav_order: 27
 ---
 
 You can run one off task with the [ufo task](https://ufoships.com/reference/ufo-task/) command.

data/docs/_docs/upgrading.md

@@ -1,6 +1,6 @@
 ---
 title: Upgrading
-nav_order: 34
+nav_order: 38
 ---
 
 <ul>

data/docs/_docs/upgrading/upgrade4.5.md

@@ -1,9 +1,9 @@
 ---
 title: Upgrading to Version 4.5
 short_title: Version 4.5
-order: 1
+order: 2
 categories: upgrading
-nav_order: 35
+nav_order: 40
 ---
 
 In ufo version 4.4 and 4.5, the default cloudformation stack names used by ufo were changed.

data/docs/_docs/upgrading/upgrade4.md

@@ -1,9 +1,9 @@
 ---
 title: Upgrading to Version 4.0
 short_title: Version 4.0
-order: 2
+order: 3
 categories: upgrading
-nav_order: 36
+nav_order: 41
 ---
 
 A major change in ufo from version 3 to 4 is that the ECS service is now created by CloudFormation. If you have an existing ECS service deployed by ufo version 3, when you deploy your app again with ufo version 4, there will be a new additional ECS service created. Here is the recommended upgrade path.

data/docs/_docs/upgrading/upgrade5.md

@@ -0,0 +1,19 @@
+---
+title: Upgrading to Version 5
+short_title: Version 5
+order: 1
+categories: upgrading
+nav_order: 39
+---
+
+In ufo v5, the ufo went from underscore key names in the [cfn settings files]({% link _docs/settings/cfn.md %}) to camelized key names. So the auto_camelize behavior is disabled for newly `ufo init` projects. This mean ufo is backwards compatiable. You can enable the v5 behavior with `auto_camelize: false`. If you have not adjusted this setting, then ufo should still work with your current `.ufo` files.
+
+## Upgrading Instructions
+
+If you want to upgrade to the latest ufo v5 default behavior.
+
+1. Adjust your .ufo/settings/cfn files so that the keys are camelized
+2. Add to your .ufo/settings.yml `auto_camelize: false`
+3. Deploy and verify that your ECS app stil works
+
+{% include prev_next.md %}

data/docs/_docs/variables.md

@@ -1,6 +1,6 @@
 ---
 title: Shared Variables
-nav_order: 17
+nav_order: 18
 ---
 
 Often, you end up using the set of common variables across your task definitions for a project.  Ufo supports a shared variables concept to support this.  You specify variables files in the `.ufo/variables` folder and they are made available to your `.ufo/task_definitions.rb` as well as your `.ufo/templates` files.

data/docs/_includes/cfn-customize.md

@@ -5,11 +5,11 @@ The properties in the file `.ufo/settings/cfn/default.yml` map directly to ufo's
 Let's customize the `AWS::ElasticLoadBalancingV2::TargetGroup` resource created by CloudFormation.  We'll adjust the `deregistration_delay.timeout_seconds` to `8`.  Here's the relevant section of the `.ufo/settings/cfn/default.yml`
 
 ```
-target_group:
+TargetGroup:
 ...
-  target_group_attributes:
-  - key: deregistration_delay.timeout_seconds
-    value: 8
+  TargetGroupAttributes:
+  - Key: deregistration_delay.timeout_seconds
+    Value: 8
 ```
 
 The value will be injected to the generated CloudFormation template under the corresponding "TargetGroup Properties".  The generated template looks something like this: