ufo 4.5.11 → 5.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 7a7401ccc5d5e288b485c4eaaf21a4ae14864a23bdc52c4e36ce9bf060f35d55
-  data.tar.gz: 2528b31c9fea0c43785a2d1c17c2529c85f3ac85dd431e9cc5de7fbca28cdc53
+  metadata.gz: ce5c8180b261636a61805a4abc5cbd3d556784f77dbf20ca0f2384d8ab50ae32
+  data.tar.gz: 9d6e1955bd7ca4b35b347c61986af5c92c916410a1d9b43b0d11a76e1dbd4fc3
 SHA512:
-  metadata.gz: 890ed4c9715cea6d1c6865e13a3895467d72e266363ea0647111e513299c1e4d3182d68ad8039212bbae2d6c59d16ab53d44629042cde185a173c4fa62f3fbe4
-  data.tar.gz: bb31a72b778cb39de850a07cb6ce94dc31c50b2f257d067c77b5fa40dcf5d1dfcc3ce50d05c9980e8a1fa298c4f229bb1d47c6df0d44ae18e24d0b218f1647fc
+  metadata.gz: 60ec0e82534f94e8daffbb9587f22753e4df92a77545ba4f220e0f9f3f1568a7dc0722f32e278f1853160ab477e80b0cfef8dbde6330c0b0af46a3e4695c8bc6
+  data.tar.gz: 4bb7540d47f271ea211b3ade315b6a69fe9d1ce23b2cf0cef82cf2ddb1905d849a6064a08aab07279e623073f985542f8396533aedde86fc0b78a39e99ca8bb0

data/CHANGELOG.md

@@ -3,6 +3,35 @@
 All notable changes to this project will be documented in this file.
 This project *tries* to adhere to [Semantic Versioning](http://semver.org/), even before v1.0.
 
+## [5.0.0]
+- #104 adjust logs default format to detailed
+- #105 major rework: build cfn template with Ruby instead of ERB for new features
+- #106 secrets support
+- Codified iam_role support with .ufo/iam_roles files: custom and managed policy support. The ECS Task definition was moved into CloudFormation to support this.
+- Allow per service security groups
+- Conventional .ufo/settings cfn and network files based on ufo env
+- Managed_security_groups_enabled=false setting.yml
+- Project custom helper methods support
+- Add image-override option for ufo ship
+- Notification ARN stack cloudformation support for compliance reasons
+- update cfn/default to use CamelCase. maintain backward compatibility with underscore. through encourage users to upgrade to CamelCase. There's less mental translation overhead.
+- remove pretty option: always pretty
+
+## [4.6.3]
+- #101 improve ufo init help
+
+## [4.6.2]
+- define texit_on_failure to remove Thor deprecation warning
+
+## [4.6.1]
+- #93 Fix firelens functionality
+- #97 ufo logs --filter-pattern option
+- #98 fix env vars helper to allow surrounding quotes
+
+## [4.6.0]
+- #95 Introduce: ufo logs command. Tail logs.
+- #96 docs and options
+
 ## [4.5.11]
 - add mfa support for normal IAM user
 

data/README.md

@@ -8,6 +8,8 @@
 [![Join the chat at https://gitter.im/tongueroo/ufo](https://badges.gitter.im/tongueroo/ufo.svg)](https://gitter.im/tongueroo/ufo?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge&utm_content=badge)
 [![Support](https://img.shields.io/badge/get-support-blue.svg)](https://boltops.com?utm_source=badge&utm_medium=badge&utm_campaign=ufo)
 
+[![BoltOps Badge](https://img.boltops.com/boltops/badges/boltops-badge.png)](https://www.boltops.com)
+
 Ufo is a tool that builds Docker images and deploys them to [AWS ECS](https://aws.amazon.com/ecs/).  The main command is `ufo ship`.  Here's summary of what it does:
 
 1. Builds a docker image.

data/docs/_docs/conventions.md

@@ -44,4 +44,4 @@ You can also use an existing ELB by specifying the target group arn as the value
 ufo ship demo-web --elb arn:aws:elasticloadbalancing:us-east-1:12345689:targetgroup/demo-web/12345
 ```
 
-{% include prev_next.md %}
+{% include prev_next.md %}

data/docs/_docs/extras/codebuild-iam-role.md

@@ -1,6 +1,6 @@
 ---
 title: CodeBuild IAM Role
-nav_order: 31
+nav_order: 32
 ---
 
 Note, the `/tmp/ecs-deploy-policy.json` policy is available at [Minimal Deploy IAM]({% link _docs/extras/minimal-deploy-iam.md %}).
@@ -43,4 +43,4 @@ Create the IAM resources:
 
 The `attach-role-policy` command attaches a Customer Managed IAM policy to the IAM role. This is a little more reusable than using an inline policy.
 
-{% include prev_next.md %}
+{% include prev_next.md %}

data/docs/_docs/extras/dockerfile-erb.md

@@ -1,6 +1,6 @@
 ---
 title: Dynamic Dockerfile.erb
-nav_order: 32
+nav_order: 33
 ---
 
 Sometimes you may need a little more dynamic control of your Dockerfile. For these cases, ufo supports dynamically creating a Dockerfile from a Dockerfile.erb.  If Dockerfile.erb exists, ufo uses it to generate a Dockerfile as a part of the build process.  These means that you should update the source Dockerfile.erb instead, as the Dockerfile will be overwritten.  If Dockerfile.erb does not exist, then ufo will use the Dockerfile instead.
@@ -57,4 +57,4 @@ Why not use [build args](https://www.jeffgeerling.com/blog/2017/use-arg-dockerfi
 
 Ufo uses a YAML file so users will not have to remember to provide the build arg. It is also easy to update the `dockerfile_variables.yml` with the `ufo docker base` command.
 
-{% include prev_next.md %}
+{% include prev_next.md %}

data/docs/_docs/extras/ecs-network-mode.md

@@ -1,6 +1,6 @@
 ---
 title: ECS Network Mode
-nav_order: 26
+nav_order: 27
 ---
 
 ## Pros and Cons: bridge network mode
@@ -34,4 +34,4 @@ awsvpc | Fine grain security group permissions for each ECS service. | The numbe
 
 It is generally recommended to use awsvpc mode with ENI trunking supported instances. You get the best of both worlds in this situation: a strong security posture as well as container density.
 
-{% include prev_next.md %}
+{% include prev_next.md %}

data/docs/_docs/extras/load-balancer.md

@@ -1,6 +1,6 @@
 ---
 title: Load Balancer Support
-nav_order: 24
+nav_order: 25
 ---
 
 Ufo can automatically create a load balancer and associate it with an ECS service.  The options:
@@ -80,4 +80,4 @@ Under the hood, ufo implements load balancer support with CloudFormation. You ca
 
 <img src="/img/docs/cloudformation-resources.png" class="doc-photo" />
 
-{% include prev_next.md %}
+{% include prev_next.md %}

data/docs/_docs/extras/minimal-deploy-iam.md

@@ -1,6 +1,6 @@
 ---
 title: Minimal Deploy IAM Policy
-nav_order: 30
+nav_order: 31
 ---
 
 The IAM user you use to run the `ufo ship` command needs a minimal set of IAM policies in order to deploy to ECS. Here is a table of the baseline services needed:
@@ -76,4 +76,4 @@ If you are using CodeBuild to deploy, you'll probably be interested the IAM poli
 This page refers to your **user** IAM policy used when running `ufo ship`. These are different from the IAM Policies associated with ECS Task.  For those iam policies refer to [IAM Roles for Tasks
 ](https://docs.aws.amazon.com/AmazonECS/latest/developerguide/task-iam-roles.html).
 
-{% include prev_next.md %}
+{% include prev_next.md %}

data/docs/_docs/extras/notification-arns.md

@@ -0,0 +1,21 @@
+---
+title: Notification ARNs
+categories: extras
+nav_order: 99
+---
+
+You can specific notification arns for CloudFormation stack related events with [configs/settings.yml]({% link _docs/settings.md %}). This may be useful for compliance purposes.
+
+## Example
+
+configs/settings.yml
+
+```yaml
+base:
+  notification_arns:
+  - arn:aws:sns:us-west-2:112233445566:my-sns-topic1
+```
+
+This will set the `notification_arns` option as the CloudFormation stack created by `ufo ship`.
+
+{% include prev_next.md %}

data/docs/_docs/extras/redirection-support.md

@@ -1,6 +1,6 @@
 ---
 title: Redirection Support
-nav_order: 29
+nav_order: 30
 ---
 
 ## Application Load Balancers
@@ -24,4 +24,4 @@ listener:
 
 Network Load Balancers work at layer 4, so they do not support redirection.  Instead you need to handle redirection within your app.
 
-{% include prev_next.md %}
+{% include prev_next.md %}

data/docs/_docs/extras/route53-support.md

@@ -1,6 +1,6 @@
 ---
 title: Route53 Support
-nav_order: 28
+nav_order: 29
 ---
 
 Ufo can create a "pretty" route53 record and set it's value to the created ELB DNS name. This is done by configuring the `.ufo/settings/cfn/default.yml` file. Example:
@@ -24,4 +24,4 @@ Results in:
     aws route53 create-hosted-zone --name mydomain.com --caller-reference $(date +%s)
     aws route53 list-hosted-zones
 
-{% include prev_next.md %}
+{% include prev_next.md %}

data/docs/_docs/extras/security-groups.md

@@ -1,6 +1,6 @@
 ---
 title: Security Groups
-nav_order: 25
+nav_order: 26
 ---
 
 Ufo creates and manages two security groups. One for the ELB and one for the ECS tasks.
@@ -33,4 +33,4 @@ In general, ports below 32768 are outside of the ephemeral port range. So an eas
 
 If you are using a network load balancer and are running bridge network mode, then you need to whitelist ports 32768 to 65535 to `0.0.0.0/0`.  This is because network load balancers operate at layer 4 of the OSI model and cannot be assigned security groups, so they use the security group of the instance.  If you feel this is too loose of permissions, you can use awsvpc mode. There are some considerations for awsvpc mode though which is discussed next.
 
-{% include prev_next.md %}
+{% include prev_next.md %}

data/docs/_docs/extras/ssl-support.md

@@ -1,6 +1,6 @@
 ---
 title: SSL Support
-nav_order: 27
+nav_order: 28
 ---
 
 You can configure SSL support by uncomment the `listener_ssl` option in `.ufo/settings/cfn/default.yml`.  Here's an example:
@@ -17,4 +17,4 @@ For the certificate arn, you will need to create a certificate with AWS ACM. To
 
 The protocol will be either HTTP or HTTPS for Application Load Balancers and TCP or TLS for Network Load Balancers. Ufo will infer the right value, so you usually don't have to configure the protocol manually.  You can configure it if required though.
 
-{% include prev_next.md %}
+{% include prev_next.md %}

data/docs/_docs/faq.md

@@ -1,6 +1,6 @@
 ---
 title: FAQ
-nav_order: 44
+nav_order: 45
 ---
 
 **Q: Is AWS ECS Fargate supported?**
@@ -97,4 +97,4 @@ Also, you might have to enable the log driver by adding the ECS_AVAILABLE_LOGGIN
 
 Hope that helps.
 
-{% include prev_next.md %}
+{% include prev_next.md %}

data/docs/_docs/helpers.md

@@ -9,14 +9,16 @@ For example, one of the helper methods provides the exposed port in the Dockerfi
 
 Helper  | Description
 ------------- | -------------
-full\_image\_name | The full docker image name that ufo builds. The "base" portion of the docker image name is defined in `settings.yml`. For example, the base portion is `tongueroo/demo-ufo` and the full image name is `tongueroo/demo-ufo:ufo-[timestamp]-[sha]`. The base name does not include the generated Docker tag, which contains a timestamp and git sha of the project.
-dockerfile\_port | Exposed port extracted from the Dockerfile of the project. 
-env_vars(text) | This method takes a block of text that contains the env values in `key=value` format and converts that block of text to the proper task definition JSON format.
-env_file(path) | This method takes a `.env` file which contains a simple key-value list of environment variables and converts the list to the proper task definition JSON format.
+full\_image\_name | The full docker image name that ufo builds. The "base" portion of the docker image name is defined in `settings.yml`. For example, the base portion is `tongueroo/demo-ufo` and the full image name is `tongueroo/demo-ufo:ufo-[timestamp]-[sha]`. The base name does not include the generated Docker tag, which contains a timestamp and git sha of the project.
+dockerfile\_port | Exposed port extracted from the Dockerfile of the project.
+env_vars(text) | This method takes a block of text that contains the env values in `key=value` format and converts that block of text to the proper task definition JSON format.
+env_file(path) | This method takes a `.env` file which contains a simple key-value list of environment variables and converts the list to the proper task definition JSON format.
+secrets_vars(text) | This method takes a block of text that contains the secrets values in `key=value` format and converts that block of text to the proper task definition JSON format.
+secrets_file(path) | This method takes a `.secrets` file which contains a simple key-value list of environment variables and converts the list to the proper task definition JSON format.
 task_definition_name | The name of the task_definition.  So if the code looks like this `task_definition "demo-web" do`, the task_definition_name is "demo-web".
 
 To call the helper in task_definitions.rb you must add `helper.` in front.  So `full_image_name` is called via `helper.full_image_name`.
 
 The 2 classes which provide these special helper methods are in [ufo/dsl.rb](https://github.com/tongueroo/ufo/blob/master/lib/ufo/dsl.rb) and [ufo/dsl/helper.rb](https://github.com/tongueroo/ufo/blob/master/lib/ufo/dsl/helper.rb). Refer to these classes for the full list of the helper methods.
 
-{% include prev_next.md %}
+{% include prev_next.md %}

data/docs/_docs/iam-roles.md

@@ -0,0 +1,111 @@
+---
+title: Task Definition IAM Roles
+---
+
+## What are ECS IAM Roles?
+
+For ECS Task Definitions, you can assign it 2 IAM roles: 1) taskRoleArn and 2) executionRoleArn. It's usually defined in the JSON structure like so:
+
+```json
+{
+  "family": "..",
+  "taskRoleArn": "...",
+  "executionRoleArn": "...",
+  "containerDefinitions": [
+    ...
+  ]
+}
+```
+
+Here's a table that explains the difference between the 2 IAM roles.
+
+Name | Purpose
+--- | ---
+taskRoleArn | This is the role that the ECS task itself uses. So this is what IAM permissions your application has access to. Think about it as the "container role".
+executionRoleArn | This is the role that the EC2 instance host uses. This allows the EC2 instance to pull from the ECR registry. Think about it as the "host role".
+
+## How to Assign IAM Roles with UFO
+
+You can assign an IAM role to the ECS Task definition in ways:
+
+1. IAM Role with Code (UFO Managed)
+2. Precreated IAM Role
+
+## IAM Role with Code (UFO Managed)
+
+UFO can automatically create the IAM and assign it to the task definition. You create these files so UFO will know to create and manage the IAM roles.
+
+    .ufo/iam_roles/execution_role.rb
+    .ufo/iam_roles/task_role.rb
+
+### Example 1
+
+You then use a DSL to create the IAM roles. Here are examples:
+
+.ufo/iam_roles/execution_role.rb
+
+```ruby
+managed_iam_policy("AmazonEC2ContainerRegistryReadOnly")
+managed_iam_policy("AmazonSSMReadOnlyAccess")
+managed_iam_policy("CloudWatchLogsFullAccess")
+managed_iam_policy("SecretsManagerReadWrite")
+managed_iam_policy("service-role/AmazonECSTaskExecutionRolePolicy")
+```
+
+.ufo/iam_roles/task_role.rb
+
+```ruby
+iam_policy("AmazonS3ReadOnlyAccess",
+  Action: [
+    "s3:Get*",
+    "s3:List*"
+  ],
+  Effect: "Allow",
+  Resource: "*"
+)
+iam_policy("CloudwatchWrite",
+  Action: [
+    "cloudwatch:PutMetricData",
+  ],
+  Effect: "Allow",
+  Resource: "*"
+)
+```
+
+### Example 2
+
+You can use the `managed_iam_policy` and `iam_policy` together. You can also group multiple statements in the `iam_policy` declaration.
+
+.ufo/iam_roles/task_role.rb
+
+```ruby
+managed_iam_policy("AmazonSSMManagedInstanceCore")
+
+iam_policy("custom-policy", [
+  {
+    Action: "ecs:UpdateContainerInstancesState",
+    Resource: "*",
+    Effect: "Allow"
+  },
+  {
+    Action: "sns:Publish",
+    Resource: "*",
+    Effect: "Allow"
+  }
+])
+```
+
+## Pre-Created IAM Role
+
+You can also assign the task definition `executionRoleArn` with pre-created IAM roles. It looks something like this in the `.ufo/templates/main.json.erb` file:
+
+```json
+{
+  "family": "<%= @family %>",
+  "taskRoleArn": "arn:aws:iam::112233445566:role/pre-created-iam-role",
+  "executionRoleArn": "arn:aws:iam::112233445566:role/pre-created-iam-role",
+  "containerDefinitions": [
+    ...
+  ]
+}
+```

data/docs/_docs/install.md

@@ -32,4 +32,4 @@ For more information about the Bolts Toolbelt or to get an installer for another
 * Docker: You will need a working version of [Docker](https://docs.docker.com/engine/installation/) installed as ufo shells out and calls the `docker` command.
 * AWS: Set up your AWS credentials at `~/.aws/credentials` and `~/.aws/config`.  This is the [AWS standard way of setting up credentials](https://aws.amazon.com/blogs/security/a-new-and-standardized-way-to-manage-credentials-in-the-aws-sdks/).
 
-{% include prev_next.md %}
+{% include prev_next.md %}

data/docs/_docs/more/auto-completion.md

@@ -1,6 +1,6 @@
 ---
 title: Auto Completion
-nav_order: 43
+nav_order: 44
 ---
 
 Ufo supports bash auto-completion.  To set it up add the following to your `~/.profile` or `.bashrc`:
@@ -21,4 +21,4 @@ Auto Completion examples:
     ufo tasks [TAB]
     ufo tasks build [TAB]
 
-{% include prev_next.md %}
+{% include prev_next.md %}

data/docs/_docs/more/automated-cleanup.md

@@ -1,6 +1,6 @@
 ---
 title: Automated Clean Up
-nav_order: 42
+nav_order: 43
 ---
 
 Ufo can be configured to automatically clean old images from the ECR registry after the deploy completes by configuring your [settings.yml]({% link _docs/settings.md %}) file like so:
@@ -11,4 +11,4 @@ ecr_keep: 30
 
 Automated Docker images clean up only works if you are using ECR registry.
 
-{% include prev_next.md %}
+{% include prev_next.md %}

data/docs/_docs/more/customize-cloudformation.md

@@ -1,6 +1,6 @@
 ---
 title: Customize CloudFormation
-nav_order: 37
+nav_order: 38
 ---
 
 Under the hood, ufo creates most of the required resources with a CloudFormation stack.  This includes the ELB, Target Group, Listener, Security Groups, ECS Service, and Route 53 records.  You might need to customize these resources.  Here are the ways to customize the resources that ufo creates.
@@ -32,4 +32,4 @@ UFO_ENV_EXTRA=2 ufo ship demo-web -\-cluster dev | demo-web-development-2
 
 The CloudFormation stack is currently generated from a template. The source code for this template is located at [cfn/stack.yml](https://github.com/tongueroo/ufo/blob/master/lib/cfn/stack.yml).  This implementation might change in the future.
 
-{% include prev_next.md %}
+{% include prev_next.md %}

data/docs/_docs/more/migrations.md

@@ -1,6 +1,6 @@
 ---
 title: Database Migrations
-nav_order: 41
+nav_order: 42
 ---
 
 A common task is to run database migrations with newer code before deploying the code. This is easily achieved with the `ufo task` command. Here's an example:
@@ -22,4 +22,4 @@ The `ufo task` command is generalized so you can run any one-off task. It is not
 2. Registers the ECS Task definition
 3. Runs a one-off ECS Task
 
-{% include prev_next.md %}
+{% include prev_next.md %}

data/docs/_docs/more/run-in-pieces.md

@@ -1,6 +1,6 @@
 ---
 title: Run in Pieces
-nav_order: 39
+nav_order: 40
 ---
 
 The `ufo ship` command goes through a few stages:
@@ -27,4 +27,4 @@ Update the service with the task definitions in `.ufo/output` untouched.
 
 Note if you use the `ufo deploy` you should ensure that you have already pushed the docker image to your docker registry.  Or else the task will not be able to spin up because the docker image does not exist.  This is one of the reasons it is recommended that you use `ufo ship`.
 
-{% include prev_next.md %}
+{% include prev_next.md %}

data/docs/_docs/more/single-task.md

@@ -1,6 +1,6 @@
 ---
 title: Run Single Task
-nav_order: 40
+nav_order: 41
 ---
 
 Sometimes you do not want to run a long running `service` but a one time task. Running Rails migrations are an example of a one off task.  Here is an example of how you would run a one time task.
@@ -22,4 +22,4 @@ You can describe that task for more details:
 
 You can check out the [ufo task](http://ufoships.com/reference/ufo-task/) reference for more details.
 
-{% include prev_next.md %}
+{% include prev_next.md %}