tfctl 1.0.0

data/docs/configuration.adoc

@@ -0,0 +1,89 @@
+// Settings:
+:idprefix:
+:idseparator: -
+ifndef::env-github[:icons: font]
+ifdef::env-github,env-browser[]
+:toc: macro
+:toclevels: 1
+endif::[]
+ifdef::env-github[]
+:branch: master
+:status:
+:outfilesuffix: .adoc
+:!toc-title:
+:caution-caption: :fire:
+:important-caption: :exclamation:
+:note-caption: :paperclip:
+:tip-caption: :bulb:
+:warning-caption: :warning:
+endif::[]
+
+= Configuration
+
+toc::[]
+
+== Overview
+
+Tfctl retrieves initial account configuration from AWS Organizations and merges
+it with configuration specified in a yaml file.
+
+The configuration is merged in the following order:
+
+* AWS Organizations data is fetched and stored in an `accounts` array.
+* `organization_root` settings are merged with all accounts.
+* `organization_units` settings are merged with accounts matching the OU.
+* `account_overrides` are merged with individual accounts matching the account name.
+
+Parameters further down the hierarchy take precedence.  For example:
+
+[source, yaml]
+----
+organization_root:
+  data:
+    example_param: 'will be overriden further down'
+
+organization_units:
+  team:
+    data:
+      example_param: 'will win in team ou'
+  team/live:
+    data:
+      example_param: 'will win in team/live ou'
+----
+
+One exception to this rule is the `profiles` parameter.  Profiles are additive:
+
+[source, yaml]
+----
+organization_root:
+  profiles:
+   - profile-one
+   - profile-two
+
+organization_units:
+  team:
+    profiles:
+      - profile-three
+----
+
+This will result in all three profiles deployed to accounts in `team` OU.
+
+TIP: You can display the fully merged configuration by running `tfctl -c
+conf/CONFIG_FILE.yaml -s`.  It's safe to run as it doesn't make any changes to
+AWS resources.  It's a good way to test your configuration.
+
+== Defining arbitrary data
+
+You can define arbitrary data under the `data:` parameter, both in the root of
+the config and in the organization sections.  It will be available in Terraform
+profiles to use by your modules.  You can use this to define things like VPC
+subnet ranges, s3 bucket names and so on.  `data:` in organization sections
+will be merged with accounts following the usual merge order as described
+above.
+
+== Handling secrets
+
+No secrets should be committed into Terraform or tfctl configuration.  Use AWS
+Secrets Manager instead and retrieve in Terraform profiles using
+https://www.terraform.io/docs/providers/aws/d/secretsmanager_secret.html[secrets
+manager data source]

data/docs/control_tower.adoc

@@ -0,0 +1,211 @@
+// Settings:
+:idprefix:
+:idseparator: -
+ifndef::env-github[:icons: font]
+ifdef::env-github,env-browser[]
+:toc: macro
+:toclevels: 1
+endif::[]
+ifdef::env-github[]
+:branch: master
+:status:
+:outfilesuffix: .adoc
+:!toc-title:
+:caution-caption: :fire:
+:important-caption: :exclamation:
+:note-caption: :paperclip:
+:tip-caption: :bulb:
+:warning-caption: :warning:
+endif::[]
+
+= Control Tower integration guide
+
+This guide will help you integrate Terraform with AWS Control Tower using the
+tfctl wrapper.  This involves setting up resources for remote state tracking,
+necessary IAM roles and a tfctl project.
+
+toc::[]
+
+== Overview
+
+For state tracking we're going to create a dedicated `shared-services` account
+under a `mgmt` organization unit.  We'll use S3 for state storage and DynamoDB
+for locking.  `TerraformState` IAM role will be created for cross account
+access to state resources from the primary account.
+
+In the primary account we'll create a `TfctlOrgAccess` role.  It gives tfctl
+read only access to AWS Organizations which is used to discover accounts and
+the organization unit structure.
+
+We'll use CloudFormation stacks and stack-sets to bootstrap these resources.
+
+For executing Terraform in spoke accounts we'll use the
+`AWSControlTowerExecution` role which is automatically created by Control Tower
+account factory and can be assumed from the primary account.
+
+We're going to create a `live` and `test` organization units in Control Tower
+and provision a couple of accounts for testing.
+
+== Prerequisites
+
+Before starting you'll need:
+
+ * Control Tower set up in your primary account.
+ * A user with `AdministratorAccess` privileges in primary account.
+ * AWS CLI tools installed on your machine.
+ * Terraform 0.12 or higher.
+
+== Configure Control Tower
+
+Create the following organization units in Control Tower:
+
+ * `mgmt`
+ * `live`
+ * `test`
+
+Then provision accounts:
+
+ * In `mgmt` OU create an account called `mgmt-shared-services`
+ * In `live` create `live-example1`
+ * In `test` create `test-example1`
+
+NOTE: Control Tower accounts need to be provisioned one at a time.  It takes
+approximately 20 mins to provision one.
+
+== Install tfctl
+
+----
+git clone git@github.com:scalefactory/tfctl.git
+cd tfctl/ && sudo make install
+----
+
+== Set up AWS resources
+
+It's assumed you have configured AWS CLI access to the primary account.
+
+We'll use CloudFormation templates in `examples/bootstrap/`.
+
+First export configuration using environment variables making sure to change to
+values to suit your set up:
+
+----
+export PRIMARY_ACCOUNT_ID=11111111
+export SHARED_SERVICES_ACCOUNT_ID=22222222
+export STATE_BUCKET_NAME='example-terraform-state'
+----
+
+Create the remote state resources stack set:
+
+----
+cd examples/bootstrap/
+
+aws cloudformation create-stack-set \
+    --stack-set-name TerraformState \
+    --template-body file://terraform-state.template \
+    --description "Resources for managing Terraform state" \
+    --capabilities CAPABILITY_NAMED_IAM CAPABILITY_IAM \
+    --execution-role-name AWSControlTowerExecution \
+    --administration-role-arn arn:aws:iam::${PRIMARY_ACCOUNT_ID}:role/service-role/AWSControlTowerStackSetRole \
+    --parameters ParameterKey=PrimaryAccountId,ParameterValue=${PRIMARY_ACCOUNT_ID} \
+    ParameterKey=TerraformStateBucket,ParameterValue=${STATE_BUCKET_NAME}
+----
+
+Create a stack set instance in you shared services account:
+
+----
+aws cloudformation create-stack-instances \
+    --stack-set-name TerraformState \
+    --accounts ${SHARED_SERVICES_ACCOUNT_ID} \
+    --regions eu-west-1
+----
+
+Check status:
+
+----
+aws cloudformation describe-stack-instance \
+    --stack-set-name TerraformState \
+    --stack-instance-account ${SHARED_SERVICES_ACCOUNT_ID} \
+    --stack-instance-region eu-west-1
+----
+
+NOTE: Initial status will be `OUTDATED`, it should change to `CURRENT` once deployed.
+
+Deploy `TfctlOrgAccess` IAM role stack:
+
+----
+aws cloudformation create-stack \
+    --stack-name TfctlOrgAccess \
+    --template-body file://tfctl-org-access.template \
+    --capabilities CAPABILITY_NAMED_IAM CAPABILITY_IAM \
+    --parameters ParameterKey=PrimaryAccountId,ParameterValue=${PRIMARY_ACCOUNT_ID}
+----
+
+Check status:
+
+----
+aws cloudformation describe-stacks --stack-name TfctlOrgAccess
+----
+
+NOTE: Successful status should read: `CREATE_COMPLETE`.
+
+== Configure tfctl
+
+Copy the example project directory `examples/control_tower` somewhere convenient
+and edit `conf/example.yaml`.
+
+You need to modify the following parameters:
+
+ * `tf_state_bucket` - set to `$STATE_BUCKET_NAME`
+ * `tf_state_role_arn` - set shared services account ID
+ * `tfctl_role_arn` - set primary account ID
+ * `primary_account` - set the primary account name.  You can find it in AWS Organizations.
+
+TIP: You should keep your project directory under version control.
+
+== Deploy example tfctl profile
+
+The example profile will create an S3 bucket in accounts under `test`, `live`
+and `mgmt` OUs.
+
+NOTE: Run tfctl commands from the root of you project directory.
+
+First dump the configuration to verify everything works:
+
+----
+tfctl -c conf/example.yaml -s
+----
+
+This will not make any changes but will print out a yaml containing the final,
+merged configuration data.  It should contain a list of discovered accounts and
+their configuration.
+
+Initialise terraform for all discovered accounts:
+
+----
+tfctl -c conf/example.yaml --all -- init
+----
+
+Tfctl will run Terraform against all accounts in parallel.
+
+Run plan:
+
+----
+tfctl -c conf/example.yaml --all -- plan
+----
+
+and apply:
+
+----
+tfctl -c conf/example.yaml --all -- apply
+----
+
+To destroy created resources run:
+
+----
+tfctl -c conf/example.yaml --all -- destroy -auto-approve
+----
+
+That's it! You can now execute terraform across your Control Tower estate.
+
+TIP: Your project directory should be under version control excluding the
+`.tfctl` directory which is automatically generated.

data/docs/creating_a_profile.adoc

@@ -0,0 +1,191 @@
+// Settings:
+:idprefix:
+:idseparator: -
+ifndef::env-github[:icons: font]
+ifdef::env-github,env-browser[]
+:toc: macro
+:toclevels: 1
+endif::[]
+ifdef::env-github[]
+:branch: master
+:status:
+:outfilesuffix: .adoc
+:!toc-title:
+:caution-caption: :fire:
+:important-caption: :exclamation:
+:note-caption: :paperclip:
+:tip-caption: :bulb:
+:warning-caption: :warning:
+endif::[]
+
+= Creating and deploying a tfctl profile
+
+This guide will show you how to create a tfctl profile, declare some resources
+in it and deploy it to to a group of accounts in an organization unit.
+
+toc::[]
+
+== Create a new profile
+
+In your tfctl project directory create a new profile:
+
+----
+mkdir profiles/example-profile
+----
+
+Withing the profile create `data.tf`:
+
+.data.tf
+[source, tf]
+----
+data "aws_caller_identity" "current" {}
+----
+
+This file contains Terraform
+https://www.terraform.io/docs/configuration/data-sources.html[data source]
+declarations.  Data sources are a way of getting data not directly managed in
+Terraform into Terraform.  In this case we're using the
+https://www.terraform.io/docs/providers/aws/d/caller_identity.html[aws_caller_identity]
+.  One of the outputs of this source is `account_id` which will
+return the id of the account Terraform is currently running in.
+
+Now create `variables.tf`:
+
+.variables.tf
+[source, tf]
+----
+variable "config" {
+  description = "Configuration generated by tfctl in string encoded JSON"
+  type = string
+}
+
+# local variables
+locals {
+  # Decode config JSON into a Terraform data structure
+  config = jsondecode(var.config)
+
+  # Get current account id from aws_caller_identity data source
+  current_account_id = "${data.aws_caller_identity.current.account_id}"
+
+  # Get tfctl configuration for the current account
+  current_account_conf = [ for account in local.config["accounts"]: account if account["id"] == local.current_account_id ][0]
+}
+----
+
+This file contains
+https://www.terraform.io/docs/configuration/variables.html[input variables] for
+the profile.
+
+The `config` variable is special and must always be declared in a tfctl
+profile.  Tfctl configuration can be accessed using this variable. This It
+includes an array of all discovered accounts as well their parameters from
+tfctl config file.
+
+TIP: You can run `tfctl -c conf/CONFIG_FILE.yaml -s` to show the config data in
+yaml format.  This exact data is available in the `config` variable in your
+profile.
+
+We also have few https://www.terraform.io/docs/configuration/locals.html[local
+variables] in the `locals` block.  We assign the current account id from the
+data source we defined previously to `current_account_id`.  This is mainly for
+convenience to make the next statement easier to read.  `current_account` loops
+over the `config` data and returns configuration for an account which matches
+the current account id (i.e. the current account configuration).
+
+Now that we have our data inputs sorted we can start declaring actual AWS
+resources to manage.
+
+Create `main.tf`:
+
+.main.tf
+[source, tf]
+----
+resource "aws_s3_bucket" "example" {
+  bucket = "tfctl-${local.current_account_conf["name"]}"
+  acl    = "private"
+}
+----
+
+This will create an S3 bucket with a name containing the current account name
+(which will vary depending on which account it's deployed to).
+
+== Assign profile to accounts
+
+Before you can deploy the new profile you need to tell `tfctl` which accounts
+to deploy it to.
+
+You have few options here:
+
+* deploy to all accounts
+* deploy to specific organization unit (OU)
+* deploy to individual account
+
+
+For the sake of this example we're going to deploy our bucket to all accounts
+in `test` OU.
+
+In tfctl config file add the profile to the `test` OU:
+
+[source, yaml]
+----
+organization_units:
+  test:
+    profiles:
+      - example-profile
+----
+
+
+== Plan
+
+To see what would happen when the change is applied run:
+
+----
+tfctl -c conf/example.yaml -o test -- init
+tfctl -c conf/example.yaml -o test -- plan
+----
+
+This will run `terraform init` to initialise terraform and then `terraform
+plan` across all accounts in the `test` OU in parallel.  It will display a diff
+of changes for each account.
+
+.example terraform plan
+----
+info: test-example: Terraform will perform the following actions:
+info: test-example:
+info: test-example:   # module.example-profile.aws_s3_bucket.example will be created
+info: test-example:   + resource "aws_s3_bucket" "example" {
+info: test-example:       + acceleration_status         = (known after apply)
+info: test-example:       + acl                         = "private"
+info: test-example:       + arn                         = (known after apply)
+info: test-example:       + bucket                      = "tfctl-test-example"
+info: test-example:       + bucket_domain_name          = (known after apply)
+info: test-example:       + bucket_regional_domain_name = (known after apply)
+info: test-example:       + force_destroy               = false
+info: test-example:       + hosted_zone_id              = (known after apply)
+info: test-example:       + id                          = (known after apply)
+info: test-example:       + region                      = (known after apply)
+info: test-example:       + request_payer               = (known after apply)
+info: test-example:       + website_domain              = (known after apply)
+info: test-example:       + website_endpoint            = (known after apply)
+info: test-example:
+info: test-example:       + versioning {
+info: test-example:           + enabled    = (known after apply)
+info: test-example:           + mfa_delete = (known after apply)
+info: test-example:         }
+info: test-example:     }
+info: test-example:
+info: test-example: Plan: 1 to add, 0 to change, 0 to destroy.
+----
+
+If there are errors in your profile, terraform will fail and usually indicate
+what went wrong.
+
+tfctl will generate a plan file automatically and use it with `apply` in the
+next step.
+
+== Apply
+
+Once you're happy with the plan, apply it.
+----
+tfctl -c conf/example.yaml -o test -- apply
+----