tfctl 0.0.2

data/docs/creating_a_profile.adoc

@@ -0,0 +1,169 @@
+== Creating and deploying a tfctl profile
+
+This guide will show you how to create a tfctl profile, declare some resources
+in it and deploy it to to a group of accounts in an organization unit.
+
+=== Create a new profile
+
+In your tfctl project directory create a new profile:
+
+----
+mkdir profiles/example-profile
+----
+
+Withing the profile create `data.tf`:
+
+.data.tf
+[source, tf]
+----
+data "aws_caller_identity" "current" {}
+----
+
+This file contains Terraform
+https://www.terraform.io/docs/configuration/data-sources.html[data source]
+declarations.  Data sources are a way of getting data not directly managed in
+Terraform into Terraform.  In this case we're using the
+https://www.terraform.io/docs/providers/aws/d/caller_identity.html[aws_caller_identity]
+.  One of the outputs of this source is `account_id` which will
+return the id of the account Terraform is currently running in.
+
+Now create `variables.tf`:
+
+.variables.tf
+[source, tf]
+----
+variable "config" {
+  description = "Configuration generated by tfctl in string encoded JSON"
+  type = string
+}
+
+# local variables
+locals {
+  # Decode config JSON into a Terraform data structure
+  config = jsondecode(var.config)
+
+  # Get current account id from aws_caller_identity data source
+  current_account_id = "${data.aws_caller_identity.current.account_id}"
+
+  # Get tfctl configuration for the current account
+  current_account_conf = [ for account in local.config["accounts"]: account if account["id"] == local.current_account_id ][0]
+}
+----
+
+This file contains
+https://www.terraform.io/docs/configuration/variables.html[input variables] for
+the profile.
+
+The `config` variable is special and must always be declared in a tfctl
+profile.  Tfctl configuration can be accessed using this variable. This It
+includes an array of all discovered accounts as well their parameters from
+tfctl config file.
+
+TIP: You can run `tfctl -c conf/CONFIG_FILE.yaml -s` to show the config data in
+yaml format.  This exact data is available in the `config` variable in your
+profile.
+
+We also have few https://www.terraform.io/docs/configuration/locals.html[local
+variables] in the `locals` block.  We assign the current account id from the
+data source we defined previously to `current_account_id`.  This is mainly for
+convenience to make the next statement easier to read.  `current_account` loops
+over the `config` data and returns configuration for an account which matches
+the current account id (i.e. the current account configuration).
+
+Now that we have our data inputs sorted we can start declaring actual AWS
+resources to manage.
+
+Create `main.tf`:
+
+.main.tf
+[source, tf]
+----
+resource "aws_s3_bucket" "example" {
+  bucket = "tfctl-${local.current_account_conf["name"]}"
+  acl    = "private"
+}
+----
+
+This will create an S3 bucket with a name containing the current account name
+(which will vary depending on which account it's deployed to).
+
+=== Assign profile to accounts
+
+Before you can deploy the new profile you need to tell `tfctl` which accounts
+to deploy it to.
+
+You have few options here:
+
+* deploy to all accounts
+* deploy to specific organization unit (OU)
+* deploy to individual account
+
+
+For the sake of this example we're going to deploy our bucket to all accounts
+in `test` OU.
+
+In tfctl config file add the profile to the `test` OU:
+
+[source, yaml]
+----
+organization_units:
+  test:
+    profiles:
+      - example-profile
+----
+
+
+=== Plan
+
+To see what would happen when the change is applied run:
+
+----
+tfctl -c conf/example.yaml -o test -- init
+tfctl -c conf/example.yaml -o test -- plan
+----
+
+This will run `terraform init` to initialise terraform and then `terraform
+plan` across all accounts in the `test` OU in parallel.  It will display a diff
+of changes for each account.
+
+.example terraform plan
+----
+info: test-example: Terraform will perform the following actions:
+info: test-example:
+info: test-example:   # module.example-profile.aws_s3_bucket.example will be created
+info: test-example:   + resource "aws_s3_bucket" "example" {
+info: test-example:       + acceleration_status         = (known after apply)
+info: test-example:       + acl                         = "private"
+info: test-example:       + arn                         = (known after apply)
+info: test-example:       + bucket                      = "tfctl-test-example"
+info: test-example:       + bucket_domain_name          = (known after apply)
+info: test-example:       + bucket_regional_domain_name = (known after apply)
+info: test-example:       + force_destroy               = false
+info: test-example:       + hosted_zone_id              = (known after apply)
+info: test-example:       + id                          = (known after apply)
+info: test-example:       + region                      = (known after apply)
+info: test-example:       + request_payer               = (known after apply)
+info: test-example:       + website_domain              = (known after apply)
+info: test-example:       + website_endpoint            = (known after apply)
+info: test-example:
+info: test-example:       + versioning {
+info: test-example:           + enabled    = (known after apply)
+info: test-example:           + mfa_delete = (known after apply)
+info: test-example:         }
+info: test-example:     }
+info: test-example:
+info: test-example: Plan: 1 to add, 0 to change, 0 to destroy.
+----
+
+If there are errors in your profile, terraform will fail and usually indicate
+what went wrong.
+
+tfctl will generate a plan file automatically and use it with `apply` in the
+next step.
+
+=== Apply
+
+Once you're happy with the plan, apply it.
+----
+tfctl -c conf/example.yaml -o test -- apply
+----

data/docs/iam_permissions.adoc

@@ -0,0 +1,18 @@
+== IAM roles
+
+Tfctl usually requires three IAM roles to be configured:
+
+ * `TfctlRole` - read only access to AWS Organizations set up in the primary account.
+ * `TerraformStateRole` - access to remote state resources (S3, DynamoDB) in the
+    account where your state is stored (can be any account).
+ * `TerraformExecutionRole` - configured in all spoke accounts and used for executing Terraform.
+
+The user executing tfctl needs permission to assume all three roles cross
+account.  Tfctl will assume roles automatically for you.
+
+It's possible to configure different Terraform execution roles in different
+spoke accounts based on OU or account names.  This can be used to restrict
+Terraform in certain accounts.
+
+We usually set those roles up using CloudFormation as part of the bootstrapping
+process.  See example templates in `examples/bootstrap/`.

data/docs/project_layout.adoc

@@ -0,0 +1,43 @@
+== Project layout
+
+Example project structure
+----
+project_dir/
+├── conf
+│   └── example.yaml
+├── modules
+│   └── s3-bucket
+│       ├── main.tf
+│       └── variables.tf
+└── profiles
+    └── example-profile
+        ├── data.tf
+        ├── main.tf
+        └── variables.tf
+----
+
+=== tfctl configuration file
+
+Assigns Terraform profiles and configuration to accounts based on:
+
+* Globally for all accounts
+* Account's organization unit
+* Individual accounts
+
+The configuration data is exposed to terraform via a profile `config` variable.
+
+It also defines Terraform and tfctl configuration such as state tracking and
+what IAM roles to use.
+
+=== profiles
+
+Profiles are re-usable collections of resources which can be applied to
+accounts.  They are implemented just like usual modules but provide an
+intermediate bridge between re-usable modules and tfctl configuration (and/or
+other data sources).  Profiles often compose multiple modules and provide
+configuration data to them.  This approach makes it possible to re-use standard
+modules (e.g. Terraform module registry).
+
+=== modules
+
+Standard Terraform modules.

data/examples/bootstrap/terraform-exec-role.template

@@ -0,0 +1,24 @@
+AWSTemplateFormatVersion: 2010-09-09
+Description: Configure Terraform execution role in spoke accounts
+
+Parameters:
+  PrimaryAccountId:
+    Type: String
+
+Resources:
+  TerraformExecutionRole:
+    Type: AWS::IAM::Role
+    Properties:
+      RoleName: TerraformExecutionRole
+      AssumeRolePolicyDocument:
+        Version: 2012-10-17
+        Statement:
+          - Effect: Allow
+            Principal:
+              AWS:
+                - !Sub 'arn:aws:iam::${PrimaryAccountId}:root'
+            Action:
+              - sts:AssumeRole
+      Path: /
+      ManagedPolicyArns:
+        - arn:aws:iam::aws:policy/AdministratorAccess

data/examples/bootstrap/terraform-state.template

@@ -0,0 +1,81 @@
+AWSTemplateFormatVersion: 2010-09-09
+Description: Resources for managing Terraform state
+
+Parameters:
+
+  PrimaryAccountId:
+    Type: String
+    Description: "Primany account ID"
+  TerraformStateBucket:
+    Type: String
+    Description: "Name of S3 bucket used for storing Terraform state"
+  TerraformDynamoDbTable:
+    Type: String
+    Description: "Name of DynamoDB table used for Terraform state locking"
+    Default: "terraform-lock"
+
+Resources:
+
+  StateBucket:
+    Type: AWS::S3::Bucket
+    Properties:
+      AccessControl: Private
+      BucketName: !Ref TerraformStateBucket
+      VersioningConfiguration:
+        Status: Enabled
+      BucketEncryption:
+        ServerSideEncryptionConfiguration:
+          - ServerSideEncryptionByDefault:
+              SSEAlgorithm: AES256
+      PublicAccessBlockConfiguration:
+        BlockPublicAcls: True
+        BlockPublicPolicy: True
+        IgnorePublicAcls: True
+        RestrictPublicBuckets: True
+
+  DynamoLockTable:
+    Type: AWS::DynamoDB::Table
+    Properties:
+      TableName: !Ref TerraformDynamoDbTable
+      AttributeDefinitions:
+        - AttributeName: LockID
+          AttributeType: S
+      KeySchema:
+        - AttributeName: LockID
+          KeyType: HASH
+      ProvisionedThroughput:
+        ReadCapacityUnits: 5
+        WriteCapacityUnits: 5
+
+  TerraformStateRole:
+    Type: AWS::IAM::Role
+    Properties:
+      RoleName: TerraformStateRole
+      AssumeRolePolicyDocument:
+        Version: 2012-10-17
+        Statement:
+          - Effect: Allow
+            Principal:
+              AWS:
+                - !Sub 'arn:aws:iam::${PrimaryAccountId}:root'
+            Action:
+              - sts:AssumeRole
+      Path: /
+      Policies:
+        - PolicyName: "terraform-state"
+          PolicyDocument:
+            Version: "2012-10-17"
+            Statement:
+              - Effect: "Allow"
+                Action:
+                  - "s3:PutObject"
+                  - "s3:GetBucketPolicy"
+                  - "s3:GetObject"
+                  - "s3:ListBucket"
+                  - "dynamodb:PutItem"
+                  - "dynamodb:DeleteItem"
+                  - "dynamodb:GetItem"
+                Resource:
+                  - !GetAtt StateBucket.Arn
+                  - !Sub "${StateBucket.Arn}/*"
+                  - !GetAtt DynamoLockTable.Arn

data/examples/bootstrap/tfctl-org-access.template

@@ -0,0 +1,42 @@
+AWSTemplateFormatVersion: 2010-09-09
+Description: Create Tfctl IAM role used to workout AWS account topolgy
+
+Parameters:
+  PrimaryAccountId:
+    Type: String
+    Description: Primary organization account ID
+
+Resources:
+  TfCtlRole:
+    Type: AWS::IAM::Role
+    Properties:
+      RoleName: TfctlRole
+      AssumeRolePolicyDocument:
+        Version: '2012-10-17'
+        Statement:
+          - Effect: Allow
+            Principal:
+              AWS:
+                - !Sub 'arn:aws:iam::${PrimaryAccountId}:root'
+            Action:
+              - sts:AssumeRole
+      Path: /
+      Policies:
+        - PolicyName: TfctlOrgAccess
+          PolicyDocument:
+            Version: '2012-10-17'
+            Statement:
+              - Effect: Allow
+                Action:
+                  - organizations:ListAccounts
+                  - organizations:ListAccountsForParent
+                  - organizations:ListChildren
+                  - organizations:ListRoots
+                  - organizations:DescribeOrganizationalUnit
+                Resource:
+                  - '*'
+
+Outputs:
+  TfCtlUserArn:
+    Description: tfctl role arn
+    Value: !GetAtt 'TfCtlRole.Arn'

data/examples/control_tower/conf/example.yaml

@@ -0,0 +1,71 @@
+#
+# Example Tfctl configuration for AWS Control Tower
+#
+# The data in this file is merged with data from AWS Organizations API to
+# create final configuration used by tfctl.  You can view the merged
+# configuration by running:
+#
+#   tfctl -c conf/example.yaml -s
+#
+
+#
+# Terraform configuration
+#
+
+# State management
+
+tf_state_bucket: 'CHANGEME'
+tf_state_dynamodb_table: 'terraform-lock'
+tf_state_region: 'eu-west-1'
+# Role for accessing state resources
+tf_state_role_arn: 'arn:aws:iam::SHARED_SERVICES_ACCOUNT_ID:role/TerraformStateRole'
+
+# Role used by tfctl to retrieve data from AWS Organizations
+# Has to be set up in the primary org account
+tfctl_role_arn: 'arn:aws:iam::PRIMARY_ACCOUNT_ID:role/TfctlRole'
+
+#
+# Organization configuration
+#
+
+# IMPORTANT: Removing a Terraform profile here will remove all of it's
+#            associated resources in AWS!
+
+# Name of the primary account.
+primary_account: CHANGEME
+
+# Configuration to apply in all accounts
+organization_root:
+
+  # Role assumed by Terraform for execution in each account
+  tf_execution_role: 'AWSControlTowerExecution'
+  region: 'eu-west-1'
+  # Bucket name used by example profile it will be prefixed with the target
+  # account number for uniqueness across accounts.
+  example_bucket_name: 'tfctl-example-bucket'
+  # Assign example-profile to all accounts in managed OUs
+  profiles:
+    - example-profile
+
+# Configuration to apply to accounts in Organization Units
+# Units not listed here will be ignored
+organization_units:
+  # Uncomment if you want to include accounts under the Core OU
+  # Core: {}
+  live: {}
+  test: {}
+  mgmt:
+    # Override the example bucket name in mgmt OU accounts
+    example_bucket_name: 'tfctl-ou-override-example'
+
+# Configuration to apply to individual accounts
+account_overrides:
+  test-example1:
+    # Override the bucket name in specific account
+    example_bucket_name: 'tfctl-account-override-example'
+
+
+# Exclude individual accounts from Terraform runs.
+#exclude_accounts:
+#   - Audit
+#   - 'Log archive'

data/examples/control_tower/modules/s3-bucket/main.tf

@@ -0,0 +1,4 @@
+resource aws_s3_bucket bucket {
+  bucket = var.name
+  acl    = "private"
+}

data/examples/control_tower/modules/s3-bucket/variables.tf

@@ -0,0 +1,4 @@
+variable "name" {
+  description = "Bucket name"
+  type = string
+}

data/examples/control_tower/profiles/example-profile/data.tf

@@ -0,0 +1 @@
+data "aws_caller_identity" "current" {}

data/examples/control_tower/profiles/example-profile/main.tf

@@ -0,0 +1,4 @@
+module "bucket" {
+  source = "../../modules/s3-bucket"
+  name   = "${local.account_id}-${local.account["example_bucket_name"]}"
+}

data/examples/control_tower/profiles/example-profile/variables.tf

@@ -0,0 +1,12 @@
+# This variable must always be present in a profile
+variable "config" {
+  description = "Configuration generated by tfctl"
+  type        = string
+}
+
+locals {
+  config     = jsondecode(var.config)
+  account_id = "${data.aws_caller_identity.current.account_id}"
+  # get current account configuration from tfctl config
+  account    = [ for account in local.config["accounts"]: account if account["id"] == local.account_id ][0]
+}

data/lib/hash.rb

@@ -0,0 +1,27 @@
+# frozen_string_literal: true
+
+# Add a deep_merge method to a Hash.
+# It unions arrays (for terraform profiles behaviour)
+class ::Hash
+    def deep_merge(second)
+        merger = proc { |key, v1, v2|
+          Hash === v1 && Hash === v2 ? v1.merge(v2, &merger) :
+          Array === v1 && Array === v2 ? v1 | v2 :
+          [:undefined, nil, :nil].include?(v2) ? v1 : v2
+        }
+        self.merge(second.to_h, &merger)
+    end
+
+    # Copied from ruby 2.6 Psych for 2.3 compatibility.
+    def symbolize_names!(result=self)
+        case result
+        when Hash
+          result.keys.each do |key|
+            result[key.to_sym] = symbolize_names!(result.delete(key))
+          end
+        when Array
+          result.map! { |r| symbolize_names!(r) }
+        end
+        result
+    end
+end

data/lib/tfctl/aws_org.rb

@@ -0,0 +1,121 @@
+# frozen_string_literal: true
+
+require_relative 'error.rb'
+require 'aws-sdk-organizations'
+
+module Tfctl
+    class AwsOrg
+
+        def initialize(role_arn)
+            @aws_org_client = Aws::Organizations::Client.new(
+              region:      'us-east-1',
+              # Assume role in primary account to read AWS organization API
+              credentials: aws_assume_role(role_arn)
+            )
+        end
+
+        # Gets account data for specified OUs from AWS Organizations API
+        def accounts(org_units)
+            output = { :accounts => [] }
+
+            aws_ou_ids = aws_ou_list
+
+            org_units.each do |ou_path|
+
+                if aws_ou_ids.has_key?(ou_path)
+                    parent_id = aws_ou_ids[ou_path]
+                else
+                    raise Tfctl::Error.new "Error: OU: #{ou_path}, does not exists in AWS organization"
+                end
+
+                @aws_org_client.list_accounts_for_parent({ parent_id: parent_id }).accounts.each do |account|
+                    if account.status == 'ACTIVE'
+
+                        output[:accounts] << {
+                            :name       => account.name,
+                            :id         => account.id,
+                            :arn        => account.arn,
+                            :email      => account.email,
+                            :ou_path    => ou_path.to_s,
+                            :ou_parents => ou_path.to_s.split('/'),
+                            :profiles   => [],
+                        }
+                    end
+                end
+            end
+            output
+        end
+
+        private
+
+        # Get a mapping of ou_name => ou_id from AWS organizations
+        def aws_ou_list()
+            output = {}
+            root_ou_id = @aws_org_client.list_roots.roots[0].id
+
+            ou_recurse = lambda do |ous|
+                ous.each do |ou_name, ou_id|
+                    children = aws_ou_list_children(ou_id, ou_name)
+                    unless children.empty?
+                        output.merge!(children)
+                        ou_recurse.call(children)
+                    end
+                end
+            end
+            ou_recurse.call({ :root => root_ou_id })
+
+            output
+        end
+
+        # Get a list of child ou's for a parent
+        def aws_ou_list_children(parent_id, parent_name)
+            output = {}
+            retries = 0
+
+            @aws_org_client.list_children( {
+                child_type: 'ORGANIZATIONAL_UNIT',
+                parent_id: parent_id,
+            }).children.each do |child|
+
+                begin
+                    ou = @aws_org_client.describe_organizational_unit({
+                        organizational_unit_id: child.id
+                    }).organizational_unit
+                rescue Aws::Organizations::Errors::TooManyRequestsException
+                    # FIXME - use logger
+                    puts 'AWS Organizations: too many requests.  Retrying in 5 secs.'
+                    sleep 5
+                    retries += 1
+                    retry if retries < 10
+                end
+
+                if parent_name == :root
+                    ou_name = ou.name.to_sym
+                else
+                    ou_name = "#{parent_name}/#{ou.name}".to_sym
+                end
+
+                output[ou_name] = ou.id
+            end
+            output
+        end
+
+        def aws_assume_role(role_arn)
+          begin
+            sts = Aws::STS::Client.new()
+
+            role_credentials = Aws::AssumeRoleCredentials.new(
+              client: sts,
+              role_arn: role_arn,
+              role_session_name: 'tfctl'
+            )
+          rescue StandardError => e
+            raise Tfctl::Error.new("Error assuming role: #{role_arn}, #{e.message}")
+            exit 1
+          end
+
+          role_credentials
+        end
+
+    end
+end