simplygenius-atmos 0.7.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: ff5c643bbfd1ad44992953679d7ace577ab60d541a0dd011c8455cb8627b5b2e
+  data.tar.gz: '017087621e3796fc796af5fd64cf3ccee30ad7025ba2ce146705e5ecf1f72235'
+SHA512:
+  metadata.gz: e6631a0c431d37ea69b7e192bba5261211716af354bff2a9d7173f975be67b351a2ead91873969d0b6b7ae74dd8502830f8f6d2da5102c0eb17741b92878c745
+  data.tar.gz: fcf3687ec3436f764c7ee4307d27df149699171cf9276b68501428379f74b9922fce5ef7b9d28c956049715c1cec9c60f1dab931fb70c8fb72de04c5d82c950c

data/CHANGELOG.md

@@ -0,0 +1,2 @@
+v0.7.0
+First public release

data/LICENSE

@@ -0,0 +1,13 @@
+Copyright (c) 2018 SimplyGenius LLC
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

data/README.md

@@ -0,0 +1,212 @@
+[![Build Status](https://travis-ci.org/simplygenius/atmos.svg?branch=master)](https://travis-ci.org/simplygenius/atmos)
+[![Coverage Status](https://coveralls.io/repos/github/simplygenius/atmos/badge.svg?branch=master)](https://coveralls.io/github/simplygenius/atmos?branch=master)
+
+# Atmos
+
+Atmos(phere) - Breathe easier with terraform
+
+Atmos provides a layer of organization on top of terraform for creating cloud system architectures with Amazon Web Services.  It handles the plumbing so you can focus on your application.  The core atmos runtime is free and open-source, with a business friendly license (Apache).  It provides some basic recipes to help get you going with a service oriented architecture implemented with AWS Elastic Container Services.  For more in-depth recipes, please check out the Atmos Pro offering.
+
+## Goals
+
+* The whole is greater than the sum of its parts.  Assist in creating a cloud infrastructure _system_ rather than just discrete infrastructure components.  Learning aws and terraform is a lot to bite off when getting started.  It's much easier to start with a working system ,and learn incrementally as you go by making changes to it.
+
+* The command line is king.  Using a CLI to iterate on and manage core infrastructure has always been more effective for me, so I aim to make things as convenient and usable as possible from there.
+
+* No lock-in.  Since atmos just provides you convenience on top of using terraform, and the templates for defining your infrastructure are 99% in terraform, its possible to migrate away (albeit giving up all the convenience atmos provides) if your goals ever diverge from those of atmos.
+
+* Your infrastructure is an important product.  It should have its own repo and be tracked under configuration management, not just clickety-click-clicked on in a UI and promptly forgotten what it is you actually did to get there.  The only guarantee you have, is that things are going to need to change, and you'll be much better off with a system that allows you to iterate easily.  Atmos gets you started with minimal up-front knowledge, but provides a path for your infrastructure to evolve.
+
+## Features
+
+* Manages AWS authentication, including MFA
+* Integrated MFA token generation for convenience.  This is technically not as secure since a laptop compromise exposes the key (vs a separate device for MFA).  Plans to add yubikey support to get both convenience and security.
+* Integrated secret management, with per-secret access permissions to minimize exposure footprint if something gets compromised
+* Manages multiple independent environments (e.g. dev, staging, production), allowing them to be as similar or divergent as desired.
+* Automates separation of environments across AWS accounts
+* Secure by default
+* Common recipe patterns to simplify maximal use of higher level AWS components. 
+* Sets up dns for your domain along with a wildcard certificate for hassle free ssl on your services
+* Free and open source core with a business friendly license (Apache)
+
+
+## Installation
+
+First install the dependencies:
+ * [Install docker](https://www.docker.com/community-edition) for deploying containers
+ * Install terraform (optional if running atmos as a docker image): e.g. `brew install terraform` on OS X 
+ * Install the aws cli (optional, useful for managing aws credentials): e.g. `brew install aws` on OS X
+
+Then install atmos:
+
+To install as a gem:
+ * gem install simplygenius-atmos
+ * verify: `atmos --help`
+
+To install/run as a docker image:
+ * curl -sL https://raw.githubusercontent.com/simplygenius/atmos/master/exe/atmos-docker > /usr/local/bin/atmos
+ * chmod +x /usr/local/bin/atmos
+ * verify: `atmos --help`
+
+Note that when running as a docker image, UI notifications get forced inline as text output as atmos no longer has access to your current OS.
+
+## Usage
+
+Usage is available via the command line: `atmos --help`
+The [terraform docs](https://www.terraform.io/docs/index.html) are excellent.
+
+## Quickstart
+
+See the [screencast](https://simplygenius.wistia.com/projects/h595iz9tbq) for a detailed walkthrough (~1 hour) of the quickstart.
+
+[Create an AWS account](https://portal.aws.amazon.com/billing/signup)
+Setup root account access keys, make note of the numeric account id
+
+It'll make your life easier dealing with multiple keys if you make use of the AWS shared credential store.  Save the access keys there with this command (picking your own name for the profile):
+
+```
+aws configure --profile <root_profile_name>
+```
+
+Create a new atmos project.  This should only contain files defining your infrastructure, and not your application:
+
+```
+mkdir my-ops-repo
+cd my-ops-repo
+atmos new
+```
+
+Initialize the atmos project for aws.  When prompted, input a short name for your organization and the AWS account id to use for the ops environment:
+
+```
+atmos generate --force aws/scaffold
+```
+
+The `--force` is optional and just prevents the prompt for every change the generator is making to files in your repo.
+
+Optionally bring up config/atmos.yml in your editor and make any other desired changes.
+
+Bootstrap your cloud provider to work with atmos.  Answer `yes` when prompted to apply the changes.
+
+```
+AWS_PROFILE=<root_profile_name> atmos -e ops bootstrap
+AWS_PROFILE=<root_profile_name> atmos -e ops apply
+```
+
+Setup a non-root user - using your email as the IAM username is convenient for email notifications in the future (e.g. per-user security validations like auto-expiry of access keys)
+
+```
+AWS_PROFILE=<root_profile_name> atmos user create -l -k -g all-users -g ops-admin your@email.address
+aws configure --profile <user_profile_name>
+```
+
+If you supply the "-m" flag, it will automatically create and activate a virtual MFA device with the user, and prompt you to save the secret to the atmos mfa keystore for integrated usage.  You can skip saving the secret and instead just copy/paste it into your MFA device of choice.  The "user create" command can also act in more of an upsert fashion, so to do something like reset a user's password and keys, you could do `atmos user create --force -l -m -k your@email.address`
+
+Login to the aws console as that user, change your password and setup MFA there if you prefer doing it that way.  Make sure you log out and back in again with MFA before you try setting up the [role switcher](#per-user-role-switcher-in-console)
+
+Now that a non-root user is created, you should be able to do everything as that user, so you can remove the root access keys if desired.  Keeping them around can be useful though, as there are some AWS operations that can only be done as the root user.  Leaving them in your shared credential store, but deactivating them in the AWS console till needed is a reasonable compromise.  
+
+While you can do everything in a single account, i've found a better practice is to use a new account for each env (dev, staging, prod, etc), and leave the ops account providing authentication duties and acting as a jumping off point to the others.  This allows for easier role/permission management down the line as well as better isolation between environments, thereby enabling safe iteration in dev environments without risking production.
+
+Create a new `dev` account, and bootstrap it to work with atmos
+
+```
+AWS_PROFILE=<user_profile_name> atmos account create dev
+AWS_PROFILE=<user_profile_name> atmos -e ops apply
+AWS_PROFILE=<user_profile_name> atmos -e dev bootstrap
+```
+
+Note that you can `export AWS_PROFILE=<user_profile_name>` in your environment, or keep using it per operation as preferred.
+
+Use the 'aws/service' template to setup an ECS Fargate based service, then apply it in the dev environment to make it active.  This template will also pull in some dependent templates to setup a vpc, dns for the provided domain and a wildcard cert to enable ssl for your service
+
+```
+atmos generate --force aws/service
+atmos -e dev apply
+
+```
+
+Setup your application repo to work with ECS by generating a Dockerfile.  For example, [here is the example app](https://github.com/simplygenius/atmos-example-app) used in the demo
+
+To deploy your app to ECS, first use docker to build an image with a tag named the same as your service name
+
+```
+# In your app repo directory
+docker build -t <service_name> .
+```
+
+Then use atmos to push and deploy that image to the ECR repo:
+
+```
+atmos -e dev container deploy -c services <service_name>
+```
+
+The atmos aws scaffold also sets up a user named deployer, with restricted permissions sufficient to do the deploy.  Add the [key/secret](https://github.com/simplygenius/atmos-recipes/blob/master/aws/scaffold/recipes/atmos-permissions.tf#L159)) to the environment for your CI to get your CI to auto deploy on successful build.
+
+```
+AWS_ACCESS_KEY_ID=<deployer_key> AWS_SECRET_ACCESS_KEY=<deployer_secret> atmos -e <env_based_on_branch> container deploy -c services <service_name>
+```
+
+To clean it all up:
+
+```
+# Applies flag to allow deleting empty buckets to existing resources
+TF_VAR_force_destroy_buckets=true atmos -e dev apply
+
+# Destroys all non-bootstrap resources create by atmos
+atmos -e dev destroy
+
+# Destroys the bootstrap resources (state, secret, lock storage and
+# cross-account access role)
+TF_VAR_force_destroy_buckets=true atmos -e dev apply --group bootstrap
+atmos -e dev destroy --group bootstrap
+
+# For normal usage you should rarely need to cleanup the ops account, but
+# included here in case you want to completely purge the atmos resources after
+# trying things out.
+
+# Cleanup non-bootstrap ops
+AWS_PROFILE=<root_profile_name> TF_VAR_force_destroy_buckets=true atmos -e ops apply
+AWS_PROFILE=<root_profile_name> atmos -e ops destroy
+
+# Cleanup ops bootstrap
+AWS_PROFILE=<root_profile_name> TF_VAR_force_destroy_buckets=true atmos -e ops apply --group bootstrap
+AWS_PROFILE=<root_profile_name> atmos -e ops destroy --group bootstrap
+
+```
+
+These are separate commands so that day-day usage where you want to tear down everything (e.g. CI spinning up then destroying while testing) doesn't compromise your ability to use atmos/terraform.  You can avoid the extra steps of applying with `TF_VAR_force_destroy_buckets=true` if you set `force_destroy_buckets: true` in atmos.yml
+
+## Per-User Role switcher in Console
+
+If you are following the account-per-environment pattern, you will need to setup a role switcher for each account in the AWS console for your user.  The AWS console seems to store these in cookies, so if you make a mistake its easy to fix by clearing them.  First, login to the AWS console with your personal aws user that was created in the ops account.  Select the dropdown with your email at top right of the page, Switch Role.  Fill the details for the environment you want to be able to access from the console:
+
+* Account number for the environment (See environments->`<env>`-> account_id in `config/atmos.yml` 
+* Role `<env>-admin` - this is the role you assume in the destination account.
+* Pick a name (e.g. DevAdmin)
+* Pick a color that you like (e.g. Red=production, Yellow=staging, Green=dev)
+
+## Managing secrets
+
+Secrets are stored in a s3 bucket unique to each environment, and automatically passed into terraform when it is executed by atmos.  The secret key should be the same as a terraform variable name defined in your terraform recipes, and if the secret exists, it will override whatever default value you have setup for the terraform variable.
+
+To set a secret:
+
+`atmos secret -e <env> set key value`
+
+For other secret usage:
+ 
+`atmos secret --help`
+
+## Contributing
+
+Bug reports and pull requests are welcome on GitHub at https://github.com/simplygenius/atmos.
+
+
+## License
+
+The gem is available as open source under the terms of the [Apache 2.0 License](https://opensource.org/licenses/apache-2.0).
+
+# About Us
+
+Simply Genius LLC is an independently run organization in Boston, MA USA.  Its Chief Everything is Matt Conway, a software engineer and executive with more than 20 years of experience in the Boston Tech Startup scene.  Atmos is his attempt at providing the world with the same tools, techniques, mindset and philosophy that he strives for when building a system architecture for a new startup.

data/exe/atmos

@@ -0,0 +1,4 @@
+#!/usr/bin/env ruby
+
+require_relative '../lib/atmos/cli'
+Atmos::CLI.run

data/exe/atmos-docker

@@ -0,0 +1,12 @@
+#!/usr/bin/env sh
+
+# fail fast
+set -e
+
+APP_VOLUME=${IMAGE_APP_VOLUME:-/app}
+
+docker run \
+  --interactive --tty --rm \
+  --volume "$(pwd)":"${APP_VOLUME}" \
+  --volume /var/run/docker.sock:/var/run/docker.sock \
+  simplygenius/atmos "$@"

data/lib/atmos.rb

@@ -0,0 +1,12 @@
+require_relative 'atmos/version'
+
+require_relative 'atmos/logging'
+Atmos::Logging.setup_logging(false, false, nil)
+
+require_relative 'atmos/config'
+require 'active_support/core_ext/string'
+require 'active_support/concern'
+
+module Atmos
+  mattr_accessor :config
+end

data/lib/atmos/cli.rb

@@ -0,0 +1,105 @@
+require_relative '../atmos'
+require_relative '../atmos/ui'
+require 'clamp'
+require 'sigdump/setup'
+
+Dir.glob(File.join(File.join(__dir__, 'commands'), '*.rb')) do |f|
+  require_relative "commands/#{File.basename(f).sub(/\.rb$/, "")}"
+end
+
+module Atmos
+
+  # The command line interface to atmos
+  class CLI < Clamp::Command
+
+    include GemLogger::LoggerSupport
+
+    def self.description
+      desc = <<-DESC
+        Atmos version #{Atmos::VERSION}
+
+        Runs The atmos command line application
+
+        e.g.
+
+        atmos --help
+      DESC
+      desc.split("\n").collect(&:strip).join("\n")
+    end
+
+    option ["-d", "--debug"],
+           :flag, "debug output\n",
+           default: false
+
+    option ["-c", "--[no-]color"],
+           :flag, "colorize output (or not)\n (default: $stdout.tty?)"
+
+    option ["-e", "--atmos-env"],
+           'ENV', "The atmos environment\n",
+           environment_variable: 'ATMOS_ENV', default: 'ops'
+
+    def default_color?
+       $stdout.tty?
+    end
+
+    option ["-l", "--[no-]log"],
+           :flag, "log to file in addition to terminal (or not)\n",
+           default: true
+
+
+    subcommand "new", "Sets up a new atmos project",
+               Atmos::Commands::New
+    subcommand "generate", "Generate recipes for the repository",
+               Atmos::Commands::Generate
+    subcommand "bootstrap", "Init cloud provider for use with atmos",
+               Atmos::Commands::Bootstrap
+    subcommand "init", "Run terraform init",
+               Atmos::Commands::Init
+    subcommand "plan", "Run terraform plan",
+               Atmos::Commands::Plan
+    subcommand "apply", "Run terraform apply",
+               Atmos::Commands::Apply
+    subcommand "destroy", "Run terraform destroy",
+               Atmos::Commands::Destroy
+    subcommand "terraform", "Run all other terraform commands",
+               Atmos::Commands::Terraform
+    subcommand "account", "Account management commands",
+               Atmos::Commands::Account
+    subcommand "user", "User management commands",
+               Atmos::Commands::User
+    subcommand "otp", "Otp tools",
+               Atmos::Commands::Otp
+    subcommand "secret", "Secret management commands",
+               Atmos::Commands::Secret
+    subcommand "auth_exec", "Authenticated exec",
+               Atmos::Commands::AuthExec
+    subcommand "container", "Container tools",
+               Atmos::Commands::Container
+
+    subcommand "version", "Display version" do
+      def execute
+        logger.info "Atmos Version #{Atmos::VERSION}"
+      end
+    end
+
+    subcommand "config", "Display expanded config for atmos_env" do
+      def execute
+        logger.info YAML.dump(Atmos.config.to_h)
+      end
+    end
+
+    # hook into clamp lifecycle to force logging setup even when we are calling
+    # a subcommand
+    def parse(arguments)
+      super
+      if Atmos.config.nil?
+        Atmos.config = Atmos::Config.new(atmos_env)
+        log = Atmos.config.is_atmos_repo? && log? ? "atmos.log" : nil
+        Atmos::Logging.setup_logging(debug?, color?, log)
+        Atmos::UI.color_enabled = color?
+      end
+    end
+
+  end
+
+end

data/lib/atmos/commands/account.rb

@@ -0,0 +1,65 @@
+require_relative 'base_command'
+require_relative '../../atmos/settings_hash'
+require 'climate_control'
+
+module Atmos::Commands
+
+  class Account < BaseCommand
+
+    def self.description
+      "Manages accounts/envs in the cloud provider"
+    end
+
+    subcommand "create", "Create a new account" do
+
+      option ["-s", "--source-env"],
+             "SOURCE_ENV", "Base the new env on a clone of the given one\n"
+
+      option ["-e", "--email"],
+             "EMAIL", "override default email used for new account\n"
+
+      option ["-n", "--name"],
+             "NAME", "override default name used for new account\n"
+
+      parameter "ENV",
+                "The name of the new env to create"
+
+      def execute
+
+        Atmos.config.provider.auth_manager.authenticate(ENV) do |auth_env|
+          ClimateControl.modify(auth_env) do
+
+            config = YAML.load_file(Atmos.config.config_file)
+
+            if config['environments'][env]
+              signal_usage_error "Env '#{env}' is already present in atmos config"
+            end
+
+            source = {}
+            if source_env.present?
+              source = config['environments'][source_env]
+              if source.blank?
+                signal_usage_error "Source env '#{source_env}' does not exist"
+              end
+              source = source.clone
+            end
+
+            account = Atmos.config.provider.account_manager.create_account(env, name: name, email: email)
+            logger.info "Account created: #{display account}"
+
+            source['account_id'] = account[:account_id].to_s
+
+            new_yml = Atmos::SettingsHash.add_config(
+                Atmos.config.config_file,
+                "environments.#{env}", source
+            )
+            logger.info("Writing out new atmos.yml containing new account")
+            File.write(Atmos.config.config_file, new_yml)
+          end
+        end
+      end
+    end
+
+  end
+
+end

data/lib/atmos/commands/apply.rb

@@ -0,0 +1,20 @@
+require_relative 'terraform'
+
+module Atmos::Commands
+
+  class Apply < Atmos::Commands::Terraform
+
+    def self.description
+      "Runs terraform apply"
+    end
+
+    def execute
+      args = ["apply"]
+      args << "--get-modules" unless Atmos.config["disable_auto_modules"].to_s == "true"
+      @terraform_arguments.insert(0, *args)
+      super
+    end
+
+  end
+
+end