simplygenius-atmos 0.11.10 → 0.13.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 53769d0519aa4f45feb52891aff6696ea1be0b054b0baf8d444799c1e93273de
-  data.tar.gz: a0a4726cf39261d4abc7d3880e06cc4b43d46dc6eeb0cedf53690d6aafb66f62
+  metadata.gz: e6e7037a18301c9dc8f8908d72b83d2390481145dcd1061d2c2d678e922bf658
+  data.tar.gz: '08858cc4a83f9af8cca448dc64c9a094ece7216c43b872c0c07bc9f8dcd32b0b'
 SHA512:
-  metadata.gz: 7fe49d0c4b8366d46f4ddd08fa3406759d43bfceed609ebe8f874e376d767e6bcb1855a97ea00dd99ca5ee082dc80edf38cd14fa87269de2814534677ec1644e
-  data.tar.gz: 3f5e5f627bb4f4f3ef5ed934bc3d0cb4675b189df6a060dbe43bb98d64c83d7987c278537d88f17a146ad61e94d18a0f9883957cd6ce4ba7faf6945c00860e33
+  metadata.gz: 75e6f5eac9061b38e12f162a31e721d74f97d532d576b142b5a4be5b39b952a91ffc013fc2cce175bd6b8fd0ee2fe8da34bbe73790a8686b930c2a6fb8368556
+  data.tar.gz: 9ebfb5234e13d3972c507c1366c1785ddebea03e23bd6dcc33b32c3057876f0dca2ec696c77bbfcd2ef7ba274aed95f56ac73809e142b52178a4c30902a73bb5

data/CHANGELOG.md

@@ -1,3 +1,67 @@
+0.13.1 (05/04/2021)
+-------------------
+
+* fix console exception, extend console startup wait loop, and add better console logging and error reporting [2d44553](../../commit/2d44553)
+* fix logging [3cc07b9](../../commit/3cc07b9)
+* allow selection of specific items with cli config dump [675ebfe](../../commit/675ebfe)
+* cli help line formatting [13b36a3](../../commit/13b36a3)
+* add container pull helper [22518f3](../../commit/22518f3)
+
+0.13.0 (01/20/2021)
+-------------------
+
+#### Notes on major changes
+
+* Now testing the atmos runtime against terraform 0.14 (and 0.11, 0.13)
+* The ability to link files into the terraform execution directory was enhanced, but still doesn't work for the .terraform.lock.hcl as it is overwritten by terraform and doesn't follow the link  
+
+#### Full changelog
+
+* exclude some new taskdef attributes for creation of new taskdef during deploy [a01b997](https://github.com/simplygenius/atmos/commit/a01b997)
+* add debug logging for aws sdk logger [55dd303](https://github.com/simplygenius/atmos/commit/55dd303)
+* directory sorting made more consistent across platforms [13622ff](https://github.com/simplygenius/atmos/commit/13622ff)
+* switch ci/cd from travis to github actions [13622ff](https://github.com/simplygenius/atmos/commit/13622ff)
+* fix test for tf 0.14 [122f861](https://github.com/simplygenius/atmos/commit/122f861)
+* replace plugin sharing with built in terraform variant [eb75266](https://github.com/simplygenius/atmos/commit/eb75266)
+* allow more options for linking in non-tf files into working dir [9878336](https://github.com/simplygenius/atmos/commit/9878336)
+* sort secret list [6b2dbfe](https://github.com/simplygenius/atmos/commit/6b2dbfe)
+* Only run terraform init when needed [d77186b](https://github.com/simplygenius/atmos/commit/d77186b)
+
+
+0.12.1 (10/14/2020)
+-------------------
+
+* fix bug with deeply nested overrides [60cb84a](https://github.com/simplygenius/atmos/commit/60cb84a)
+
+0.12.0 (10/12/2020)
+-------------------
+
+#### Notes on major changes
+
+* Now testing the atmos runtime against terraform 0.13 (and 0.11, 0.12), but no changes were necessary, so it should work if you upgrade aside from recipe changes
+* The fix to to make overrides (keys like ^foo to replace foo rather than add to it) work no matter where they appear in the config load chain could break compatibility if you were depending on the old behavior, i.e. you have multiple overrides for the same key with only the last one winning, or you have an override earlier in the change that was previously not taking effect
+
+#### Full changelog
+
+* Make overrides work no matter where they appear in the config load chain [47d73ba](https://github.com/simplygenius/atmos/commit/47d73ba)
+* allow interpolation in (remote_)config_source references, e.g. for supplying an access key from ~/.atmos.yml in the url [43f0c4d](https://github.com/simplygenius/atmos/commit/43f0c4d)
+* fix docker build [f4b3e2e](https://github.com/simplygenius/atmos/commit/f4b3e2e)
+* fix stdin test [5bd3f95](https://github.com/simplygenius/atmos/commit/5bd3f95)
+* exit with code on terraform error [2d2af2a](https://github.com/simplygenius/atmos/commit/2d2af2a)
+* test against tf 0.1[123] [1bbfb3b](https://github.com/simplygenius/atmos/commit/1bbfb3b)
+* add 'none' provider for tests (or if one wants to only run against terraform without aws) [991cefe](https://github.com/simplygenius/atmos/commit/991cefe)
+* fix activesupport cve [3d8b30c](https://github.com/simplygenius/atmos/commit/3d8b30c)
+* Add the ability to fetch atmos config from remote sources [b4b97fb](https://github.com/simplygenius/atmos/commit/b4b97fb)
+* Add convenience (`atmos account setup_credentials`)for populating aws local credential stores with the aws accounts managed by atmos [05466bd](https://github.com/simplygenius/atmos/commit/05466bd)
+* Add error msg for when task definition does't exist [0f8291e](https://github.com/simplygenius/atmos/commit/0f8291e)
+* Fix `jsonify` to convert the correct value to JSON. (#4) [1bda229](https://github.com/simplygenius/atmos/commit/1bda229)
+* Use ruby 2.4 for legacy testing since 2.3 is eol [78d2862](https://github.com/simplygenius/atmos/commit/78d2862)
+
+0.11.11 (02/20/2020)
+--------------------
+
+* Add the ability to setup a version requirement for forcing teams to use a consistent version of atmos [5c44ef3](https://github.com/simplygenius/atmos/commit/5c44ef3)
+
 0.11.10 (02/19/2020)
 --------------------
 

data/README.md

@@ -1,4 +1,4 @@
-[![Build Status](https://travis-ci.org/simplygenius/atmos.svg?branch=master)](https://travis-ci.org/simplygenius/atmos)
+[![Build Status](https://github.com/simplygenius/atmos/workflows/CD/badge.svg)](https://github.com/simplygenius/atmos/actions)
 [![Coverage Status](https://coveralls.io/repos/github/simplygenius/atmos/badge.svg?branch=master)](https://coveralls.io/github/simplygenius/atmos?branch=master)
 [![Gitter](https://badges.gitter.im/simplygenius/atmos.svg)](https://gitter.im/simplygenius/atmos?utm_source=badge&utm_medium=badge&utm_campaign=pr-badge)
 

data/lib/simplygenius/atmos.rb

@@ -18,4 +18,7 @@ module SimplyGenius
 end
 
 require_relative 'atmos/logging'
+# Need to setup logging before loading any other files
+SimplyGenius::Atmos::Logging.setup_logging(:info, false, nil)
+
 require_relative 'atmos/config'

data/lib/simplygenius/atmos/cli.rb

@@ -2,6 +2,7 @@ require_relative '../atmos'
 require_relative '../atmos/ui'
 require 'clamp'
 require 'sigdump/setup'
+require 'open-uri'
 
 Dir.glob(File.join(File.join(__dir__, 'commands'), '*.rb')) do |f|
   require_relative "commands/#{File.basename(f).sub(/\.rb$/, "")}"
@@ -29,22 +30,22 @@ module SimplyGenius
       end
 
       option ["-d", "--debug"],
-             :flag, "debug output\n",
+             :flag, "debug output",
              default: false
 
       option ["-q", "--quiet"],
-             :flag, "suppress output\n",
+             :flag, "suppress output",
              default: false
 
       option ["-c", "--[no-]color"],
-             :flag, "colorize output (or not)\n (default: $stdout.tty?)"
+             :flag, "colorize output (or not) (default: $stdout.tty?)"
 
       option ["-e", "--atmos-env"],
-             'ENV', "The atmos environment\n",
+             'ENV', "The atmos environment",
              environment_variable: 'ATMOS_ENV', default: 'ops'
 
       option ["-g", "--atmos-group"],
-             'GROUP', "The atmos working group\n for selecting recipe groups\n",
+             'GROUP', "The atmos working group for selecting recipe groups",
              default: 'default'
 
       option ["-p", "--load-path"],
@@ -52,7 +53,7 @@ module SimplyGenius
              multivalued: true
 
       option ["-o", "--override"],
-             "KEYVALUE", "overrides atmos configuration\nin the form 'some.config=value' where value can\nbe expressed in yaml form for complex types\ne.g. foo=1 foo=abc, foo=[x, y], foo={x: y}",
+             "KEYVALUE", "overrides atmos configuration in the form 'some.config=value' where value can be expressed in yaml form for complex types e.g. foo=1 foo=abc, foo=[x, y], foo={x: y}",
              multivalued: true
 
       option ["-v", "--version"],
@@ -63,7 +64,7 @@ module SimplyGenius
       end
 
       option ["-l", "--[no-]log"],
-             :flag, "log to file in addition to terminal (or not)\n",
+             :flag, "log to file in addition to terminal (or not)",
              default: true
 
 
@@ -109,16 +110,69 @@ module SimplyGenius
         option ["-j", "--json"],
                :flag, "Dump config as json instead of yaml"
 
+        parameter "PATH",
+                  "The dot notation path of a specific config item to get",
+                  required: false
+
         def execute
+          if path
+            result = Atmos.config[path]
+            result = case result
+              when Hash
+                result.to_h
+              when Array
+                result.to_a
+              else
+                result
+            end
+          else
+            result = Atmos.config.to_h
+          end
+
           if json?
-            output = JSON.pretty_generate(Atmos.config.to_h)
+            output = JSON.pretty_generate(result)
           else
-            output = YAML.dump(Atmos.config.to_h)
+            output = YAML.dump(result).sub(/^\s*---\s*/, '')
           end
           logger.info output
         end
       end
 
+      def fetch_latest_version
+        begin
+          latest_ver = JSON.parse(URI.open("https://rubygems.org/api/v1/versions/simplygenius-atmos/latest.json").read)['version']
+        rescue => e
+          latest_ver = "[Version Fetch Failed]"
+          logger.log_exception(e, "Couldn't check latest atmos gem version", level: :debug)
+        end
+        latest_ver
+      end
+
+      def version_check(atmos_version)
+
+        required_ver = Atmos.config["atmos.version_requirement"]
+        if required_ver.present?
+          case required_ver
+
+          when "latest"
+            latest_ver = fetch_latest_version
+
+            if latest_ver != atmos_version
+              raise "The atmos version (#{atmos_version}) does not match the given requirement (latest: #{latest_ver})"
+            end
+
+          when /[~<>=]*\s*[\d\.]*/
+            if ! Gem::Dependency.new('', required_ver).match?('', atmos_version)
+              raise "The atmos version (#{atmos_version}) does not match the given requirement (#{required_ver})"
+            end
+
+          else
+            raise "Invalid atmos.version_requirement, should be 'latest' or in a gem dependency form"
+          end
+        end
+
+      end
+
       # hook into clamp lifecycle to force logging setup even when we are calling
       # a subcommand
       def parse(arguments)
@@ -149,6 +203,8 @@ module SimplyGenius
             logger.info "Atmos Version #{VERSION}"
             exit(0)
           end
+
+          version_check(VERSION)
         end
       end
 
@@ -175,3 +231,37 @@ module SimplyGenius
 
   end
 end
+
+# Hack to make clamp usage less of a pain to get long lines to fit within a
+# standard terminal width
+class Clamp::Help::Builder
+
+  def word_wrap(text, line_width:)
+    text.split("\n").collect do |line|
+      line.length > line_width ? line.gsub(/(.{1,#{line_width}})(\s+|$)/, "\\1\n").strip.split("\n") : line
+    end.flatten
+  end
+
+  def string
+    line_width = 79
+    indent_size = 4
+    indent = " " * indent_size
+    StringIO.new.tap do |out|
+      lines.each do |line|
+        case line
+        when Array
+          out << indent
+          out.puts(line[0])
+          formatted_line = line[1].gsub(/\((default|required)/, "\n\\0")
+          word_wrap(formatted_line, line_width: (line_width - indent_size * 2)).each do |l|
+            out << (indent * 2)
+            out.puts(l)
+          end
+        else
+          out.puts(line)
+        end
+      end
+    end.string
+  end
+
+end

data/lib/simplygenius/atmos/commands/account.rb

@@ -15,13 +15,13 @@ module SimplyGenius
         subcommand "create", "Create a new account" do
 
           option ["-s", "--source-env"],
-                 "SOURCE_ENV", "Base the new env on a clone of the given one\n"
+                 "SOURCE_ENV", "Base the new env on a clone of the given one"
 
           option ["-e", "--email"],
-                 "EMAIL", "override default email used for new account\n"
+                 "EMAIL", "override default email used for new account"
 
           option ["-n", "--name"],
-                 "NAME", "override default name used for new account\n"
+                 "NAME", "override default name used for new account"
 
           parameter "ENV",
                     "The name of the new env to create"
@@ -62,6 +62,42 @@ module SimplyGenius
           end
         end
 
+        subcommand "setup_credentials", "Convenience that adds accounts to the local aws credentials store" do
+
+          option ["-u", "--user"],
+                 "USERNAME", "The username in the cloud provider", required: true
+
+          option ["-k", "--key"],
+                 "KEY", "The access key in the cloud provider"
+
+          option ["-s", "--secret"],
+                 "SECRET", "The access secret in the cloud provider"
+
+          option ["-d", "--default"],
+                 :flag, "Sets as default credentials"
+
+          option ["-f", "--force"],
+                 :flag, "Forces overwrites of existing"
+
+          option ["-n", "--nowrite"],
+                 :flag, "Trial run without writing results to files"
+
+          def execute
+
+            if ! key.present?
+              key = ask("Input your access key: ") { |q| q.echo = "*" }
+              raise ArgumentError.new("An access key is required") if key.blank?
+            end
+            if ! secret.present?
+              secret = ask("Input your access secret: ") { |q| q.echo = "*" }
+              raise ArgumentError.new("An access secret is required") if secret.blank?
+            end
+
+            Atmos.config.provider.account_manager.setup_credentials(username: user, access_key: key, access_secret: secret,
+                                                                    become_default: default?, force: force?, nowrite: nowrite?)
+          end
+        end
+
       end
 
     end

data/lib/simplygenius/atmos/commands/auth_exec.rb

@@ -12,7 +12,7 @@ module SimplyGenius
         end
 
         option ["-r", "--role"],
-               'ROLE', "overrides assume role name\n"
+               'ROLE', "overrides assume role name"
 
         parameter "COMMAND ...", "command to exec", :attribute_name => :command
 

data/lib/simplygenius/atmos/commands/bootstrap.rb

@@ -11,7 +11,7 @@ module SimplyGenius
         end
 
         option ["-f", "--force"],
-               :flag, "forces bootstrap\n"
+               :flag, "forces bootstrap"
 
         def execute
           orig_config = Atmos.config
@@ -68,6 +68,7 @@ module SimplyGenius
             rescue TerraformExecutor::ProcessFailed => e
               logger.error(e.message)
               logger.error(rebootstrap_msg)
+              exit(1)
             end
           end
         end

data/lib/simplygenius/atmos/commands/container.rb

@@ -15,22 +15,44 @@ module SimplyGenius
         end
 
         option ["-c", "--cluster"],
-               "CLUSTER", "The cluster name\n",
+               "CLUSTER", "The cluster name",
                required: true
 
         option ["-r", "--role"],
-               "ROLE", "The role to assume when deploying\n"
+               "ROLE", "The role to assume when deploying"
+
+        subcommand "pull", "Pulls a container image from repository" do
+
+          option ["-v", "--revision"],
+                 "REVISION", "Use as the remote image revision"
+
+          parameter "NAME",
+                    "The name of the service (or task) to pull the image for"
+
+          def execute
+            Atmos.config.provider.auth_manager.authenticate(ENV, role: role) do |auth_env|
+              ClimateControl.modify(auth_env) do
+                mgr = Atmos.config.provider.container_manager
+
+                result = mgr.pull(name, revision: revision)
+
+                logger.info "Container pulled:\n #{display result}"
+              end
+            end
+          end
+
+        end
 
         subcommand "push", "Only push a container image without activating it" do
 
           option ["-i", "--image"],
-                 "IMAGE", "The local container image to deploy\nDefaults to service/task name"
+                 "IMAGE", "The local container image to deploy Defaults to service/task name"
 
           option ["-v", "--revision"],
-                 "REVISION", "Use as the remote image revision\n"
+                 "REVISION", "Use as the remote image revision"
 
           parameter "NAME ...",
-                    "The name of the service (or task) to deploy\nWhen multiple, the first is the primary, and\nthe rest get deployed with its image"
+                    "The name of the service (or task) to deploy. When multiple, the first is the primary, and the rest get deployed with its image"
 
           def default_image
             name_list.first
@@ -55,22 +77,22 @@ module SimplyGenius
         subcommand "activate", "Activate a container that has already been pushed" do
 
           option ["-v", "--revision"],
-                 "REVISION", "Use the given revision of the pushed image\nto activate\n"
+                 "REVISION", "Use the given revision of the pushed image to activate"
 
           option ["-w", "--wait"],
-                 :flag, "Wait for service to become stable after deploy\nnon-zero exit on fail"
+                 :flag, "Wait for service to become stable after deploy non-zero exit on fail"
 
           option ["-l", "--list"],
-                 :flag, "List the most recent pushed images\n"
+                 :flag, "List the most recent pushed images"
 
           option ["-t", "--tagcount"],
-                 "N", "Only show the last N items when listing\n",
+                 "N", "Only show the last N items when listing",
                  default: 10 do |s|
             Integer(s)
           end
 
           parameter "NAME ...",
-                    "The name of the service (or task) to activate\nWhen multiple, the first is the primary, and\nthe rest get activated with its image"
+                    "The name of the service (or task) to activate. When multiple, the first is the primary, and the rest get activated with its image"
 
           def execute
             Atmos.config.provider.auth_manager.authenticate(ENV, role: role) do |auth_env|
@@ -125,16 +147,16 @@ module SimplyGenius
         subcommand "deploy", "Push and activate a container" do
 
           option ["-i", "--image"],
-                 "IMAGE", "The local container image to deploy\nDefaults to service/task name"
+                 "IMAGE", "The local container image to deploy. Defaults to service/task name"
 
           option ["-w", "--wait"],
-                 :flag, "Wait for service to become stable after deploy\nnon-zero exit on fail"
+                 :flag, "Wait for service to become stable after deploy non-zero exit on fail"
 
           option ["-v", "--revision"],
-                 "REVISION", "Use as the remote image revision\n"
+                 "REVISION", "Use as the remote image revision"
 
           parameter "NAME ...",
-                    "The name of the service (or task) to deploy\nWhen multiple, the first is the primary, and\nthe rest get deployed with its image"
+                    "The name of the service (or task) to deploy. When multiple, the first is the primary, and the rest get deployed with its image"
 
           def default_image
             name_list.first
@@ -171,10 +193,10 @@ module SimplyGenius
         subcommand "console", "Spawn a console and attach to it" do
 
           option ["-p", "--persist"],
-                 :flag, "Leave the task running after disconnect\n"
+                 :flag, "Leave the task running after disconnect"
 
           parameter "NAME",
-                    "The name of the service (or task) to attach\nthe console to"
+                    "The name of the service (or task) to attach the console to"
 
           def execute
             Atmos.config.provider.auth_manager.authenticate(ENV, role: role) do |auth_env|
@@ -191,8 +213,12 @@ module SimplyGenius
                 logger.debug "Run task result: #{result}"
                 begin
                   match = result[:log_match]
-                  local_command = local_command.collect {|c| match.names.each {|n| c = c.gsub("<#{n}>", match[n]) }; c }
-                  system(*local_command)
+                  if match.blank?
+                    logger.error("Aborting, the console task failed to produce the expected output: #{log_pattern}")
+                  else
+                    local_command = local_command.collect {|c| match.names.each {|n| c = c.gsub("<#{n}>", match[n]) }; c }
+                    system(*local_command)
+                  end
                 ensure
                   if persist?
                     logger.info "Console disconnected, you can reconnect with: #{local_command.join(" ")}"