sonic-screwdriver 1.4.0 → 2.0.0

data/lib/sonic/help.rb

@@ -0,0 +1,9 @@
+module Sonic::Help
+  class << self
+    def text(namespaced_command)
+      path = namespaced_command.to_s.gsub(':','/')
+      path = File.expand_path("../help/#{path}.md", __FILE__)
+      IO.read(path) if File.exist?(path)
+    end
+  end
+end

data/lib/sonic/help/command/send.md

@@ -0,0 +1,10 @@
+Run as a command across a list of servers. A filter must be provided.  The filter can be a mix of instance ids and ec2 tags. This command can also take a path to a file. To specify a path to a file use file:// at the beginning of your file.
+
+Examples:
+
+    $ sonic command send hi-web-prod uptime
+    $ sonic command send hi-web-prod,hi-worker-prod,hi-clock-prod uptime
+    $ sonic command send i-030033c20c54bf149,i-030033c20c54bf150 uname -a
+    $ sonic command send i-030033c20c54bf149 file://hello.sh
+
+You cannot mix instance ids and tag names in the filter.

data/lib/sonic/help/completion.md

@@ -0,0 +1,22 @@
+Example:
+
+    sonic completion
+
+Prints words for TAB auto-completion.
+
+Examples:
+
+    sonic completion
+    sonic completion execute
+    sonic completion list
+
+To enable, TAB auto-completion add the following to your profile:
+
+    eval $(sonic completion_script)
+
+Auto-completion example usage:
+
+    sonic [TAB]
+    sonic exe[TAB]
+    sonic execute [TAB]
+    sonic list [TAB]

data/lib/sonic/help/completion_script.md

@@ -0,0 +1,3 @@
+To use, add the following to your `~/.bashrc` or `~/.profile`
+
+    eval $(sonic completion_script)

data/lib/sonic/help/ecs/exec.md

@@ -0,0 +1,8 @@
+Ssh into an ECS container instance, finds a running docker container associated
+with the service and docker exec's into it.
+
+Examples:
+
+    $ sonic ecs exec my-service --cluster my-cluster
+
+You can use a variety of identifiers.  These include the ECS service name and task id.

data/lib/sonic/help/ecs/sh.md

@@ -0,0 +1,13 @@
+Ssh into an ECS container instance and runs a docker container using the same
+environment and image as the specified running service.
+
+Examples:
+
+    $ sonic ecs sh --cluster my-cluster my-service
+    $ sonic ecs sh --cluster my-cluster my-service
+
+# If there are flags in the command that you want to pass down to the docker
+run command you will need to put the command in single quotes.  This is due to
+the way Thor (what this tool uses) parses options.
+
+    $ sonic ecs sh --cluster production-hi hi-web 'rake -T'

data/lib/sonic/help/execute.md

@@ -0,0 +1,60 @@
+Run as a command across a list of servers. A filter must be provided.  The filter can be a mix of instance ids and ec2 tags. This command can also take a path to a file. To specify a path to a file use file:// at the beginning of your file.
+
+## Examples Summary
+
+    $ sonic execute hi-web-prod uptime
+    $ sonic execute hi-web-prod,hi-worker-prod,hi-clock-prod uptime
+    $ sonic execute i-030033c20c54bf149,i-030033c20c54bf150 uname -a
+    $ sonic execute i-030033c20c54bf149 file://hello.sh
+
+You cannot mix instance ids and tag names in the filter.
+
+## Example Detailed
+
+Here's a command example output in detailed:
+
+```sh
+$ sonic execute i-0bf51a000ab4e73a8 uptime
+Sending command to SSM with options:
+---
+instance_ids:
+- i-0bf51a000ab4e73a8
+document_name: AWS-RunShellScript
+comment: sonic execute i-0bf51a000ab4e73a8 uptime
+parameters:
+  commands:
+  - uptime
+output_s3_region: us-east-1
+output_s3_bucket_name: [reacted]
+output_s3_key_prefix: ssm/commands/sonic
+
+Command sent to AWS SSM. To check the details of the command:
+  aws ssm list-commands --command-id 0bb18d58-6436-49fd-9bfd-0c4b6c51c7a2
+  aws ssm get-command-invocation --command-id 0bb18d58-6436-49fd-9bfd-0c4b6c51c7a2 --instance-id i-0bf51a000ab4e73a8
+
+Waiting for ssm command to finish.....
+Command finished.
+
+Displaying output for i-0bf51a000ab4e73a8.
+Command status: Success
+Command standard output:
+ 01:08:10 up 8 days,  6:41,  0 users,  load average: 0.00, 0.00, 0.00
+
+To see the more output details visit:
+  https://us-west-2.console.aws.amazon.com/systems-manager/run-command/0bb18d58-6436-49fd-9bfd-0c4b6c51c7a2
+
+Pro tip: the console url is already in your copy/paste clipboard.
+$
+```
+
+Notice the conveniences of `sonic execute`, it:
+
+1. Showed the parameters that will be sent as part of the send_command call to SSM.
+2. Sent the command to SSM.
+3. Waited for the command to finish.
+4. Displayed the output of the command.
+5. Provided the console url that visit to view more details about the SSM command.
+
+The AWS SSM console looks like this:
+
+<img src="/img/tutorials/ec2-console-run-command.png" class="doc-photo" />

data/lib/sonic/help/list.md

@@ -0,0 +1,17 @@
+List ec2 servers. A filter must be provided.  The filter can be a mix of instance ids and ec2 tags. sonic list will auto-detect the what type of filter it is filter appropriately.  The filter for listing is optional.
+
+Examples:
+
+    $ sonic list
+    $ sonic list hi-web-prod
+    $ sonic list hi-web-prod,hi-clock-prod
+    $ sonic list i-09482b1a6e330fbf7
+
+Example Output:
+
+    $ sonic list --header i-09482b1a6e330fbf7
+    Instance Id Public IP Private IP  Type
+    i-09482b1a6e330fbf7 54.202.152.168  172.31.21.108 t2.small
+    $
+
+You cannot mix instance ids and tag names in the filter.

data/lib/sonic/help/ssh.md

@@ -0,0 +1,60 @@
+* EC2 instance id. Example: i-067c5e3f026c1e801
+* EC2 tag value. Any tag value is search, the tag key does not matter only the tag value matters. Example: hi-web
+* ECS service. Example: my-ecs-service
+* ECS container instance id. Example: 7fbc8c75-4675-4d39-a5a4-0395ff8cd474
+* ECS task id. Example: 1ed12abd-645c-4a05-9acf-739b9d790170
+
+When using ecs identifiers the `--cluster` option is required or can be set in ~/.sonic/settings.yml.
+
+Examples:
+
+    $ sonic ssh i-067c5e3f026c1e801
+    $ sonic ssh hi-web
+    $ sonic ssh --cluster my-cluster my-ecs-service
+    $ sonic ssh 7fbc8c75-4675-4d39-a5a4-0395ff8cd474
+    $ sonic ssh 1ed12abd-645c-4a05-9acf-739b9d790170
+
+Sonic ssh builds up the ssh command and shells out to it. For example, the following commands:
+
+    sonic ssh i-027363802c6ff314f
+
+Translates to:
+
+    ssh ec2-user@ec2-52-24-216-170.us-west-2.compute.amazonaws.com
+
+You can also tack on any command to be run at the end of the command. Example:
+
+    $ sonic ssh i-027363802c6ff314f uptime
+    => ssh ec2-user@ec2-52-24-216-170.us-west-2.compute.amazonaws.com uptime
+ 15:57:02 up 18:21,  0 users,  load average: 0.00, 0.01, 0.00
+
+## Specifying pem keys
+
+The recommended way to specify custom private keys is to use ssh-agent as covered here: https://blog.boltops.com/2017/09/21/3-ssh-tips-ssh-agent-tunnel-and-escaping-from-the-dead
+
+But you can also specify the pem key to use with the -i option.  Example:
+
+    $ sonic ssh -i ~/.ssh/id_rsa-custom ec2-user@ec2-52-24-216-170.us-west-2.compute.amazonaws.com
+
+## Retry option
+
+For newly launched instances, the instance's ssh access might not be quite ready. Typically, you must press up enter repeatedly until the instance is ready.  Sonic ssh has a retry option that automates this. Example:
+
+    $ sonic ssh -r i-027363802c6ff314f
+
+Bastion Host Support
+
+Sonic ssh also supports a bastion host.
+
+    $ sonic ssh --bastion bastion.domain.com i-027363802c6ff314f
+    $ sonic ssh --bastion user@bastion.domain.com i-027363802c6ff314f
+
+Here's what the output looks like:
+
+    $ sonic ssh --bastion 34.211.223.3 i-0f7f833131a51ce35 uptime
+    => ssh -At ec2-user@34.211.223.3 ssh ec2-user@10.10.110.135 uptime
+     17:57:59 up 37 min,  0 users,  load average: 0.00, 0.02, 0.00
+    Connection to 34.211.223.3 closed.
+    $
+
+You can also set the bastion host and other options with a [settings file](http://sonic-screwdriver.cloud/docs/settings/).

data/lib/sonic/list.rb

@@ -1,6 +1,6 @@
 module Sonic
   class List
-    include AwsServices
+    include AwsService
 
     def initialize(options)
       @options = options
@@ -22,6 +22,9 @@ module Sonic
         # ERROR: The instance ID 'i-066b140d9479e9682' does not exist
         UI.error(e.message)
         exit 1
+      rescue Aws::EC2::Errors::InvalidInstanceIDMalformed => e
+        UI.error(e.message)
+        exit 1
       end
       instances
     end

data/lib/sonic/setting.rb

@@ -0,0 +1,47 @@
+require 'yaml'
+require 'memoist'
+require 'active_support/core_ext/hash'
+
+module Sonic
+  class Setting
+    extend Memoist
+
+    def data
+      settings_file = Sonic.profile || 'default'
+      settings_file += ".yml"
+
+      project = yaml_file("#{Sonic.root}/.sonic/#{settings_file}")
+      user = yaml_file("#{home}/.sonic/#{settings_file}")
+      default_file = File.expand_path("../default/settings.yml", __FILE__)
+      default = yaml_file(default_file)
+
+      data = merge(default, user, project)
+
+      if ENV['DEBUG_SETTINGS']
+        puts "settings data:"
+        pp data
+      end
+      data
+    end
+    memoize :data
+
+    def merge(*hashes)
+      hashes.inject({}) do |result, hash|
+        # note: important to compact for keys with nil value
+        result.deep_merge(hash.compact)
+      end
+    end
+
+    # Any empty file will result in "false".  Lets ensure that an empty file
+    # loads an empty hash instead.
+    def yaml_file(path)
+      return {} unless File.exist?(path)
+      YAML.load_file(path) || {}
+    end
+
+    def home
+      # hack but fast
+      ENV['TEST'] ? "spec/fixtures/home" : ENV['HOME']
+    end
+  end
+end

data/lib/sonic/ssh.rb

@@ -5,7 +5,7 @@ module Sonic
     autoload :IdentifierDetector, 'sonic/ssh/identifier_detector'
     autoload :CliOptions, 'sonic/ssh/cli_options'
 
-    include AwsServices
+    include AwsService
     include CliOptions
 
     def initialize(identifier, options)
@@ -14,11 +14,12 @@ module Sonic
       @user, @identifier = extract_user!(identifier) # extracts/strips user from identifier
       # While --user option is supported at the class level, don't expose at the CLI level
       # to encourage users to use user@host notation.
-      @user ||= options[:user] || settings.data["user"]
+      @user ||= options[:user] || settings["bastion"]["user"]
 
       @service = @identifier # always set service even though it's not always used as the identifier
-      @cluster = options[:cluster] || settings.default_cluster(@service)
-      @bastion = options[:bastion] || settings.default_bastion(@bastion)
+      map = settings["ecs_service_cluster_map"]
+      @cluster = options[:cluster] || map[@service] || map["default"] || "default"
+      @bastion = options[:bastion] || settings["bastion"]["host"]
     end
 
     def run
@@ -47,28 +48,34 @@ module Sonic
     def build_ssh_host
       return @identifier if ENV['TEST']
 
-      detector = Ssh::IdentifierDetector.new(@cluster, @service, @identifier, @options)
       instance_id = detector.detect!
       instance_hostname(instance_id)
     end
 
+    def detector
+      @detector ||= Ssh::IdentifierDetector.new(@cluster, @service, @identifier, @options)
+    end
+
+
     def instance_hostname(ec2_instance_id)
       begin
         resp = ec2.describe_instances(instance_ids: [ec2_instance_id])
       rescue Aws::EC2::Errors::InvalidInstanceIDNotFound => e
         # e.message: The instance ID 'i-027363802c6ff3141' does not exist
-        UI.error(e.message)
+        UI.error e.message
+        exit 1
+      rescue Aws::Errors::NoSuchEndpointError, SocketError
+        UI.error "It doesnt look like you have an internet connection. Please double check that you have an internet connection."
         exit 1
       end
       instance = resp.reservations[0].instances[0]
       # struct Aws::EC2::Types::Instance
       # http://docs.aws.amazon.com/sdkforruby/api/Aws/EC2/Types/Instance.html
-      host = if @bastion
-              instance.private_ip_address
-            else
-              instance.public_ip_address
-            end
-      "#{@user}@#{host}"
+      if @bastion
+        instance.private_ip_address
+      else
+        instance.public_ip_address
+      end
     end
 
     # Will use Kernel.exec so that the ssh process takes over this ruby process.
@@ -82,18 +89,30 @@ module Sonic
     end
 
 private
+    # direct access to settings data
     def settings
-      @settings ||= Settings.new(@options[:project_root])
+      @settings ||= Setting.new.data
     end
 
     # Returns Array of flags.
     # Example:
     #   ["-o", "StrictHostKeyChecking=no", "-o", "UserKnownHostsFile=/dev/null"]
     def ssh_options
-      host_key_check_options = settings.host_key_check_options
       keys_option + host_key_check_options
     end
 
+    # By default bypass strict host key checking for convenience.
+    # But user can overrride this.
+    def host_key_check_options
+      if settings["bastion"]["host_key_check"] == true
+        []
+      else
+        # settings["bastion"]["host_key_check"] nil will disable checking also
+        # disables host key checking
+        %w[-o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null]
+      end
+    end
+
     # Will prepend the bastion host if required
     # When bastion set
     #   ssh [options] -At [bastion_host] ssh -At [ssh_host]
@@ -111,12 +130,14 @@ private
     # -A = Enables forwarding of the authentication agent connection
     # -t = Force pseudo-terminal allocati
     def build_ssh_command
-      ssh = ["ssh"] + ssh_options
-      ssh += ["-At", bastion_host, "ssh"] + ssh_options if @bastion
-      # ssh_host is internal ip when bastion is set
-      # ssh_host is public ip when bastion is not set
-      ssh += ["-At", ssh_host]
-      ssh
+      command = ["ssh", "-t"] + ssh_options
+      if @bastion
+        # https://en.wikibooks.org/wiki/OpenSSH/Cookbook/Proxies_and_Jump_Hosts
+        # -J xxx is -o ProxyJump=xxx
+        proxy_jump = ["-J", bastion_host]
+        command += proxy_jump
+      end
+      command += ["-t", "#{@user}@#{ssh_host}"]
     end
 
     # Private: Extracts and strips the user from the identifier.

data/lib/sonic/ssh/identifier_detector.rb

@@ -4,7 +4,7 @@ class Ssh
   class IdentifierDetector
 
     include Ec2Tag
-    include AwsServices
+    include AwsService
     include Checks
 
     def initialize(cluster, service, identifier, options)
@@ -17,7 +17,7 @@ class Ssh
     # Returns exactly 1 instance_id or exits the program.
     # The bang check_* methods can exit early.
     def detect!
-      instance_id = case detected_type
+      @instance_id ||= case detected_type
         when :ecs_container_instance_or_task_arn
           check_cluster_exists! unless @options[:noop]
 
@@ -36,6 +36,10 @@ class Ssh
         end
     end
 
+    def instance_id
+      detect!
+    end
+
     # Any of these methods either returns exactly 1 instance_id or exit the program.
     #
     # Returns detected_type

data/lib/sonic/version.rb

@@ -1,3 +1,3 @@
 module Sonic
-  VERSION = "1.4.0"
+  VERSION = "2.0.0"
 end

data/sonic.gemspec

@@ -1,34 +1,39 @@
 # coding: utf-8
-lib = File.expand_path('../lib', __FILE__)
+lib = File.expand_path("../lib", __FILE__)
 $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
-require 'sonic/version'
+require "sonic/version"
 
 Gem::Specification.new do |spec|
   spec.name          = "sonic-screwdriver"
   spec.version       = Sonic::VERSION
   spec.authors       = ["Tung Nguyen"]
   spec.email         = ["tung@boltops.com"]
-  spec.description   = %q{Multi-functional tool to manage AWS infrastructure}
-  spec.summary       = %q{Multi-functional tool to manage AWS infrastructure}
+  spec.summary       = "Multi-functional tool to manage AWS infrastructure"
   spec.homepage      = "http://sonic-screwdriver.cloud/"
   spec.license       = "MIT"
 
   spec.files         = `git ls-files`.split($/)
-  spec.executables   = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
+  spec.bindir        = "exe"
+  spec.executables   = spec.files.grep(%r{^exe/}) { |f| File.basename(f) }
   spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
   spec.require_paths = ["lib"]
 
-  spec.add_dependency "thor"
-  spec.add_dependency "hashie"
-  spec.add_dependency "colorize"
+  spec.add_dependency "activesupport"
   spec.add_dependency "aws-sdk-ec2"
   spec.add_dependency "aws-sdk-ecs"
+  spec.add_dependency "aws-sdk-s3"
   spec.add_dependency "aws-sdk-ssm"
+  spec.add_dependency "colorize"
+  spec.add_dependency "hashie"
+  spec.add_dependency "memoist"
+  spec.add_dependency "thor"
   spec.add_dependency "tty-prompt"
 
   spec.add_development_dependency "bundler"
-  spec.add_development_dependency "rake"
+  spec.add_development_dependency "byebug"
   spec.add_development_dependency "guard"
   spec.add_development_dependency "guard-bundler"
   spec.add_development_dependency "guard-rspec"
+  spec.add_development_dependency "rake"
+  spec.add_development_dependency "cli_markdown"
 end