sonic-screwdriver 1.2.0 → 1.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: bab148a2ed5638be1af0462d0e8d8e594bb17b8b
-  data.tar.gz: ab33bc905251629ebfe0afb313b1467652ec14f5
+  metadata.gz: dd37ca876dccfeb67a1540e2af0f312fafcde6a3
+  data.tar.gz: c728fd9cf1da7019cc2738912d372f937e36704d
 SHA512:
-  metadata.gz: 448ee390aeaf1d33395515f30d0a7aafd1347a6c8cf07230325441cd4a38027f7d3eb0a7dd361d0698b96a4a36088a0eadc8631050c56e303bac017dd6efc7f1
-  data.tar.gz: 5f346a3756cd24f8025eafd1ac227fc075d3f9948164ba076544f7bafb7905a5a35e01d3cad32d3a5a991605f456da72dec052ad11b93ce6f4acaeb6801b337a
+  metadata.gz: 04f2e8e50729ca19e70a5fdd720cdcef5bdbe1bef20f14c93287811ba44ca8864760715ab55af8a93ce609d3b4759cbb1c304640e981d765b72143f49deec99d
+  data.tar.gz: 67357d39582de07b37672ae14a101515afe8eb9be1569199a2eaa0b9f59226577c444a25530f8067427973351f04309fc0ec5ca7a56ca2d478462f988a19df24

data/CHANGELOG.md

@@ -3,6 +3,9 @@
 All notable changes to this project will be documented in this file.
 This project *tries* to adhere to [Semantic Versioning](http://semver.org/), even before v1.0.
 
+## [1.3.0]
+- support for different bastion cluster host mapping in settings
+
 ## [1.2.0]
 - add old instance id support
 - update docs and help

data/docs/_docs/settings.md

@@ -11,7 +11,11 @@ You can adjust the behavior of sonic and set some handy default values with `set
 Here is an `settings.yml` example:
 
 ```yaml
-bastion: bastion.mydomain.com
+bastion: # cluster_host mapping
+  default: # default is nil - which means a bastion host wont be used
+  # Examples:
+  # prod: bastion.mydomain.com
+  # stag: ubuntu@bastion-stag.mydomain.com
 host_key_check: false
 service_cluster:
   default: # defaults to nil
@@ -28,13 +32,32 @@ The following table covers the different setting options:
 
 Setting  | Description | Default
 ------------- | ------------- | -------------
-bastion  | Bastion host to use as the jump host. Examples: bastion.mydomain.com, myuser@bastion.myuser.com or 123.123.123.123. | (no value)
+bastion  | Bastion mapping allows you to set a bastion host on a per ECS cluster basis.  The bastion host is used as the jump host. Examples: bastion.mydomain.com, myuser@bastion.myuser.com or 123.123.123.123. | (no value)
 host_key_check  | Controls whether or not use the strict host checking ssh option.  Since EC2 server host changes often the default value is false. | false
 service_cluster  | Service to cluster mapping.  This is a Hash structure that maps the service name to cluster names. | (no value)
 user  | User to ssh into the server with. This can be overriden at the CLI with the user@host notation but can be set in the settings.yml file also. | ec2-user
 
 The default settings are located tool source code at [lib/sonic/default/settings.yml](https://github.com/boltopslabs/sonic/blob/master/lib/sonic/default/settings.yml).
 
+### Bastion cluster to host mapping
+
+Provided this example:
+
+```yaml
+bastion: # cluster_host mapping
+  default: ec2-user@bastion.mydomain.com
+  prod: ec2-user@bastion.mydomain.com
+  stag: ubuntu@bastion-stag.mydomain.com
+```
+
+This results in
+
+```sh
+sonic ssh --cluster prod [IDENTIFER] # ec2-user@bastion.mydomain.com used as the bastion host
+sonic ssh --cluster stag [IDENTIFER] # ubuntu@bastion-stag.mydomain.com used as the bastion host
+sonic ssh --cluster whatever [IDENTIFER] # ec2-user@bastion.mydomain.com used as the bastion host
+```
+
 ### Service to Cluster Mapping
 
 One of the useful options is the `service_cluster`.  This option maps service names to cluster names.  This saves you from  typing the `--cluster` option over and over.  Here is an example `~/.sonic/settings.yml`:
@@ -68,6 +91,6 @@ sonic ssh hi-worker-stag --cluster stag
 ```
 
 
-<a id="prev" class="btn btn-basic" href="{% link _docs/tutorial-execute.md %}">Back</a>
+<a id="prev" class="btn btn-basic" href="{% link _docs/tutorial-list.md %}">Back</a>
 <a id="next" class="btn btn-primary" href="{% link _docs/help.md %}">Next Step</a>
 <p class="keyboard-tip">Pro tip: Use the <- and -> arrow keys to move back and forward.</p>

data/docs/_docs/tutorial-execute.md

@@ -15,7 +15,6 @@ Examples:
 ```sh
 sonic execute hi-web-stag uptime
 sonic execute hi-web-prod uptime
-sonic execute hi-web-prod,hi-worker-prod,hi-clock-prod uptime
 sonic execute i-030033c20c54bf149,i-030033c20c54bf150 uname -a
 sonic execute i-030033c20c54bf149 file://hello.sh
 ```
@@ -42,12 +41,12 @@ The output of the commands ran are also showed in the EC2 Run Command Console.
 
 ### Polymorphic Filter
 
-The `sonic execute` command can understand a variety of different filters.  The filters can be a list of instances ids or a list of EC2 tag names. Note, ECS service names are *not* supported for the filter.
+The `sonic execute` command can understand a variety of different filters.  The filters can be a list of instances ids or one EC2 tag value.  Note, ECS service names are *not* supported for the filter.
 
-Here is an example, where the uptime command will run on any server with a tag value of hi-web-stag,hi-clock-stag, or hi-worker-stag.
+Here is an example, where the uptime command will run on both i-030033c20c54bf149 and i-030033c20c54bf150 instances.
 
 ```sh
-sonic execute hi-web-stag,hi-clock-stag,hi-worker-stag uptime
+sonic execute i-066b140d9479e9681,i-09482b1a6e330fbf7 uptime
 ```
 
 ### Run Scripts
@@ -75,5 +74,5 @@ The `sonic execute` command relies on EC2 Run Manager. So you will need to have
 * You can read on [Why EC2 Run Manager]({% link _docs/why-ec2-run-command.md %}) is used also.
 
 <a id="prev" class="btn btn-basic" href="{% link _docs/tutorial-ecs-run.md %}">Back</a>
-<a id="next" class="btn btn-primary" href="{% link _docs/settings.md %}">Next Step</a>
+<a id="next" class="btn btn-primary" href="{% link _docs/tutorial-list.md %}">Next Step</a>
 <p class="keyboard-tip">Pro tip: Use the <- and -> arrow keys to move back and forward.</p>

data/docs/_docs/tutorial-list.md

@@ -0,0 +1,50 @@
+---
+title: Sonic List
+---
+
+Sonic provides a quick way to list your instances.
+
+```sh
+sonic list [FILTER]
+```
+
+Example:
+
+```sh
+sonic list i-066b140d9479e9681,i-09482b1a6e330fbf7
+sonic list ec2-tag-1,ec2-tag-2
+```
+
+You should see something like this:
+
+```sh
+$ sonic list i-066b140d9479e9681,i-09482b1a6e330fbf7
+i-09482b1a6e330fbf7 prod-20170416110240 54.202.152.168  172.31.21.108 t2.small
+i-066b140d9479e9681 docker-20170428071833 34.211.144.113  172.31.11.250 m3.medium
+$
+```
+
+You can include a header with the `--header` option:
+
+```sh
+$ sonic list i-066b140d9479e9681,i-09482b1a6e330fbf7 --header
+Instance Id Name  Public IP Private IP  Type
+i-09482b1a6e330fbf7 prod-20170416110240 54.202.152.168  172.31.21.108 t2.small
+i-066b140d9479e9681 docker-20170428071833 34.211.144.113  172.31.11.250 m3.medium
+$
+```
+
+The list command can be helpful if you want to list out the instances and pipe it back into other tools. Here's a simple example:
+
+```sh
+$ for i in $(sonic list i-066b140d9479e9681,i-09482b1a6e330fbf7 | awk '{print $3}') ; do echo $i ; ssh ec2-user@$i uptime ; done
+54.202.152.168
+ 17:39:14 up 6 days,  1:24,  0 users,  load average: 0.00, 0.00, 0.00
+34.211.144.113
+ 17:39:14 up 3 days, 12:03,  0 users,  load average: 0.00, 0.00, 0.00
+$
+```
+
+<a id="prev" class="btn btn-basic" href="{% link _docs/tutorial-execute.md %}">Back</a>
+<a id="next" class="btn btn-primary" href="{% link _docs/settings.md %}">Next Step</a>
+<p class="keyboard-tip">Pro tip: Use the <- and -> arrow keys to move back and forward.</p>

data/docs/_includes/subnav.html

@@ -13,6 +13,7 @@
                 <li><a href="{% link _docs/tutorial-ecs-exec.md %}">ECS Exec</a></li>
                 <li><a href="{% link _docs/tutorial-ecs-run.md %}">ECS Run</a></li>
                 <li><a href="{% link _docs/tutorial-execute.md %}">Execute</a></li>
+                <li><a href="{% link _docs/tutorial-list.md %}">List</a></li>
             </ul>
         </li>
         <li><a href="{% link _docs/settings.md %}">Settings</a></li>

data/lib/sonic/cli.rb

@@ -30,6 +30,7 @@ module Sonic
 
     desc "execute [FILTER] [COMMAND]", "runs command across fleet of servers via AWS Run Command"
     long_desc Help.execute
+    option :zero_warn, type: :boolean, default: true, desc: "Warns user when no instances found"
     # filter - Filter ec2 instances by tag name or instance_ids separated by commas
     def execute(filter, *command)
       Execute.new(command, options.merge(filter: filter)).execute

data/lib/sonic/default/settings.yml

@@ -1,4 +1,8 @@
-bastion: # defaults to nil
+bastion: # cluster_host mapping
+  default: # default is nil - which means a bastion host wont be used
+  # Examples:
+  # prod: bastion.mydomain.com
+  # stag: ubuntu@bastion-stag.mydomain.com
 host_key_check: false
 service_cluster:
   default: # defaults to nil

data/lib/sonic/execute.rb

@@ -1,3 +1,4 @@
+require "byebug"
 module Sonic
   class Execute
     include AwsServices
@@ -19,16 +20,23 @@ module Sonic
       if @options[:noop]
         UI.noop = true
         command_id = "fake command id"
+        success = true # fake it for specs
       else
-        resp = ssm.send_command(ssm_options)
-        command_id = resp.command.command_id
+        instances_count = check_instances
+        return unless instances_count > 0
+
+        success = nil
+        begin
+          resp = ssm.send_command(ssm_options)
+          command_id = resp.command.command_id
+          success = true
+        rescue Aws::SSM::Errors::InvalidInstanceId => e
+          ssm_invalid_instance_error_message(e)
+        end
       end
-      UI.say "Command sent to AWS SSM. To check the details of the command:"
-      list_command = "aws ssm list-commands --command-id #{command_id}"
-      UI.say list_command
-      if RUBY_PLATFORM =~ /darwin/
-        system("echo '#{list_command}' | pbcopy")
-        UI.say "Pro tip: the aws ssm command is already in your copy/paste clipboard."
+      if success
+        UI.say "Command sent to AWS SSM. To check the details of the command:"
+        display_list_command(command_id)
       end
     end
 
@@ -44,36 +52,6 @@ module Sonic
       )
     end
 
-    def build_command(command)
-      if file_path?(command)
-        path = file_path(command)
-        if File.exist?(path)
-          IO.readlines(path).map {|s| s.strip}
-        else
-          UI.error("File #{path} could not be found. Are you sure it exist?")
-          exit 1
-        end
-      else
-        # The script is being feed inline so just join the command together into one script.
-        # Still keep in an array form because that's how ssn.send_command with AWS-RunShellScript
-        # usually reads the command.
-        [command.join(" ")]
-      end
-    end
-
-    def file_path?(command)
-      return false unless command.size == 1
-      possible_path = command.first
-      possible_path.include?("file://")
-    end
-
-    def file_path(command)
-      path = command.first
-      path = path.sub('file://', '')
-      path = "#{@options[:project_root]}/#{path}" if @options[:project_root]
-      path
-    end
-
     #
     # Public: Transform the filter to the ssm send_command equivalent options
     #
@@ -119,6 +97,69 @@ module Sonic
       end
     end
 
+    def build_command(command)
+      if file_path?(command)
+        path = file_path(command)
+        if File.exist?(path)
+          IO.readlines(path).map {|s| s.strip}
+        else
+          UI.error("File #{path} could not be found. Are you sure it exist?")
+          exit 1
+        end
+      else
+        # The script is being feed inline so just join the command together into one script.
+        # Still keep in an array form because that's how ssn.send_command works with AWS-RunShellScript
+        # usually reads the command.
+        [command.join(" ")]
+      end
+    end
+
+    # e = Aws::SSM::Errors::InvalidInstanceId
+    def ssm_invalid_instance_error_message(e)
+      # e.message is an empty string so not very helpful
+      ssm_describe_command = 'aws ssm describe-instance-information --output text --query "InstanceInformationList[*]"'
+      message = <<-EOS
+One of the instance ids: #{@filter.join(",")} is invalid according to SSM.
+This might be because the SSM agent on the instance has not yet checked in.
+You can use the following command to check registered instances to SSM.
+#{ssm_describe_command}
+      EOS
+      UI.warn(message)
+      copy_paste_clipboard(ssm_describe_command)
+    end
+
+    def file_path?(command)
+      return false unless command.size == 1
+      possible_path = command.first
+      possible_path.include?("file://")
+    end
+
+    def file_path(command)
+      path = command.first
+      path = path.sub('file://', '')
+      path = "#{@options[:project_root]}/#{path}" if @options[:project_root]
+      path
+    end
+
+    # Counts the number of instances found using the filter and displays a helpful
+    # message to the user if 0 found.
+    def check_instances
+      return if @options[:zero_warn] == false
+
+      # The list options is a superset of the execute options so we can pass
+      # it right through
+      instances = List.new(@options).instances
+      if instances.count == 0
+        message = <<~EOL
+          Unable to find any instances with filter #{@filter.join(',')}.
+          Are you sure you specify the filter with either a EC2 tag or list instance ids?
+          If you are using ECS identifiers, they are not supported with this command.
+        EOL
+        UI.warn(message)
+      end
+      instances.count
+    end
+
     # TODO: make configurable
     def tag_name
       "Name"
@@ -129,5 +170,17 @@ module Sonic
       # old format is 8 characters long after i-
       text =~ /i-.{17}/ || text =~ /i-.{8}/
     end
+
+    def display_list_command(command_id)
+      list_command = "aws ssm list-commands --command-id #{command_id}"
+      UI.say list_command
+      copy_paste_clipboard(list_command)
+    end
+
+    def copy_paste_clipboard(command)
+      return unless RUBY_PLATFORM =~ /darwin/
+      system("echo '#{command}' | pbcopy")
+      UI.say "Pro tip: the aws ssm command is already in your copy/paste clipboard."
+    end
   end
 end

data/lib/sonic/list.rb

@@ -8,20 +8,22 @@ module Sonic
     end
 
     def run
+      display(instances)
+    end
+
+    def instances
+      return [] if @options[:noop]
+
       filter_options = transform_filter_option(@filter)
-      if @options[:noop]
-        instances = []
-      else
-        begin
-          instances = ec2_resource.instances(filter_options)
-          instances.count # force eager loading
-        rescue Aws::EC2::Errors::InvalidInstanceIDNotFound => e
-          # ERROR: The instance ID 'i-066b140d9479e9682' does not exist
-          UI.error(e.message)
-          exit 1
-        end
+      begin
+        instances = ec2_resource.instances(filter_options)
+        instances.count # force eager loading
+      rescue Aws::EC2::Errors::InvalidInstanceIDNotFound => e
+        # ERROR: The instance ID 'i-066b140d9479e9682' does not exist
+        UI.error(e.message)
+        exit 1
       end
-      display(instances)
+      instances
     end
 
     def display(instances)

data/lib/sonic/settings.rb

@@ -19,7 +19,8 @@ module Sonic
       default = YAML.load_file(default_file)
 
       @data = default.merge(user.merge(project))
-      ensure_default_cluster(@data)
+      ensure_default_cluster!(@data)
+      ensure_default_bastion!(data)
       @data
     end
 
@@ -55,13 +56,46 @@ module Sonic
     # When user's .sonic/settings.yml lack the default cluster, we add it on.
     # Otherwise the user get confusing and scary aws-sdk-core/param_validator errors:
     # Example: https://gist.github.com/sonic/67b9a68a77363b908d1c36047bc2709a
-    def ensure_default_cluster(data)
+    def ensure_default_cluster!(data)
       unless data["service_cluster"]["default"]
         data["service_cluster"]["default"] = "default"
       end
       data
     end
 
+    # Public: Returns default bastion host.
+    #
+    # cluster  - cluster provided by user
+    #
+    # The settings.yml format:
+    #
+    # bastion:
+    #   default: bastion.mydomain.com
+    #   prod: bastion.mydomain.com
+    #   stag: bastion-stag.mydomain.com
+    #
+    # Examples
+    #
+    #   default_bastion('stag')
+    #   # => 'bastion-stag.mydomain.com'
+    #   default_bastion('whatever')
+    #   # => 'bastion.mydomain.com'
+    #
+    # Returns the bastion host that is mapped to the cluster
+    def default_bastion(cluster)
+      bastion = data["bastion"]
+      bastion[cluster] || bastion["default"]
+    end
+
+    # When user's .sonic/settings.yml lack the default cluster, we add it on.
+    def ensure_default_bastion!(data)
+      unless data["bastion"] && data["bastion"].has_key?("default")
+        data["bastion"] ||= {}
+        data["bastion"]["default"] = nil
+      end
+      data
+    end
+
     def host_key_check_options
       if data["host_key_check"]
         # no options by default enables strict host key checking

data/lib/sonic/ssh.rb

@@ -16,7 +16,7 @@ module Sonic
 
       @service = @identifier # always set service even though it's not always used as the identifier
       @cluster = options[:cluster] || settings.default_cluster(@service)
-      @bastion = options[:bastion] || settings.data["bastion"]
+      @bastion = options[:bastion] || settings.default_bastion(@bastion)
     end
 
     def run

data/lib/sonic/ui.rb

@@ -21,6 +21,9 @@ module Sonic
       def error(msg='')
         say "ERROR: #{msg}".colorize(:red)
       end
+      def warn(msg='')
+        say "WARN: #{msg}".colorize(:yellow)
+      end
     end
   end
 end

data/lib/sonic/version.rb

@@ -1,3 +1,3 @@
 module Sonic
-  VERSION = "1.2.0"
+  VERSION = "1.3.0"
 end

data/spec/lib/sonic/execute_spec.rb

@@ -4,7 +4,7 @@ describe Sonic::Execute do
   before(:each) do
     @options = {
       project_root: "spec/fixtures/project",
-      filter: "hi-web-stag,hi-clock-stag"
+      filter: "i-066b140d9479e9681,i-09482b1a6e330fbf7"
     }
   end
 
@@ -12,8 +12,7 @@ describe Sonic::Execute do
     it "should build up options for ssm send command with inline command" do
       execute = Sonic::Execute.new(["uname", "-a"], @options)
       options = execute.build_ssm_options
-      expect(options[:instance_ids]).to eq nil
-      expect(options[:targets]).to eq [{:key=>"tag:Name", :values=>%w[hi-web-stag hi-clock-stag]}]
+      expect(options[:instance_ids]).to eq %w[i-066b140d9479e9681 i-09482b1a6e330fbf7]
       expect(options[:document_name]).to eq "AWS-RunShellScript"
       expect(options[:comment]).to include "sonic "
       expect(options[:parameters]["commands"]).to eq ["uname -a"]
@@ -22,8 +21,7 @@ describe Sonic::Execute do
     it "should build up options for ssm send command with file" do
       execute = Sonic::Execute.new(["file://command.txt"], @options)
       options = execute.build_ssm_options
-      expect(options[:instance_ids]).to eq nil
-      expect(options[:targets]).to eq [{:key=>"tag:Name", :values=>%w[hi-web-stag hi-clock-stag]}]
+      expect(options[:instance_ids]).to eq %w[i-066b140d9479e9681 i-09482b1a6e330fbf7]
       expect(options[:document_name]).to eq "AWS-RunShellScript"
       expect(options[:comment]).to include "sonic "
       expect(options[:parameters]["commands"]).to eq([

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: sonic-screwdriver
 version: !ruby/object:Gem::Version
-  version: 1.2.0
+  version: 1.3.0
 platform: ruby
 authors:
 - Tung Nguyen
@@ -183,6 +183,7 @@ files:
 - docs/_docs/tutorial-ecs-exec.md
 - docs/_docs/tutorial-ecs-run.md
 - docs/_docs/tutorial-execute.md
+- docs/_docs/tutorial-list.md
 - docs/_docs/tutorial-ssh.md
 - docs/_docs/tutorial.md
 - docs/_docs/why-ec2-run-command.md