sonic-screwdriver 1.4.0 → 2.0.0

data/docs/_docs/install.md

@@ -30,14 +30,14 @@ For more information about the Bolts Toolbelt or to get an installer for another
 
 For a small set of the commands there are server side dependencies.
 
-#### sonic ecs-* dependencies
+#### sonic ecs dependencies
 
-For the `sonic ecs-*` commands to work `jq` is required on the server side. This is covered in the [How It Works]({% link _docs/how-it-works.md %}) section.
+For the `sonic ecs` commands to work `jq` is required on the server side. This is covered in the [How It Works]({% link _docs/how-it-works.md %}) section.
 
 One way to install `jq` quickly is by using the `sonic execute` command.  For example:
 
 ```sh
-sonic execute hi-web-stag yum install -y jq
+sonic execute hi-web yum install -y jq
 ```
 
 It is recommended that you install `jq` with the UserData script or bake it into the AMI though.

data/docs/_docs/settings.md

@@ -2,92 +2,76 @@
 title: Settings
 ---
 
-You can adjust the behavior of sonic and set some handy default values with `settings.yml` files.  There can exist multiple settings files which all get loaded and merged. The options from the files follow the following precedence rules:
+You can adjust the behavior of sonic and set some handy default values with settings files.  The settings files used are determined by the value of environment variable `SONIC_PROFILE` or `AWS_PROFILE`. The value determines the settings profile to use.  There can exist multiple settings files which all get loaded and merged. The options from the files follow the following precedence rules:
 
-1. current folder - The current folder's `.sonic/settings.yml` values take the highest precedence. The current folder is typically the project folder.
-2. user - The user's `~/.sonic/settings.yml` values take the second highest precedence.
+1. current folder - The current folder's `.sonic/[AWS_PROFILE].yml` values take the highest precedence. The current folder is typically the project folder.
+2. user - The user's `~/.sonic/[AWS_PROFILE].yml` values take the second highest precedence.
 3. default - The [default settings](https://github.com/boltopslabs/sonic/blob/master/lib/sonic/default/settings.yml) bundled with the tool takes the lowest precedence.
 
-Here is an `settings.yml` example:
+A concrete example helps explain it. Let's say `AWS_PROFILE=prod-profile` with the following files:
 
-```yaml
-bastion: # cluster_host mapping
-  default: # default is nil - which means a bastion host wont be used
-  # Examples:
-  # prod: bastion.mydomain.com
-  # stag: ubuntu@bastion-stag.mydomain.com
-host_key_check: false
-service_cluster:
-  default: # defaults to nil
-  hi-web-prod: prod
-  hi-clock-prod: prod
-  hi-worker-prod: prod
-  hi-web-stag: stag
-  hi-clock-stag: stag
-  hi-worker-stag: stag
-user: ec2-user
-```
+* in current folder: `.sonic/prod-profile.yml`
+* in user home folder: `~/.sonic/prod-profile.yml`
 
-The following table covers the different setting options:
-
-Setting  | Description | Default
-------------- | ------------- | -------------
-bastion  | Bastion mapping allows you to set a bastion host on a per ECS cluster basis.  The bastion host is used as the jump host. Examples: bastion.mydomain.com, myuser@bastion.myuser.com or 123.123.123.123. | (no value)
-host_key_check  | Controls whether or not use the strict host checking ssh option.  Since EC2 server host changes often the default value is false. | false
-service_cluster  | Service to cluster mapping.  This is a Hash structure that maps the service name to cluster names. | (no value)
-user  | User to ssh into the server with. This can be overriden at the CLI with the user@host notation but can be set in the settings.yml file also. | ec2-user
+Then the `.sonic/prod-profile.yml` gets merged into `~/.sonic/prod-profile.yml` and that in turn gets merged into sonic's [default settings](https://github.com/boltopslabs/sonic/blob/master/lib/sonic/default/settings.yml).
 
-The default settings are located tool source code at [lib/sonic/default/settings.yml](https://github.com/boltopslabs/sonic/blob/master/lib/sonic/default/settings.yml).
+You can also use the `SONIC_PROFILE=prod-profile` environment variable instead of the `AWS_PROFILE` environment variable. The `SONIC_PROFILE` wins over the `AWS_PROFILE` value.
 
-### Bastion cluster to host mapping
+## Full Example
 
-Provided this example:
+Here is an `prod-profile.yml` example:
 
 ```yaml
 bastion: # cluster_host mapping
-  default: ec2-user@bastion.mydomain.com
-  prod: ec2-user@bastion.mydomain.com
-  stag: ubuntu@bastion-stag.mydomain.com
+  user: ec2-user
+  host: # default is nil - which means a bastion host wont be used
+  host_key_check: false
+
+# used with `sonic ecs ssh` command
+ecs_service_cluster_map:
+  default: default # default cluster
+  hi-web: production
+  hi-clock: production
+  hi-worker: production
 ```
 
-This results in
+The following table covers the different setting options:
 
-```sh
-sonic ssh --cluster prod [IDENTIFER] # ec2-user@bastion.mydomain.com used as the bastion host
-sonic ssh --cluster stag [IDENTIFER] # ubuntu@bastion-stag.mydomain.com used as the bastion host
-sonic ssh --cluster whatever [IDENTIFER] # ec2-user@bastion.mydomain.com used as the bastion host
-```
+Setting  | Description | Default
+------------- | ------------- | -------------
+bastion.user  | User to ssh into the server with. This can be overriden at the CLI with the user@host notation but can be set in the settings.yml file also. | ec2-user
+bastion.host  | Bastion mapping allows you to set a bastion host on a per ECS cluster basis.  The bastion host is used as the jump host. Examples: bastion.mydomain.com, myuser@bastion.myuser.com or 123.123.123.123. | (no value)
+bastion.host_key_check  | Controls whether or not use the strict host checking ssh option.  Since EC2 server host changes often the default value is false. | false
+ecs_service_cluster_map  | Service to cluster mapping.  This is a Hash structure that maps the service name to cluster names. | (no value)
+
+The default settings are located tool source code at [lib/sonic/default/settings.yml](https://github.com/boltopslabs/sonic/blob/master/lib/sonic/default/settings.yml).
 
 ### Service to Cluster Mapping
 
-One of the useful options is the `service_cluster`.  This option maps service names to cluster names.  This saves you from  typing the `--cluster` option over and over.  Here is an example `~/.sonic/settings.yml`:
+A useful option is the `ecs_service_cluster_map`.  This option maps ecs service names to cluster names.  This saves you from  typing the `--cluster` option repeatedly.  Here is an example `~/.sonic/settings.yml`:
 
 ```yaml
-user: ec2-user
-service_cluster:
+ecs_service_cluster_map:
   default: my-default-cluster
-  hi-web-prod: prod
-  hi-clock-prod: prod
-  hi-worker-prod: prod
-  hi-web-stag: stag
-  hi-clock-stag: stag
-  hi-worker-stag: stag
+  hi-web: production
+  hi-clock: production
+  hi-worker: production
 ```
 
 This results in shorter commands:
 
 ```
-sonic ssh hi-web-prod
-sonic ssh hi-clock-prod
-sonic ssh hi-worker-stag
+sonic ecs ssh hi-web
+sonic ecs ssh hi-clock
+sonic ecs ssh hi-worker
 ```
 
 Instead of what you normally would have to type:
 
 ```
-sonic ssh hi-web-prod --cluster prod
-sonic ssh hi-clock-prod --cluster prod
-sonic ssh hi-worker-stag --cluster stag
+sonic ecs ssh hi-web --cluster production
+sonic ecs ssh hi-clock --cluster production
+sonic ecs ssh hi-worker --cluster production
 ```
 
 

data/docs/_docs/tutorial-ecs-exec.md

@@ -8,26 +8,24 @@ One of the additional things `sonic` can do is hop one more level and get you al
 
 It does this with a variety of scripts and trickery and is covered in [How It Works]({% link _docs/how-it-works.md %}).  Let's go through examples of how sonic can help you get into an running ECS docker container quickly.
 
-### sonic ecs-exec
+### sonic ecs exec
 
 ```sh
-sonic ecs-exec [ECS_SERVICE] --cluster [ECS_CLUSTER]
+sonic ecs exec [ECS_SERVICE] --cluster [ECS_CLUSTER]
 ```
 
 Here's a concrete example:
 
 ```sh
-sonic ecs-exec hi-web-stag --cluster stag
+sonic ecs exec hi-web --cluster staging
 ```
 
 You should see something like this:
 
 ```sh
-$ sonic ecs-exec hi-web-stag --cluster stag
+$ sonic ecs exec hi-web --cluster staging
 Running: scp -r /tmp/sonic ec2-user@34.211.195.71:/tmp/sonic  > /dev/null
-Warning: Permanently added '34.211.195.71' (ECDSA) to the list of known hosts.
 => ssh -t ec2-user@34.211.195.71 bash /tmp/sonic/bash_scripts/docker-exec.sh
-Warning: Permanently added '34.211.195.71' (ECDSA) to the list of known hosts.
 root@fc4035f90bdc:/app#
 ```
 
@@ -38,7 +36,7 @@ What you see in the last line above is a bash prompt because you are in a bash s
 Here are examples to show what is possible:
 
 ```
-$ sonic ecs-exec hi-web-stag bash
+$ sonic ecs exec hi-web bash
 # You're in the docker container now
 $ ls # check out some files to make sure you're the right place
 $ ps auxxx | grep puma # is the web process up?
@@ -49,7 +47,7 @@ $ bundle exec rails c # start up a rails console to debug
 You can also pass in bundle exec rails console if you want to get to that as quickly as possible.
 
 ```
-$ sonic ecs-exec hi-web-stag bundle exec rails console
+$ sonic ecs exec hi-web bundle exec rails console
 # You're a rails console in the docker container now
 > User.count
 ```
@@ -57,36 +55,34 @@ $ sonic ecs-exec hi-web-stag bundle exec rails console
 You can also use the container instance id or instance id in place of the service name:
 
 ```
-sonic ecs-exec 9f1dadc7-4f67-41da-abec-ec08810bfbc9 bash
-sonic ecs-exec i-006a097bb10643e20 bash
+sonic ecs exec 9f1dadc7-4f67-41da-abec-ec08810bfbc9 bash
+sonic ecs exec i-006a097bb10643e20 bash
 ```
 
-### Settings - service_cluster mapping
+### Settings - ecs_service_cluster_map
 
 As mentioned in the [previous section]({% link _docs/tutorial-ssh.md %}) and also in the [Settings documentation]({% link _docs/settings.md %}) you can configure a `~/.sonic/settings.yml` file which shortens the command further.  Let's add this to your settings:
 
 ```yaml
-service_cluster:
-  default: stag
-  hi-web-stag: stag
+ecs_service_cluster_map:
+  default: staging
+  hi-web: staging
 ```
 
 This makes the command consise and memorable.
 
 ```sh
-sonic ecs-exec hi-web-stag
+sonic ecs exec hi-web
 ```
 
 The rest of this section assumes that you have the `~/.sonic/settings.yml` set up.
 
-You can also tack on a command at the end of the `ecs-exec` command to be run as a one off instead of starting a bash shell. Example:
+You can also tack on a command at the end of the `ecs exec` command to be run as a one off instead of starting a bash shell. Example:
 
 ```
-$ sonic ecs-exec hi-web-stag uname -a
+$ sonic ecs exec hi-web uname -a
 Running: scp -r /tmp/sonic ec2-user@34.211.195.71:/tmp/sonic  > /dev/null
-Warning: Permanently added '34.211.195.71' (ECDSA) to the list of known hosts.
 => ssh -t ec2-user@34.211.195.71 bash /tmp/sonic/bash_scripts/docker-exec.sh uname -a
-Warning: Permanently added '34.211.195.71' (ECDSA) to the list of known hosts.
 Linux fc4035f90bdc 4.4.51-40.58.amzn1.x86_64 #1 SMP Tue Feb 28 21:57:17 UTC 2017 x86_64 GNU/Linux
 Connection to 34.211.195.71 closed.
 $
@@ -95,5 +91,5 @@ $
 Remember the command runs within the running docker container.
 
 <a id="prev" class="btn btn-basic" href="{% link _docs/tutorial-ssh.md %}">Back</a>
-<a id="next" class="btn btn-primary" href="{% link _docs/tutorial-ecs-run.md %}">Next Step</a>
+<a id="next" class="btn btn-primary" href="{% link _docs/tutorial-ecs-sh.md %}">Next Step</a>
 <p class="keyboard-tip">Pro tip: Use the <- and -> arrow keys to move back and forward.</p>

data/docs/_docs/tutorial-ecs-sh.md

@@ -0,0 +1,73 @@
+---
+title: ECS Run
+---
+
+The nice thing about the previous `ecs exec` command we covered is that it allows you to get into the actual running container and debug with the exact environment that is on production.  The cavaet with doing this is that we are affecting a live process that could be in actual use. If you do something inadvertently wrong on the server it could affect users.  Sometimes it is nice to start up a new container with the exact same environment as the other running containers but be isolated so you cannot affect live requests.
+
+The `sonic ecs sh` command is similar to the `sonic ecs exec` command except it'll run a brand new container with the same environment variables as the task associated with the service. This allows you to debug in a container with the exact environment variables as the running tasks/containers without affecting the live service. So this is safer since you will not be able to mess up a live container that is in service.
+
+### sonic ecs sh
+
+```sh
+sonic ecs sh [ECS_SERVICE] --cluster [ECS_CLUSTER]
+```
+
+Here's an example:
+
+```sh
+sonic ecs sh hi-web
+```
+
+You see something like this:
+
+```sh
+$ sonic ecs sh hi-web
+Running: scp -r /tmp/sonic ec2-user@34.211.195.71:/tmp/sonic  > /dev/null
+=> ssh -t ec2-user@34.211.195.71 bash /tmp/sonic/bash_scripts/docker-run.sh
++ exec docker exec -ti 385b643c7a895231d2b193574368b0c6c6bebce487267c3c175d0acea3082d4c bash
+root@29e7c1253c46:/app#
+$
+```
+
+You are now in a docker container running exactly the same environment as the other running containers with the `hi-web` service. While this looks similiar to the `ecs exec` command this container is a brand new process and is isolated from any live request. You can do whatever you want in this container and experiment to your heart's content.
+
+We can prove that this is a brand new docker container that is outside of ECS' knowledge. Let's ssh into the same instance and take a look at all the running docker containers in another terminal.
+
+```sh
+$ sonic ssh hi-web docker ps
+=> ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ec2-user@34.211.195.71 docker ps
+CONTAINER ID        IMAGE                                          COMMAND             CREATED             STATUS              PORTS                     NAMES
+29e7c1253c46        tongueroo/hi:ufo-2017-06-13T14-48-08-0a9eea5   "bash"              54 seconds ago      Up 53 seconds       3000/tcp                  cocky_goldstine
+fc4035f90bdc        tongueroo/hi:ufo-2017-06-13T14-48-08-0a9eea5   "bin/web"           About an hour ago   Up About an hour    0.0.0.0:32768->3000/tcp   ecs-hi-web-11-web-9eb081978abad89a9701
+bf646ae7789a        amazon/amazon-ecs-agent:latest                 "/agent"            About an hour ago   Up About an hour                              ecs-agent
+$
+```
+
+The output shows that there is this extra runnning container called `cocky_goldstine`.  This name does not look like the typical ECS managed running docker container: `ecs-hi-web-11-web-9eb081978abad89a9701`.  This is how we can tell that this is a container outside of ECS control.
+
+```sh
+$ sonic ecs sh hi-web bash
+Running: scp -r /tmp/sonic ec2-user@34.211.195.71:/tmp/sonic  > /dev/null
+=> ssh -t ec2-user@34.211.195.71 bash /tmp/sonic/bash_scripts/docker-run.sh bash
+root@29e7c1253c46:/app# exit
+exit
+Connection to 34.211.195.71 closed.
+$
+```
+
+Let's exit out of the first terminal where you ran the original `sonic ecs sh` command and then list the running containers again.
+
+```sh
+$ sonic ssh hi-web docker ps
+=> ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ec2-user@34.211.195.71 docker ps
+CONTAINER ID        IMAGE                                          COMMAND             CREATED             STATUS              PORTS                     NAMES
+fc4035f90bdc        tongueroo/hi:ufo-2017-06-13T14-48-08-0a9eea5   "bin/web"           About an hour ago   Up About an hour    0.0.0.0:32768->3000/tcp   ecs-hi-web-11-web-9eb081978abad89a9701
+bf646ae7789a        amazon/amazon-ecs-agent:latest                 "/agent"            About an hour ago   Up About an hour                              ecs-agent
+$
+```
+
+Zapped!  The `cocky_goldstine` container that was created with `sonic ecs sh` is no more.
+
+<a id="prev" class="btn btn-basic" href="{% link _docs/tutorial-ecs-exec.md %}">Back</a>
+<a id="next" class="btn btn-primary" href="{% link _docs/tutorial-execute.md %}">Next Step</a>
+<p class="keyboard-tip">Pro tip: Use the <- and -> arrow keys to move back and forward.</p>

data/docs/_docs/tutorial-execute.md

@@ -4,38 +4,68 @@ title: Sonic Execute
 
 ### Run One Liners
 
-Sonic provides a way to execute commands remotely and securely across a list of AWS servers.  It does this by leveraging [Amazon EC2 Run Command](https://aws.amazon.com/ec2/execute/).  Sonic hides any complexity and provides a simple interface for you.   The command is called `sonic execute`:
+Sonic provides a way to execute commands remotely and securely across a list of AWS servers.  It does this by leveraging [Amazon EC2 Run Command](https://aws.amazon.com/ec2/execute/).  Sonic a simple interface and some conveniences for you.   The command is called `sonic execute`:
 
 ```sh
 sonic execute [FILTER] [COMMAND]
 ```
 
-Examples:
+## Examples Summary
 
 ```sh
-sonic execute hi-web-stag uptime
+sonic execute hi-web uptime
 sonic execute hi-web-prod uptime
 sonic execute i-030033c20c54bf149,i-030033c20c54bf150 uname -a
 sonic execute i-030033c20c54bf149 file://hello.sh
 ```
 
-Let's do something more useful:
+## Example Detailed
 
-```sh
-sonic execute hi-web-stag yum install -y curl
-```
-
-The output of the command will show a useful `aws ssm list-commands` command to get status of the requested command.
+Here's a command example output in detailed:
 
 ```sh
-$ sonic execute hi-web-stag uptime
+$ sonic execute i-0bf51a000ab4e73a8 uptime
+Sending command to SSM with options:
+---
+instance_ids:
+- i-0bf51a000ab4e73a8
+document_name: AWS-RunShellScript
+comment: sonic execute i-0bf51a000ab4e73a8 uptime
+parameters:
+  commands:
+  - uptime
+output_s3_region: us-east-1
+output_s3_bucket_name: [reacted]
+output_s3_key_prefix: ssm/commands/sonic
+
 Command sent to AWS SSM. To check the details of the command:
-aws ssm list-commands --command-id 4133e5eb-aa18-40dd-be25-a176eb15e515
-Pro tip: the aws ssm command is already in your copy/paste clipboard.
+  aws ssm list-commands --command-id 0bb18d58-6436-49fd-9bfd-0c4b6c51c7a2
+  aws ssm get-command-invocation --command-id 0bb18d58-6436-49fd-9bfd-0c4b6c51c7a2 --instance-id i-0bf51a000ab4e73a8
+
+Waiting for ssm command to finish.....
+Command finished.
+
+Displaying output for i-0bf51a000ab4e73a8.
+Command status: Success
+Command standard output:
+ 01:08:10 up 8 days,  6:41,  0 users,  load average: 0.00, 0.00, 0.00
+
+To see the more output details visit:
+  https://us-west-2.console.aws.amazon.com/systems-manager/run-command/0bb18d58-6436-49fd-9bfd-0c4b6c51c7a2
+
+Pro tip: the console url is already in your copy/paste clipboard.
 $
 ```
 
-The output of the commands ran are also showed in the EC2 Run Command Console.  Here's an example:
+Notice the conveniences of `sonic execute`, it:
+
+1. Showed the parameters that will be sent as part of the send_command call to SSM.
+2. Sent the command to SSM.
+3. Waited for the command to finish.
+4. Displayed the output of the command.
+5. Provided the console url that visit to view more details about the SSM command.
+
+The AWS SSM console looks like this:
 
 <img src="/img/tutorials/ec2-console-run-command.png" class="doc-photo" />
 
@@ -43,13 +73,59 @@ The output of the commands ran are also showed in the EC2 Run Command Console.
 
 The `sonic execute` command can understand a variety of different filters.  The filters can be a list of instances ids or one EC2 tag value.  Note, ECS service names are *not* supported for the filter.
 
-Here is an example, where the uptime command will run on both i-030033c20c54bf149 and i-030033c20c54bf150 instances.
+Here is an example, where the uptime command will run on both `i-030033c20c54bf149` and `i-030033c20c54bf150` instances.
 
 ```sh
 sonic execute i-066b140d9479e9681,i-09482b1a6e330fbf7 uptime
 ```
 
-### Run Scripts
+## Windows Support
+
+Windows is also supported. When running a command sonic will first attempt to use the `AWS-RunShellScript` run command, and if it detects that the instance's platform does not support `AWS-RunShellScript`, it will run the command with the `AWS-RunPowerShellScript` run command.  Here's an example:
+
+```
+$ sonic execute i-0917ad61b10fa1059 pwd
+Sending command to SSM with options:
+---
+instance_ids:
+- i-0917ad61b10fa1059
+document_name: AWS-RunShellScript
+comment: sonic execute i-0917ad61b10fa1059 pwd
+parameters:
+  commands:
+  - pwd
+output_s3_region: us-east-1
+output_s3_bucket_name: boltops-infra-stag
+output_s3_key_prefix: ssm/commands/sonic
+
+Cannot perform operation for instance id i-0917ad61b10fa1059 of platform type Windows
+Retrying with document_name AWS-RunPowerShellScript
+Retries: 1
+Command sent to AWS SSM. To check the details of the command:
+  aws ssm list-commands --command-id 8a196058-445e-4960-9efb-be746ecf98dc
+  aws ssm get-command-invocation --command-id 8a196058-445e-4960-9efb-be746ecf98dc --instance-id i-0917ad61b10fa1059
+
+Waiting for ssm command to finish......
+Command finished.
+
+Displaying output for i-0917ad61b10fa1059.
+Command status: Success
+Command standard output:
+
+Path
+----
+C:\Windows\system32
+
+
+
+To see the more output details visit:
+  https://us-east-1.console.aws.amazon.com/systems-manager/run-command/8a196058-445e-4960-9efb-be746ecf98dc
+
+Pro tip: the console url is already in your copy/paste clipboard.
+$
+```
+
+## Run Scripts
 
 Sometimes you might want to run more than just a one-liner command. If you need to run a full script, you can provide the file path to the script by designating it with `file://`.  For example, here's a file called `hi.sh`:
 
@@ -61,7 +137,7 @@ echo "hello world"
 Here's how you run that file:
 
 ```sh
-sonic execute hi-web-stag file://hi.sh
+sonic execute hi-web file://hi.sh
 ```
 
 The file gets read by `sonic execute` and sent to EC2 Run Command to be executed.
@@ -73,6 +149,6 @@ The `sonic execute` command relies on EC2 Run Manager. So you will need to have
 * You can follow the [installation guide]({% link _docs/install.md %}) to install EC2 Run Manager.
 * You can read on [Why EC2 Run Manager]({% link _docs/why-ec2-run-command.md %}) is used also.
 
-<a id="prev" class="btn btn-basic" href="{% link _docs/tutorial-ecs-run.md %}">Back</a>
+<a id="prev" class="btn btn-basic" href="{% link _docs/tutorial-ecs-sh.md %}">Back</a>
 <a id="next" class="btn btn-primary" href="{% link _docs/tutorial-list.md %}">Next Step</a>
 <p class="keyboard-tip">Pro tip: Use the <- and -> arrow keys to move back and forward.</p>