sonic-screwdriver 1.4.0 → 2.2.1

data/docs/_docs/tutorial-ecs-exec.md

@@ -8,26 +8,24 @@ One of the additional things `sonic` can do is hop one more level and get you al
 
 It does this with a variety of scripts and trickery and is covered in [How It Works]({% link _docs/how-it-works.md %}).  Let's go through examples of how sonic can help you get into an running ECS docker container quickly.
 
-### sonic ecs-exec
+### sonic ecs exec
 
 ```sh
-sonic ecs-exec [ECS_SERVICE] --cluster [ECS_CLUSTER]
+sonic ecs exec [ECS_SERVICE] --cluster [ECS_CLUSTER]
 ```
 
 Here's a concrete example:
 
 ```sh
-sonic ecs-exec hi-web-stag --cluster stag
+sonic ecs exec hi-web --cluster staging
 ```
 
 You should see something like this:
 
 ```sh
-$ sonic ecs-exec hi-web-stag --cluster stag
+$ sonic ecs exec hi-web --cluster staging
 Running: scp -r /tmp/sonic ec2-user@34.211.195.71:/tmp/sonic  > /dev/null
-Warning: Permanently added '34.211.195.71' (ECDSA) to the list of known hosts.
 => ssh -t ec2-user@34.211.195.71 bash /tmp/sonic/bash_scripts/docker-exec.sh
-Warning: Permanently added '34.211.195.71' (ECDSA) to the list of known hosts.
 root@fc4035f90bdc:/app#
 ```
 
@@ -38,7 +36,7 @@ What you see in the last line above is a bash prompt because you are in a bash s
 Here are examples to show what is possible:
 
 ```
-$ sonic ecs-exec hi-web-stag bash
+$ sonic ecs exec hi-web bash
 # You're in the docker container now
 $ ls # check out some files to make sure you're the right place
 $ ps auxxx | grep puma # is the web process up?
@@ -49,7 +47,7 @@ $ bundle exec rails c # start up a rails console to debug
 You can also pass in bundle exec rails console if you want to get to that as quickly as possible.
 
 ```
-$ sonic ecs-exec hi-web-stag bundle exec rails console
+$ sonic ecs exec hi-web bundle exec rails console
 # You're a rails console in the docker container now
 > User.count
 ```
@@ -57,36 +55,34 @@ $ sonic ecs-exec hi-web-stag bundle exec rails console
 You can also use the container instance id or instance id in place of the service name:
 
 ```
-sonic ecs-exec 9f1dadc7-4f67-41da-abec-ec08810bfbc9 bash
-sonic ecs-exec i-006a097bb10643e20 bash
+sonic ecs exec 9f1dadc7-4f67-41da-abec-ec08810bfbc9 bash
+sonic ecs exec i-006a097bb10643e20 bash
 ```
 
-### Settings - service_cluster mapping
+### Settings - ecs_service_cluster_map
 
 As mentioned in the [previous section]({% link _docs/tutorial-ssh.md %}) and also in the [Settings documentation]({% link _docs/settings.md %}) you can configure a `~/.sonic/settings.yml` file which shortens the command further.  Let's add this to your settings:
 
 ```yaml
-service_cluster:
-  default: stag
-  hi-web-stag: stag
+ecs_service_cluster_map:
+  default: staging
+  hi-web: staging
 ```
 
 This makes the command consise and memorable.
 
 ```sh
-sonic ecs-exec hi-web-stag
+sonic ecs exec hi-web
 ```
 
 The rest of this section assumes that you have the `~/.sonic/settings.yml` set up.
 
-You can also tack on a command at the end of the `ecs-exec` command to be run as a one off instead of starting a bash shell. Example:
+You can also tack on a command at the end of the `ecs exec` command to be run as a one off instead of starting a bash shell. Example:
 
 ```
-$ sonic ecs-exec hi-web-stag uname -a
+$ sonic ecs exec hi-web uname -a
 Running: scp -r /tmp/sonic ec2-user@34.211.195.71:/tmp/sonic  > /dev/null
-Warning: Permanently added '34.211.195.71' (ECDSA) to the list of known hosts.
 => ssh -t ec2-user@34.211.195.71 bash /tmp/sonic/bash_scripts/docker-exec.sh uname -a
-Warning: Permanently added '34.211.195.71' (ECDSA) to the list of known hosts.
 Linux fc4035f90bdc 4.4.51-40.58.amzn1.x86_64 #1 SMP Tue Feb 28 21:57:17 UTC 2017 x86_64 GNU/Linux
 Connection to 34.211.195.71 closed.
 $
@@ -95,5 +91,5 @@ $
 Remember the command runs within the running docker container.
 
 <a id="prev" class="btn btn-basic" href="{% link _docs/tutorial-ssh.md %}">Back</a>
-<a id="next" class="btn btn-primary" href="{% link _docs/tutorial-ecs-run.md %}">Next Step</a>
+<a id="next" class="btn btn-primary" href="{% link _docs/tutorial-ecs-sh.md %}">Next Step</a>
 <p class="keyboard-tip">Pro tip: Use the <- and -> arrow keys to move back and forward.</p>

data/docs/_docs/tutorial-ecs-sh.md

@@ -0,0 +1,73 @@
+---
+title: ECS Run
+---
+
+The nice thing about the previous `ecs exec` command we covered is that it allows you to get into the actual running container and debug with the exact environment that is on production.  The cavaet with doing this is that we are affecting a live process that could be in actual use. If you do something inadvertently wrong on the server it could affect users.  Sometimes it is nice to start up a new container with the exact same environment as the other running containers but be isolated so you cannot affect live requests.
+
+The `sonic ecs sh` command is similar to the `sonic ecs exec` command except it'll run a brand new container with the same environment variables as the task associated with the service. This allows you to debug in a container with the exact environment variables as the running tasks/containers without affecting the live service. So this is safer since you will not be able to mess up a live container that is in service.
+
+### sonic ecs sh
+
+```sh
+sonic ecs sh [ECS_SERVICE] --cluster [ECS_CLUSTER]
+```
+
+Here's an example:
+
+```sh
+sonic ecs sh hi-web
+```
+
+You see something like this:
+
+```sh
+$ sonic ecs sh hi-web
+Running: scp -r /tmp/sonic ec2-user@34.211.195.71:/tmp/sonic  > /dev/null
+=> ssh -t ec2-user@34.211.195.71 bash /tmp/sonic/bash_scripts/docker-run.sh
++ exec docker exec -ti 385b643c7a895231d2b193574368b0c6c6bebce487267c3c175d0acea3082d4c bash
+root@29e7c1253c46:/app#
+$
+```
+
+You are now in a docker container running exactly the same environment as the other running containers with the `hi-web` service. While this looks similiar to the `ecs exec` command this container is a brand new process and is isolated from any live request. You can do whatever you want in this container and experiment to your heart's content.
+
+We can prove that this is a brand new docker container that is outside of ECS' knowledge. Let's ssh into the same instance and take a look at all the running docker containers in another terminal.
+
+```sh
+$ sonic ssh hi-web docker ps
+=> ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ec2-user@34.211.195.71 docker ps
+CONTAINER ID        IMAGE                                          COMMAND             CREATED             STATUS              PORTS                     NAMES
+29e7c1253c46        tongueroo/hi:ufo-2017-06-13T14-48-08-0a9eea5   "bash"              54 seconds ago      Up 53 seconds       3000/tcp                  cocky_goldstine
+fc4035f90bdc        tongueroo/hi:ufo-2017-06-13T14-48-08-0a9eea5   "bin/web"           About an hour ago   Up About an hour    0.0.0.0:32768->3000/tcp   ecs-hi-web-11-web-9eb081978abad89a9701
+bf646ae7789a        amazon/amazon-ecs-agent:latest                 "/agent"            About an hour ago   Up About an hour                              ecs-agent
+$
+```
+
+The output shows that there is this extra runnning container called `cocky_goldstine`.  This name does not look like the typical ECS managed running docker container: `ecs-hi-web-11-web-9eb081978abad89a9701`.  This is how we can tell that this is a container outside of ECS control.
+
+```sh
+$ sonic ecs sh hi-web bash
+Running: scp -r /tmp/sonic ec2-user@34.211.195.71:/tmp/sonic  > /dev/null
+=> ssh -t ec2-user@34.211.195.71 bash /tmp/sonic/bash_scripts/docker-run.sh bash
+root@29e7c1253c46:/app# exit
+exit
+Connection to 34.211.195.71 closed.
+$
+```
+
+Let's exit out of the first terminal where you ran the original `sonic ecs sh` command and then list the running containers again.
+
+```sh
+$ sonic ssh hi-web docker ps
+=> ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ec2-user@34.211.195.71 docker ps
+CONTAINER ID        IMAGE                                          COMMAND             CREATED             STATUS              PORTS                     NAMES
+fc4035f90bdc        tongueroo/hi:ufo-2017-06-13T14-48-08-0a9eea5   "bin/web"           About an hour ago   Up About an hour    0.0.0.0:32768->3000/tcp   ecs-hi-web-11-web-9eb081978abad89a9701
+bf646ae7789a        amazon/amazon-ecs-agent:latest                 "/agent"            About an hour ago   Up About an hour                              ecs-agent
+$
+```
+
+Zapped!  The `cocky_goldstine` container that was created with `sonic ecs sh` is no more.
+
+<a id="prev" class="btn btn-basic" href="{% link _docs/tutorial-ecs-exec.md %}">Back</a>
+<a id="next" class="btn btn-primary" href="{% link _docs/tutorial-execute.md %}">Next Step</a>
+<p class="keyboard-tip">Pro tip: Use the <- and -> arrow keys to move back and forward.</p>

data/docs/_docs/tutorial-execute.md

@@ -4,65 +4,133 @@ title: Sonic Execute
 
 ### Run One Liners
 
-Sonic provides a way to execute commands remotely and securely across a list of AWS servers.  It does this by leveraging [Amazon EC2 Run Command](https://aws.amazon.com/ec2/execute/).  Sonic hides any complexity and provides a simple interface for you.   The command is called `sonic execute`:
+Sonic provides a way to execute commands remotely and securely across a list of AWS servers.  It does this by leveraging [Amazon EC2 Run Command](https://aws.amazon.com/ec2/execute/).  Sonic a simple interface and some conveniences for you.   The command is called `sonic execute`:
 
-```sh
-sonic execute [FILTER] [COMMAND]
-```
+    sonic execute [FILTER] [COMMAND]
 
-Examples:
+## Examples Summary
 
-```sh
-sonic execute hi-web-stag uptime
-sonic execute hi-web-prod uptime
-sonic execute i-030033c20c54bf149,i-030033c20c54bf150 uname -a
-sonic execute i-030033c20c54bf149 file://hello.sh
-```
+    sonic execute --tags Name=demo-web uptime
+    sonic execute --tags Name=demo-web,demo-worker uptime # multiple tag values
+    sonic execute --instance-ids i-030033c20c54bf149,i-030033c20c54bf150 uname -a
+    sonic execute --instance-ids i-030033c20c54bf149 file://hello.sh
 
-Let's do something more useful:
+## Example Detailed
 
-```sh
-sonic execute hi-web-stag yum install -y curl
-```
+Here's a command example output in detailed:
 
-The output of the command will show a useful `aws ssm list-commands` command to get status of the requested command.
+    $ sonic execute --instance-ids i-0bf51a000ab4e73a8 uptime
+    Sending command to SSM with options:
+    ---
+    instance_ids:
+    - i-0bf51a000ab4e73a8
+    document_name: AWS-RunShellScript
+    comment: sonic execute --instance-ids i-0bf51a000ab4e73a8 uptime
+    parameters:
+      commands:
+      - uptime
+    output_s3_region: us-east-1
+    output_s3_bucket_name: [reacted]
+    output_s3_key_prefix: ssm/commands/sonic
 
-```sh
-$ sonic execute hi-web-stag uptime
-Command sent to AWS SSM. To check the details of the command:
-aws ssm list-commands --command-id 4133e5eb-aa18-40dd-be25-a176eb15e515
-Pro tip: the aws ssm command is already in your copy/paste clipboard.
-$
-```
+    Command sent to AWS SSM. To check the details of the command:
+      aws ssm list-commands --command-id 0bb18d58-6436-49fd-9bfd-0c4b6c51c7a2
+      aws ssm get-command-invocation --command-id 0bb18d58-6436-49fd-9bfd-0c4b6c51c7a2 --instance-id i-0bf51a000ab4e73a8
+
+    Waiting for ssm command to finish.....
+    Command finished.
+
+    Displaying output for i-0bf51a000ab4e73a8.
+    Command status: Success
+    Command standard output:
+     01:08:10 up 8 days,  6:41,  0 users,  load average: 0.00, 0.00, 0.00
 
-The output of the commands ran are also showed in the EC2 Run Command Console.  Here's an example:
+    To see the more output details visit:
+      https://us-west-2.console.aws.amazon.com/systems-manager/run-command/0bb18d58-6436-49fd-9bfd-0c4b6c51c7a2
+
+    Pro tip: the console url is already in your copy/paste clipboard.
+    $
+
+Notice the conveniences of `sonic execute`, it:
+
+1. Showed the parameters that will be sent as part of the send_command call to SSM.
+2. Sent the command to SSM.
+3. Waited for the command to finish.
+4. Displayed the output of the command.
+5. Provided the console url that visit to view more details about the SSM command.
+
+The AWS SSM console looks like this:
 
 <img src="/img/tutorials/ec2-console-run-command.png" class="doc-photo" />
 
-### Polymorphic Filter
+### Filter Options
+
+The `sonic execute` command can understand a variety of different filters: `--tags` and `--instance-ids`.  Note, ECS service names are *not* supported for the filter.
+
+Here is an example, where the uptime command will run on both `i-030033c20c54bf149` and `i-030033c20c54bf150` instances.
+
+    sonic execute --instance-ids i-066b140d9479e9681,i-09482b1a6e330fbf7 uptime
 
-The `sonic execute` command can understand a variety of different filters.  The filters can be a list of instances ids or one EC2 tag value.  Note, ECS service names are *not* supported for the filter.
+Here is an example, where the uptime command will run on instances tagged with `Name=demo-web`
 
-Here is an example, where the uptime command will run on both i-030033c20c54bf149 and i-030033c20c54bf150 instances.
+    sonic execute --tags Name=demo-web uptime
+
+## Windows Support
+
+Windows is also supported. When running a command sonic will first attempt to use the `AWS-RunShellScript` run command, and if it detects that the instance's platform does not support `AWS-RunShellScript`, it will run the command with the `AWS-RunPowerShellScript` run command.  Here's an example:
 
-```sh
-sonic execute i-066b140d9479e9681,i-09482b1a6e330fbf7 uptime
 ```
+$ sonic execute --instance-ids i-0917ad61b10fa1059 pwd
+Sending command to SSM with options:
+---
+instance_ids:
+- i-0917ad61b10fa1059
+document_name: AWS-RunShellScript
+comment: sonic execute --instance-ids i-0917ad61b10fa1059 pwd
+parameters:
+  commands:
+  - pwd
+output_s3_region: us-east-1
+output_s3_bucket_name: boltops-infra-stag
+output_s3_key_prefix: ssm/commands/sonic
+
+Cannot perform operation for instance id i-0917ad61b10fa1059 of platform type Windows
+Retrying with document_name AWS-RunPowerShellScript
+Retries: 1
+Command sent to AWS SSM. To check the details of the command:
+  aws ssm list-commands --command-id 8a196058-445e-4960-9efb-be746ecf98dc
+  aws ssm get-command-invocation --command-id 8a196058-445e-4960-9efb-be746ecf98dc --instance-id i-0917ad61b10fa1059
+
+Waiting for ssm command to finish......
+Command finished.
+
+Displaying output for i-0917ad61b10fa1059.
+Command status: Success
+Command standard output:
+
+Path
+----
+C:\Windows\system32
 
-### Run Scripts
 
-Sometimes you might want to run more than just a one-liner command. If you need to run a full script, you can provide the file path to the script by designating it with `file://`.  For example, here's a file called `hi.sh`:
 
-```bash
-#!/bin/bash
-echo "hello world"
+To see the more output details visit:
+  https://us-east-1.console.aws.amazon.com/systems-manager/run-command/8a196058-445e-4960-9efb-be746ecf98dc
+
+Pro tip: the console url is already in your copy/paste clipboard.
+$
 ```
 
+## Run Scripts
+
+Sometimes you might want to run more than just a one-liner command. If you need to run a full script, you can provide the file path to the script by designating it with `file://`.  For example, here's a file called `hi.sh`:
+
+    #!/bin/bash
+    echo "hello world"
+
 Here's how you run that file:
 
-```sh
-sonic execute hi-web-stag file://hi.sh
-```
+    sonic execute demo-web file://hi.sh
 
 The file gets read by `sonic execute` and sent to EC2 Run Command to be executed.
 
@@ -73,6 +141,6 @@ The `sonic execute` command relies on EC2 Run Manager. So you will need to have
 * You can follow the [installation guide]({% link _docs/install.md %}) to install EC2 Run Manager.
 * You can read on [Why EC2 Run Manager]({% link _docs/why-ec2-run-command.md %}) is used also.
 
-<a id="prev" class="btn btn-basic" href="{% link _docs/tutorial-ecs-run.md %}">Back</a>
+<a id="prev" class="btn btn-basic" href="{% link _docs/tutorial-ecs-sh.md %}">Back</a>
 <a id="next" class="btn btn-primary" href="{% link _docs/tutorial-list.md %}">Next Step</a>
 <p class="keyboard-tip">Pro tip: Use the <- and -> arrow keys to move back and forward.</p>

data/docs/_docs/tutorial-ssh.md

@@ -2,8 +2,6 @@
 title: SSH
 ---
 
-### SSH
-
 Sonic allows you to ssh into an instance quickly.
 
 Often when working with AWS EC2 it is helpful to ssh into an instance to debug.  To ssh into an instance, the first thing you do is go to the EC2 Console and grab the public IP address.
@@ -32,7 +30,7 @@ The above command effectively translates to:
 ssh ec2-user@52.24.216.170
 ```
 
-By default the user that sonic uses to log in to the server is `ec2-user`. You can override the user easily like so:
+By default the user that sonic uses to log in to the server is `ec2-user`. You can override the user as part of the sonic command like so:
 
 ```sh
 sonic ssh ubuntu@i-0f7f833131a51ce35
@@ -41,22 +39,23 @@ sonic ssh ubuntu@i-0f7f833131a51ce35
 The default user can also be configured with a `~/.sonic/settings.yml` or the project's `.sonic/settings.yml` file like so:
 
 ```yaml
-user: ec2-user
+ssh:
+  user: ec2-user
 ```
 
 More information about sonic settings in available in the docs: [Settings]({% link _docs/settings.md %}).
 
 ### Polymorphic Identifiers
 
-The `sonic ssh` command can auto-detect the proper IP address with a variety of different identifiers.  The identifier is not just limited to the instance id. The identifier can also be an EC2 tag-value filter, ECS service name, ECS container id or ECS task id.
+The `sonic ssh` command can auto-detect the proper IP address with a variety of different identifiers.  The identifier is not limited to the instance id. The identifier can be an EC2 tag-value filter, ECS service name, ECS container id or ECS task id.
 
 Polymorphic identifiers are convenient in case you happen to be on a dashboard with another identifier close by and handy.  Here are examples of identifiers that `sonic ssh` understands.
 
 ```
 sonic ssh EC2_TAG_FILTER
-sonic ssh ECS_SERVICE --cluster stag
-sonic ssh ECS_CONTAINER_ID --cluster stag
-sonic ssh ECS_TASK_ID --cluster stag
+sonic ssh ECS_SERVICE --cluster staging
+sonic ssh ECS_CONTAINER_ID --cluster staging
+sonic ssh ECS_TASK_ID --cluster staging
 ```
 
 The EC2 tag filter uses the 'tag-value' filter as described in the [AWS describe-instances](http://docs.aws.amazon.com/cli/latest/reference/ec2/describe-instances.html) filter docs. This filter is independent of the tag-key filter, which means any EC2 tag value will match, regardless of the tag key name.  Sonic does not support spaces in the EC2 tag filter.
@@ -64,14 +63,11 @@ The EC2 tag filter uses the 'tag-value' filter as described in the [AWS describe
 Notice, that when the `sonic ssh` is passed an ECS identifier then it also requires the ECS cluster name. The commands above with the ECS identifier are normally shorten further by configuring the a [settings]({% link _docs/settings.md %}) file.  Here's an example:
 
 ```yaml
-service_cluster:
+ecs_service_cluster_map:
   default: my-default-cluster
-  hi-web-prod: prod
-  hi-clock-prod: prod
-  hi-worker-prod: prod
-  hi-web-stag: stag
-  hi-clock-stag: stag
-  hi-worker-stag: stag
+  hi-web: production
+  hi-clock: production
+  hi-worker: production
 ```
 
 With these settings in place, the ECS identifier commands get shortened to become:
@@ -82,16 +78,16 @@ sonic ssh ECS_SERVICE
 sonic ssh ECS_TASK_ID
 ```
 
-It then becomes effortless to ssh into an EC2 Container Instance with the ECS service name.  For example, if the ECS service name is `hi-web-stag` then the command becomes.
+It then becomes effortless to ssh into an EC2 Container Instance with the ECS service name.  For example, if the ECS service name is `hi-web` then the command becomes.
 
 ```sh
-$ sonic ssh hi-web-stag
+$ sonic ssh hi-web
 # now you are on the container instance
 $ docker ps
 $ curl -s http://localhost:51678/v1/meta | jq .
 ```
 
-The `hi-web-stag` can be running on multiple container instances.  The `sonic ssh` command chooses the first container instance that it finds.  If you need to ssh into a specific container instance, then use the `sonic ssh` command with an instance id instead.
+The `hi-web` can be running on multiple container instances.  The `sonic ssh` command chooses the first container instance that it finds.  If you need to ssh into a specific container instance, then use the `sonic ssh` command with an instance id instead.
 
 You can also use the ECS container instance arn or task id to ssh into the machine.  Examples:
 
@@ -120,7 +116,7 @@ Can't ssh into the server yet.  Retrying until success.
 
 ### Specifying Custom Pem or Private Keys
 
-It is recommended that you use ssh-agent to specify a custom private key, covered here [3 SSH tips: Ssh-agent, Tunnel, and Escaping from the Dead](https://blog.boltops.com/2017/09/21/3-ssh-tips-ssh-agent-tunnel-and-escaping-from-the-dead).  You can specify the private key if you prefer with the `-i` option though.  Example:
+It is recommended that you use ssh-agent to specify a custom private key, covered here [3 SSH tips: Ssh-agent, Tunnel, and Escaping from the Dead](https://blog.boltops.com/2017/09/21/3-ssh-tips-ssh-agent-tunnel-and-escaping-from-the-dead).  But you can specify the private key if you prefer with the `-i` option though.  Example:
 
 ```sh
 $ sonic ssh -i ~/.ssh/id_rsa-custom i-0b21da68fff89937b

data/docs/_docs/why-ec2-run-command.md

@@ -9,7 +9,7 @@ Why use Amazon EC2 Run Command vs just using a multi-ssh session?
 * The EC2 Run Manager has the ability to run the command in "blue/green" fashion with concurrency controls. Say you have 100 servers, you can tell EC2 Run Manager to run the command on one server first and the expodentially roll it out to the rest of the servers until the command has successfully ran on all servers.  If it the command errors on one server then it halts execution and does not run on the rest of the servers.
 * This is all provided for free by using EC2 Run Manager.
 
-The iniitial ertia of setting up EC2 Run Manager is actually very little.  The [installation instructions]({% link _docs/install.md %}) demonstrate that installing EC2 Run Manager is literally one command.
+The initial ertia of setting up EC2 Run Manager is actually very little.  The [installation instructions]({% link _docs/install.md %}) demonstrate that installing EC2 Run Manager is literally one command.
 
 <a id="prev" class="btn btn-basic" href="{% link _docs/why.md %}">Back</a>
 <a id="next" class="btn btn-primary" href="{% link _docs/how-it-works.md %}">Next Step</a>