sonic-screwdriver 0.0.1 → 1.0.0

data/docs/_docs/commands.md

@@ -0,0 +1,10 @@
+---
+title: Commands
+---
+
+The [tutorial]({% link _docs/tutorial.md %}) does a great job of covering the main sonic commands and overall usage.  In the next sections we'll cover the sonic commands in more detail.
+
+<a id="prev" class="btn btn-basic" href="{% link _docs/tutorial-execute.md %}">Back</a>
+<a id="next" class="btn btn-primary" href="{% link _docs/sonic-ssh.md %}">Next Step</a>
+<p class="keyboard-tip">Pro tip: Use the <- and -> arrow keys to move back and forward.</p>
+

data/docs/_docs/how-it-works.md

@@ -0,0 +1,34 @@
+---
+title: How It Works
+---
+
+### Internals
+
+The process that I outline in the [Why]({% link _docs/why.md %}) about clicking around is close to the logic that actually takes place in the tool but it is actually slightly different. Here's an overview of what actually happens internally for those who are interested.
+
+Steps:
+
+1. list-tasks: list all the tasks for their task_arns (scoped to service).  this is all tasks on the service.  We already know the service name!
+2. describe-tasks: Using the task_arns from list-tasks. This will provide the container instance scoped the service since list-tasks was scoped to service.  Keep the task arn for step 7. Also describe-task-definition and capture env vars and image for step 8b.
+3. describe-container-instances: Using container\_instance\_arn from step 2. This will provide the instance_id to ssh into.
+4. ec2 describe-instances to get the dns name or IP address.
+5. Copy over files with required info over to the server with scp.
+6. ssh into the machine with IP address.
+7. Use the ecs metadata and pass it the task_arn from step 2. This will provide the map to the container id.
+8. Run docker command
+    1. docker exec -ti CONTAINER_ID `options[:command]`
+    2. docker run `options[:run_options]` IMAGE `options[:command]`
+
+In order to pass info over from your local machine to the container instance a file is generated and copied in step 5. File contains:
+
+* Options all the way from the original cli call like command to run. This is in json form.
+A bash script is also copied.
+* Bash script gets the container id using the task_arn. It also will run the docker exec or run command.
+* So bash script does steps 7 and 8.1 or 8.2.
+
+NOTE: I thought it would be possible to map the container instance info from `aws ecs describe-services` but it is not possible. But we can map to the container instance DNS name starting from `aws ecs list-tasks`.
+
+
+<a id="prev" class="btn btn-basic" href="{% link _docs/why.md %}">Back</a>
+<a id="next" class="btn btn-primary" href="{% link _docs/next-steps.md %}">Next Step</a>
+<p class="keyboard-tip">Pro tip: Use the <- and -> arrow keys to move back and forward.</p>

data/docs/_docs/install.md

@@ -0,0 +1,75 @@
+---
+title: Installation
+---
+
+### Install with Bolts Toolbelt
+
+If you want to quickly install sonic without having to worry about sonic's dependency you can simply install the Bolts Toolbelt which has sonic included.
+
+```sh
+brew cask install boltopslabs/software/bolts
+```
+
+For more information about the Bolts Toolbelt or to get an installer for another operating system visit: [https://boltops.com/toolbelt](https://boltops.com/toolbelt)
+
+### Install with RubyGems
+
+If you prefer to install sonic via RubyGems follow the instructions:
+
+```sh
+gem install sonic-screwdriver
+```
+
+Or you can add sonic to your Gemfile in your project if you are working with a ruby project.  It is not required for your project to be a ruby project to use sonic.
+
+```ruby
+gem "sonic-screwdriver"
+```
+
+### Additional Server Side Dependencies
+
+#### sonic ecs-* dependencies
+
+In order for the `sonic ecs-*` trickery to work `jq` is used on the server side.  This is covered in the [How It Works]({% link _docs/how-it-works.md %}) section. So you must have `jq` installed on the servers for the `sonic ecs-*` commands to work.
+
+One way to install `jq` quickly is by using the `sonic execute` command.  For example:
+
+```sh
+sonic execute -f hi-web-stag yum install -y jq
+```
+
+It is recommended that you install `jq` with the UserData script or bake it into the AMI though.
+
+#### sonic execute dependencies
+
+The `sonic execute` works alongside [Amazon EC2 Run Command](https://aws.amazon.com/ec2/execute/).  So this is the required to be installed on the servers for `sonic execute` to work.
+
+#### Amazon EC2 Run Manager Installation
+
+Installing the EC2 Run Manager agent on your linux servers is super simple and often only one command.  The recommended instructions are the offical Amazon EC2 Systems Manager [Install SSM Agent](http://docs.aws.amazon.com/systems-manager/latest/userguide/ssm-agent.html) documentation.
+
+The trickest part of installing is likely making sure that the agent on the server has successfully checked into the SSM service.  Verify it by tailing `/var/log/amazon/ssm/errors/errors.log`.
+
+If you are having issues, it is most likely IAM issues.  Amazon also provides [Configuring Security Roles](http://docs.aws.amazon.com/systems-manager/latest/userguide/systems-manager-access.html) docs to fix any IAM issues.
+
+You can verify the instances that have successfully checked into SSM with `aws ssm describe-instance-information`:
+
+```sh
+aws ssm describe-instance-information --output text --query "InstanceInformationList[*]"
+```
+
+Here's an exmaple of the output:
+
+```sh
+$ aws ssm describe-instance-information --output text --query "InstanceInformationList[*]"
+2.0.822.0 ip-10-10-41-38  10.10.41.38 i-030033c20c54bf149 True  1497482505.12 Online  Amazon Linux AMI  Linux 2017.03 EC2Instance
+2.0.822.0 ip-10-10-110-135  10.10.110.135 i-0f7f833131a51ce35 True  1497482686.53 Online  Amazon Linux AMI  Linu2016.09 EC2Instance
+$
+```
+
+More information is provided in the AWS Run Command Walkthrough [Using the AWS CLI](http://docs.aws.amazon.com/systems-manager/latest/userguide/walkthrough-cli.html).
+
+<a id="prev" class="btn btn-basic" href="{% link docs.md %}">Back</a>
+<a id="next" class="btn btn-primary" href="{% link _docs/tutorial.md %}">Next Step</a>
+<p class="keyboard-tip">Pro tip: Use the <- and -> arrow keys to move back and forward.</p>
+

data/docs/_docs/next-steps.md

@@ -0,0 +1,18 @@
+---
+title: Next Steps
+---
+
+This concludes the tutorial guide for sonic. Hopefully you are now more comfortable with sonic's basic usage, concepts, and have a feel for the workflow.
+
+From here, there are a few resources that can help you continue along:
+
+* Check out the [sonic](https://github.com/boltopslabs/sonic) repo on GitHub
+* Submit an issue
+* Write a blog post describing how you are using sonic, or an interesting problem sonic has allowed you to solve
+
+Everyone can contribute to make sonic better, including the documentation. These docs are of the same sonic repo in the [docs folder](https://github.com/boltopslabs/sonic/tree/master/docs). Please fork the project and open a pull request!  We love your pull requests. Contributions are encouraged and welcomed!
+
+<a id="prev" class="btn btn-basic" href="{% link _docs/how-it-works.md %}">Back</a>
+<!-- <a id="next" class="btn btn-primary" href="{% link articles.md %}">Next Step</a> -->
+<p class="keyboard-tip">Pro tip: Use the <- and -> arrow keys to move back and forward.</p>
+

data/docs/_docs/settings.md

@@ -0,0 +1,73 @@
+---
+title: Settings
+---
+
+You can adjust the behavior of sonic and set some handy default values with `settings.yml` files.  There can exist multiple settings files which get loaded and merged. The options from the files follow the the following precedence rules:
+
+1. project - The project's `.sonic/settings.yml` values take the highest precedence.
+2. user - The user's `~/.sonic/settings.yml` values take the second highest precedence.
+3. default - The [default settings](https://github.com/boltopslabs/sonic/blob/master/lib/sonic/default/settings.yml) bundled with the tool takes the lowest precedence.
+
+Here is an `settings.yml` example:
+
+```yaml
+bastion: bastion.mydomain.com
+host_key_check: false
+service_cluster:
+  default: # defaults to nil
+  hi-web-prod: prod
+  hi-clock-prod: prod
+  hi-worker-prod: prod
+  hi-web-stag: stag
+  hi-clock-stag: stag
+  hi-worker-stag: stag
+user: ec2-user
+```
+
+The following table covers the different setting options:
+
+Setting  | Description | Default
+------------- | ------------- | -------------
+bastion  | Bastion host to use as the jump host. Examples: bastion.mydomain.com, myuser@bastion.myuser.com or 123.123.123.123. | (no value)
+host_key_check  | Controls whether or not use the strict host checking ssh option.  Since EC2 server host changes often the default value is false. | false
+service_cluster  | Service to cluster mapping.  This is a Hash structure that maps the service name to cluster names. | (no value)
+user  | User to ssh into the server with. This can be overriden at the CLI with the user@host notation but can be set in the settings.yml file also. | ec2-user
+
+The default settings are located within the package tool at [lib/sonic/default/settings.yml](https://github.com/boltopslabs/sonic/blob/master/lib/sonic/default/settings.yml).
+
+### Service to Cluster Mapping
+
+One of the useful options is the `service_clsuter`.  This option maps service names to cluster names.  This saves you from  typing the `--cluster` option over and over.  Here is an example `~/.sonic/settings.yml`:
+
+```yaml
+user: ec2-user
+service_cluster:
+  default: my-default-cluster
+  hi-web-prod: prod
+  hi-clock-prod: prod
+  hi-worker-prod: prod
+  hi-web-stag: stag
+  hi-clock-stag: stag
+  hi-worker-stag: stag
+```
+
+This results in shorter commands:
+
+```
+sonic ssh hi-web-prod
+sonic ssh hi-clock-prod
+sonic ssh hi-worker-stag
+```
+
+Instead of what you normally would have to type:
+
+```
+sonic ssh hi-web-prod --cluster prod
+sonic ssh hi-clock-prod --cluster prod
+sonic ssh hi-worker-stag --cluster stag
+```
+
+
+<a id="prev" class="btn btn-basic" href="{% link _docs/tutorial-execute.md %}">Back</a>
+<a id="next" class="btn btn-primary" href="{% link _docs/why.md %}">Next Step</a>
+<p class="keyboard-tip">Pro tip: Use the <- and -> arrow keys to move back and forward.</p>

data/docs/_docs/sonic-ecs-exec.md

@@ -0,0 +1,7 @@
+---
+title: sonic ecs-exec
+---
+
+<a id="prev" class="btn btn-basic" href="{% link _docs/sonic-ssh.md %}">Back</a>
+<a id="next" class="btn btn-primary" href="{% link _docs/sonic-ecs-run.md %}">Next Step</a>
+<p class="keyboard-tip">Pro tip: Use the <- and -> arrow keys to move back and forward.</p>

data/docs/_docs/sonic-ecs-run.md

@@ -0,0 +1,7 @@
+---
+title: sonic ecs run
+---
+
+<a id="prev" class="btn btn-basic" href="{% link _docs/sonic-ecs-exec.md %}">Back</a>
+<a id="next" class="btn btn-primary" href="{% link _docs/sonic-list.md %}">Next Step</a>
+<p class="keyboard-tip">Pro tip: Use the <- and -> arrow keys to move back and forward.</p>

data/docs/_docs/sonic-execute.md

@@ -0,0 +1,7 @@
+---
+title: sonic execute
+---
+
+<a id="prev" class="btn btn-basic" href="{% link _docs/sonic-list.md %}">Back</a>
+<a id="next" class="btn btn-primary" href="{% link _docs/sonic-help.md %}">Next Step</a>
+<p class="keyboard-tip">Pro tip: Use the <- and -> arrow keys to move back and forward.</p>

data/docs/_docs/sonic-help.md

@@ -0,0 +1,7 @@
+---
+title: sonic help
+---
+
+<a id="prev" class="btn btn-basic" href="{% link _docs/sonic-execute.md %}">Back</a>
+<a id="next" class="btn btn-primary" href="{% link _docs/settings.md %}">Next Step</a>
+<p class="keyboard-tip">Pro tip: Use the <- and -> arrow keys to move back and forward.</p>

data/docs/_docs/sonic-list.md

@@ -0,0 +1,7 @@
+---
+title: sonic list
+---
+
+<a id="prev" class="btn btn-basic" href="{% link _docs/sonic-ecs-run.md %}">Back</a>
+<a id="next" class="btn btn-primary" href="{% link _docs/sonic-execute.md %}">Next Step</a>
+<p class="keyboard-tip">Pro tip: Use the <- and -> arrow keys to move back and forward.</p>

data/docs/_docs/sonic-ssh.md

@@ -0,0 +1,7 @@
+---
+title: sonic-ssh
+---
+
+<a id="prev" class="btn btn-basic" href="{% link _docs/commands.md %}">Back</a>
+<a id="next" class="btn btn-primary" href="{% link _docs/sonic-ecs-exec.md %}">Next Step</a>
+<p class="keyboard-tip">Pro tip: Use the <- and -> arrow keys to move back and forward.</p>

data/docs/_docs/tutorial-ecs-exec.md

@@ -0,0 +1,69 @@
+---
+title: ECS Exec
+---
+
+In the previous section we showed you how to use `sonic ssh` to quickly ssh into an instance.  Some of the identifiers used were ECS identifiers.  As you can see sonic is very much ECS aware.  It was built with ECS in mind.
+
+One of the things `sonic` can do is go from your local machine, ssh into an EC2 Container Instance, find the running docker instance and jump into the docker container via `docker exec`.
+
+It does this with a variety of scripts and trickery and is covered in [How It Works]({% link _docs/how-it-works.md %}).  Let's through examples of how sonic can help you get into an docker container running on ECS quickly.
+
+### sonic ecs-exec
+
+```sh
+sonic ecs-exec [ECS_SERVICE] --cluster [ECS_CLUSTER]
+```
+
+Here's a concrete example:
+
+```sh
+sonic ecs-exec hi-web-stag --cluster stag
+```
+
+You should see something like this:
+
+```sh
+$ sonic ecs-exec hi-web-stag --cluster stag
+Running: scp -r /tmp/sonic ec2-user@34.211.195.71:/tmp/sonic  > /dev/null
+Warning: Permanently added '34.211.195.71' (ECDSA) to the list of known hosts.
+=> ssh -t ec2-user@34.211.195.71 bash /tmp/sonic/bash_scripts/docker-exec.sh
+Warning: Permanently added '34.211.195.71' (ECDSA) to the list of known hosts.
+root@fc4035f90bdc:/app#
+```
+
+What you see above is a bash shell within the docker container!  With one command you have placed yourself into the running container 🎉
+
+As mentioned in the [previous section]({% link _docs/tutorial-ssh.md %}) and also in the [Settings documentation]({% link _docs/settings.md %}) you can configure a `~/.sonic/settings.yml` file which shortens the command further.  Let's add this to your settings:
+
+```yaml
+service_cluster:
+  default: stag
+  hi-web-stag: stag
+```
+
+This makes the command consise and memorable.
+
+```sh
+sonic ecs-exec hi-web-stag
+```
+
+The rest of this section assumes that you have the `~/.sonic/settings.yml` set up.
+
+You can also tack on a command at the end of the `ecs-exec` command to be run as a one off instead of starting a bash shell. Example:
+
+```
+$ sonic ecs-exec hi-web-stag uname -a
+Running: scp -r /tmp/sonic ec2-user@34.211.195.71:/tmp/sonic  > /dev/null
+Warning: Permanently added '34.211.195.71' (ECDSA) to the list of known hosts.
+=> ssh -t ec2-user@34.211.195.71 bash /tmp/sonic/bash_scripts/docker-exec.sh uname -a
+Warning: Permanently added '34.211.195.71' (ECDSA) to the list of known hosts.
+Linux fc4035f90bdc 4.4.51-40.58.amzn1.x86_64 #1 SMP Tue Feb 28 21:57:17 UTC 2017 x86_64 GNU/Linux
+Connection to 34.211.195.71 closed.
+$
+```
+
+Remember the command runs within the running docker container.
+
+<a id="prev" class="btn btn-basic" href="{% link _docs/tutorial-ssh.md %}">Back</a>
+<a id="next" class="btn btn-primary" href="{% link _docs/tutorial-ecs-run.md %}">Next Step</a>
+<p class="keyboard-tip">Pro tip: Use the <- and -> arrow keys to move back and forward.</p>

data/docs/_docs/tutorial-ecs-run.md

@@ -0,0 +1,94 @@
+---
+title: ECS Run
+---
+
+The nice thing about the previous `ecs-exec` command we covered is that it allows you to get into the actual running container and debug with the exact enviornment that is on production.  The cavaet with doing this is that we are affecting a live process in actual use. If you do something inadvertently wrong on the server it could affect users.  Sometimes it is nice to start up a new container with the exact same environment as the other running containers but be isolated so you cannot affect live requests.
+
+### sonic ecs-run
+
+Sonic provides a `sonic ecs-run` command that allows you to start up a new container with the same environment as one of the running containers. Here's an example:
+
+```sh
+sonic ecs-run hi-web-stag
+```
+
+You see something like this:
+
+```sh
+$ sonic ecs-run hi-web-stag
+Running: scp -r /tmp/sonic ec2-user@34.211.195.71:/tmp/sonic  > /dev/null
+Warning: Permanently added '34.211.195.71' (ECDSA) to the list of known hosts.
+=> ssh -t ec2-user@34.211.195.71 bash /tmp/sonic/bash_scripts/docker-run.sh
+Warning: Permanently added '34.211.195.71' (ECDSA) to the list of known hosts.
++ exec bundle exec rails server -b 0.0.0.0
+=> Booting WEBrick
+=> Rails 4.2.3 application starting in development on http://0.0.0.0:3000
+=> Run `rails server -h` for more startup options
+=> Ctrl-C to shutdown server
+[2017-06-14 19:01:30] INFO  WEBrick 1.3.1
+[2017-06-14 19:01:30] INFO  ruby 2.3.3 (2016-11-21) [x86_64-linux]
+[2017-06-14 19:01:30] INFO  WEBrick::HTTPServer#start: pid=1 port=3000
+^C[2017-06-14 19:01:34] INFO  going to shutdown ...
+[2017-06-14 19:01:34] INFO  WEBrick::HTTPServer#start done.
+Exiting
+Connection to 34.211.195.71 closed.
+$
+```
+
+In the above output a WEBrick server gets started.  The reason this happens is because the Dockerfile default `CMD` in this project happens to start a webserver.  Most of the time you probably want shell to debug.  To start a bash shell just tack the bash command at the end.
+
+```sh
+$ sonic ecs-run hi-web-stag bash
+Running: scp -r /tmp/sonic ec2-user@34.211.195.71:/tmp/sonic  > /dev/null
+Warning: Permanently added '34.211.195.71' (ECDSA) to the list of known hosts.
+=> ssh -t ec2-user@34.211.195.71 bash /tmp/sonic/bash_scripts/docker-run.sh bash
+Warning: Permanently added '34.211.195.71' (ECDSA) to the list of known hosts.
+root@56a495dbd5cd:/app#
+```
+
+You are now in a docker container running exactly the same environment as the other running containers with the `hi-web-stag` service. While this looks similiar to the `ecs-exec` command this is docker container is a new process and is isolated from any live request. You can do whatever you want to this container and experiment to your heart's content.
+
+We can prove that this is a brand new docker container that is outside of ECS' knowledge. Let's ssh into the same instance and take a look at all the running docker containers in another terminal.
+
+```sh
+$ sonic ssh hi-web-stag docker ps
+=> ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ec2-user@34.211.195.71 docker ps
+Warning: Permanently added '34.211.195.71' (ECDSA) to the list of known hosts.
+CONTAINER ID        IMAGE                                          COMMAND             CREATED             STATUS              PORTS                     NAMES
+29e7c1253c46        tongueroo/hi:ufo-2017-06-13T14-48-08-0a9eea5   "bash"              54 seconds ago      Up 53 seconds       3000/tcp                  cocky_goldstine
+fc4035f90bdc        tongueroo/hi:ufo-2017-06-13T14-48-08-0a9eea5   "bin/web"           About an hour ago   Up About an hour    0.0.0.0:32768->3000/tcp   ecs-hi-web-stag-11-web-9eb081978abad89a9701
+bf646ae7789a        amazon/amazon-ecs-agent:latest                 "/agent"            About an hour ago   Up About an hour                              ecs-agent
+$
+```
+
+The output shows that there is this extra runnning container called `cocky_goldstine`.  This name does not look like the typical ECS managed running docker container: `ecs-hi-web-stag-11-web-9eb081978abad89a9701`
+
+```sh
+$ sonic ecs-run hi-web-stag bash
+Running: scp -r /tmp/sonic ec2-user@34.211.195.71:/tmp/sonic  > /dev/null
+Warning: Permanently added '34.211.195.71' (ECDSA) to the list of known hosts.
+=> ssh -t ec2-user@34.211.195.71 bash /tmp/sonic/bash_scripts/docker-run.sh bash
+Warning: Permanently added '34.211.195.71' (ECDSA) to the list of known hosts.
+root@29e7c1253c46:/app# exit
+exit
+Connection to 34.211.195.71 closed.
+$
+```
+
+Let's exit out of the first terminate where you ran the original `sonic ecs-run` command and then list the running containers again.
+
+```sh
+$ sonic ssh hi-web-stag docker ps
+=> ssh -o StrictHostKeyChecking=no -o UserKnownHostsFile=/dev/null ec2-user@34.211.195.71 docker ps
+Warning: Permanently added '34.211.195.71' (ECDSA) to the list of known hosts.
+CONTAINER ID        IMAGE                                          COMMAND             CREATED             STATUS              PORTS                     NAMES
+fc4035f90bdc        tongueroo/hi:ufo-2017-06-13T14-48-08-0a9eea5   "bin/web"           About an hour ago   Up About an hour    0.0.0.0:32768->3000/tcp   ecs-hi-web-stag-11-web-9eb081978abad89a9701
+bf646ae7789a        amazon/amazon-ecs-agent:latest                 "/agent"            About an hour ago   Up About an hour                              ecs-agent
+$
+```
+
+Zapped!  The container that was created with `sonic ecs-run` is no more.
+
+<a id="prev" class="btn btn-basic" href="{% link _docs/tutorial-ecs-exec.md %}">Back</a>
+<a id="next" class="btn btn-primary" href="{% link _docs/tutorial-execute.md %}">Next Step</a>
+<p class="keyboard-tip">Pro tip: Use the <- and -> arrow keys to move back and forward.</p>

data/docs/_docs/tutorial-execute.md

@@ -0,0 +1,38 @@
+---
+title: Sonic Execute
+---
+
+Sonic provides a way to execute commands remotely and securely across a list of AWS servers.  It does this by relying on [Amazon EC2 Run Command](https://aws.amazon.com/ec2/execute/).  Sonic hides any complexity and provides a simplier interface for you. Example:
+
+```sh
+sonic execute --filter hi-web-stag "uptime"
+```
+
+Let's do something more useful:
+
+```sh
+sonic execute --filter hi-web-stag "yum install -y curl"
+```
+
+The output of the commands ran are showed in the EC2 Run Command Console.  Here's an example:
+
+```
+$ sonic execute --filter hi-web-stag uptime
+Command sent to AWS SSM. To check the details of the command:
+aws ssm list-commands --command-id 4133e5eb-aa18-40dd-be25-a176eb15e515
+Pro tip: the aws ssm command is already in your copy/paste clipboard.
+$
+```
+
+### Why Amazon EC2 Run Command
+
+Why use Amazon EC2 Run Command vs just using a multi-ssh session?
+
+* Some times it is not possible to use ssh across several servers.  For example, really secured networks might have [MFA setup](TODO) so you need to authorized the requests via your phone before the command actually gets ran. In this case, you would get annoying confirmation notifications on your phone over and over as you approve each request for each of your servers.
+* EC2 Run Command provides auditability. Any command that runs the EC2 Run Command gets logged and is tracked.
+* The EC2 Run Manager has the ability to run the command in "blue/green" fashion with concurrency controls. Say you have 100 servers, you can tell EC2 Run Manager to run the command on one server first and the expodentially roll it out to the rest of the servers until the command has successfully ran on all servers.  If it the command errors then it execute can be told to halt.
+* This is all provided for free by using EC2 Run Manager.
+
+<a id="prev" class="btn btn-basic" href="{% link _docs/tutorial-ecs-run.md %}">Back</a>
+<a id="next" class="btn btn-primary" href="{% link _docs/settings.md %}">Next Step</a>
+<p class="keyboard-tip">Pro tip: Use the <- and -> arrow keys to move back and forward.</p>