aws-ec2 0.5.1 → 0.5.2

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 5a7783fbdf32f51f3a71ac3d71b14a6806bef0f865aca87f6a68a3f952148398
-  data.tar.gz: 2cac98bb451c4576b52ac1503af2101ddd5aa6659af016b012484ae03bfd9b3a
+  metadata.gz: 0d2b33737beb7cc7f0b0f62447883b3cb7ad5235a4b534359f2128b42cde928b
+  data.tar.gz: b49727f0247691d3f21370fad4ba618064d3c7d69031500241e322edf33ce4bf
 SHA512:
-  metadata.gz: a07313a039186ba63a3b0f833586d8824799f37abfc71674781a1405d50819d85b1b278d511e71836ef932b7e9d92c383874dd6787e78e778138040beaf02411
-  data.tar.gz: b038d96c8ba36fe40550163af47cee2cc4dc89469680c941cb04830d62ba45ee44d6e7ae1b9d8c92a16818a2df75bd95c6d9cf0124ef3e4d99295d108528fa6d
+  metadata.gz: fe62f5a533ba85f766c956e500c3ca5b7771df36936948c1f89c44f359fec08218343dcbbedef841f9a005f3f7127907cda62d3d8b1c9635232985c4741309db
+  data.tar.gz: e0c4b8db0bf9195693e3be9321376ffa752740d4ee6ac6584b2f44ccbd95f2162e4b489546de2a7672b06b3424f8038dd13d630414d2a2f87255c48b6e327aa7

data/CHANGELOG.md

@@ -3,6 +3,9 @@
 All notable changes to this project will be documented in this file.
 This project *tries* to adhere to [Semantic Versioning](http://semver.org/), even before v1.0.
 
+## [0.5.2]
+- remove byebug debugging
+
 ## [0.5.1]
 - show a friendly user message if not an aws-ec2 project
 

data/Gemfile.lock

@@ -1,7 +1,7 @@
 PATH
   remote: .
   specs:
-    aws-ec2 (0.3.0)
+    aws-ec2 (0.5.1)
       activesupport
       aws-sdk-ec2
       colorize

data/README.md

@@ -3,7 +3,7 @@
 Simple tool to create AWS ec2 instances consistently with pre-configured settings.  The pre-configured settings are stored in the profiles folder of the current directory.
 Example:
 
-* profiles/default.yml: default settings.  If there is no other matching profile.
+* profiles/default.yml: default settings.
 * profiles/myserver.yml: myserver settings.
 
 ## Usage
@@ -14,12 +14,21 @@ aws-ec2 create myserver --profile myserver
 
 In a nutshell, the profile parameters are passed to the ruby aws-sdk [AWS::EC2::Client#run_instances](https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/EC2/Client.html#run_instances-instance_method) method.  So you can specify any parameter you wish that is available there. To check out what a profile looks like look at the [example default](example/profiles/default.yml)
 
-You can use ERB in the profile files. Some useful helper methods in the profile files are:
+You can use ERB in the profile files. Some useful helper methods in the profile files are documented here:
 
-* user_data: a helper method that allows you to embed a generated user_data script.  More details on the user_data helper are provided below.
-* config:
+Helper  | Description
+------------- | -------------
+user_data | Allows you to embed a generated user_data script.  More details on the are provided in the user data section below.
+config | Access to the variables set in config/[AWS_EC2_ENV].yml.
+latest_ami | Returns an AMI id by searching the ami name pattern and sorting in reverse older.  Example: `latest_ami("ruby-2.5.0_*")`
+search_ami | Returns a collection of AMI image objects based on a search pattern. The query searches on the AMI name.
 
-The template helpers defined in [template_helper.rb](lib/aws_ec2/template_helper.rb).
+The template helpers defined in:
+
+* [aws_ec2/template_helper.rb](lib/aws_ec2/template_helper.rb).
+* [aws_ec2/template_helper](lib/aws_ec2/template_helper).
+
+You can also define your own custom helpers in the `app/helpers` folder as ruby modules with the naming convention `'*_helper.rb`.  Example, the module FooHelper  should be defined in `app/helpers/foo_helper.rb`.  The custom helpers are first class citizens and have access to the same variables and methods as built in helpers.
 
 ### Convention
 
@@ -29,29 +38,90 @@ By convention, the profile name matches the first parameter after the create com
 aws-ec2 create myserver
 ```
 
+## Noop mode
+
+You can do a test run with the `--noop` flag.  This will print out what settings will be used to launch the instance.  This is a good way to inspect the generated user-data script.
+
+```sh
+aws-ec2 create myserver --profile myserver --noop
+```
+
+## Project Structure
+
+Directory  | Description
+------------- | -------------
+app/helpers  | Your own custom helpers methods.  Define them in modules and the methods made available to your `config`, `profiles`, `app/scripts`, `app/user-data` files.
+app/partials  | Your partials that can be use to be included in other scripts.  This is used in conjunction with the partial helper method.
+app/scripts  | Where you define common scripts that can be used to configure the server.
+app/user-data  | Your user-data scripts that are used to bootstrap EC2 instance.
+config/[AWS_EC2_ENV].yml  | AWS_EC2_ENV can be development, staging or production. Use this config file to set configs that you want available in your templating logic.
+profiles  | Your profile files.  These files mainly contain parameters that are passed to the aws-sd2 run_instances API.
+tmp  | Where the generated scripts get compiled to. You can manually invoke the complilation via `aws-ec2 compile`.
+
 ## User-Data
 
-You can provide user-data script to customize the server upon launch.  The user-data scripts are under the app/user-data folder.
+You can provide user-data script to customize the server upon launch.  The user-data scripts are under the `app/user-data` folder.
 
 * app/user-data/myserver.yml
 
-The user-data script is generated on the machine that is running the aws-ec2 command. If this is your local macosx machine, then the context is your local macosx machine is available. To see the generated user-data script, you can use the `aws userdata NAME`.  Example:
+The user-data script is generated on the machine that is running the aws-ec2 command. If this is your local macosx machine, then the context is your local macosx machine is available. To see the generated user-data script, you can use the run the create command in noop mode and then inspect the generated script.  Example:
+
+```sh
+aws create myserver --noop
+cat /tmp/aws-ec2/user-data.txt
+```
 
-* aws userdata myserver # shows a preview of the user-data script
+To use the user-data script when creating an EC2 instance, you can use the helper method in the profile.  Here's a grep of a profile that users the helper:
 
-To use the user-data script when creating an EC2 instance, you can use the helper method in the profile.
+```
+$ grep user_data profiles/default.yml
+user_data: "<%= user_data("bootstrap") %>"
+```
 
 ### Config
 
-You can set a config file and define variables in there that are available to in your profiles and user_data scripts.
+You can set a config file and define variables in there that are available to in your profiles and user_data scripts.  Example `config/development.yml`:
+
+```yaml
+---
+vpc_id: vpc-123
+db_subnet_group_name: default
+  - subnet-123
+  - subnet-456
+  - subnet-789
+security_group_ids:
+  - sg-123
+s3_bucket: mybucket # for the user-data shared scripts
+```
 
-## Noop mode
+The variables are accessible via the `config` helper method. Example (only showing the part of the profile), `profiles/default.yml`:
 
-You can do a test run with the `--noop` flag.  This will print out what settings will be used to launch the instance.  This is a good way to inspect the generated user-data script.
+```yaml
+image_id: ami-4fffc834 # Amazon Lambda AMI
+instance_type: t2.medium
+security_group_ids: <%= config["security_group_ids"] %>
+subnet_id: <%= config["subnets"].shuffle %>
+...
+```
+
+## Dotenv File Support
+
+You can set and configure environment variables in `.env*` files.  Examples of this is in the [doc/example](doc/example) project.
+
+### Hooks
+
+There is only one hook: before_run_instances.  You can configure this with `config/hooks.yml`:  Example:
 
-```sh
-aws-ec2 create myserver --profile myserver --noop
 ```
+---
+before_run_instances: /path/to/my/script.sh
+```
+
+## AMI Creation
+
+To create AMIs you can use the `aws-ec2 ami` command.  This command will run launch an EC2 instance with one of the profiles and create an AMI after the user-data script completes successfully. It does this by adding an AMI creation script at the end of the user-data script.  It is recommended to use the `set -e` option so that any error halts the script and the AMI does not get created.
+
+After the AMI is created successfully, the instance will also terminate itself automatically so you do not have to worry about cleanup.  For more help run `aws-ec2 ami help`.
 
 ## Spot Instance Support
 
@@ -64,11 +134,12 @@ Spot instance support natively supported by the AWS run_instances command.  Simp
 
 ```sh
 aws-ec2 create help
-aws-ec2 userdata help
+aws-ec2 ami help
+aws-ec2 compile help
 aws-ec2 help # general help
 ```
 
-Examples are in the [example](example) folder.  You will have to update settings like your subnet and security group ids.
+Examples are in the [doc/example](doc/example) folder.  You will have to update settings like your subnet and security group ids.
 
 ## Installation
 
@@ -76,6 +147,10 @@ Examples are in the [example](example) folder.  You will have to update settings
 gem install aws-ec2
 ```
 
+### Dependencies
+
+This tool mainly uses the ruby aws-sdk but it does use the aws cli to check your region: `aws configure get region`. So it is dependent on the the `aws cli`.
+
 ## Contributing
 
 1. Fork it

data/lib/aws-ec2.rb

@@ -16,7 +16,7 @@ module AwsEc2
   autoload :Core, "aws_ec2/core"
   autoload :Dotenv, "aws_ec2/dotenv"
   autoload :Hook, "aws_ec2/hook"
-  autoload :CompileScripts, "aws_ec2/compile_scripts"
+  autoload :Compile, "aws_ec2/compile"
 
   extend Core
 end

data/lib/aws_ec2/cli.rb

@@ -20,10 +20,10 @@ module AwsEc2
       Ami.new(options.merge(name: name)).run
     end
 
-    desc "compile_scripts", "compiles app/scripts and app/user-data to tmp folder"
-    long_desc Help.text(:compile_scripts)
-    def compile_scripts
-      CompileScripts.new(options).compile
+    desc "compile", "compiles app/scripts and app/user-data to tmp folder"
+    long_desc Help.text(:compile)
+    def compile
+      Compile.new(options).compile
     end
   end
 end

data/lib/aws_ec2/{compile_scripts.rb → compile.rb}

@@ -1,7 +1,7 @@
 require 'fileutils'
 
 module AwsEc2
-  class CompileScripts < Base
+  class Compile < Base
     include TemplateHelper
     BUILD_ROOT = "tmp"
 

data/lib/aws_ec2/create/params.rb

@@ -1,6 +1,3 @@
-require "byebug"
-
-
 class AwsEc2::Create
   class Params
     include AwsEc2::TemplateHelper

data/lib/aws_ec2/help/compile.md

@@ -0,0 +1,5 @@
+Examples:
+
+  $ aws-ec2 compile
+
+Compiles the scripts in app/scripts and app/user-data to the tmp folder. Useful for inspection.

data/lib/aws_ec2/template_helper/ami_helper.rb

@@ -11,13 +11,17 @@ module AwsEc2::TemplateHelper::AmiHelper
   #
   # Returns latest ami ami
   def latest_ami(query, owners=["self"])
+    images = search_ami(query, owners)
+    image = images.sort_by(&:name).reverse.first
+    image.image_id
+  end
+
+  def search_ami(query, owners=["self"])
     images = ec2.describe_images(
       owners: owners,
       filters: [
         {name: "name", values: [query]}
       ]
     ).images
-    image = images.sort_by(&:name).reverse.first
-    image.image_id
   end
 end

data/lib/aws_ec2/version.rb

@@ -1,3 +1,3 @@
 module AwsEc2
-  VERSION = "0.5.1"
+  VERSION = "0.5.2"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: aws-ec2
 version: !ruby/object:Gem::Version
-  version: 0.5.1
+  version: 0.5.2
 platform: ruby
 authors:
 - Tung Nguyen
@@ -197,9 +197,6 @@ files:
 - README.md
 - Rakefile
 - aws-ec2.gemspec
-- example/profiles/default.yml
-- example/profiles/spot.yml
-- example/profiles/user-data/dev.sh
 - exe/aws-ec2
 - lib/aws-ec2.rb
 - lib/aws_ec2/ami.rb
@@ -207,7 +204,7 @@ files:
 - lib/aws_ec2/base.rb
 - lib/aws_ec2/cli.rb
 - lib/aws_ec2/command.rb
-- lib/aws_ec2/compile_scripts.rb
+- lib/aws_ec2/compile.rb
 - lib/aws_ec2/config.rb
 - lib/aws_ec2/core.rb
 - lib/aws_ec2/create.rb
@@ -215,8 +212,8 @@ files:
 - lib/aws_ec2/dotenv.rb
 - lib/aws_ec2/help.rb
 - lib/aws_ec2/help/ami.md
+- lib/aws_ec2/help/compile.md
 - lib/aws_ec2/help/create.md
-- lib/aws_ec2/help/user_data.md
 - lib/aws_ec2/hook.rb
 - lib/aws_ec2/script.rb
 - lib/aws_ec2/scripts/ami_creation.sh

data/example/profiles/default.yml

@@ -1,12 +0,0 @@
----
-image_id: ami-97785bed
-instance_type: t2.medium
-key_name: default
-max_count: 1
-min_count: 1
-security_group_ids:
-- sg-111
-subnet_id: <%= %w[subnet-111 subnet-222].shuffle.first %>
-user_data: "<%= user_data("dev") %>"
-iam_instance_profile:
-  name: IAMProfileName

data/example/profiles/spot.yml

@@ -1,9 +0,0 @@
----
-instance_market_options:
-  market_type: spot
-  # https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_LaunchTemplateSpotMarketOptionsRequest.html
-  # https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/EC2/Types/SpotMarketOptions.html
-  spot_options:
-    max_price: "0.02"
-    spot_instance_type: one-time
-    # instance_interruption_behavior: hibernate

data/example/profiles/user-data/dev.sh

@@ -1,47 +0,0 @@
-#!/bin/bash -exu
-
-exec > >(tee "/var/log/user-data.log" | logger -t user-data -s 2>/dev/console) 2>&1
-export HOME=/root # user-data env runs in weird shell where user is root but HOME is not set
-<% pubkey_path = "#{ENV['HOME']}/.ssh/id_rsa.pub" -%>
-<% if File.exist?(pubkey_path) -%>
-<% pubkey = IO.read(pubkey_path).strip -%>
-# Automatically add user's public key
-echo <%= pubkey %> >> ~/.ssh/authorized_keys
-echo <%= pubkey %> >> /home/ec2-user/.ssh/authorized_keys
-chown ec2-user:ec2-user /home/ec2-user/.ssh/authorized_keys
-<% else %>
-# WARN: unable to find a ~/.ssh/id_rsa.pub locally on your machine.  user: <%= ENV['USER'] %>
-# Unable to automatically add the public key
-<% end -%>
-
-sudo yum install -y postgresql
-
-# https://gist.github.com/juno/1330165
-# Install developer tools
-yum install -y git gcc make readline-devel openssl-devel
-
-# Install ruby-build system-widely
-git clone git://github.com/sstephenson/ruby-build.git /tmp/ruby-build
-cd /tmp/ruby-build
-./install.sh
-echo 'export PATH="/usr/local/bin:$PATH"' >> ~/.bashrc
-
-# Install rbenv for root
-git clone git://github.com/sstephenson/rbenv.git ~/.rbenv
-echo 'export PATH="$HOME/.rbenv/bin:$PATH"' >> ~/.bashrc
-echo 'eval "$(rbenv init -)"' >> ~/.bashrc
-set +u
-source ~/.bashrc
-set -u
-
-# Install and enable ruby
-rbenv install 2.5.0
-
-# Install ruby for ec2-user also
-cp -R ~/.rbenv /home/ec2-user/
-chown -R ec2-user:ec2-user /home/ec2-user/.rbenv
-echo 'export PATH="$HOME/.rbenv/bin:$PATH"' >> /home/ec2-user/.bashrc
-echo 'eval "$(rbenv init -)"' >> /home/ec2-user/.bashrc
-echo '2.5.0' > /home/ec2-user/.ruby-version
-
-uptime | tee /var/log/boot-time.log

data/lib/aws_ec2/help/user_data.md

@@ -1,13 +0,0 @@
-Displays the generated user data script. Useful for debugging since ERB can be ran on the user-data scripts.
-
-Given a user data script in app/user-data/myscript.sh, run:
-
-  $ aws-ec2 userdata myscript
-
-You can have an ami creation snippet of code added to the end of the user data script with the `--ami` option.
-
-  $ aws-ec2 userdata myscript --ami myname
-
-If you want to include a timestamp in the name you can use this:
-
-  $ aws-ec2 userdata myscript --ami '`date "+myname_%Y-%m-%d-%H-%M"`'