forger 1.5.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 5b3aa8428858adf323d614a2ad2d417ef111ecba07ed47f921ded9971b687e32
+  data.tar.gz: 954cade32be1581361774f9ef33d2aac9a6b4cc37006b7a60fb28eddf2f3fde8
+SHA512:
+  metadata.gz: 26d8d1882ce393975e3b0721eacaec3c36e6abfeba8b72093260872b1d56ab158c967669d8991b7d7030938f7bd2e1b9906b607d7257ba9b3e5faa110f0e561e
+  data.tar.gz: 5cf53ac7391f7ae681b0f8951e3b7b923e11e05e29f65437ea7769db3c858b149f915a295f013a9a84f26916d1514e6633734c7a177fd962b3d052c0cb1e99ad

data/.gitignore

@@ -0,0 +1,16 @@
+*.gem
+*.rbc
+.bundle
+.config
+.yardoc
+_yardoc
+coverage
+doc/
+InstalledFiles
+lib/bundler/man
+pkg
+rdoc
+spec/reports
+test/tmp
+test/version_tmp
+tmp

data/.rspec

@@ -0,0 +1,3 @@
+--color
+--format documentation
+--require spec_helper

data/CHANGELOG.md

@@ -0,0 +1,147 @@
+# Change Log
+
+All notable changes to this project will be documented in this file.
+This project *tries* to adhere to [Semantic Versioning](http://semver.org/), even before v1.0.
+
+## [1.5.0]
+- rename to forger
+
+## [1.4.9]
+- fix /var/log/auto-terminate.log puts
+
+## [1.4.8]
+- show auto-terminate log also in AWS_EC2_CW mode
+
+## [1.4.7]
+- add AWS_EC2_CW to show cw tail command
+
+## [1.4.6]
+- enable awslogs on reboot
+- improve message when auto-terminate gets called from a previous ami
+
+## [1.4.5]
+- fix cloudwatch python install for ubuntu
+
+## [1.4.4]
+- add AWS_EC2_REGION env to set the region for the cloudwatch log displayed url
+
+## [1.4.3]
+- fix get_region when aws-ec2 called from an ec2 instance
+
+## [1.4.2]
+- fix cloudwatch for ubuntu
+
+## [1.4.1]
+- fix cloudwatch support check and add add_to_clipboard url
+- fix copy_to_clipboard
+- remove normalize_os debugging call
+
+## [1.4.0]
+- Merge pull request #8 from tongueroo/cloudwatch
+- cloudwatch support for amazonlinux2 and ubuntu
+
+## [1.3.2]
+- add noop mode for clean ami command
+
+## [1.3.1]
+- fix cleaning so it cleans out the oldest images
+
+## [1.3.0]
+- Merge pull request #7 from tongueroo/clean: aws-ec2 clean ami command
+
+## [1.2.2]
+- add render_me_pretty as gem instead of submodule
+
+## [1.2.1]
+- fix formatting after found ami with puts
+
+## [1.2.0]
+- Merge pull request #6 from tongueroo/wait: aws-ec2 wait ami command
+- Fix dependencies: add aws-sdk-s3 dependency
+- require render_me_pretty as vendor submodule for now
+
+## [1.1.0]
+- Merge pull request #5 from tongueroo/auto-terminate
+- allow AWS_RDS_CODE_VERSION env variable
+- restructure way scripts work for sanity
+- bash scripts: ami_creation.sh and auto_terminate.sh
+- support only amazonlinux2 and ubuntu
+
+## [1.0.1]
+- update ordering of the info displayed
+
+## [1.0.0]
+- Merge pull request #1 from tongueroo/cli-template-upgrade
+- Merge pull request #2 from tongueroo/render_me_pretty
+- Merge pull request #3 from tongueroo/s3-upload
+- Merge pull request #4 from tongueroo/layout-support
+- add --randomize option
+- add extract_scripts and add_ssh_key helpers
+- conventionally use name of server as profile if profile exists
+- introduce settings.yml
+- latest_ami: exit if image cannot be found
+
+## [0.9.0]
+- much improved error messaging
+- rename docs folder
+- update readme
+
+## [0.8.4]
+- add doc/example
+- rename to commands to compile and upload
+- rename to core_helper, only require specfic activesupport dependencies
+- generate user-data in tmp/user-data.txt instead
+
+## [0.8.3]
+- change compile_keep to compile_clean option instead
+
+## [0.8.2]
+- add compile_keep option
+
+## [0.8.1]
+- reorganize template helpers into core and update readme
+
+## [0.8.0]
+- aws-ec2 upload_scripts command
+- rename aws-ec2 compile to aws-ec2 compile_scripts
+
+## [0.7.0]
+- Rid of name to profile convention, check profile exists and report to user
+  if it does not exist. This is a more expected interface.
+
+## [0.6.0]
+- add scripts_s3_bucket config option
+- halt script if hooks fail
+
+## [0.5.2]
+- remove byebug debugging
+
+## [0.5.1]
+- show a friendly user message if not an aws-ec2 project
+
+## [0.5.0]
+- compile_scripts compiles both app/user-data and app/scripts
+
+## [0.4.0]
+- aws-ec2 ami command
+- create: add --source-ami options
+- compile_scripts command
+- custom helper support
+- dotenv support
+- hook support
+- latest_ami helper
+- partial support
+- starter specs: spec for ami
+- remove aws-ec2 spot command
+- remove aws-ec2 userdata command, sanity rspec passing
+
+## [0.3.0]
+- Do not merge profile to default profile.  This was pretty confusing usage.
+- Add --ami option which result in automatically creating an ami at the end of
+  the user-data script.
+
+## [0.2.0]
+- Add config files support. example: config/development.yml.
+
+## [0.1.0]
+- Initial release.

data/Gemfile

@@ -0,0 +1,6 @@
+source "https://rubygems.org"
+
+# Specify your gem dependencies in forger.gemspec
+gemspec
+
+gem "codeclimate-test-reporter", group: :test, require: nil

data/Gemfile.lock

@@ -0,0 +1,136 @@
+PATH
+  remote: .
+  specs:
+    forger (1.4.9)
+      activesupport
+      aws-sdk-ec2
+      aws-sdk-s3
+      colorize
+      dotenv
+      filesize
+      hashie
+      render_me_pretty
+      thor
+
+GEM
+  remote: https://rubygems.org/
+  specs:
+    activesupport (5.2.0)
+      concurrent-ruby (~> 1.0, >= 1.0.2)
+      i18n (>= 0.7, < 2)
+      minitest (~> 5.1)
+      tzinfo (~> 1.1)
+    aws-partitions (1.80.0)
+    aws-sdk-core (3.19.0)
+      aws-partitions (~> 1.0)
+      aws-sigv4 (~> 1.0)
+      jmespath (~> 1.0)
+    aws-sdk-ec2 (1.29.0)
+      aws-sdk-core (~> 3)
+      aws-sigv4 (~> 1.0)
+    aws-sdk-kms (1.5.0)
+      aws-sdk-core (~> 3)
+      aws-sigv4 (~> 1.0)
+    aws-sdk-s3 (1.9.0)
+      aws-sdk-core (~> 3)
+      aws-sdk-kms (~> 1)
+      aws-sigv4 (~> 1.0)
+    aws-sigv4 (1.0.2)
+    byebug (10.0.1)
+    codeclimate-test-reporter (1.0.8)
+      simplecov (<= 0.13)
+    coderay (1.1.2)
+    colorize (0.8.1)
+    concurrent-ruby (1.0.5)
+    diff-lcs (1.3)
+    docile (1.1.5)
+    dotenv (2.2.1)
+    ffi (1.9.23)
+    filesize (0.1.1)
+    formatador (0.2.5)
+    guard (2.14.2)
+      formatador (>= 0.2.4)
+      listen (>= 2.7, < 4.0)
+      lumberjack (>= 1.0.12, < 2.0)
+      nenv (~> 0.1)
+      notiffany (~> 0.0)
+      pry (>= 0.9.12)
+      shellany (~> 0.0)
+      thor (>= 0.18.1)
+    guard-bundler (2.1.0)
+      bundler (~> 1.0)
+      guard (~> 2.2)
+      guard-compat (~> 1.1)
+    guard-compat (1.2.1)
+    guard-rspec (4.7.3)
+      guard (~> 2.1)
+      guard-compat (~> 1.1)
+      rspec (>= 2.99.0, < 4.0)
+    hashie (3.5.7)
+    i18n (1.0.1)
+      concurrent-ruby (~> 1.0)
+    jmespath (1.4.0)
+    json (2.1.0)
+    listen (3.1.5)
+      rb-fsevent (~> 0.9, >= 0.9.4)
+      rb-inotify (~> 0.9, >= 0.9.7)
+      ruby_dep (~> 1.2)
+    lumberjack (1.0.12)
+    method_source (0.9.0)
+    minitest (5.11.3)
+    nenv (0.3.0)
+    notiffany (0.1.1)
+      nenv (~> 0.1)
+      shellany (~> 0.0)
+    pry (0.11.3)
+      coderay (~> 1.1.0)
+      method_source (~> 0.9.0)
+    rake (12.3.1)
+    rb-fsevent (0.10.3)
+    rb-inotify (0.9.10)
+      ffi (>= 0.5.0, < 2)
+    render_me_pretty (0.8.1)
+      activesupport
+      colorize
+      tilt
+    rspec (3.7.0)
+      rspec-core (~> 3.7.0)
+      rspec-expectations (~> 3.7.0)
+      rspec-mocks (~> 3.7.0)
+    rspec-core (3.7.1)
+      rspec-support (~> 3.7.0)
+    rspec-expectations (3.7.0)
+      diff-lcs (>= 1.2.0, < 2.0)
+      rspec-support (~> 3.7.0)
+    rspec-mocks (3.7.0)
+      diff-lcs (>= 1.2.0, < 2.0)
+      rspec-support (~> 3.7.0)
+    rspec-support (3.7.1)
+    ruby_dep (1.5.0)
+    shellany (0.0.1)
+    simplecov (0.13.0)
+      docile (~> 1.1.0)
+      json (>= 1.8, < 3)
+      simplecov-html (~> 0.10.0)
+    simplecov-html (0.10.2)
+    thor (0.20.0)
+    thread_safe (0.3.6)
+    tilt (2.0.8)
+    tzinfo (1.2.5)
+      thread_safe (~> 0.1)
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  bundler
+  byebug
+  codeclimate-test-reporter
+  forger!
+  guard
+  guard-bundler
+  guard-rspec
+  rake
+
+BUNDLED WITH
+   1.16.1

data/Guardfile

@@ -0,0 +1,19 @@
+guard "bundler", cmd: "bundle" do
+  watch("Gemfile")
+  watch(/^.+\.gemspec/)
+end
+
+guard :rspec, cmd: "bundle exec rspec" do
+  require "guard/rspec/dsl"
+  dsl = Guard::RSpec::Dsl.new(self)
+
+  # RSpec files
+  rspec = dsl.rspec
+  watch(rspec.spec_helper) { rspec.spec_dir }
+  watch(rspec.spec_support) { rspec.spec_dir }
+  watch(rspec.spec_files)
+
+  # Ruby files
+  ruby = dsl.ruby
+  dsl.watch_spec_files_for(ruby.lib_files)
+end

data/LICENSE.txt

@@ -0,0 +1,22 @@
+Copyright (c) 2018 Tung Nguyen
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,249 @@
+# Forger
+
+[![CircleCI](https://circleci.com/gh/tongueroo/forger.svg?style=svg)](https://circleci.com/gh/tongueroo/forger)
+
+Tool to create AWS ec2 instances consistently with pre-configured settings.  The pre-configured settings are stored in the profiles folder of the current project directory.
+Example:
+
+* profiles/default.yml: Default settings. Used when no profile is specified.
+* profiles/myserver.yml: myserver profile.  Used when `--profile myserver` is specified.
+
+## Usage
+
+```sh
+forger create NAME --profile PROFILE
+forger create myserver --profile myserver
+```
+
+In a nutshell, the profile parameters are passed to the ruby aws-sdk [AWS::EC2::Client#run_instances](https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/EC2/Client.html#run_instances-instance_method) method.  This allows you to specify any parameter you wish that is available in the aws-sdk. To check out what a profile looks like check out [example default](docs/example/profiles/default.yml)
+
+## Noop mode
+
+You can do a test run with the `--noop` flag.  This will print out what settings will be used to launch the instance.  This is one good way to inspect the generated user-data script.
+
+```sh
+forger create myserver --profile myserver --noop
+cat tmp/user-data.txt # to view generated user-data script
+```
+
+## Conventional Profile Name
+
+If there is a profile name that matches the ec2 specified instance name, you can omit the `--profile` flag. Example
+
+```sh
+forger create webserver --profile webserver
+forger create webserver # same as above
+```
+
+It is useful to add a random string to the end of your server name, but not use it for the `--profile` flag.  Example:
+
+```
+forger create myserver-abc --profile myserver
+forger create myserver-123 --profile myserver
+```
+
+You can use the `--randomize` option to do this automatically:
+
+```
+forger create myserver --randomize
+```
+
+## Project Structure
+
+Directory  | Description
+------------- | -------------
+app/helpers  | Custom helpers methods.  Define them as modules and their methods are made available whenever ERB is available: `profiles`, `app/scripts`, `app/user-data` files, etc. For example, you would define a `module FooHelper` in `app/helpers/foo_helper.rb`.
+app/partials  | Your partials that can to be included in other scripts.  This is used in conjunction with the `partial` helper method. With great power comes great responsibility.  It is recommended to use partials sparely to keep scripts more straightforward.
+app/scripts  | Where you define common scripts that can be used to configure the server. These scripts can be automatically uploaded to an s3 bucket for later downloading in your user-data script by setting the `s3_folder` settings option.
+app/user-data  | Your user-data scripts that are used to bootstrap EC2 instance.
+app/user-data/layouts  | user-data scripts support layouts. You user-data layouts go in here.
+config/[AWS_EC2_ENV].yml  | The config file where you set configs that you want available in your templating logic.  Examples are: `config/development.yml` and `config/production.yml`. You access the config variables with the `<%= config["var"] %>` helper.
+profiles  | Your profile files.  These files mainly contain parameters that are passed to the aws-sdk run_instances API method.
+tmp  | Where the generated scripts get compiled to. You can manually invoke the compilation via `forger compile` to inspect what is generated.  This is automatically done as part of the `forger` create command.
+
+## Helpers
+
+You can use ERB in the profile files. Some useful helper methods are documented here:
+
+Helper  | Description
+------------- | -------------
+user_data | Allows you to embed a generated user_data script.  More details on the user-data are provided in the user data section below.
+config | Access to the variables set in config/[AWS\_EC2\_ENV].yml.  Examples are `config/development.yml` and `config/production.yml`.
+latest_ami | Returns an AMI id by searching the AMI name pattern and sorting in reverse order.  Example: `latest_ami("ruby-2.5.0_*")` would return the latest ruby AMIs are named with timestamps at the end like so: `ruby-2.5.0_2018-01-30-05-36-02` and `ruby-2.5.0_2018-01-29-05-36-02`.
+search_ami | Returns a collection of AMI image objects based on a search pattern. The query searches on the AMI name.
+extract_scripts | Use this in your bash script to extract the `app/scripts` files that get uploaded to s3.
+
+For a full list of all the template helpers check out: [lib/forger/template/helper](lib/forger/template/helper).
+
+You can also define custom helpers in the `app/helpers` folder as ruby modules with the naming convention `*_helper.rb`.  For example, you would define a `module FooHelper` in `app/helpers/foo_helper.rb`.  Custom helpers are first-class citizens and have access to the same variables, methods, and scope as built-in helpers.
+
+## User-Data
+
+You can provide a user-data script to customize the server upon launch.  The user-data scripts are located under the `app/user-data` folder.  Example:
+
+* app/user-data/myserver.yml
+
+The user-data script is generated on the machine that is running the forger command. If this is your local macosx machine, then the context of your local macosx machine is available. To see the generated user-data script, you can run the create command in `--noop` mode and then inspect the generated script.  Example:
+
+```sh
+forger create myserver --noop
+cat tmp/user-data.txt
+```
+
+Another way to view the generated user-data scripts is the `forger compile` command.  It generates the files in the `tmp` folder.  Example:
+
+```
+forger compile # generates files in tmp folder
+```
+
+To use the user-data script when creating an EC2 instance, use the `user_data` helper method in the profile file.  Here's a grep of an example profile that uses the helper to show you want it looks like. Be sure to surround the ERB call with quotes because the user-data script context is base64 encoded.
+
+```
+$ grep user_data profiles/default.yml
+user_data: "<%= user_data("bootstrap") %>"
+```
+
+### User-Data Layouts
+
+User-data scripts support layouts.  This is useful if you have common setup and finish code with your user-data scripts. Here's an example: `app/user-data/layouts/default.sh`:
+
+```bash
+#!/bin/bash
+# do some setup
+<%= yield %>
+# finish work
+```
+
+And `app/user-data/box.sh`:
+
+```
+yum install -y vim
+```
+
+The resulting generated user-data script will be:
+
+```bash
+#!/bin/bash
+# do some setup
+yum install -y vim
+# finish work
+```
+
+You can specify the layout to use when you call the `user_data` helper method in your profile. Example: `profiles/box.yml`:
+
+```yaml
+---
+...
+user_data: <%= user_data("box.sh", layout: "mylayout" ) %>
+...
+```
+
+If there's a `layouts/default.sh`, then it will automatically be used without having to specify the layout option.  You can disable this behavior by passing in `layout: false` or by deleting the `layouts/default.sh` file.
+
+### Config
+
+You can set variables in a config file and they are available when ERB is available: profiles, user-data, scripts, etc.  Example `config/development.yml`:
+
+```yaml
+---
+vpc_id: vpc-123
+subnets:
+  - subnet-123
+  - subnet-456
+  - subnet-789
+security_group_ids:
+  - sg-123
+```
+
+The variables are accessed via the `config` helper method. Here's a filtered example where it shows the relevant part of a profile: `profiles/default.yml`:
+
+```yaml
+image_id: ami-4fffc834 # Amazon Lambda AMI
+instance_type: t2.medium
+security_group_ids: <%= config["security_group_ids"] %>
+subnet_id: <%= config["subnets"].shuffle %>
+...
+```
+
+### Settings
+
+A `config/settings.yml` file controls the internal behavior of forger. It is different from config files which are meant for user defined varibles.  Settings variables are for internal use.  Example:
+
+```yaml
+development:
+  # By setting s3_folder, forger will automatically tarball and upload your scripts
+  # to set. You then can then use the extract_scripts helper method to download
+  # the scripts onto the server.
+  s3_folder: boltops-infra-stag/ec2
+  # compile_clean: true # uncomment to clean at the end of a compile
+  # extract_scripts:
+  #   to: "/opt"
+  #   as: "ec2-user"
+
+production:
+```
+
+### Hooks
+
+There is only one hook: `before_run_instances`.  You can configure this with `config/hooks.yml`:  Example:
+
+```
+---
+before_run_instances: /path/to/my/script.sh
+```
+
+This will run `/path/to/my/script.sh` as a shelled out command before the `run_instances` call.
+
+## Dotenv File Support
+
+You can set and configure environment variables in `.env*` files.  Examples of this are in the [example](docs/example) project.
+
+## AMI Creation
+
+To create AMIs you can use the `forger ami` command.  This command launches an EC2 instance with the specified profile and creates an AMI after the user-data script successfully completes. It does this by appending an AMI creation script at the end of the user-data script.  It is recommended to use the `set -e` option in your user-data script so that any error halts the script and the AMI does not get created.
+
+After the AMI is successfully created, the instance will also terminate itself automatically so you do not have to worry about cleanup.  This is also done with an appended script. For more help run `forger ami help`.
+
+For the instance to image and terminate itself, the EC2 IAM role for the instance requires IAM permissions for:
+
+* aws ec2 create-image
+* aws ec2 cancel-spot-instance-requests # in case a spot instance was used
+* aws ec2 terminate-instances
+
+## Spot Instance Support
+
+Spot instance is natively supported by the AWS `run_instances` command by adding the `instance_market_options` to the parameters in the profile file.  The available spot market options are available here:
+
+* [https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_LaunchTemplateSpotMarketOptionsRequest.html](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_LaunchTemplateSpotMarketOptionsRequest.html)
+* [https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/EC2/Types/SpotMarketOptions.html](https://docs.aws.amazon.com/sdk-for-ruby/v3/api/Aws/EC2/Types/SpotMarketOptions.html)
+
+An example of a spot instance profile is provided in [example/profiles/spot.yml](docs/example/profiles/spot.yml).
+
+## More Help
+
+```sh
+forger create help
+forger ami help
+forger compile help
+forger help # general help
+```
+
+Examples are in the [example](docs/example) folder.  You will have to update settings like your subnet and security group ids.
+
+## Installation
+
+```sh
+gem install forger
+```
+
+### Dependencies
+
+This tool mainly uses the ruby aws-sdk. Though it does use the aws cli to check your region: `aws configure get region`. It also the uses `aws s3 sync` to perform the scripts upload. So it is dependent on the the `aws cli`.
+
+## Contributing
+
+1. Fork it
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am "Add some feature"`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create new Pull Request