aws_ec2_environment 0.1.0

data/README.md

@@ -0,0 +1,296 @@
+# AwsEc2Environment
+
+A gem that makes it easier to interact with and deploy Ruby projects that are
+hosted on EC2 instances in AWS.
+
+## Installation
+
+Install the gem and add to the application's Gemfile by executing:
+
+    $ bundle add aws_ec2_environment
+
+If bundler is not being used to manage dependencies, install the gem by
+executing:
+
+    $ gem install aws_ec2_environment
+
+## Usage
+
+Use `AwsEc2Environment.from_yaml_file` to create a new representation of your
+EC2 environment from a config file:
+
+```ruby
+ec2_env = AwsEc2Environment.from_yaml_file("./aws.yml", :production)
+
+# this will ensure that any post-connection cleanup is handled, such as terminating
+# any SSM port forwarding sessions that are active
+at_exit { ec2_env.stop_ssh_port_forwarding_sessions } if ec2_env.config.use_ssm
+
+# this will return a list of hosts for sshing, handling any pre-connection setup
+# such as starting port forwarding sessions for each instance if SSM is enabled.
+ec2_env.hosts_for_sshing
+```
+
+### Configuration
+
+This is the most basic configuration you can have:
+
+```yaml
+production:
+  aws_region: ap-southeast-2
+  ssh_user: deploy
+  filters:
+    - name: 'instance-state-name'
+      values: ['running']
+    - name: 'tag:Name'
+      values: ['MyWebsiteProductionAppServerAsg']
+```
+
+All the top level properties are required, and the `filters` key holds an array
+of filters that are used with the
+[`DescribeInstances` API endpoint](https://docs.aws.amazon.com/AWSEC2/latest/APIReference/API_DescribeInstances.html).
+
+### With bastion hosts
+
+You can specify filters for a bastion instance too:
+
+```yaml
+production:
+  aws_region: ap-southeast-2
+  ssh_user: deploy
+  filters:
+    - name: 'instance-state-name'
+      values: ['running']
+    - name: 'tag:Name'
+      values: ['MyWebsiteProductionAppServerAsg']
+  bastion_instance:
+    ssh_user: bastion
+    filters:
+      - name: 'instance-state-name'
+        values: ['running']
+      - name: 'tag:Name'
+        values: ['MyWebsiteProductionBastionAsg']
+```
+
+Note that the filters should result in _one_ instance being returned, otherwise
+an error will be thrown.
+
+If you use the same user as your application servers, you can pass an array of
+filters as the value of the top-level property:
+
+```yaml
+production:
+  aws_region: ap-southeast-2
+  ssh_user: deploy
+  filters:
+    - name: 'instance-state-name'
+      values: ['running']
+    - name: 'tag:Name'
+      values: ['MyWebsiteProductionAppServerAsg']
+  bastion_instance:
+    - name: 'instance-state-name'
+      values: ['running']
+    - name: 'tag:Name'
+      values: ['MyWebsiteProductionBastionAsg']
+```
+
+#### With SSM
+
+If your instances have the
+[SSM Agent](https://docs.aws.amazon.com/systems-manager/latest/userguide/ssm-agent.html)
+(preinstalled on some
+[AMIs](https://docs.aws.amazon.com/systems-manager/latest/userguide/ami-preinstalled-agent.html)),
+you can use SSM to connect directly to instances even if they're in a private
+subnet, via port forwarding:
+
+```yaml
+production:
+  aws_region: ap-southeast-2
+  ssh_user: deploy
+  ssm_host: 'ec2.#{id}.local.ackama.app'
+  use_ssm: true
+  filters:
+    - name: 'instance-state-name'
+      values: ['running']
+    - name: 'tag:Name'
+      values: ['MyWebsiteProductionAppServerAsg']
+```
+
+> This requires the
+> [`aws`](https://docs.aws.amazon.com/cli/latest/userguide/cli-chap-welcome.html)
+> CLI and
+> [`session-manager-plugin`](https://github.com/aws/session-manager-plugin) to
+> be installed locally. These both come preinstalled on GitHub Actions runners,
+> and are otherwise easy to install manually.
+>
+> - [Installing `aws` cli](https://docs.aws.amazon.com/cli/latest/userguide/getting-started-install.html)
+> - [Installing `session-manager-plugin`](https://docs.aws.amazon.com/systems-manager/latest/userguide/session-manager-working-with-install-plugin.html)
+
+You can also specify an alternative hostname to use instead of `127.0.0.1` with
+the `ssm_host` property - this is useful when working with tools like Capistrano
+that only log the host _name_, so this property can let you ensure each instance
+can be identified in the logs.
+
+This property should be a host that resolves to `127.0.0.1`, and you can inject
+the instance id with `#{id}`.
+
+> Ackama provides `ec2.*.local.ackama.app` for this
+
+### With Capistrano
+
+```ruby
+# ./Capfile
+# ...
+require "aws_ec2_environment"
+
+# ./config/deploy/production.rb
+set :rails_env, "production"
+set :branch, "production"
+
+ec2_env = AwsEc2Environment.from_yaml_file("./aws.yml", :production)
+
+at_exit { ec2_env.stop_ssh_port_forwarding_sessions } if ec2_env.config.use_ssm
+
+ssh_options = {}
+
+if ec2_env.use_bastion_server?
+  ssh_options[:proxy] = Net::SSH::Proxy::Command.new(ec2_env.build_ssh_bastion_proxy_command)
+end
+
+set :ssh_options, ssh_options
+
+role(:app, ec2_env.hosts_for_sshing, user: ec2_env.config.ssh_user)
+```
+
+### With custom port forwarding
+
+You can also use the `SsmPortForwardingSession` class directly to do port
+forwarding, which can be useful for things like custom rake tasks:
+
+```ruby
+require "aws_ec2_environment"
+
+task :forward_port, %i[instance_id remote_port local_port] => :environment do |_, args|
+  # trap ctl+c to make things a bit nicer (otherwise we'll get an ugly stacktrace)
+  # since we expect this to be used to terminate the command
+  trap("SIGINT") { exit }
+
+  logger = Logger.new($stdout)
+
+  instance_id = args.fetch(:instance_id)
+  remote_port = args.fetch(:remote_port)
+  local_port = args.fetch(:local_port, nil)
+
+  session = AwsEc2Environment::SsmPortForwardingSession.new(
+    instance_id,
+    remote_port,
+    local_port:,
+    logger:
+  )
+
+  at_exit { session.close }
+
+  local_port = session.wait_for_local_port
+
+  local_alias = "ec2.#{instance_id}.local.ackama.app:#{local_port}"
+  logger.info "Use #{local_alias} to communicate with port #{remote_port} on #{instance_id}"
+
+  loop { sleep 1 }
+end
+```
+
+### AWS Authentication and Permissions
+
+Since this gem interacts with AWS, it must be configured with credentials - see
+[here](https://docs.aws.amazon.com/sdk-for-ruby/v3/developer-guide/setup-config.html)
+for how to do that.
+
+> We recommend using
+> [OpenID Connect](https://docs.github.com/en/actions/deployment/security-hardening-your-deployments/configuring-openid-connect-in-amazon-web-services)
+> to authenticate with AWS when running in GitHub Actions.
+
+The credentials must be for an identity that is allowed to perform the
+`ec2:DescribeInstances` action. If you're using SSM you must also allow the
+`ssm:StartSession` and `ssm:TerminateSession` actions.
+
+Here is a sample IAM policy document that grants these actions conditionally in
+accordance with the principle of least privilege:
+
+```json
+{
+  "Version": "2012-10-17",
+  "Statement": [
+    {
+      "Sid": "AllowDescribingInstances",
+      "Effect": "Allow",
+      "Action": "ec2:DescribeInstances",
+      "Resource": "*"
+    },
+    {
+      "Sid": "AllowStartingPortForwardingSessions",
+      "Effect": "Allow",
+      "Action": "ssm:StartSession",
+      "Resource": "arn:aws:ssm:*::document/AWS-StartPortForwardingSession"
+    },
+    {
+      "Sid": "AllowStartingNewSessionsOnTaggedEC2Instances",
+      "Effect": "Allow",
+      "Action": "ssm:StartSession",
+      "Resource": "arn:aws:ec2:*:account-id:instance/*",
+      "Condition": {
+        "StringEquals": {
+          "ssm:resourceTag/Environment": "Production",
+          "ssm:resourceTag/Name": "MyWebsiteProductionAppServerAsg"
+        }
+      }
+    },
+    {
+      "Sid": "AllowTerminatingOwnSessions",
+      "Effect": "Allow",
+      "Action": "ssm:TerminateSession",
+      "Resource": "arn:aws:ssm:*:account-id:session/*",
+      "Condition": {
+        "StringLike": {
+          "ssm:resourceTag/aws:ssmmessages:session-id": "${aws:username}"
+        }
+      }
+    }
+  ]
+}
+```
+
+> Remember to replace "account-id" in the above document with the ID of your AWS
+> account!
+
+> If you are using a federated identity (such as GitHub's OpenID Connect
+> provider), then you will need to replace `${aws:username}` with
+> `${aws:userid}` - see
+> [here](https://aws.amazon.com/premiumsupport/knowledge-center/iam-policy-variables-federated/)
+> for more.
+
+## Development
+
+After checking out the repo, run `bin/setup` to install dependencies. Then, run
+`rake spec` to run the tests. You can also run `bin/console` for an interactive
+prompt that will allow you to experiment.
+
+To install this gem onto your local machine, run `bundle exec rake install`. To
+release a new version, update the version number in `version.rb`, and then run
+`bundle exec rake release`, which will create a git tag for the version, push
+git commits and the created tag, and push the `.gem` file to
+[rubygems.org](https://rubygems.org).
+
+## Contributing
+
+Contributions are welcome. Please see the
+[contribution guidelines](CONTRIBUTING.md) for detailed instructions.
+
+## License
+
+The gem is available as open source under the terms of the
+[MIT License](https://opensource.org/licenses/MIT).
+
+## Code of Conduct
+
+Everyone interacting in this project's codebases, issue trackers, chat rooms and
+mailing lists is expected to follow the [code of conduct](CODE_OF_CONDUCT.md).

data/Rakefile

@@ -0,0 +1,12 @@
+# frozen_string_literal: true
+
+require "bundler/gem_tasks"
+require "rspec/core/rake_task"
+
+RSpec::Core::RakeTask.new(:spec)
+
+require "rubocop/rake_task"
+
+RuboCop::RakeTask.new
+
+task default: %i[spec rubocop]

data/lib/aws_ec2_environment/ci_service.rb

@@ -0,0 +1,151 @@
+class AwsEc2Environment
+  class CiService # rubocop:disable Metrics/ClassLength
+    CI_SERVICES = [
+      {
+        name: "AppVeyor",
+        detect: "APPVEYOR",
+        build_id_var: "APPVEYOR_BUILD_NUMBER"
+      },
+      {
+        name: "Azure Pipelines",
+        detect: "BUILD_BUILDURI",
+        build_id_var: "BUILD_BUILDNUMBER"
+      },
+      {
+        name: "Bamboo",
+        detect: "bamboo_agentId",
+        build_id_var: "bamboo_buildNumber"
+      },
+      {
+        name: "BitBucket Pipelines",
+        detect: "BITBUCKET_BUILD_NUMBER",
+        build_id_var: "BITBUCKET_BUILD_NUMBER"
+      },
+      {
+        name: "Buddy",
+        detect: "BUDDY_WORKSPACE_ID",
+        build_id_var: "BUDDY_EXECUTION_ID"
+      },
+      {
+        name: "Buildkite",
+        detect: "BUILDKITE",
+        build_id_var: "BUILDKITE_BUILD_NUMBER"
+      },
+      {
+        name: "CircleCI",
+        detect: "CIRCLECI",
+        build_id_var: "CIRCLE_BUILD_NUM"
+      },
+      {
+        name: "Cirrus",
+        detect: "CIRRUS_CI",
+        build_id_var: "CIRRUS_BUILD_ID"
+      },
+      {
+        name: "CodeBuild",
+        detect: "CODEBUILD_BUILD_ID",
+        build_id_var: "CODEBUILD_BUILD_ID"
+      },
+      {
+        name: "Codefresh",
+        detect: "CF_BUILD_ID",
+        build_id_var: "CF_BUILD_ID"
+      },
+      {
+        name: "CodeShip",
+        detect: -> { ENV.fetch("CI_NAME", "") == "codeship" },
+        build_id_var: "CI_BUILD_NUMBER"
+      },
+      {
+        name: "Drone",
+        detect: "DRONE",
+        build_id_var: "DRONE_BUILD_NUMBER"
+      },
+      {
+        name: "GitHub Actions",
+        detect: "GITHUB_ACTIONS",
+        build_id_var: "GITHUB_RUN_ID"
+      },
+      {
+        name: "GitLab",
+        detect: "GITLAB_CI",
+        build_id_var: "CI_PIPELINE_ID"
+      },
+      {
+        name: "Jenkins",
+        detect: "JENKINS_URL",
+        build_id_var: "BUILD_NUMBER"
+      },
+      {
+        name: "JetBrains Spaces",
+        detect: "JB_SPACE_EXECUTION_NUMBER",
+        build_id_var: "JB_SPACE_EXECUTION_NUMBER"
+      },
+      {
+        name: "Puppet",
+        detect: "DISTELLI_APPNAME",
+        build_id_var: "DISTELLI_BUILDNUM"
+      },
+      {
+        name: "Scrutinizer",
+        detect: "SCRUTINIZER",
+        build_id_var: "SCRUTINIZER_INSPECTION_UUID"
+      },
+      {
+        name: "Semaphore",
+        detect: "SEMAPHORE",
+        build_id_var: "SEMAPHORE_JOB_ID"
+      },
+      {
+        name: "Shippable",
+        detect: "SHIPPABLE",
+        build_id_var: "BUILD_NUMBER"
+      },
+      {
+        name: "TeamCity",
+        detect: "TEAMCITY_VERSION",
+        build_id_var: "BUILD_NUMBER"
+      },
+      {
+        name: "Travis",
+        detect: "TRAVIS",
+        build_id_var: "TRAVIS_BUILD_NUMBER"
+      },
+      {
+        name: "Vela",
+        detect: "VELA",
+        build_id_var: "VELA_BUILD_NUMBER"
+      },
+      {
+        name: "Wercker",
+        detect: "WERCKER_MAIN_PIPELINE_STARTED",
+        build_id_var: "WERCKER_MAIN_PIPELINE_STARTED"
+      },
+      {
+        name: "Woodpecker",
+        detect: -> { ENV.fetch("CI", "") == "woodpecker" },
+        build_id_var: "CI_BUILD_NUMBER"
+      }
+    ].freeze
+
+    # Attempts to determine if the current process is running on a CI service,
+    # and if so returns the name of the service and the id of the current build
+    # which generally can be used to find the details and logs for this build.
+    def self.detect
+      service = CI_SERVICES.find do |details|
+        if details[:detect].is_a? String
+          ENV.key? details[:detect]
+        else
+          details[:detect].call
+        end
+      end
+
+      return nil if service.nil?
+
+      {
+        name: service[:name],
+        build_id: ENV.fetch(service[:build_id_var])
+      }
+    end
+  end
+end

data/lib/aws_ec2_environment/config.rb

@@ -0,0 +1,47 @@
+# Holds the details about an application environment composed primarily of EC2 instances, including
+#   - what region they're in
+#   - the user to use for sshing
+#   - how to identify those instances
+#   - how to identify the bastion instance to use to connect (if any)
+#   - if SSM should be used to connect to the instances
+class AwsEc2Environment
+  class Config
+    attr_reader :env_name,
+                :aws_region,
+                :ssh_user,
+                :instance_filters,
+                :bastion_ssh_user,
+                :bastion_filters,
+                :use_ssm,
+                :ssm_host
+
+    # @param [Symbol] env_name
+    # @param [Hash] attrs
+    def initialize(env_name, attrs)
+      @env_name = env_name
+      @aws_region = attrs.fetch("aws_region")
+      @use_ssm = attrs.fetch("use_ssm", false)
+      @ssm_host = attrs.fetch("ssm_host", "127.0.0.1")
+      @ssh_user = attrs.fetch("ssh_user")
+      @instance_filters = attrs.fetch("filters")
+
+      @bastion_filters, @bastion_ssh_user = fetch_bastion_details(attrs)
+    end
+
+    private
+
+    def fetch_bastion_details(attrs)
+      bastion_instance = attrs.fetch("bastion_instance", nil)
+      bastion_ssh_user = ssh_user
+
+      # if the bastion_instance is a hash, then we need to fetch specific keys
+      unless bastion_instance.nil? || bastion_instance.is_a?(Array)
+        bastion_ssh_user = bastion_instance.fetch("ssh_user", ssh_user)
+        # this is required when using the longhand
+        bastion_instance = bastion_instance.fetch("filters")
+      end
+
+      [bastion_instance, bastion_ssh_user]
+    end
+  end
+end

data/lib/aws_ec2_environment/ssm_port_forwarding_session.rb

@@ -0,0 +1,136 @@
+require "aws-sdk-ssm"
+require "pty"
+require "timeout"
+require "shellwords"
+require "json"
+
+class AwsEc2Environment
+  class SsmPortForwardingSession
+    class SessionIdNotFoundError < Error; end
+    class SessionTimedOutError < Error; end
+    class SessionProcessError < Error; end
+
+    # @return [String]
+    attr_reader :instance_id
+
+    # @return [Number]
+    attr_reader :remote_port
+
+    # @return [String]
+    attr_reader :pid
+
+    # rubocop:disable Metrics/ParameterLists
+    def initialize(
+      instance_id, remote_port,
+      local_port: nil, logger: Logger.new($stdout),
+      timeout: 15, reason: nil
+    )
+      # rubocop:enable Metrics/ParameterLists
+      @logger = logger
+      @instance_id = instance_id
+      @remote_port = remote_port
+      @local_port = nil
+      @timeout = timeout
+
+      @reader, @writer, @pid = PTY.spawn(ssm_port_forward_cmd(local_port, reason))
+
+      @cmd_output = ""
+      @session_id = wait_for_session_id
+
+      @logger.info("SSM session #{@session_id} opening, forwarding port #{remote_port} on #{instance_id}")
+    end
+
+    def close
+      @logger.info "Terminating SSM session #{@session_id}..."
+      resp = Aws::SSM::Client.new.terminate_session({ session_id: @session_id })
+      @logger.info "Terminated SSM session #{resp.session_id} successfully"
+
+      @reader.close
+      @writer.close
+    end
+
+    def wait_for_local_port
+      _, local_port = expect_cmd_output(/Port (\d+) opened for sessionId #{@session_id}.\r?\n/, @timeout) || []
+
+      if local_port.nil?
+        raise(
+          SessionTimedOutError,
+          "SSM session #{@session_id} did not become ready within #{@timeout} seconds (maybe increase the timeout?)"
+        )
+      end
+
+      local_port.to_i
+    end
+
+    private
+
+    def ssm_port_forward_cmd(local_port, reason)
+      document_name = "AWS-StartPortForwardingSession"
+      parameters = { "portNumber" => [remote_port.to_s] }
+      parameters["localPortNumber"] = [local_port.to_s] unless local_port.nil?
+      flags = [
+        ["--target", instance_id],
+        ["--document-name", document_name],
+        ["--parameters", parameters.to_json]
+      ]
+
+      flags << ["--reason", reason] unless reason.nil?
+      flags = flags.map { |(flag, value)| "#{flag} #{Shellwords.escape(value)}" }.join(" ")
+
+      "aws ssm start-session #{flags}"
+    end
+
+    # Checks the cmd process output until either the given +pattern+ matches or the +timeout+ is over.
+    #
+    # It returns an array with the result of the match or otherwise +nil+.
+    #
+    # This is effectively a re-implementation of the +File#expect+ method except it captures
+    # the cmd process output over time so we can include it in the case of errors
+    def expect_cmd_output(pattern, timeout)
+      Timeout.timeout(timeout) do
+        loop do
+          update_cmd_output
+
+          match = @cmd_output.match(pattern)
+
+          return match.to_a unless match.nil?
+        end
+      end
+    rescue Timeout::Error
+      nil
+    end
+
+    # Updates the tracked output of the cmd process with any new data that is available in the buffer,
+    # until the next read will block at which point this method returns.
+    def update_cmd_output
+      loop do
+        @cmd_output << @reader.read_nonblock(1)
+
+        # next unless @cmd_output[-1] == "\n"
+        #
+        # last_newline = @cmd_output.rindex("\n", -2) || 0
+        # puts @cmd_output.slice(last_newline + 1, @cmd_output.length) if @cmd_output[-1] == "\n"
+      end
+    rescue IO::EAGAINWaitReadable
+      # do nothing as we don't want to block
+    rescue Errno::EIO
+      output = @cmd_output.strip
+      output = "<nothing was outputted by process>" if output.empty?
+
+      raise SessionProcessError, output
+    end
+
+    def wait_for_session_id
+      _, session_id = expect_cmd_output(/Starting session with SessionId: ([=,.@\w-]+)\r?\n/, @timeout) || []
+
+      if session_id.nil?
+        raise(
+          SessionIdNotFoundError,
+          "could not find session id within #{@timeout} seconds - SSM plugin output: #{@cmd_output}"
+        )
+      end
+
+      session_id
+    end
+  end
+end

data/lib/aws_ec2_environment/version.rb

@@ -0,0 +1,5 @@
+# frozen_string_literal: true
+
+class AwsEc2Environment
+  VERSION = "0.1.0"
+end