codebuild 0.1.0 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: eddc827d24d5707f612e5fc445c838a5ff90293e21518ecb3b65f33211601788
-  data.tar.gz: ee1885f87b9a060622957287f4d35f772d02924acb3de20ad9e08faa442063e9
+  metadata.gz: 57f656279e8afad7619113d3834b6af34c1dd7d583934b2829f7a621e8a07c60
+  data.tar.gz: e72dcd6f7db6f3b9569074e41f0c478aaf808def00360a00eabf7eea4aeee4fc
 SHA512:
-  metadata.gz: 8f49961753e2106d88252b038bfdc268517be497f2a4361f7edc8a1694bcc4c2c146ec872ae8fa94ac1d42168f250c92b4aa2e41f812ae3cf4cde3b9c6bc9710
-  data.tar.gz: 98133343e44e59f1dcb33db2136995a4caf121c865b51eb2ac978e2277984bbd930bcd8565bbe6c7673608810d00568f98e6febf4e3f47a6c158f125b446b32b
+  metadata.gz: 64c0580f8f1dfdfefc7494cdb33a4d7805d2867158b76334f5725cc5f5fdf1103e116b160b912c1a60a8da6dcec39a2a8e6a8fd0d372e57e367e55c073028378
+  data.tar.gz: 856cf81c1c5c6eda69a87933727c9924ae9548c398adefed1c12503fd6f92bcc67518dfaa23b0f9c04ba80d269a75db3f800297cd9bf1258133c0e67f41dafe7

data/.codebuild/buildspec.yml

@@ -0,0 +1,8 @@
+version: 0.2
+
+phases:
+  build:
+    commands:
+      - echo Build started on `date`
+      - bundle
+      - bundle exec rspec

data/.codebuild/project.rb

@@ -0,0 +1,16 @@
+# For methods, refer to the properties of the CloudFormation CodeBuild::Project https://amzn.to/2UTeNlr
+# For convenience methods, refer to the source https://github.com/tongueroo/codebuild/blob/master/lib/codebuild/dsl/project.rb
+
+# Note: for the oauth_token, you have to set this parameter in ssm parameter store
+# One way to create an GitHub oauth token:
+# Go to GitHub / Settings / Developer Settings / Personal access tokens
+# If using webhook, the oauth token needs admin:repo_hook
+
+name("codebuild")
+github_url("https://github.com/tongueroo/codebuild")
+linux_image("aws/codebuild/ruby:2.5.3-1.7.0")
+environment_variables(
+  JETS_ENV: "test",
+)
+triggers(webhook: true)
+local_cache(false)

data/.gitignore

@@ -14,3 +14,5 @@ spec/reports
 test/tmp
 test/version_tmp
 tmp
+
+Gemfile.lock

data/.rspec

@@ -1,2 +1,3 @@
 --color
 --format documentation
+--require spec_helper

data/CHANGELOG.md

@@ -3,5 +3,12 @@
 All notable changes to this project will be documented in this file.
 This project *tries* to adhere to [Semantic Versioning](http://semver.org/), even before v1.0.
 
+## [0.2.0]
+- First good release.
+- Project and Role DSL
+- codebuild commands: init, deploy, start
+- lookup path support
+- Docs in main readme and readme folder
+
 ## [0.1.0]
 - Initial release.

data/README.md

@@ -1,14 +1,128 @@
 # Codebuild
 
-CodeBuild tool.
+![Build Status](https://codebuild.us-west-2.amazonaws.com/badges?uuid=eyJlbmNyeXB0ZWREYXRhIjoidHFFaithL1pLZWFEUzBXbk5LY05Mc0FrZW56NDVJWTArbUlOdzBUalVPWWZ5a1ZYUEFtTkhlbFBjeURRZEd1Q292WTI1RUJwWkcvdEgxUXhSYnBqVU9VPSIsIml2UGFyYW1ldGVyU3BlYyI6IjJ0dnpqMC9XMzQ4VExCMGgiLCJtYXRlcmlhbFNldFNlcmlhbCI6MX0%3D&branch=master)
 
-## Usage
+Tool creates a CodeBuild project with some reasonable defaults. It provides a DSL that can be used to create and override setting if required.
+
+## Quick Start
 
     codebuild init
-    codebuild create
-    codebuild update
+    codebuild deploy
     codebuild start
 
+## Usage
+
+1. **init**: generate starter .codebuild files.
+2. **deploy**: deploy the CodeBuild project on AWS.
+3. **start**: kick off a CodeBuild project run.
+
+### Init
+
+First, run `codebuild init` to generate a starter .codebuild structure.
+
+    $ tree .codebuild
+    .codebuild
+    ├── buildspec.yml
+    ├── project.rb
+    └── role.rb
+
+File | Description
+--- | ---
+buildspec.yml | The build commands to run.
+project.rb | The codebuild project defined as a DSL.
+role.rb | The IAM role assocaited with the codebuild project defined as a DSL.
+
+### Deploy
+
+Adjust the files in `.codebuild` to fit your needs. When you're ready, deploy the CodeBuild project with:
+
+    codebuild deploy STACK_NAME
+
+More examples:
+
+    codebuild deploy # infers the CloudFormation name from the parent folder
+    codebuild deploy stack-name # explicitly specify stack name
+
+It is useful to just see the generated CloudFormation template with `--noop` mode:
+
+    codebuild deploy --noop # see generated CloudFormation template
+
+For more help:
+
+    codebuild deploy -h
+
+### Start
+
+When you are ready to start a codebuild project run, you can use `codebuild start`. Examples:
+
+    codebuild start # infers the name from the parent folder
+    codebuild start stack-name # looks up project via CloudFormation stack
+    codebuild start demo-project # looks up project via CodeBuild project name
+
+The `codebuild start` command understands multiple identifiers. It will look up the codebuild project either via CloudFormation or the CodeBuild project name.
+
+## Project DSL
+
+The tool provides a DSL to create a codebuild project.  Here's an example.
+
+.codebuild/project.rb:
+
+```ruby
+name("demo")
+github_url("https://github.com/tongueroo/demo-ufo")
+linux_image("aws/codebuild/ruby:2.5.3-1.7.0")
+environment_variables(
+  UFO_ENV: "development",
+  API_KEY: "ssm:/codebuild/demo/api_key"
+)
+```
+
+Here's a list of some of the convenience shorthand DSL methods:
+
+* github_url(url)
+* github_source(options={})
+* linux_image(name)
+* linux_environment(options={})
+* environment_variables(vars)
+* local_cache(enable=true)
+
+Please refer to [lib/codebuild/dsl/project.rb](lib/codebuild/dsl/project.rb) for the full list.
+
+More slightly more control, you may be interested in the `github_source` and `linux_environment` methods.  For even more control, see [Full DSL docs](readme/full_dsl.md).
+
+## IAM Role DSL
+
+The codebuild tool can create the IAM service role associated with the codebuild project. Here's an example:
+
+.codebuild/role.rb:
+
+```ruby
+iam_policy("logs", "ssm")
+```
+
+For more control, here's a longer form:
+
+```ruby
+iam_policy(
+  action: [
+    "logs:CreateLogGroup",
+    "logs:CreateLogStream",
+    "logs:PutLogEvents",
+    "ssm:*",
+  ],
+  effect: "Allow",
+  resource: "*"
+)
+```
+
+## Full DSL
+
+The convenience DSL methods shown above are short and clean.  They merely wrap a DSL that map to the properties of CloudFormation resources like [AWS::CodeBuild::Project](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-codebuild-project.html) and [AWS::IAM::Role](https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-resource-iam-role.html). Refer the [Full DSL docs](readme/full_dsl.md) for more info.
+
+## Lookup Paths
+
+By default, the codebuild tool looks up files in the `.codebuild` folder.  You can affect the behavior of the lookup logic with the `--lookup` option.  More info [Lookup docs](readme/lookup.md).
+
 ## Installation
 
 Add this line to your application's Gemfile:

data/codebuild.gemspec

@@ -8,7 +8,7 @@ Gem::Specification.new do |spec|
   spec.version       = Codebuild::VERSION
   spec.authors       = ["Tung Nguyen"]
   spec.email         = ["tongueroo@gmail.com"]
-  spec.summary       = "Codebuild tool"
+  spec.summary       = "CodeBuild DSL Tool to Quickly Create CodeBuild Project"
   spec.homepage      = "https://github.com/tongueroo/codebuild"
   spec.license       = "MIT"
 
@@ -19,9 +19,12 @@ Gem::Specification.new do |spec|
   spec.require_paths = ["lib"]
 
   spec.add_dependency "activesupport"
+  spec.add_dependency "aws-sdk-cloudformation"
+  spec.add_dependency "aws-sdk-codebuild"
+  spec.add_dependency "aws-sdk-ssm"
+  spec.add_dependency "cfn_camelizer"
   spec.add_dependency "rainbow"
   spec.add_dependency "thor"
-  spec.add_dependency "aws-sdk-codebuild"
 
   spec.add_development_dependency "bundler"
   spec.add_development_dependency "byebug"

data/lib/codebuild/aws_services/helpers.rb

@@ -0,0 +1,52 @@
+module Codebuild::AwsServices
+  module Helpers
+    def stack_exists?(stack_name)
+      return false if ENV['TEST']
+
+      exist = nil
+      begin
+        # When the stack does not exist an exception is raised. Example:
+        # Aws::CloudFormation::Errors::ValidationError: Stack with id blah does not exist
+        cfn.describe_stacks(stack_name: stack_name)
+        exist = true
+      rescue Aws::CloudFormation::Errors::ValidationError => e
+        if e.message =~ /does not exist/
+          exist = false
+        elsif e.message.include?("'stackName' failed to satisfy constraint")
+          # Example of e.message when describe_stack with invalid stack name
+          # "1 validation error detected: Value 'instance_and_route53' at 'stackName' failed to satisfy constraint: Member must satisfy regular expression pattern: [a-zA-Z][-a-zA-Z0-9]*|arn:[-a-zA-Z0-9:/._+]*"
+          puts "Invalid stack name: #{stack_name}"
+          puts "Full error message: #{e.message}"
+          exit 1
+        else
+          raise # re-raise exception  because unsure what other errors can happen
+        end
+      end
+      exist
+    end
+
+    def inferred_stack_name
+      File.basename(Dir.pwd).gsub('_','-').gsub(/[^0-9a-zA-Z,-]/, '')
+    end
+
+    def are_you_sure?(stack_name, action)
+      if @options[:sure]
+        sure = 'y'
+      else
+        message = case action
+        when :update
+          "Are you sure you want to want to update the #{stack_name.color(:green)} stack with the changes? (y/N)"
+        when :delete
+          "Are you sure you want to want to delete the #{stack_name.color(:green)} stack? (y/N)"
+        end
+        puts message
+        sure = $stdin.gets
+      end
+
+      unless sure =~ /^y/
+        puts "Whew! Exiting without running #{action}."
+        exit 0
+      end
+    end
+  end
+end

data/lib/codebuild/aws_services.rb

@@ -0,0 +1,17 @@
+require "aws-sdk-codebuild"
+require "aws-sdk-cloudformation"
+
+module Codebuild
+  module AwsServices
+    autoload :Helpers, "codebuild/aws_services/helpers"
+    include Helpers
+
+    def codebuild
+      @codebuild ||= Aws::CodeBuild::Client.new
+    end
+
+    def cfn
+      @cfn ||= Aws::CloudFormation::Client.new
+    end
+  end
+end

data/lib/codebuild/cli.rb

@@ -5,18 +5,64 @@ module Codebuild
 
     desc "init", "Initialize project with .codebuild files"
     long_desc Help.text(:init)
-    def init
-      puts "init called"
+    Init.cli_options.each do |args|
+      option(*args)
+    end
+    register(Init, "init", "init", "Set up initial ufo files.")
+
+    desc "evaluate", "Evaluate the .codebuild/project.rb DSL."
+    long_desc Help.text(:evaluate)
+    def evaluate
+      Dsl.new(options).evaluate
+    end
+
+    deploy_options = Proc.new do
+      option :lookup, desc: "folder to use within .codebuild folder for extra lookups of files"
+    end
+
+    desc "create", "Create codebuild project."
+    long_desc Help.text(:create)
+    deploy_options.call
+    def create(stack_name=nil)
+      Create.new(options.merge(stack_name: stack_name)).run
+    end
+
+    desc "update", "Update codebuild project."
+    long_desc Help.text(:update)
+    deploy_options.call
+    def update(stack_name=nil)
+      Update.new(options.merge(stack_name: stack_name)).run
+    end
+
+    desc "deploy", "Deploy codebuild project."
+    long_desc Help.text(:deploy)
+    deploy_options.call
+    def deploy(stack_name=nil)
+      Deploy.new(options.merge(stack_name: stack_name)).run
+    end
+
+    desc "delete", "Delete codebuild project."
+    long_desc Help.text(:delete)
+    option :sure, desc: "Bypass are you sure prompt"
+    def delete(stack_name=nil)
+      Delete.new(options.merge(stack_name: stack_name)).run
+    end
+
+    desc "start", "start codebuild project."
+    long_desc Help.text(:start)
+    option :source_version, default: "master", desc: "git branch"
+    def start(identifier=nil)
+      Start.new(options.merge(identifier: identifier)).run
     end
 
     desc "completion *PARAMS", "Prints words for auto-completion."
-    long_desc Help.text("completion")
+    long_desc Help.text(:completion)
     def completion(*params)
       Completer.new(CLI, *params).run
     end
 
     desc "completion_script", "Generates a script that can be eval to setup auto-completion."
-    long_desc Help.text("completion_script")
+    long_desc Help.text(:completion_script)
     def completion_script
       Completer::Script.generate
     end

data/lib/codebuild/create.rb

@@ -0,0 +1,12 @@
+module Codebuild
+  class Create < Stack
+    def perform
+      cfn.create_stack(
+        stack_name: @stack_name,
+        template_body: YAML.dump(@template),
+        capabilities: ["CAPABILITY_IAM"]
+      )
+      puts "Creating stack #{@stack_name}. Check CloudFormation console for status."
+    end
+  end
+end

data/lib/codebuild/delete.rb

@@ -0,0 +1,26 @@
+module Codebuild
+  class Delete
+    include AwsServices
+
+    def initialize(options)
+      @options = options
+      @stack_name = options[:stack_name] || inferred_stack_name
+    end
+
+    def run
+      message = "Deleted #{@stack_name} stack."
+      if @options[:noop]
+        puts "NOOP #{message}"
+      else
+        are_you_sure?(@stack_name, :delete)
+
+        if stack_exists?(@stack_name)
+          cfn.delete_stack(stack_name: @stack_name)
+          puts message
+        else
+          puts "#{@stack_name.inspect} stack does not exist".color(:red)
+        end
+      end
+    end
+  end
+end

data/lib/codebuild/deploy.rb

@@ -0,0 +1,11 @@
+module Codebuild
+  class Deploy < Stack
+    def run
+      if stack_exists?(@stack_name)
+        Update.new(@options).run
+      else
+        Create.new(@options).run
+      end
+    end
+  end
+end

data/lib/codebuild/dsl/project/ssm.rb

@@ -0,0 +1,22 @@
+require "aws-sdk-ssm"
+
+module Codebuild::Dsl::Project
+  module Ssm
+    # This method grabs the ssm parameter store value at "compile" time vs
+    # CloudFormation run time. In case we need it as part of the DSL compile phase.
+    def ssm(name)
+      resp = ssm_client.get_parameter(name: name)
+      if resp.parameter.type == "SecureString"
+        resp = ssm_client.get_parameter(name: name, with_decryption: true)
+      end
+
+      resp.parameter.value
+    rescue Aws::SSM::Errors::ParameterNotFound
+      puts "WARN: #{name} found on AWS SSM.".color(:yellow)
+    end
+
+    def ssm_client
+      @ssm_client ||= Aws::SSM::Client.new
+    end
+  end
+end

data/lib/codebuild/dsl/project.rb

@@ -0,0 +1,116 @@
+module Codebuild::Dsl
+  module Project
+    autoload :Ssm, "codebuild/dsl/project/ssm"
+    include Ssm
+
+    PROPERTIES = %w[
+      artifacts
+      badge_enabled
+      cache
+      description
+      encryption_key
+      environment
+      logs_config
+      name
+      queued_timeout_in_minutes
+      secondary_artifacts
+      secondary_sources
+      service_role
+      source
+      tags
+      timeout_in_minutes
+      triggers
+      vpc_config
+    ]
+    PROPERTIES.each do |prop|
+      define_method(prop) do |v|
+        @properties[prop.to_sym] = v
+      end
+    end
+
+    # Convenience wrapper methods
+    def github_url(url)
+      @properties[:source][:location] = url
+    end
+
+    # So it looks like the auth resource property doesnt really get used.
+    # Instead an account level credential is worked.  Refer to:
+    # https://github.com/tongueroo/codebuild/blob/master/readme/github_oauth.md
+    #
+    # Keeping this method around in case the CloudFormation method works one day,
+    # or end up figuring out to use it properly.
+    def github_token(token)
+      @properties[:source][:auth][:resource] = token
+    end
+
+    def github_source(options={})
+      source = {
+        type: "GITHUB",
+        location: options[:location],
+        git_clone_depth: 1,
+        git_submodules_config: { fetch_submodules: true },
+        build_spec: options[:buildspec] || ".codebuild/buildspec.yml",
+        report_build_status: true,
+      }
+
+      if options[:oauth_token]
+        source[:auth] = {
+          type: "OAUTH",
+          resource: options[:oauth_token],
+        }
+      end
+
+      @properties[:source] = source
+    end
+
+    def linux_image(name)
+      linux_environment(image: name)
+    end
+
+    def linux_environment(options={})
+      image = options[:image] || "aws/codebuild/ruby:2.5.3-1.7.0"
+      env = {
+        compute_type: options[:compute_type] || "BUILD_GENERAL1_SMALL",
+        image_pull_credentials_type: "CODEBUILD", # https://docs.aws.amazon.com/AWSCloudFormation/latest/UserGuide/aws-properties-codebuild-project-environment.html#cfn-codebuild-project-environment-imagepullcredentialstype
+        privileged_mode: true,
+        image: image,
+        type: "LINUX_CONTAINER"
+      }
+      # @mapped_env_vars is in memory
+      env[:environment_variables] = @mapped_env_vars if @mapped_env_vars
+      # options has highest precedence
+      env[:environment_variables] = options[:environment_variables] if options[:environment_variables]
+      @properties[:environment] = env
+    end
+
+    def environment_variables(vars)
+      @env_vars = vars
+      @mapped_env_vars = @env_vars.map { |k,v|
+        k = k.to_s
+        if v =~ /^ssm:/
+          { type: "PARAMETER_STORE", name: k, value: v.sub('ssm:','') }
+        else
+          { type: "PLAINTEXT", name: k, value: v }
+        end
+      }
+      @properties[:environment] ||= {}
+      @properties[:environment][:environment_variables] = @mapped_env_vars
+    end
+
+    def local_cache(enable=true)
+      cache = if enable
+        {
+          type: "LOCAL",
+          modes: [
+              "LOCAL_DOCKER_LAYER_CACHE",
+              "LOCAL_SOURCE_CACHE",
+              "LOCAL_CUSTOM_CACHE"
+          ]
+        }
+      else
+        {type: "NO_CACHE"}
+      end
+      @properties[:cache] = cache
+    end
+  end
+end

data/lib/codebuild/dsl/role.rb

@@ -0,0 +1,40 @@
+module Codebuild::Dsl
+  module Role
+    PROPERTIES = %w[
+      assume_role_policy_document
+      managed_policy_arns
+      max_session_duration
+      path
+      permissions_boundary
+      policies
+      role_name
+    ]
+    PROPERTIES.each do |prop|
+      define_method(prop) do |v|
+        @properties[prop.to_sym] = v
+      end
+    end
+
+    # convenience wrapper methods
+    def iam_policy(*definitions)
+      @iam_statements = definitions.map { |definition| standardize(definition) }
+    end
+
+    # Returns standarized IAM statement
+    def standardize(definition)
+      case definition
+      when String
+        # Expands simple string from: logs => logs:*
+        definition = "#{definition}:*" unless definition.include?(':')
+        {
+          action: [definition],
+          effect: "Allow",
+          resource: "*",
+        }
+      when Hash
+        definition
+      end
+    end
+
+  end
+end

data/lib/codebuild/dsl.rb

@@ -0,0 +1,8 @@
+require "yaml"
+
+module Codebuild
+  module Dsl
+    autoload :Project, "codebuild/dsl/project"
+    autoload :Role, "codebuild/dsl/role"
+  end
+end

data/lib/codebuild/evaluate.rb

@@ -0,0 +1,52 @@
+module Codebuild
+  module Evaluate
+    def evaluate(path)
+      source_code = IO.read(path)
+      begin
+        instance_eval(source_code, path)
+      rescue Exception => e
+        if e.class == SystemExit # allow exit to happen normally
+          raise
+        else
+          task_definition_error(e)
+          puts "\nFull error:"
+          raise
+        end
+      end
+    end
+
+  private
+    # Prints out a user friendly task_definition error message
+    def task_definition_error(e)
+      error_info = e.backtrace.first
+      path, line_no, _ = error_info.split(':')
+      line_no = line_no.to_i
+      puts "Error evaluating #{path}:".color(:red)
+      puts e.message
+      puts "Here's the line in #{path} with the error:\n\n"
+
+      contents = IO.read(path)
+      content_lines = contents.split("\n")
+      context = 5 # lines of context
+      top, bottom = [line_no-context-1, 0].max, line_no+context-1
+      spacing = content_lines.size.to_s.size
+      content_lines[top..bottom].each_with_index do |line_content, index|
+        line_number = top+index+1
+        if line_number == line_no
+          printf("%#{spacing}d %s\n".color(:red), line_number, line_content)
+        else
+          printf("%#{spacing}d %s\n", line_number, line_content)
+        end
+      end
+    end
+
+    def lookup_codebuild_file(name)
+      folder_path = [".codebuild", @options[:lookup], name].compact.join("/")
+      if File.exist?(folder_path)
+        folder_path
+      else
+        ".codebuild/#{name}" # default
+      end
+    end
+  end
+end

data/lib/codebuild/help/deploy.md

@@ -0,0 +1,8 @@
+## Examples
+
+    codebuild deploy # infers the CloudFormation name from the parent folder
+    codebuild deploy stack-name # explicitly specify stack name
+
+It is useful to just see the generated CloudFormation template with `--noop` mode:
+
+    codebuild deploy --noop # see generated CloudFormation template