buildkite-builder 1.0.0.beta.4 → 1.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: f1c9b9e735b6130fa12f1424bc3a6fecdb280edd9b600e003acb406c3462c114
-  data.tar.gz: 82ab940a585b5021d0337dba8fda4a9a8c2d5b69214aae65271799c53918f406
+  metadata.gz: 6684f24ab3c156884bcd3055fbddbb7831afa03be9e0b0b2056fbedef023084b
+  data.tar.gz: 64753e7a15224a0b3c161335cfbc9046ecbaa5e015692ace78001eeef550eea3
 SHA512:
-  metadata.gz: 39db63fa323f822549a64045d6100fb646956ffc077c16752af9b24f9e8467cf5d73538a074d3d91e94b526de8a6ad37da08125475b4fad9d80d6c61823711ed
-  data.tar.gz: 700414499cec1e93162eddbe075015316e253ab9e2d73d328d8cc7a1a09bf5685f18d6dcdc5ef516651a9596302f8483a3dc73fa78419b601cb51422f30c01f8
+  metadata.gz: 21b7f849f79acd215f260644f75767143bb10f2eeab88e0edee7c3728024f128b12f421ddd632ed4163fc0501f44051bc065594906d28643642839be934ac152
+  data.tar.gz: 5bfe120d20cb914525c61dba9ddec9b298058e6ccb01a5b6a4a55cccc95cff1fa993753cf65ba1041615d9adcb735ffb4545c7a58987a004cc3638b11d4eeba3

data/CHANGELOG.md

@@ -0,0 +1,4 @@
+## 1.3.0
+* Add ability for step to store data in step context
+* Move `upload` from `BuildKite::Builder::Commands::Run` to `BuildKite::Builder::Context`
+* Add ability to set custom artifacts in context and uplaod before `pipeile upload`

data/README.md

@@ -1,39 +1,143 @@
-# Buildkite::Builder
+# Buildkite Builder [![Build status](https://badge.buildkite.com/a26bf804e9a93fb118d29824d5695a601a248ceec51591be23.svg?branch=main)](https://buildkite.com/gusto-open-source/buildkite-builder/builds?branch=main)
 
-Welcome to your new gem! In this directory, you'll find the files you need to be able to package up your Ruby library into a gem. Put your Ruby code in the file `lib/buildkite/builder`. To experiment with that code, run `bin/console` for an interactive prompt.
+## Introduction
+Buildkite Builder (BKB) is a Buildkite pipeline builder written in Ruby. It allows you to build your pipeline with a Ruby DSL for dynamically generated pipeline steps.
 
-TODO: Delete this and the text above, and describe your gem
+## Gem Installation (optional)
 
-## Installation
+There are 2 components to this toolkit. The `buildkite-builder` Rubygem and the `buildkite-builder` Docker image. You technically only need the image to use Buildkite Builder, but installing the gem in your repo helps you preview your pipeline during development.
 
-Add this line to your application's Gemfile:
+To install the gem, add this line to your application's Gemfile:
 
 ```ruby
 gem 'buildkite-builder'
 ```
 
-And then execute:
+The gem provides a command line tool that lets you perform various operations on your pipelines:
 
-    $ bundle install
+```shell
+  $ buildkite-builder help
+```
+
+## Pipeline Installation
 
-Or install it yourself as:
+As with every Buildkite pipeline, you'll need to define the initial pipeline step. You can do this directly in the Pipeline Settings or with a `.buildkite/pipeline.yml` file in your repository. You'll need to define a single step to kick off Buildkite Builder:
+
+```yaml
+steps:
+  - label: ":toolbox:"
+    plugins:
+      - docker#v3.7.0:
+          image: gusto/buildkite-builder:1.2.0
+          propagate-environment: true
+```
 
-    $ gem install buildkite-builder
+Some things to note:
+  - The `label` can be whatever you like.
+  - You'll want to update the `docker` plugin version from time to time.
+  - You can update the `buildkite-builder` version by bumping the Docker image tag.
 
 ## Usage
 
-TODO: Write usage instructions here
+💡 We have a [Showcase pipeline](https://buildkite.com/gusto-open-source/showcase/builds/latest?branch=main) (defined in [`.buildkite/pipelines/showcase/pipeline.rb`](https://github.com/Gusto/buildkite-builder/blob/main/.buildkite/pipelines/showcase/pipeline.rb)) that, well, showcases some of the features and possibilities with Buildkite Builder. Sometimes the best way to learning something is seeing how it's used.
+
+At its core, BKB is really just a YAML builder. This tool allows you to scale your needs when it comes to building a Buildkite pipeline. Your pipeline can be as straight forward as you'd like, or as complex as you need. Since you have Ruby at your disposal, you can do some cool things like:
+  - Perform pre-build code/diff analysis to determine whether or not to to add a step to the pipeline.
+  - Reorder pipeline steps dynamically.
+  - Augment your pipeline steps with BKB processors.
+  
+### Pipeline Files
+Your repo can contain as many pipeline definitions as you'd like. By convention, pipeline file structure are as such:
+
+```
+.buildkite/
+  pipelines/
+    <your-pipeline1-slug>/
+      pipeline.rb
+    <your-pipeline2-slug>/
+      pipeline.rb
+```
+
+For an example, refer to the [dummy pipeline in the fixtures directory](https://github.com/Gusto/buildkite-builder/blob/main/spec/fixtures/basic/.buildkite/pipelines/dummy/pipeline.rb).
+
+### Defining Steps
+
+Buildkite Builder was designed to be as intuitive as possible by making DSL match Buildkite's attributes and [step types](https://buildkite.com/docs/pipelines/defining-steps#step-types). Here's a basic pipeline:
+
+```ruby
+Buildkite::Builder.pipeline do
+  command do
+    label "Rspec", emoji: :rspec
+    command "bundle exec rspec"
+  end
+  
+  wait
+  
+  trigger do
+    trigger "deploy-pipeline"
+  end
+end
+```
+
+Which generates:
+
+```yaml
+steps:
+  - label: ":rspec: RSpec"
+    command: "bundle exec rspec"
+  - wait
+  - trigger: deploy-pipeline
+```
+
+If the step type or attribute exists in Buildkite docs, then it should exist in the DSL. **The only exception is the `if` attribute**. Since `if` is a ruby keyword, we've mapped it to `condition`.
+
+### Step Templates
+
+If your pipeline has a lot of steps, you should consider using Step Templates. Templates allow you to break out your build steps into reusable template files.
+
+```
+.buildkite/
+  pipelines/
+    foobar-widget/
+      pipeline.rb
+      templates/
+        rspec.rb
+        rubocop.rb
+```
+
+A template is basically a step that was extracted from the pipeline:
+
+`.buildkite/pipelines/foobar-widget/templates/rspec.rb`
+```ruby
+Buildkite::Builder.template do
+  label "Rspec", emoji: :rspec
+  commmand "bundle exec rspec"
+end
+```
+
+You can then include the template into the the pipeline once or as many time as you need. The template name will be the name of the file (without the extension).
+
+`.buildkite/pipelines/foobar-widget/pipeline.rb`
+```ruby
+Buildkite::Builder.pipeline do
+  command(:rspec)
+  
+  # Reuse and agument templates on the fly.
+  command(:rspec) do
+    label "Run RSpec again!"
+  end
+end
+```
 
 ## Development
 
 After checking out the repo, run `bin/setup` to install dependencies. Then, run `rake spec` to run the tests. You can also run `bin/console` for an interactive prompt that will allow you to experiment.
 
-To install this gem onto your local machine, run `bundle exec rake install`. To release a new version, update the version number in `version.rb`, and then run `bundle exec rake release`, which will create a git tag for the version, push git commits and tags, and push the `.gem` file to [rubygems.org](https://rubygems.org).
+To install this gem onto your local machine, run `bundle exec rake install`.
 
 ## Contributing
 
-Bug reports and pull requests are welcome on GitHub at https://github.com/[USERNAME]/buildkite-builder. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [code of conduct](https://github.com/[USERNAME]/buildkite-builder/blob/master/CODE_OF_CONDUCT.md).
-
+Bug reports and pull requests are welcome on GitHub at https://github.com/gusto/buildkite-builder. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [code of conduct](https://github.com/gusto/buildkite-builder/blob/main/CODE_OF_CONDUCT.md).
 
 ## License
 
@@ -41,4 +145,4 @@ The gem is available as open source under the terms of the [MIT License](https:/
 
 ## Code of Conduct
 
-Everyone interacting in the Buildkite::Builder project's codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/[USERNAME]/buildkite-builder/blob/master/CODE_OF_CONDUCT.md).
+Everyone interacting in the Buildkite::Builder project's codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/gusto/buildkite-builder/blob/main/CODE_OF_CONDUCT.md).

data/VERSION

@@ -0,0 +1 @@
+1.3.0

data/lib/buildkite/builder.rb

@@ -1,14 +1,11 @@
 # frozen_string_literal: true
 
-require 'bundler'
-require 'logger'
 require 'pathname'
 
 module Buildkite
   module Builder
-    VERSION = '1.0.0.beta.4'
-
     autoload :Commands, File.expand_path('builder/commands', __dir__)
+    autoload :Context, File.expand_path('builder/context', __dir__)
     autoload :Definition, File.expand_path('builder/definition', __dir__)
     autoload :FileResolver, File.expand_path('builder/file_resolver', __dir__)
     autoload :Github, File.expand_path('builder/github', __dir__)
@@ -17,9 +14,8 @@ module Buildkite
     autoload :Manifest, File.expand_path('builder/manifest', __dir__)
     autoload :Processors, File.expand_path('builder/processors', __dir__)
     autoload :Rainbow, File.expand_path('builder/rainbow', __dir__)
-    autoload :Runner, File.expand_path('builder/runner', __dir__)
 
-    BUILDKITE_DIRECTORY_NAME = '.buildkite/'
+    BUILDKITE_DIRECTORY_NAME = Pathname.new('.buildkite').freeze
 
     class << self
       def root(start_path: Dir.pwd, reset: false)
@@ -27,6 +23,16 @@ module Buildkite
         @root ||= find_buildkite_directory(start_path)
       end
 
+      def template(&block)
+        Definition::Template.new(&block) if block_given?
+      end
+
+      def pipeline(&block)
+        Definition::Pipeline.new(&block) if block_given?
+      end
+
+      private
+
       def find_buildkite_directory(start_path)
         path = Pathname.new(start_path)
         until path.join(BUILDKITE_DIRECTORY_NAME).exist? && path.join(BUILDKITE_DIRECTORY_NAME).directory?
@@ -37,18 +43,6 @@ module Buildkite
         path.expand_path
       end
 
-      def expand_path(path)
-        path = Pathname.new(path)
-        path.absolute? ? path : root.join(path)
-      end
-
-      def template(&block)
-        Definition::Template.new(&block) if block_given?
-      end
-
-      def pipeline(&block)
-        Definition::Pipeline.new(&block) if block_given?
-      end
     end
   end
 end

data/lib/buildkite/builder/commands/abstract.rb

@@ -6,6 +6,18 @@ module Buildkite
   module Builder
     module Commands
       class Abstract
+        PIPELINES_DIRECTORY = 'pipelines'
+        POSSIBLE_PIPELINE_PATHS = [
+          File.join('.buildkite', Context::PIPELINE_DEFINITION_FILE),
+          File.join('buildkite', Context::PIPELINE_DEFINITION_FILE),
+          File.join(Context::PIPELINE_DEFINITION_FILE)
+        ].freeze
+        POSSIBLE_PIPELINES_PATHS = [
+          File.join('.buildkite', PIPELINES_DIRECTORY),
+          File.join('buildkite', PIPELINES_DIRECTORY),
+          File.join(PIPELINES_DIRECTORY)
+        ].freeze
+
         class << self
           attr_accessor :description
 
@@ -35,6 +47,8 @@ module Buildkite
           if options[:help]
             puts options[:help]
             return
+          elsif !pipeline_path
+            abort "Unable to find pipeline"
           end
 
           run
@@ -42,6 +56,10 @@ module Buildkite
 
         private
 
+        def pipeline_slug
+          ARGV.last || Buildkite.env&.pipeline_slug
+        end
+
         def command_name
           Commands::COMMANDS.key(self.class.name.split('::').last.to_sym)
         end
@@ -51,12 +69,39 @@ module Buildkite
           # Subclasses should override to parse options.
         end
 
-        def available_pipelines
-          @available_pipelines ||= pipelines_path.children.select(&:directory?).map { |dir| dir.basename.to_s }
+        def log
+          @log ||= begin
+            Logger.new($stdout).tap do |logger|
+              logger.formatter = proc do |_severity, _datetime, _progname, msg|
+                "#{msg}\n"
+              end
+            end
+          end
         end
 
-        def pipelines_path
-          Buildkite::Builder.root.join(Runner::PIPELINES_PATH)
+        def pipeline_path
+          @pipeline_path ||=
+            find_root_by_main_pipeline ||
+            find_root_by_multi_pipeline
+        end
+
+        def find_root_by_main_pipeline
+          POSSIBLE_PIPELINE_PATHS.map { |path| Builder.root.join(path) }.find(&:exist?)&.dirname
+        end
+
+        def find_root_by_multi_pipeline
+          pipelines_path = POSSIBLE_PIPELINES_PATHS.map { |path| Builder.root.join(path) }.find(&:directory?)
+          
+          if pipelines_path
+            if pipeline_slug
+              path = pipelines_path.join(pipeline_slug)
+              path if path.directory?
+            elsif pipelines_path.children.one?
+              pipelines_path.children.first
+            else
+              raise 'Your project has multiple pipelines, please specify one.'
+            end
+          end
         end
       end
     end

data/lib/buildkite/builder/commands/preview.rb

@@ -9,19 +9,7 @@ module Buildkite
         self.description = 'Outputs the pipeline YAML.'
 
         def run
-          unless pipeline
-            raise 'You must specify a pipeline'
-          end
-
-          puts Runner.new(pipeline: pipeline).run.to_yaml
-        end
-
-        def pipeline
-          @pipeline ||= ARGV.last || begin
-            if available_pipelines.one?
-              available_pipelines.first
-            end
-          end
+          puts Context.build(pipeline_path).pipeline.to_yaml
         end
       end
     end

data/lib/buildkite/builder/commands/run.rb

@@ -5,11 +5,31 @@ module Buildkite
     module Commands
       class Run < Abstract
         private
+        include LoggingUtils
+        using Rainbow
 
         self.description = 'Builds and uploads the generated pipeline.'
 
         def run
-          Builder::Runner.run
+          relative_pipeline_path = pipeline_path.relative_path_from(Builder.root)
+
+          # This entrypoint is for running on CI. It expects certain environment
+          # variables to be set. It also uploads the pipeline to Buildkite.
+          log.info "#{'+++ ' if Buildkite.env}🧰 " + 'Buildkite Builder'.color(:springgreen) + " ─ #{relative_pipeline_path.to_s.yellow}"
+          Context.new(pipeline_path, logger: log).upload
+        end
+
+        private
+
+        def pipeline_path
+          pipeline_path_override || super
+        end
+
+        def pipeline_path_override
+          if ENV['BUILDKITE_BUILDER_PIPELINE_PATH']
+            path = Pathname.new(ENV['BUILDKITE_BUILDER_PIPELINE_PATH'])
+            path.absolute? ? path : Builder.root.join(path)
+          end
         end
       end
     end

data/lib/buildkite/builder/context.rb

@@ -0,0 +1,108 @@
+require 'logger'
+require 'tempfile'
+
+module Buildkite
+  module Builder
+    class Context
+      include Definition::Helper
+      include LoggingUtils
+      using Rainbow
+
+      PIPELINE_DEFINITION_FILE = Pathname.new('pipeline.rb').freeze
+
+      attr_reader :logger
+      attr_reader :root
+      attr_reader :pipeline
+      attr_reader :artifacts
+
+      def self.build(root, logger: nil)
+        context = new(root, logger: logger)
+        context.build
+        context
+      end
+
+      def initialize(root, logger: nil)
+        @root = root
+        @logger = logger || Logger.new(File::NULL)
+        @artifacts = []
+      end
+
+      def build
+        results = benchmark("\nDone (%s)".color(:springgreen)) do
+          unless @pipeline
+            @pipeline = Pipelines::Pipeline.new
+
+            load_manifests
+            load_templates
+            load_processors
+            load_pipeline
+            run_processors
+          end
+        end
+        logger.info(results)
+
+        @pipeline
+      end
+
+      def upload
+        build unless @pipeline
+
+        logger.info '+++ :paperclip: Uploading artifacts'
+        upload_artifacts
+
+        # Upload the pipeline.
+        Tempfile.create(['pipeline', '.yml']) do |file|
+          file.sync = true
+          file.write(pipeline.to_yaml)
+
+          logger.info '+++ :paperclip: Uploading pipeline.yml as artifact'
+          Buildkite::Pipelines::Command.artifact!(:upload, file.path)
+          logger.info '+++ :pipeline: Uploading pipeline'
+          Buildkite::Pipelines::Command.pipeline!(:upload, file.path)
+        end
+      end
+
+      private
+
+      def load_manifests
+        Loaders::Manifests.load(root).each do |name, asset|
+          Manifest[name] = asset
+        end
+      end
+
+      def load_templates
+        Loaders::Templates.load(root).each do |name, asset|
+          pipeline.template(name, &asset)
+        end
+      end
+
+      def load_processors
+        Loaders::Processors.load(root)
+      end
+
+      def run_processors
+        pipeline.processors.each do |processor|
+          processor.process(self)
+        end
+      end
+
+      def upload_artifacts
+        return if artifacts.empty?
+
+        artifacts.each do |path|
+          if File.exist?(path)
+            Buildkite::Pipelines::Command.artifact!(:upload, path)
+          end
+        end
+      end
+
+      def load_pipeline
+        pipeline.instance_eval(&pipeline_definition)
+      end
+
+      def pipeline_definition
+        @pipeline_definition ||= load_definition(root.join(PIPELINE_DEFINITION_FILE), Definition::Pipeline)
+      end
+    end
+  end
+end