buildkite-builder 2.0.0.beta4 → 2.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 41faf27b8584ef0a615dfafb727dcac55254a4a2d95107518f550130bc470e0a
-  data.tar.gz: 2e778aca984b052a3b4be40c015a969b9fc7d329ad0ca8a28ef9eb605d42457b
+  metadata.gz: 2c6eaf580c5dea018a834cf11f76e5d1d1aa13792fda9b9d513993e53f56f0a7
+  data.tar.gz: 5958862296476f8421ccb62fb0f23e7fef86302fdf354b00c8db9885f71691f0
 SHA512:
-  metadata.gz: 638f4fe0bf8638a2525b6eea0cc778bb078cdc44d81b23b73dd8c29364404cb0a70bc08441b04ab5eb10f670de33cc1d55ff8e2136684679f4499326b9d77c27
-  data.tar.gz: 4408c929ec57aaef0159a1966d7d6dd10d33bc05ce43b382f94a29c12a4eb719418360e4a18717c4324377a424e73b9840186a6b1cc894c368d5ba21e389e59f
+  metadata.gz: 42719a13b6515e91edd391ea18e3f3e18b8db49ec4e4b36d81d0388a701a7567c63992b72860c20f1041b378c6b7a5475c2f5c33a2e454886c3f8a8a678d0b75
+  data.tar.gz: f19d241a0507f434f778560bb4bf5b10dacbe1f8e743718257ac235b433cffd69f472b4d8eb30f9b3ceb28401cb241bda022743a29ffa8f72449964b3a091fa2

data/CHANGELOG.md

@@ -1,3 +1,22 @@
+## 2.3.0
+* Improve BKB step idempotency.
+
+## 2.2.0
+* Add `.buildkite/lib` directory to $LOAD_PATH if it exists.
+
+## 2.1.0
+* Fix a bug introduced in 2.0.0 where artifacts were being uploaded before extensions had a chance to do work.
+* Remove `SortedSet` dependency.
+* Add `annotate` pipeline command helper.
+* Add `StepCollection#find` and `StepCollection#find!` for ease of finding a step by its key in extensions.
+* `group` now supports the `emoji:` helper. (Eg. `group "foobar", emoji: :smile`)
+
+## 2.0.0
+* Add support for `group`.
+* `Processor`s has been renamed to `Extension`. Extensions add more capabilities (will document separately).
+* `plugin` no longer takes 2 arguments (source, version). It's simply 1 arg that is both source and version, separated by a `#`. This is more akin to Buildkite's usage.
+* Full refactor of pipeline code allowing for extensions to extend DSL methods.
+
 ## 1.5.0
 * Merge `BuildKite::Builder::Context` and `BuildKite::Pipelines::Pipeline` to `BuildKite::Builder::Pipeline` (#37)
 

data/README.md

@@ -1,11 +1,12 @@
 # Buildkite Builder [![Build status](https://badge.buildkite.com/a26bf804e9a93fb118d29824d5695a601a248ceec51591be23.svg?branch=main)](https://buildkite.com/gusto-open-source/buildkite-builder/builds?branch=main)
 
 ## Introduction
+
 Buildkite Builder (BKB) is a Buildkite pipeline builder written in Ruby. It allows you to build your pipeline with a Ruby DSL for dynamically generated pipeline steps.
 
 ## Gem Installation (optional)
 
-There are 2 components to this toolkit. The `buildkite-builder` Rubygem and the `buildkite-builder` Docker image. You technically only need the image to use Buildkite Builder, but installing the gem in your repo helps you preview your pipeline during development.
+There are 2 components to this toolkit. The `buildkite-builder` RubyGem and the `buildkite-builder` Docker image. You technically only need the image to use Buildkite Builder, but installing the gem in your repo helps you preview your pipeline during development.
 
 To install the gem, add this line to your application's Gemfile:
 
@@ -16,7 +17,7 @@ gem 'buildkite-builder'
 The gem provides a command line tool that lets you perform various operations on your pipelines:
 
 ```shell
-  $ buildkite-builder help
+  buildkite-builder help
 ```
 
 ## Pipeline Installation
@@ -28,28 +29,31 @@ steps:
   - label: ":toolbox:"
     plugins:
       - docker#v3.7.0:
-          image: gusto/buildkite-builder:1.2.0
+          image: gusto/buildkite-builder:2.1.0
           propagate-environment: true
 ```
 
 Some things to note:
-  - The `label` can be whatever you like.
-  - You'll want to update the `docker` plugin version from time to time.
-  - You can update the `buildkite-builder` version by bumping the Docker image tag.
+
+- The `label` can be whatever you like.
+- You'll want to update the `docker` plugin version from time to time.
+- You can update the `buildkite-builder` version by bumping the Docker image tag.
 
 ## Usage
 
 💡 We have a [Showcase pipeline](https://buildkite.com/gusto-open-source/showcase/builds/latest?branch=main) (defined in [`.buildkite/pipelines/showcase/pipeline.rb`](https://github.com/Gusto/buildkite-builder/blob/main/.buildkite/pipelines/showcase/pipeline.rb)) that, well, showcases some of the features and possibilities with Buildkite Builder. Sometimes the best way to learning something is seeing how it's used.
 
 At its core, BKB is really just a YAML builder. This tool allows you to scale your needs when it comes to building a Buildkite pipeline. Your pipeline can be as straight forward as you'd like, or as complex as you need. Since you have Ruby at your disposal, you can do some cool things like:
-  - Perform pre-build code/diff analysis to determine whether or not to to add a step to the pipeline.
-  - Reorder pipeline steps dynamically.
-  - Augment your pipeline steps with BKB processors.
+
+- Perform pre-build code/diff analysis to determine whether or not to to add a step to the pipeline.
+- Reorder pipeline steps dynamically.
+- Augment your pipeline steps with BKB processors.
   
 ### Pipeline Files
+
 Your repo can contain as many pipeline definitions as you'd like. By convention, pipeline file structure are as such:
 
-```
+```console
 .buildkite/
   pipelines/
     <your-pipeline1-slug>/
@@ -95,7 +99,7 @@ If the step type or attribute exists in Buildkite docs, then it should exist in
 
 If your pipeline has a lot of steps, you should consider using Step Templates. Templates allow you to break out your build steps into reusable template files.
 
-```
+```console
 .buildkite/
   pipelines/
     foobar-widget/
@@ -108,6 +112,7 @@ If your pipeline has a lot of steps, you should consider using Step Templates. T
 A template is basically a step that was extracted from the pipeline:
 
 `.buildkite/pipelines/foobar-widget/templates/rspec.rb`
+
 ```ruby
 Buildkite::Builder.template do
   label "Rspec", emoji: :rspec
@@ -118,6 +123,7 @@ end
 You can then include the template into the the pipeline once or as many time as you need. The template name will be the name of the file (without the extension).
 
 `.buildkite/pipelines/foobar-widget/pipeline.rb`
+
 ```ruby
 Buildkite::Builder.pipeline do
   command(:rspec)
@@ -137,7 +143,7 @@ To install this gem onto your local machine, run `bundle exec rake install`.
 
 ## Contributing
 
-Bug reports and pull requests are welcome on GitHub at https://github.com/gusto/buildkite-builder. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [code of conduct](https://github.com/gusto/buildkite-builder/blob/main/CODE_OF_CONDUCT.md).
+Bug reports and pull requests are welcome on GitHub at <https://github.com/gusto/buildkite-builder>. This project is intended to be a safe, welcoming space for collaboration, and contributors are expected to adhere to the [code of conduct](https://github.com/gusto/buildkite-builder/blob/main/CODE_OF_CONDUCT.md).
 
 ## License
 
@@ -145,4 +151,4 @@ The gem is available as open source under the terms of the [MIT License](https:/
 
 ## Code of Conduct
 
-Everyone interacting in the Buildkite::Builder project's codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/gusto/buildkite-builder/blob/main/CODE_OF_CONDUCT.md).
+Everyone interacting in the `Buildkite::Builder` project's codebases, issue trackers, chat rooms and mailing lists is expected to follow the [code of conduct](https://github.com/gusto/buildkite-builder/blob/main/CODE_OF_CONDUCT.md).

data/VERSION

@@ -1 +1 @@
-2.0.0.beta4
+2.3.0

data/lib/buildkite/builder/commands/run.rb

@@ -15,8 +15,13 @@ module Buildkite
 
           # This entrypoint is for running on CI. It expects certain environment
           # variables to be set. It also uploads the pipeline to Buildkite.
-          log.info "#{'+++ ' if Buildkite.env}🧰 " + 'Buildkite Builder'.color(:springgreen) + " ─ #{relative_pipeline_path.to_s.yellow}"
-          Pipeline.new(pipeline_path, logger: log).upload
+          log.info "+++ 🧰 #{'Buildkite Builder'.color(:springgreen)} ─ #{relative_pipeline_path.to_s.yellow}"
+
+          if Buildkite::Pipelines::Command.meta_data(:exists, Builder::META_DATA.fetch(:job))
+            log.info 'Pipeline already uploaded'.color(:dimgray)
+          else
+            Pipeline.new(pipeline_path, logger: log).upload
+          end
         end
 
         private

data/lib/buildkite/builder/extensions/lib.rb

@@ -0,0 +1,15 @@
+module Buildkite
+  module Builder
+    module Extensions
+      class Lib < Extension
+        def prepare
+          lib_dir = Buildkite::Builder.root.join(Buildkite::Builder::BUILDKITE_DIRECTORY_NAME, 'lib')
+
+          if lib_dir.directory? && !$LOAD_PATH.include?(lib_dir)
+            $LOAD_PATH.unshift(lib_dir)
+          end
+        end
+      end
+    end
+  end
+end

data/lib/buildkite/builder/extensions/steps.rb

@@ -10,9 +10,14 @@ module Buildkite
         end
 
         dsl do
-          def group(label = nil, &block)
+          def group(label = nil, emoji: nil, &block)
             raise "Group does not allow nested in another Group" if context.is_a?(Group)
 
+            if emoji
+              emoji = Array(emoji).map { |name| ":#{name}:" }.join
+              label = [emoji, label].compact.join(' ')
+            end
+
             context.data.steps.push(Buildkite::Builder::Group.new(label, context.data.steps, &block))
           end
 

data/lib/buildkite/builder/extensions.rb

@@ -4,6 +4,7 @@ module Buildkite
   module Builder
     module Extensions
       autoload :Env, File.expand_path('extensions/env', __dir__)
+      autoload :Lib, File.expand_path('extensions/lib', __dir__)
       autoload :Notify, File.expand_path('extensions/notify', __dir__)
       autoload :Steps, File.expand_path('extensions/steps', __dir__)
       autoload :Use, File.expand_path('extensions/use', __dir__)

data/lib/buildkite/builder/file_resolver.rb

@@ -20,7 +20,7 @@ module Buildkite
       end
 
       def initialize
-        @modified_files = SortedSet.new(pull_request? ? files_from_pull_request : files_from_git)
+        @modified_files = Set.new(pull_request? ? files_from_pull_request.sort! : files_from_git.sort!)
       end
 
       private

data/lib/buildkite/builder/manifest/rule.rb

@@ -36,7 +36,7 @@ module Buildkite
           matched.map! { |file| Pathname.new(file) }
           matched.reject!(&:directory?)
           matched.map! { |file| file.relative_path_from(Builder.root) }
-          SortedSet.new(matched)
+          Set.new(matched.sort!)
         end
       end
 

data/lib/buildkite/builder/manifest.rb

@@ -2,7 +2,6 @@
 
 require 'digest/md5'
 require 'pathname'
-require 'sorted_set'
 
 module Buildkite
   module Builder
@@ -56,7 +55,7 @@ module Buildkite
       end
 
       def files
-        @files ||= inclusion_rules.map(&:files).reduce(SortedSet.new, :merge) - exclusion_rules.map(&:files).reduce(SortedSet.new, :merge)
+        @files ||= (inclusion_rules.map(&:files).reduce(Set.new, :merge) - exclusion_rules.map(&:files).reduce(Set.new, :merge)).sort.to_set
       end
 
       def digest

data/lib/buildkite/builder/pipeline.rb

@@ -33,6 +33,7 @@ module Buildkite
         @data = Data.new
 
         use(Extensions::Use)
+        use(Extensions::Lib)
         use(Extensions::Env)
         use(Extensions::Notify)
         use(Extensions::Steps)
@@ -40,18 +41,22 @@ module Buildkite
       end
 
       def upload
-        logger.info '+++ :paperclip: Uploading artifacts'
+        # Generate the pipeline YAML first.
+        contents = to_yaml
+
         upload_artifacts
 
         # Upload the pipeline.
         Tempfile.create(['pipeline', '.yml']) do |file|
           file.sync = true
-          file.write(to_yaml)
+          file.write(contents)
 
           logger.info '+++ :paperclip: Uploading pipeline.yml as artifact'
           Buildkite::Pipelines::Command.artifact!(:upload, file.path)
           logger.info '+++ :pipeline: Uploading pipeline'
           Buildkite::Pipelines::Command.pipeline!(:upload, file.path)
+          logger.info "+++ :toolbox: Setting job meta-data to #{Buildkite.env.job_id.color(:yellow)}"
+          Buildkite::Pipelines::Command.meta_data!(:set, Builder::META_DATA.fetch(:job), Buildkite.env.job_id)
         end
       end
 
@@ -84,6 +89,8 @@ module Buildkite
       def upload_artifacts
         return if artifacts.empty?
 
+        logger.info "+++ :paperclip: Uploading #{artifacts.size.to_s.color(:yellow)} artifact#{'s' if artifacts.size != 1}"
+
         artifacts.each do |path|
           if File.exist?(path)
             Buildkite::Pipelines::Command.artifact!(:upload, path)

data/lib/buildkite/builder/step_collection.rb

@@ -27,6 +27,14 @@ module Buildkite
         end
       end
 
+      def find(key)
+        @steps.find { |step| step.has?(:key) && step.key == key.to_s }
+      end
+
+      def find!(key)
+        find(key) || raise(ArgumentError, "Can't find step with key: #{key}")
+      end
+
       def add(step_class, template = nil, **args, &block)
         @steps.push(step_class.new(self, template, **args, &block)).last
       end

data/lib/buildkite/builder.rb

@@ -27,6 +27,9 @@ module Buildkite
     autoload :PluginManager, File.expand_path('builder/plugin_manager', __dir__)
 
     BUILDKITE_DIRECTORY_NAME = Pathname.new('.buildkite').freeze
+    META_DATA = {
+      job: 'buildkite-builder:job'
+    }.freeze
 
     class << self
       def root(start_path: Dir.pwd, reset: false)
@@ -46,7 +49,7 @@ module Buildkite
 
       def find_buildkite_directory(start_path)
         path = Pathname.new(start_path)
-        until path.join(BUILDKITE_DIRECTORY_NAME).exist? && path.join(BUILDKITE_DIRECTORY_NAME).directory?
+        until path.join(BUILDKITE_DIRECTORY_NAME).directory?
           raise "Unable to find #{BUILDKITE_DIRECTORY_NAME} from #{start_path}" if path == path.parent
 
           path = path.parent

data/lib/buildkite/pipelines/command.rb

@@ -4,21 +4,35 @@ module Buildkite
   module Pipelines
     class Command
       BIN_PATH = 'buildkite-agent'
+      COMMANDS = %w(
+        pipeline
+        artifact
+        annotate
+        meta_data
+      )
 
-      def self.pipeline!(*args)
-        abort unless pipeline(*args)
-      end
+      class << self
+        def pipeline(subcommand, *args)
+          new(:pipeline, subcommand, *args).run
+        end
 
-      def self.pipeline(subcommand, *args)
-        new(:pipeline, subcommand, *args).run
-      end
+        def artifact(subcommand, *args)
+          new(:artifact, subcommand, *args).run
+        end
 
-      def self.artifact!(*args)
-        abort unless artifact(*args)
+        def annotate(body, *args)
+          new(:annotate, body, *args).run
+        end
+
+        def meta_data(subcommand, *args)
+          new(:'meta-data', subcommand, *args).run
+        end
       end
 
-      def self.artifact(subcommand, *args)
-        new(:artifact, subcommand, *args).run
+      COMMANDS.each do |command|
+        define_singleton_method("#{command}!") do |*args|
+          abort unless public_send(command, *args)
+        end
       end
 
       def initialize(command, subcommand, *args)

metadata

@@ -1,30 +1,16 @@
 --- !ruby/object:Gem::Specification
 name: buildkite-builder
 version: !ruby/object:Gem::Version
-  version: 2.0.0.beta4
+  version: 2.3.0
 platform: ruby
 authors:
 - Ngan Pham
 - Andrew Lee
-autorequire:
+autorequire: 
 bindir: exe
 cert_chain: []
-date: 2021-08-25 00:00:00.000000000 Z
+date: 2021-11-12 00:00:00.000000000 Z
 dependencies:
-- !ruby/object:Gem::Dependency
-  name: sorted_set
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
-  type: :runtime
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: '0'
 - !ruby/object:Gem::Dependency
   name: rainbow
   requirement: !ruby/object:Gem::Requirement
@@ -140,6 +126,7 @@ files:
 - lib/buildkite/builder/extension_manager.rb
 - lib/buildkite/builder/extensions.rb
 - lib/buildkite/builder/extensions/env.rb
+- lib/buildkite/builder/extensions/lib.rb
 - lib/buildkite/builder/extensions/notify.rb
 - lib/buildkite/builder/extensions/steps.rb
 - lib/buildkite/builder/extensions/use.rb
@@ -194,7 +181,7 @@ metadata:
   source_code_uri: https://github.com/Gusto/buildkite-builder
   changelog_uri: https://github.com/Gusto/buildkite-builder/blob/master/CHANGELOG.md
   bug_tracker_uri: https://github.com/Gusto/buildkite-builder/issues
-post_install_message:
+post_install_message: 
 rdoc_options: []
 require_paths:
 - lib
@@ -205,12 +192,12 @@ required_ruby_version: !ruby/object:Gem::Requirement
       version: 2.3.0
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
-  - - ">"
+  - - ">="
     - !ruby/object:Gem::Version
-      version: 1.3.1
+      version: '0'
 requirements: []
-rubygems_version: 3.2.2
-signing_key:
+rubygems_version: 3.1.6
+signing_key: 
 specification_version: 4
 summary: A gem for programmatically creating Buildkite pipelines.
 test_files: []