heroku_hatchet 7.4.0 → 8.0.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 4e33ffcf73b42d412f81525fdce9b77baa34da4083a6662cb14ddb098a0535dc
-  data.tar.gz: 3d73f585c00dd74cd4733802af979ba75e2c04e4cb93118128fdaf6efe9bc53b
+  metadata.gz: f4233a137df7685f1e411471db8c53a72c9c6e77e5b9a901d0d8bb1c0147f06a
+  data.tar.gz: ffd93abf69b7a46ccbdda5148710e237b9c41171abef209ea56f187d7a52c501
 SHA512:
-  metadata.gz: f9ef93379dd21e7808f30d9a2ed6179a76aa12679d5a1129a1ed394682c3e046abcc6c8b87ae8d3112edc5ff77965f90d42550be6d09a49ed89b7ad7270fdf14
-  data.tar.gz: 6cd06944d64ca9ae96ed8b25744000de748a5235dc52fab838368b9afc75c28af6dc9c1432106a99f972b4044d94762ce573fcaef2fa91f062a68aad2000b05a
+  metadata.gz: 61cc153424acbdf88c61b41ef9b4206fbb1e59c171ad30fded428d40bfe2edc728cdb2db72ca95526d1d5c5225d38eacacb6794b8c811b40988cac58480b0e74
+  data.tar.gz: 615e586767703298020a5d2a0172b9793a681a7cc681f45e6c6e42082e9db8c4a32f6e647fb006d8e9b7d8618b6c0dace7df6109cbca20e5e7fe0c34f991ed62

data/CHANGELOG.md

@@ -1,5 +1,18 @@
 ## HEAD
 
+## 8.0.1
+
+- Bugfix: Lock and sleep and refresh API when duplicate app deletion detected (https://github.com/heroku/hatchet/pull/198)
+
+## 8.0.0
+
+- Breaking change: Delete apps on teardown. Previously hatchet would delete apps lazily to help with debugging. This behavior allowed developers to inspect logs and `heroku run bash` in the event of an unexpected failure. In practice, it is rarely needed and causes accounts to retain apps indefinitely. Previously there was no cost to retaining applications, but now `basic` applications incur a charge. Change details:
+  - The application teardown process now deletes applications directly.
+  - To skip destroying applications on teardown, set `HEROKU_DEBUG_EXPENSIVE=1`. This env var will cause `App#teardown!` to skip deletion so you can introspect why one failed.
+  - When hatchet needs a new application, it will first delete all applications that are created at least `HATCHET_ALIVE_TTL_MINUTES` ago. If it cannot delete any applications and the account is already at `HATCHET_APP_LIMIT`, it will sleep and try again later.
+  - Introduce `--older-than` flag to `hatchet destroy` CLI command. For example, the command `hatchet destroy --older-than=7`will remove any apps older than the provided value (in minutes).
+- Add support for GitHub Actions env vars (https://github.com/heroku/hatchet/pull/189)
+
 ## 7.4.0
 
 - Supports "basic" scaledown (https://github.com/heroku/hatchet/pull/193)

data/README.md

@@ -266,16 +266,17 @@ Hatchet::Runner.new("minimal_webpacker", buildpacks: buildpacks).deploy do |app,
   expect($?.exitstatus).to eq(0)
   expect($?.success?).to be_truthy
 
-  # In Ruby all objects except `nil` and `false` are "truthy" in this case it could also be tested using `be_true` but
-  # it's best practice to use this test helper in rspec
+  # In Ruby all objects except `nil` and `false` are "truthy"
+  # in this case it could also be tested using `be_true` but
+  # it's best practice to use this `be_truthy` test helper instead
 end
 ```
 
-You can disable this behavior [see how to do it in the reference tests](https://github.com/heroku/hatchet/blob/master/spec/hatchet/app_spec.rb)
+You can disable error on exit status behavior, [see how to do it in the reference tests](https://github.com/heroku/hatchet/blob/master/spec/hatchet/app_spec.rb)
 
 - **Escaping and raw mode:**
 
-By default `app.run()` will escape the input so you can safely call `app.run("cmd && cmd")` and it works as expected. But if you want to do something custom, you can enable raw mode by passing in `raw: true` [see how to do it in the reference tests](https://github.com/heroku/hatchet/blob/master/spec/hatchet/app_spec.rb)
+By default `app.run()` will escape the input so you can safely call `app.run("cmd && cmd")` and it works as expected. But if you want to do something custom, you can enable raw mode by passing in `raw: true` [see how to do it in the reference tests](https://github.com/heroku/hatchet/blob/master/spec/hatchet/app_spec.rb). If you do that, you'll need to ensure your inputs are propperly shell escaped.
 
 - **Heroku options:**
 
@@ -358,21 +359,9 @@ And later:
 Destroying "hatchet-t-fd25e3626b". Hatchet app limit: 80
 ```
 
-By default, Hatchet does not destroy your app at the end of the test run, that way if your test failed unexpectedly if it's not destroyed yet, you can:
+If an app is not deleted for some reason, it will be deleted by a future test run. Applications not deleted by calling `teardown` will be allowed to live for `HATCHET_ALIVE_TTL_MINUTES`. This behavior allows multiple test runs on the same hatchet account without one test run deleting apps that another one still needs.
 
-```
-$ heroku run bash -a hatchet-t-bed73940a6
-```
-
-And use that to debug. Hatchet deletes old apps on demand. You tell it what your limits are and it will stay within those limits:
-
-```
-HATCHET_APP_LIMIT=20
-```
-
-With these env vars, Hatchet will "reap" older hatchet apps when it sees there are 20 or more hatchet apps. For CI, it's recommended that you increase the `HATCHET_APP_LIMIT` to 80-100. Hatchet will mark apps as safe for deletion once they've finished, and the `teardown!` method has been called on them (it tracks this by enabling maintenance mode on apps). Hatchet only tracks its apps. Hatchet uses a regex pattern on the name of apps to see which ones it can manage. If your account has reached the maximum number of global Heroku apps, you'll need to remove some manually.
-
-If an app is not marked as being in maintenance mode for some reason, it can be deleted, but only after it has been allowed to live for some time. This behavior is configured by the `HATCHET_ALIVE_TTL_MINUTES` env var. For example, if you set it for `7`, Hatchet will ensure that any apps that are not marked as being in maintenance mode are allowed to live for at least seven minutes. This should give the app time to finish the test's execution, so it is not deleted mid-deploy. When this deletion happens, you'll see a warning in your output. It could indicate you're not properly cleaning up and calling `teardown!` on some of your apps, or it could mean that you're attempting to execute more tests concurrently than your `HATCHET_APP_LIMIT` allows. This deletion-mid-test behavior might otherwise be triggered if you have multiple CI runs executing at the same time.
+For example, if you set `HATCHET_ALIVE_TTL_MINUTES=7`, Hatchet will ensure that any apps that are older than 7 minutes will be deleted. If all of your tests finish in under 7 minutes then apps won't be deleted mid-deploy. When apps are deleted, you'll see a warning in your output. It could indicate you're not properly cleaning up and calling `teardown!` on some of your apps, or it could mean that you're attempting to execute more tests concurrently than your `HATCHET_APP_LIMIT` allows. Or that something else prevented apps from getting deleted on teardown, such as a Heroku API outage.
 
 It's recommended you don't use your personal Heroku API key for running tests on a CI server since the hatchet apps count against your account maximum limits. Running tests using your account locally is fine for debugging one or two tests.
 
@@ -684,7 +673,9 @@ end
 - `app.platform_api`: Returns an instance of the [platform-api Heroku client](https://github.com/heroku/platform-api). If Hatchet doesn't give you access to a part of Heroku that you need, you can likely do it with the platform-api client.
 - `app.push!`: Push code to your Heroku app. It can be used inside of a `deploy` block to re-deploy.
 - `app.run_ci`: Runs Heroku CI against the app returns a TestRun object in the block
-- `app.teardown!`: This method is called automatically when using `app.deploy` in block mode after the deploy block finishes. When called it will clean up resources, mark the app as being finished (by setting `{"maintenance" => true}` on the app) so that the reaper knows it is safe to delete later. Here is an example of a test that creates and deploys an app manually, then later tears it down manually. If you deploy an application without calling `teardown!` then Hatchet will not know it is safe to delete and may keep it around for much longer than required for the test to finish.
+- `app.teardown!`: This method is called automatically when using `app.deploy` in block mode after the deploy block finishes. When called it will delete the application.
+
+Here is an example of a test that creates and deploys an app manually (without a block), then later tears it down manually. If you deploy an application without calling `teardown!` then hatchet will leak apps.
 
 ```ruby
 before(:each) do
@@ -693,7 +684,7 @@ before(:each) do
 end
 
 after(:each) do
-  @app.teardown! if @app
+  @app&.teardown!
 end
 
 it "uses ruby" do
@@ -718,6 +709,7 @@ HATCHET_ALIVE_TTL_MINUTES=7
 
 # HATCHET_RUN_MULTI=1      # WARNING: Setting this env var will incur charges against your account. To use this env var you must also enable `HATCHET_EXPENSIVE_MODE`
 # HATCHET_EXPENSIVE_MODE=1 # WARNING: Do not set this environment variable unless you're okay with possibly large bills
+# HEROKU_DEBUG_EXPENSIVE=1 # WARNING: This will prevent apps from being cleaned up automatically and you will be billed for any apps left running
 ```
 
 > The syntax to set an env var in Ruby is `ENV["HATCHET_RETRIES"] = "2"` all env vars are strings.
@@ -725,12 +717,13 @@ HATCHET_ALIVE_TTL_MINUTES=7
 - `HATCHET_BUILDPACK_BASE`: This is the URL where Hatchet can find your buildpack. It must be public for Heroku to be able to use your buildpack.
 - `HATCHET_BUILDPACK_BRANCH`: By default, Hatchet will use your current git branch name. If, for some reason, git is not available or you want to manually specify it like `ENV["HATCHET_BUILDPACK_BRANCH'] = ENV[`MY_CI_BRANCH`]` then you can.
 - `HATCHET_RETRIES` If the `ENV['HATCHET_RETRIES']` is set to a number, deploys are expected to work and automatically retry that number of times. Due to testing using a network and random failures, setting this value to `3` retries seems to work well. If an app cannot be deployed within its allotted number of retries, an error will be raised. The downside of a larger number is that your suite will keep running for much longer when there are legitimate failures.
-- `HATCHET_APP_LIMIT`: The maximum number of **hatchet** apps that Hatchet will allow in the given account before running the reaper. For local execution, keep this low as you don't want your account dominated by hatchet apps. For CI, you want it to be much larger, 80-100 since it's not competing with non-hatchet apps. Your test runner account needs to be a dedicated account.
+- `HATCHET_APP_LIMIT`: The maximum number of applications that hatchet is allowed to utilize on your account. If you hit this limit mid-test then Hatchet will need to wait until other tests finish and delete their apps. Keep in mind that you may have several concurrent CI executions on the same account competing for the same limited number of apps.
 - `HEROKU_API_KEY`: The API key of your test account user. If you run locally without this set, it will use your personal credentials.
 - `HEROKU_API_USER`: The email address of your user account. If you run locally without this set, it will use your personal credentials.
 - `HATCHET_DEFAULT_STACK`: The default Heroku stack to be used when an explicit `stack` is not passed to `App.new`. If this is not set, apps will instead use Heroku platform's [default stack](https://devcenter.heroku.com/articles/stack#default-stack).
 - `HATCHET_RUN_MULTI`: If enabled, this will scale up deployed apps to "standard-1x" once deployed instead of running on the free tier. This enables the `run_multi` method capability, however scaling up is not free. WARNING: Setting this env var will incur charges to your Heroku account. We recommended never to enable this setting unless you work for Heroku. To use this you must also set `HATCHET_EXPENSIVE_MODE=1`
 - `HATCHET_EXPENSIVE_MODE`: This is intended to be a "safety" environment variable. If it is not set, Hatchet will prevent you from using the `run_multi: true` setting or the `HATCHET_RUN_MULTI` environment variables. There are still ways to incur charges without this feature, but unless you're absolutely confident your test setup will not leave "orphan" apps that are billing you, do not enable this setting. Even then, only set this value if you work for Heroku. To recap WARNING: setting this is expensive.
+- `HEROKU_DEBUG_EXPENSIVE`: If set, hatchet will not delete applications on teardown. This can be used to introspect build logs or run `heroku run bash` on the application created for the failed test. Note that if another hatchet process runs after `HATCHET_ALIVE_TTL_MINUTES` then it will delete your applications.
 
 ## Basic
 
@@ -1037,7 +1030,7 @@ Ruby is full of multitudes, this isn't even close to being exhaustive, just enou
 Hatchet has a CLI for installing and maintaining external repos you're
 using to test against. If you have Hatchet installed as a gem run
 
-    $ Hatchet --help
+    $ hatchet --help
 
 For more info on commands. If you're using the source code you can run
 the command by going to the source code directory and running:

data/bin/hatchet

@@ -106,20 +106,31 @@ class HatchetCLI < Thor
     end
   end
 
-  desc "destroy [NAME]", "Deletes applications"
-  option :all, :type => :boolean
-  def destroy(name=nil)
+  desc "destroy", "Deletes application(s)"
+  option :all, type: :boolean, desc: "Delete ALL hatchet apps"
+  option :older_than, type: :numeric, desc: "Delete all hatchet apps older than N minutes"
+  def destroy
     api_key      = ENV['HEROKU_API_KEY'] || `heroku auth:token`.chomp
     platform_api = PlatformAPI.connect_oauth(api_key, cache: Moneta.new(:Null))
     api_rate_limit = ApiRateLimit.new(platform_api)
     reaper       = Hatchet::Reaper.new(api_rate_limit: api_rate_limit)
 
-    if options[:all]
+    case
+    when options[:all]
+      puts "Destroying ALL apps"
       reaper.destroy_all
-    elsif !name.nil?
-      reaper.destroy_by_name(name)
+      puts "Done"
+    when options[:older_than]
+      minutes = options[:older_than].to_i
+      puts "Destroying apps older than #{minutes}m"
+      reaper.destroy_older_apps(
+        minutes: minutes,
+        force_refresh: true,
+        on_conflict: :refresh_api_and_continue
+      )
+      puts "Done"
     else
-      raise "Must provide an app name or --all"
+      raise "No flags given run `hatchet help destroy` for options"
     end
   end
 

data/lib/hatchet/app.rb

@@ -11,7 +11,7 @@ module Hatchet
         "https://github.com/heroku/heroku-buildpack-ruby.git"
       }
     }
-    HATCHET_BUILDPACK_BRANCH = -> { ENV['HATCHET_BUILDPACK_BRANCH'] || ENV['HEROKU_TEST_RUN_BRANCH'] || Hatchet.git_branch }
+    HATCHET_BUILDPACK_BRANCH = -> { ENV['HATCHET_BUILDPACK_BRANCH'] || Hatchet.git_branch }
 
     attr_reader :name, :stack, :repo_name, :app_config, :buildpacks, :reaper, :max_retries_count
 
@@ -67,6 +67,7 @@ module Hatchet
       @repo_name     = repo_name
       @directory     = self.config.path_for_name(@repo_name)
       @name          = name
+      @heroku_id     = nil
       @stack         = stack
       @debug         = debug || debugging
       @allow_failure = allow_failure
@@ -82,7 +83,6 @@ module Hatchet
       end
       @run_multi_array = []
       @already_in_dir = nil
-      @app_is_setup = nil
 
       @before_deploy_array = []
       @before_deploy_array << before_deploy if before_deploy
@@ -280,11 +280,31 @@ module Hatchet
     def create_app
       3.times.retry do
         begin
+          # Remove any obviously old apps first
+          # Try to use existing cache of apps to
+          # minimize API calls
+          @reaper.destroy_older_apps(
+            force_refresh: false,
+            on_conflict: :stop_if_under_limit,
+          )
           hash = { name: name, stack: stack }
           hash.delete_if { |k,v| v.nil? }
-          heroku_api_create_app(hash)
+          result = heroku_api_create_app(hash)
+          @heroku_id = result["id"]
         rescue => e
-          @reaper.cycle(app_exception_message: e.message)
+          # If we can't create an app assume
+          # it might be due to resource constraints
+          #
+          # Try to delete existing apps
+          @reaper.destroy_older_apps(
+            force_refresh: true,
+            on_conflict: :stop_if_under_limit,
+          )
+          # If we're still not under the limit, sleep a bit
+          # retry later.
+          @reaper.sleep_if_over_limit(
+            reason: "Could not create app #{e.message}"
+          )
           raise e
         end
       end
@@ -301,7 +321,7 @@ module Hatchet
 
     # creates a new heroku app via the API
     def setup!
-      return self if @app_is_setup
+      return self if @heroku_id
       puts "Hatchet setup: #{name.inspect} for #{repo_name.inspect}"
       create_app
       set_labs!
@@ -309,7 +329,6 @@ module Hatchet
       api_rate_limit.call.buildpack_installation.update(name, updates: buildpack_list)
       set_config @app_config
 
-      @app_is_setup = true
       self
     end
     alias :setup :setup!
@@ -356,16 +375,11 @@ module Hatchet
     end
 
     def teardown!
-      return false unless @app_is_setup
-
-      if @run_multi_is_setup
         @run_multi_array.map(&:join)
-        platform_api.formation.update(name, "web", {"size" => "basic"})
-      end
-
     ensure
-      @app_update_info = platform_api.app.update(name, { maintenance: true }) if @app_is_setup
-      @reaper.cycle if @app_is_setup
+      if @heroku_id && !ENV["HEROKU_DEBUG_EXPENSIVE"]
+        @reaper.destroy_with_log(name: @name, id: @heroku_id, reason: "teardown")
+      end
     end
 
     def in_directory

data/lib/hatchet/reaper.rb

@@ -1,138 +1,172 @@
 require 'tmpdir'
 
 module Hatchet
-  # This class lazilly deletes hatchet apps
+  # Delete apps
   #
-  # When the reaper is called, it will check if the system has too many apps (Bassed off of HATCHET_APP_LIMIT), if so it will attempt
-  # to delete an app to free up capacity. The goal of lazilly deleting apps is to temporarilly keep
-  # apps around for debugging if they fail.
+  # Delete a single app:
   #
-  # When App#teardown! is called on an app it is marked as being in a "finished" state by turning
-  # on maintenance mode. The reaper will delete these in order (oldest first).
+  #   @reaper.destroy_with_log(id: id, name: name, reason: "console")
   #
-  # If no apps are marked as being "finished" then the reaper will check to see if the oldest app
-  # has been alive for a long enough period for it's tests to finish (configured by HATCHET_ALIVE_TTL_MINUTES env var).
-  # If the "unfinished" app has been alive that long it will be deleted. If not, the system will sleep for a period of time
-  # in an attempt to allow other apps to move to be "finished".
+  # Clear out all apps older than HATCHET_ALIVE_TTL_MINUTES:
+  #
+  #   @reaper.destroy_older_apps
+  #
+  # If you need to clear up space or wait for space to be cleared
+  # up then:
+  #
+  #   @reaper.clean_old_or_sleep
   #
-  # This class only limits and the number of "hatchet" apps on the system. Prevously there was a maximum of 100 apps on a
-  # Heroku account. Now a user can belong to multiple orgs and the total number of apps they have access to is no longer
-  # fixed at 100. Instead of hard coding a maximum limit, this failure mode is handled by forcing deletion of
-  # an app when app creation fails. In the future we may find a better way of detecting this failure mode
   #
   # Notes:
   #
   # - The class uses a file mutex so that multiple processes on the same machine do not attempt to run the
   #   reaper at the same time.
-  # - AlreadyDeletedError will be raised if an app has already been deleted (possibly by another test run on
-  #   another machine). When this happens, the system will automatically attempt to reap another app.
   class Reaper
     class AlreadyDeletedError < StandardError; end
 
     HATCHET_APP_LIMIT = Integer(ENV["HATCHET_APP_LIMIT"] || 20)  # the number of apps hatchet keeps around
     DEFAULT_REGEX = /^#{Regexp.escape(Hatchet::APP_PREFIX)}[a-f0-9]+/
+    TTL_MINUTES = ENV.fetch("HATCHET_ALIVE_TTL_MINUTES", "7").to_i
+
+    # Protect against parallel deletion on the same machine
+    # via concurrent processes
+    #
+    # Does not protect against distributed systems on different
+    # machines trying to delete the same applications
+    MUTEX_FILE = File.open(File.join(Dir.tmpdir(), "hatchet_reaper_mutex"), File::CREAT)
 
     attr_accessor :io, :hatchet_app_limit
 
     def initialize(api_rate_limit: , regex: DEFAULT_REGEX, io: STDOUT, hatchet_app_limit:  HATCHET_APP_LIMIT, initial_sleep: 10)
-      @api_rate_limit = api_rate_limit
-      @regex = regex
       @io = io
-      @finished_hatchet_apps = []
-      @unfinished_hatchet_apps = []
-      @app_count = 0
-      @hatchet_app_limit = hatchet_app_limit
+      @apps = []
+      @regex = regex
+      @limit = hatchet_app_limit
+      @api_rate_limit = api_rate_limit
       @reaper_throttle = ReaperThrottle.new(initial_sleep: initial_sleep)
     end
 
-    def cycle(app_exception_message: false)
-      # Protect against parallel deletion of the same app on the same system
-      mutex_file = File.open("#{Dir.tmpdir()}/hatchet_reaper_mutex", File::CREAT)
-      mutex_file.flock(File::LOCK_EX)
-
-      refresh_app_list if @finished_hatchet_apps.empty?
-
-      # To be safe try to delete an app even if we're not over the limit
-      # since the exception may have been caused by going over the maximum account limit
-      if app_exception_message
+    def sleep_if_over_limit(reason: )
+      if @apps.length >= @limit
+        age = AppAge.new(created_at: @apps.last["created_at"], ttl_minutes: TTL_MINUTES)
+        @reaper_throttle.call(max_sleep: age.sleep_for_ttl) do |sleep_for|
           io.puts <<-EOM.strip_heredoc
-            WARNING: Running reaper due to exception on app
-                     #{stats_string}
-                     Exception: #{app_exception_message}
+            WARNING: Hatchet app limit reached (#{@apps.length}/#{@limit})
+            All known apps are younger than #{TTL_MINUTES} minutes.
+            Sleeping (#{sleep_for}s)
+
+            Reason: #{reason}
           EOM
-        reap_once
-      end
 
-      while over_limit?
-        reap_once
+          sleep(sleep_for)
+        end
       end
-    ensure
-      mutex_file.close
     end
 
-    def stats_string
-      "total_app_count: #{@app_count}, hatchet_app_count: #{hatchet_app_count}/#{HATCHET_APP_LIMIT}, finished: #{@finished_hatchet_apps.length}, unfinished: #{@unfinished_hatchet_apps.length}"
-    end
-
-    def over_limit?
-      hatchet_app_count > hatchet_app_limit
+    # Destroys apps that are older than the given argument (expecting integer minutes)
+    #
+    # This method might be running concurrently on multiple processes or multiple
+    # machines.
+    #
+    # When a duplicate destroy is detected we can move forward with a conflict strategy:
+    #
+    # - `:refresh_api_and_continue`: Sleep to see if another process will clean up everything for
+    #   us and then re-populate apps from the API and continue.
+    # - `:stop_if_under_limit`: Sleep to allow other processes to continue. Then if apps list
+    #   is under the limit, assume someone else is already cleaning up for us and that we're
+    #   good to move ahead to try to create an app. Otherwise if we're at or
+    #   over the limit sleep, refresh the app list, and continue attempting to delete apps.
+    def destroy_older_apps(minutes: TTL_MINUTES, force_refresh: @apps.empty?, on_conflict: :refresh_api_and_continue)
+      MUTEX_FILE.flock(File::LOCK_EX)
+
+      refresh_app_list if force_refresh
+
+      while app = @apps.pop
+        age = AppAge.new(created_at: app["created_at"], ttl_minutes: minutes)
+        if !age.can_delete?
+          @apps.push(app)
+          break
+        else
+          begin
+            destroy_with_log(
+              id: app["id"],
+              name: app["name"],
+              reason: "app age (#{age.in_minutes}m) is older than #{minutes}m"
+            )
+          rescue AlreadyDeletedError => e
+            if handle_conflict(
+              strategy: on_conflict,
+              conflict_message: e.message,
+            ) == :stop
+              break
+            end
+          end
+        end
+      end
+    ensure
+      MUTEX_FILE.flock(File::LOCK_UN)
     end
 
     # No guardrails, will delete all apps that match the hatchet namespace
-    def destroy_all
-      refresh_app_list
+    def destroy_all(force_refresh: @apps.empty?)
+      MUTEX_FILE.flock(File::LOCK_EX)
 
-      (@finished_hatchet_apps + @unfinished_hatchet_apps).each do |app|
+      refresh_app_list if force_refresh
+
+      while app = @apps.pop
         begin
-          destroy_with_log(name: app["name"], id: app["id"])
-        rescue AlreadyDeletedError
-          # Ignore, keep going
+          destroy_with_log(name: app["name"], id: app["id"], reason: "destroy all")
+        rescue AlreadyDeletedError => e
+          handle_conflict(
+            conflict_message: e.message,
+            strategy: :refresh_api_and_continue
+          )
         end
       end
+    ensure
+      MUTEX_FILE.flock(File::LOCK_UN)
     end
 
-    private def reap_once
-      refresh_app_list if @finished_hatchet_apps.empty?
-
-      if (app = @finished_hatchet_apps.pop)
-        destroy_with_log(name: app["name"], id: app["id"])
-      elsif (app = @unfinished_hatchet_apps.pop)
-        destroy_if_old_enough(app)
+    # Will sleep with backoff and emit a warning message
+    # returns :continue or :stop symbols
+    # :stop indicates execution should stop
+    private def handle_conflict(conflict_message:, strategy:)
+      message = String.new(<<-EOM.strip_heredoc)
+        WARNING: Possible race condition detected: #{conflict_message}
+        Hatchet app limit (#{@apps.length}/#{@limit}), using strategy #{strategy}
+      EOM
+
+      conflict_state = if :refresh_api_and_continue ==  strategy
+        message << "\nSleeping, refreshing app list, and continuing."
+        :continue
+      elsif :stop_if_under_limit == strategy && @apps.length >= @limit
+        message << "\nSleeping, refreshing app list, and continuing. Not under limit."
+        :continue
+      elsif :stop_if_under_limit == strategy
+        message << "\nHalting deletion of older apps. Under limit."
+        :stop
+      else
+        raise "No such strategy: #{strategy}, plese use :stop_if_under_limit or :refresh_api_and_continue"
       end
-    rescue AlreadyDeletedError
-      retry
-    end
 
-    # Checks to see if the given app is older than the HATCHET_ALIVE_TTL_MINUTES
-    # if so, then the app is deleted, otherwise the reaper sleeps for a period of time after which
-    # It can try again to delete another app. The hope is that some apps will be marked as finished
-    # in that time
-    private def destroy_if_old_enough(app)
-      age = AppAge.new(
-        created_at: app["created_at"],
-        ttl_minutes: ENV.fetch("HATCHET_ALIVE_TTL_MINUTES", "7").to_i
-      )
-      if age.can_delete?
-        io.puts "WARNING: Destroying an app without maintenance mode on, app: #{app["name"]}, app_age: #{age.in_minutes} minutes"
-
-        destroy_with_log(name: app["name"], id: app["id"])
-      else
-        # We're not going to delete it yet, so put it back
-        @unfinished_hatchet_apps << app
+      @reaper_throttle.call(max_sleep: TTL_MINUTES) do |sleep_for|
+        io.puts <<-EOM.strip_heredoc
+          #{message}
+          Sleeping (#{sleep_for}s)
+        EOM
 
-        # Sleep, try again later
-        @reaper_throttle.call(max_sleep: age.sleep_for_ttl) do |sleep_for|
-          io.puts <<-EOM.strip_heredoc
-            WARNING: Attempting to destroy an app without maintenance mode on, but it is not old enough. app: #{app["name"]}, app_age: #{age.in_minutes} minutes
-                     This can happen if App#teardown! is not called on an application, which will leave it in an 'unfinished' state
-                     This can also happen if you're trying to run more tests concurrently than your currently set value for HATCHET_APP_COUNT
-                     Sleeping: #{sleep_for} seconds before trying to find another app to reap"
-                     #{stats_string}, HATCHET_ALIVE_TTL_MINUTES=#{age.ttl_minutes}
-          EOM
+        sleep(sleep_for)
+      end
 
-          sleep(sleep_for)
-        end
+      case conflict_state
+      when :continue
+        refresh_app_list
+      when :stop
+      else
+        raise "Unknown state #{conflict_state}"
       end
+
+      conflict_state
     end
 
     private def get_heroku_apps
@@ -140,28 +174,15 @@ module Hatchet
     end
 
     private def refresh_app_list
-      apps = get_heroku_apps.
+      @apps = get_heroku_apps.
+        filter {|app| app["name"].match(@regex) }.
         map {|app| app["created_at"] = DateTime.parse(app["created_at"].to_s); app }.
         sort_by { |app| app["created_at"] }.
         reverse # Ascending order, oldest is last
-
-      @app_count = apps.length
-
-      @finished_hatchet_apps.clear
-      @unfinished_hatchet_apps.clear
-      apps.each do |app|
-        next unless app["name"].match(@regex)
-
-        if app["maintenance"]
-          @finished_hatchet_apps << app
-        else
-          @unfinished_hatchet_apps << app
-        end
-      end
     end
 
-    private def destroy_with_log(name:, id:)
-      message = "Destroying #{name.inspect}: #{id}, #{stats_string}"
+    def destroy_with_log(name:, id:, reason: )
+      message = "Destroying #{name.inspect}: #{id}, (#{@apps.length}/#{@limit}) reason: #{reason}"
 
       @api_rate_limit.call.app.delete(id)
 
@@ -170,19 +191,15 @@ module Hatchet
       body = e.response.body
       request_id = e.response.headers["Request-Id"]
       if body =~ /Couldn\'t find that app./
-        io.puts "Duplicate destroy attempted #{name.inspect}: #{id}, status: 404, request_id: #{request_id}"
-        raise AlreadyDeletedError.new
+        message = "Duplicate destroy attempted #{name.inspect}: #{id}, status: 404, request_id: #{request_id}"
+        raise AlreadyDeletedError.new(message)
       else
         raise e
       end
     rescue Excon::Error::Forbidden => e
       request_id = e.response.headers["Request-Id"]
-      io.puts "Duplicate destroy attempted #{name.inspect}: #{id}, status: 403, request_id: #{request_id}"
-      raise AlreadyDeletedError.new
-    end
-
-    private def hatchet_app_count
-      @finished_hatchet_apps.length + @unfinished_hatchet_apps.length
+      message = "Duplicate destroy attempted #{name.inspect}: #{id}, status: 403, request_id: #{request_id}"
+      raise AlreadyDeletedError.new(message)
     end
   end
 end

data/lib/hatchet/version.rb

@@ -1,3 +1,3 @@
 module Hatchet
-  VERSION = "7.4.0"
+  VERSION = "8.0.1"
 end

data/lib/hatchet.rb

@@ -21,20 +21,43 @@ require 'hatchet/api_rate_limit'
 require 'hatchet/init_project'
 require 'hatchet/heroku_run'
 
+class DefaultCIBranch
+  def initialize(env: ENV)
+    @env = env
+  end
+
+  def call
+    # https://circleci.com/docs/variables
+    return @env['CIRCLE_BRANCH'] if @env['CIRCLE_BRANCH']
+    # https://docs.github.com/en/actions/learn-github-actions/environment-variables
+    # GITHUB_HEAD_REF is provided for PRs, but blank for branch actions.
+    return @env['GITHUB_HEAD_REF'] if @env['GITHUB_HEAD_REF'] && !@env['GITHUB_HEAD_REF']&.empty?
+    # GITHUB_REF_NAME is incorrect on PRs (`1371/merge`), but correct for branch actions.
+    return @env['GITHUB_REF_NAME'] if @env['GITHUB_REF_NAME']
+    # https://devcenter.heroku.com/articles/heroku-ci#immutable-environment-variables
+    return @env['HEROKU_TEST_RUN_BRANCH'] if @env['HEROKU_TEST_RUN_BRANCH']
+    # TRAVIS_BRANCH works fine unless the build is a pull-request. In that case, it will contain the target branch
+    # not the actual pull-request branch! TRAVIS_PULL_REQUEST_BRANCH contains the correct branch but will be empty
+    # for push builds. See: https://docs.travis-ci.com/user/environment-variables/
+    return @env['TRAVIS_PULL_REQUEST_BRANCH'] if @env['TRAVIS_PULL_REQUEST_BRANCH'] && !@env['TRAVIS_PULL_REQUEST_BRANCH']&.empty?
+    return @env['TRAVIS_BRANCH'] if @env['TRAVIS_BRANCH']
+  end
+end
+
 module Hatchet
   RETRIES = Integer(ENV['HATCHET_RETRIES']   || 1)
   Runner  = Hatchet::GitApp
 
   def self.git_branch
-    # TRAVIS_BRANCH works fine unless the build is a pull-request. In that case, it will contain the target branch
-    # not the actual pull-request branch! TRAVIS_PULL_REQUEST_BRANCH contains the correct branch but will be empty
-    # for push builds. See: https://docs.travis-ci.com/user/environment-variables/
-    return ENV['TRAVIS_PULL_REQUEST_BRANCH'] if ENV['TRAVIS_PULL_REQUEST_BRANCH'] && !ENV['TRAVIS_PULL_REQUEST_BRANCH'].empty?
-    return ENV['TRAVIS_BRANCH'] if ENV['TRAVIS_BRANCH']
+    branch = DefaultCIBranch.new.call
 
-    out = `git rev-parse --abbrev-ref HEAD`.strip
-    raise "Attempting to find current branch name. Error: Cannot describe git: #{out}" unless $?.success?
-    out
+    if branch
+      branch
+    else
+      out = `git rev-parse --abbrev-ref HEAD`.strip
+      raise "Attempting to find current branch name. Error: Cannot describe git: #{out}" unless $?.success?
+      out
+    end
   end
 
   if ENV["HATCHET_DEBUG_DEADLOCK"]

data/spec/hatchet/app_spec.rb

@@ -43,14 +43,14 @@ describe "AppTest" do
 
     reaper = app.reaper
 
-    def reaper.cycle(app_exception_message: ); @app_exception_message = app_exception_message; end
-    def reaper.recorded_app_exception_message; @app_exception_message; end
+    def reaper.destroy_older_apps(*args, **kwargs, &block); @app_exception_message = true; end
+    def reaper.clean_old_was_called?; @app_exception_message; end
 
     expect {
       app.create_app
     }.to raise_error("made you look")
 
-    expect(reaper.recorded_app_exception_message).to match("made you look")
+    expect(reaper.clean_old_was_called?).to be_truthy
   end
 
   it "app with default" do
@@ -82,37 +82,6 @@ describe "AppTest" do
     end
   end
 
-  it "marks itself 'finished' when done in block mode" do
-    app = Hatchet::Runner.new("default_ruby")
-
-    def app.push_with_retry!; nil; end
-    app.deploy do |app|
-      expect(app.platform_api.app.info(app.name)["maintenance"]).to be_falsey
-    end
-
-    # After the app is updated, there's no guarantee it will still exist
-    # so we cannot rely on an api call to determine maintenance mode
-    app_update_info = app.instance_variable_get(:"@app_update_info")
-    expect(app_update_info["name"]).to eq(app.name)
-    expect(app_update_info["maintenance"]).to be_truthy
-  end
-
-  it "marks itself 'finished' when done in non-block mode" do
-    app = Hatchet::Runner.new("default_ruby")
-
-    def app.push_with_retry!; nil; end
-    app.deploy
-    expect(app.platform_api.app.info(app.name)["maintenance"]).to be_falsey
-
-    app.teardown!
-
-    # After the app is updated, there's no guarantee it will still exist
-    # so we cannot rely on an api call to determine maintenance mode
-    app_update_info = app.instance_variable_get(:"@app_update_info")
-    expect(app_update_info["name"]).to eq(app.name)
-    expect(app_update_info["maintenance"]).to be_truthy
-  end
-
   describe "before deploy" do
     it "dir" do
       @called = false
@@ -263,16 +232,14 @@ describe "AppTest" do
     @run_count = AtomicCount.new(0)
     app = Hatchet::GitApp.new("default_ruby", run_multi: true)
     app.deploy do
-      app.run_multi("ls") { |out| expect(out).to include("Gemfile"); @run_count.add(1) }
-      app.run_multi("blerg -v") { |_, status| expect(status.success?).to be_falsey; @run_count.add(1) }
+      app.run_multi("ls") { |out| expect(out).to include("Gemfile"); @run_count.add(1); }
+      app.run_multi("blerg -v") { |_, status| expect(status.success?).to be_falsey; @run_count.add(1); }
       app.run_multi("ruby -v") do |out, status|
         expect(out).to include("ruby")
         expect(status.success?).to be_truthy
 
         @run_count.add(1)
       end
-
-      expect(app.platform_api.formation.list(app.name).detect {|ps| ps["type"] == "web"}["size"].downcase).to_not eq("free")
     end
 
     # After the deploy block exits `teardown!` is called

data/spec/hatchet/ci_spec.rb

@@ -15,15 +15,7 @@ describe "CIFourTest" do
       expect(test_run.output).to match(string)
       expect(test_run.output).to match("Using rake")
       expect(test_run.output).to_not match("Installing rake")
-
-      expect(app.platform_api.app.info(app.name)["maintenance"]).to be_falsey
     end
-
-    # After the app is updated, there's no guarantee it will still exist
-    # so we cannot rely on an api call to determine maintenance mode
-    app_update_info = app.instance_variable_get(:"@app_update_info")
-    expect(app_update_info["name"]).to eq(app.name)
-    expect(app_update_info["maintenance"]).to be_truthy
   end
 
   it "error with bad app" do

data/spec/unit/default_ci_branch_spec.rb

@@ -0,0 +1,27 @@
+
+require "spec_helper"
+
+describe "DefaultCIBranch" do
+  it "doesn't error on empty env" do
+    out = DefaultCIBranch.new(env: {}).call
+    expect(out).to be_nil
+  end
+
+  it "GitHub PRs" do
+    out = DefaultCIBranch.new(env: {"GITHUB_HEAD_REF" => "iAmaPR"}).call
+    expect(out).to eq("iAmaPR")
+
+    out = DefaultCIBranch.new(env: {"GITHUB_HEAD_REF" => ""}).call
+    expect(out).to be_nil
+  end
+
+  it "GitHub branches" do
+    out = DefaultCIBranch.new(env: {"GITHUB_REF_NAME" => "iAmaBranch"}).call
+    expect(out).to eq("iAmaBranch")
+  end
+
+  it "heroku" do
+    out = DefaultCIBranch.new(env: {"HEROKU_TEST_RUN_BRANCH" => "iAmaBranch"}).call
+    expect(out).to eq("iAmaBranch")
+  end
+end

data/spec/unit/reaper_spec.rb

@@ -18,7 +18,7 @@ describe "Reaper" do
   end
 
   describe "cycle" do
-    it "does not delete anything if under the limit" do
+    it "does not delete anything if no old apps" do
       reaper = Hatchet::Reaper.new(api_rate_limit: Object.new, hatchet_app_limit: 1, io: StringIO.new)
 
       def reaper.get_heroku_apps
@@ -29,46 +29,11 @@ describe "Reaper" do
       def reaper.check_get_heroku_apps_called; @called_get_heroku_apps ; end
       def reaper.reap_once; raise "should not be called"; end
 
-      reaper.cycle
+      reaper.destroy_older_apps
 
       expect(reaper.check_get_heroku_apps_called).to be_truthy
     end
 
-    it "deletes a maintenance mode app on error" do
-      reaper = Hatchet::Reaper.new(api_rate_limit: Object.new, hatchet_app_limit: 1, io: StringIO.new)
-
-      def reaper.get_heroku_apps
-        @mock_apps ||= [
-          {"name" => "hatchet-t-unfinished", "id" => 2, "maintenance" => false, "created_at" => Time.now.to_s},
-          {"name" => "hatchet-t-foo", "id" => 1, "maintenance" => true, "created_at" => Time.now.to_s}
-        ]
-      end
-      def reaper.destroy_with_log(name: , id: )
-        @reaper_destroy_called_with = {"name" => name, "id" => id}
-      end
-      def reaper.destroy_called_with; @reaper_destroy_called_with; end
-
-      reaper.cycle(app_exception_message: true)
-
-      expect(reaper.destroy_called_with).to eq({"name" => "hatchet-t-foo", "id" => 1})
-    end
-
-    it "deletes maintenance mode app when over limit" do
-      reaper = Hatchet::Reaper.new(api_rate_limit: Object.new, hatchet_app_limit: 0, io: StringIO.new)
-
-      def reaper.get_heroku_apps
-        @mock_apps ||= [{"name" => "hatchet-t-foo", "id" => 1, "maintenance" => true, "created_at" => Time.now.to_s}]
-      end
-      def reaper.destroy_with_log(name: , id: )
-        @reaper_destroy_called_with = {"name" => name, "id" => id}
-      end
-      def reaper.destroy_called_with; @reaper_destroy_called_with; end
-
-      reaper.cycle
-
-      expect(reaper.destroy_called_with).to eq({"name" => "hatchet-t-foo", "id" => 1})
-    end
-
     it "deletes an old app that is past TLL" do
       reaper = Hatchet::Reaper.new(api_rate_limit: Object.new, hatchet_app_limit: 0, io: StringIO.new)
 
@@ -76,25 +41,30 @@ describe "Reaper" do
         two_days_ago = DateTime.now.new_offset(0) - 2
         @mock_apps ||= [{"name" => "hatchet-t-foo", "id" => 1, "maintenance" => false, "created_at" => two_days_ago.to_s }]
       end
-      def reaper.destroy_with_log(name: , id: )
+      def reaper.destroy_with_log(name: , id: , reason: )
         @reaper_destroy_called_with = {"name" => name, "id" => id}
       end
       def reaper.destroy_called_with; @reaper_destroy_called_with; end
 
-      reaper.cycle
+      reaper.destroy_older_apps
 
       expect(reaper.destroy_called_with).to eq({"name" => "hatchet-t-foo", "id" => 1})
     end
 
     it "sleeps, refreshes app list, and tries again when an old app is not past TTL" do
       warning = StringIO.new
-      reaper = Hatchet::Reaper.new(api_rate_limit: Object.new, hatchet_app_limit: 1, initial_sleep: 0, io: warning)
+      reaper = Hatchet::Reaper.new(
+        io: warning,
+        initial_sleep: 0,
+        api_rate_limit: Object.new,
+        hatchet_app_limit: 0,
+      )
 
       def reaper.get_heroku_apps
         now = DateTime.now.new_offset(0)
         @mock_apps ||= [{"name" => "hatchet-t-foo", "id" => 1, "maintenance" => false, "created_at" => now.to_s }]
       end
-      def reaper.destroy_with_log(name: , id: )
+      def reaper.destroy_with_log(name: , id: , reason: )
         @reaper_destroy_called_with = {"name" => name, "id" => id}
       end
       def reaper.destroy_called_with; @reaper_destroy_called_with; end
@@ -104,13 +74,13 @@ describe "Reaper" do
 
       def reaper.get_slept_for_val; @_slept_for; end
 
-      reaper.cycle(app_exception_message: true)
+      reaper.destroy_older_apps
+      reaper.sleep_if_over_limit(reason: "test")
 
       expect(reaper.get_slept_for_val).to eq(0)
       expect(reaper.destroy_called_with).to eq(nil)
 
-      expect(warning.string).to match("WARNING")
-      expect(warning.string).to match("total_app_count: 1, hatchet_app_count: 1/#{Hatchet::Reaper::HATCHET_APP_LIMIT}, finished: 0, unfinished: 1")
+      expect(warning.string).to include("WARNING: Hatchet app limit reached (1/0)")
     end
   end
 
@@ -154,16 +124,4 @@ describe "Reaper" do
       end
     end
   end
-
-  it "over limit" do
-    reaper = Hatchet::Reaper.new(api_rate_limit: -> (){}, io: StringIO.new)
-    def reaper.hatchet_app_count; Hatchet::Reaper::HATCHET_APP_LIMIT + 1; end
-
-    expect(reaper.over_limit?).to be_truthy
-
-    reaper = Hatchet::Reaper.new(api_rate_limit: -> (){}, io: StringIO.new)
-    def reaper.hatchet_app_count; Hatchet::Reaper::HATCHET_APP_LIMIT - 1; end
-
-    expect(reaper.over_limit?).to be_falsey
-  end
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: heroku_hatchet
 version: !ruby/object:Gem::Version
-  version: 7.4.0
+  version: 8.0.1
 platform: ruby
 authors:
 - Richard Schneeman
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2023-01-23 00:00:00.000000000 Z
+date: 2023-03-01 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: platform-api
@@ -206,6 +206,7 @@ files:
 - spec/hatchet/local_repo_spec.rb
 - spec/hatchet/lock_spec.rb
 - spec/spec_helper.rb
+- spec/unit/default_ci_branch_spec.rb
 - spec/unit/heroku_run_spec.rb
 - spec/unit/init_spec.rb
 - spec/unit/reaper_spec.rb
@@ -245,6 +246,7 @@ test_files:
 - spec/hatchet/local_repo_spec.rb
 - spec/hatchet/lock_spec.rb
 - spec/spec_helper.rb
+- spec/unit/default_ci_branch_spec.rb
 - spec/unit/heroku_run_spec.rb
 - spec/unit/init_spec.rb
 - spec/unit/reaper_spec.rb