heroku_hatchet 6.0.0 → 7.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: ce2fc5243218285fa0505cf41a9fc51d1f7cdfd3cb69ff57e83c11be6137f57e
-  data.tar.gz: f61248e6cfd590ce04ca68b0a5233336a56bb3b1f135580eb10ff5016b59b36b
+  metadata.gz: 89dec86ebf0199547ebf527242257a88b26c2741b272aa73945b4c907b72f51a
+  data.tar.gz: fe0bd8c520fc7102e13ad6f91da96b837158f0ba8eaa324c846e3acc92cb233f
 SHA512:
-  metadata.gz: 269805f1a98ccf64e772e66a503ad9e0558635eca7a7c04b12567b1335f059b7dfa17e4e704b8fca1c088d38672eafce1bd8f290c564a62d44911059f5eb7700
-  data.tar.gz: f0d1d0a640b5de56214acd122aaa918e3038ecc65ceef526c86c44585cd5ad253061f05252c40154c7edac57aa84d038a8216843198373c7a9b1d67b1662fb12
+  metadata.gz: bad4717807a5cddb7b0cdd2750bdd1811b8b012e767d92173f364795eb1da3a8af41fddb28269d14c67ad8d2cfe7e7d4c7f9cefbbafd575172408787cca41142
+  data.tar.gz: b2f7905e07091c4b66ad7b6ba4ac00b8e304e32f10c01a259723cb5e12e592160f9ca30cb66f5485c3bc59dffa26d99bbba070f73be161022d85e63c6afd4409

data/.circleci/config.yml

@@ -4,6 +4,8 @@ references:
     run:
       name: Run test suite
       command: PARALLEL_SPLIT_TEST_PROCESSES=25 bundle exec parallel_split_test spec/
+      environment:
+        HATCHET_EXPENSIVE_MODE: 1 # !!!! WARNING !!!! ONLY RUN THIS IF YOU WORK FOR HEROKU !!!! WARNING !!!!
   restore: &restore
     restore_cache:
       keys:

data/CHANGELOG.md

@@ -1,5 +1,20 @@
 ## HEAD
 
+## 7.0.0
+
+- ActiveSupport's Object#blank? and Object#present? are no longer provided by default (https://github.com/heroku/hatchet/pull/107)
+- Remove deprecated support for passing a block to `App#run` (https://github.com/heroku/hatchet/pull/105)
+- Ignore  403 on app delete due to race condition (https://github.com/heroku/hatchet/pull/101)
+- The hatchet.lock file can now be locked to "main" in addition to "master" (https://github.com/heroku/hatchet/pull/86)
+- Allow concurrent one-off dyno runs with the `run_multi: true` flag on apps (https://github.com/heroku/hatchet/pull/94)
+- Apps are now marked as being "finished" by enabling maintenance mode on them when `teardown!` is called. Finished apps can be reaped immediately (https://github.com/heroku/hatchet/pull/97)
+- Applications that are not marked as "finished" will be allowed to live for a HATCHET_ALIVE_TTL_MINUTES duration before they're deleted by the reaper to protect against deleting an app mid-deploy, default is seven minutes (https://github.com/heroku/hatchet/pull/97)
+- The HEROKU_APP_LIMIT env var no longer does anything, instead hatchet application reaping is manually executed if an app cannot be created (https://github.com/heroku/hatchet/pull/97)
+- App#deploy without a block will no longer run `teardown!` automatically (https://github.com/heroku/hatchet/pull/97)
+- Calls to `git push heroku` are now rate throttled (https://github.com/heroku/hatchet/pull/98)
+- Calls to `app.run` are now rate throttled (https://github.com/heroku/hatchet/pull/99)
+- Deployment now raises and error when the release failed (https://github.com/heroku/hatchet/pull/93)
+
 ## 6.0.0
 
 - Rate throttling is now provided directly by `platform-api` (https://github.com/heroku/hatchet/pull/82)
@@ -248,7 +263,7 @@
 
 ## 1.1.1
 
-- Ensure external commands run inside of `bundle_exec` block are run with propper environment variables set.
+- Ensure external commands run inside of `bundle_exec` block are run with proper environment variables set.
 
 ## 1.1.0
 

data/README.md

@@ -2,52 +2,130 @@
 
 ![](http://f.cl.ly/items/2M2O2Q2I2x0e1M1P2936/Screen%20Shot%202013-01-06%20at%209.59.38%20PM.png)
 
-Hatchet is a an integration testing library for developing Heroku buildpacks.
+Hatchet is an integration testing library for developing Heroku buildpacks.
 
 ## Install
 
-First run:
+To get started, add this gem to your Gemfile:
 
-    $ bundle install
+```
+gem "heroku_hatchet"
+```
+
+Then run:
 
-This library uses the heroku CLI and API. You will need to make your API key available to the system. If you're running on a CI platform, you'll need to generate an OAuth token and make it available on the system you're running on see the "CI" section below.
+```
+$ bundle install
+```
 
-## Run the Tests
+This library uses the Heroku CLI and API. You will need to make your API key available to the system (`$ heroku login`). If you're running on a CI platform, you'll need to generate an OAuth token and make it available on the system you're running on see the "CI" section below.
 
-    $ bundle exec rake test
+## About Hatchet
 
-## Why Test a Buildpack?
+### Why Test a Buildpack?
 
 Testing a buildpack prevents regressions, and pushes out new features faster and easier.
 
-## What can Hatchet Test?
+### What can Hatchet test?
+
+Hatchet can easily test certain operations: deployment of buildpacks, getting the build output and running arbitrary interactive processes (e.g. `heroku run bash`). Hatchet can also test running Heroku CI against an app.
 
-Hatchet can easily test certain operations: deployment of buildpacks, getting the build output, and running arbitrary interactive processes (e.g. `heroku run bash`). Hatchet can also test running CI against an app.
+### How does Hatchet test a buildpack?
 
-## Writing Tests
+To be able to check the behavior of a buildpack, you have to execute it. Hatchet does this by creating new Heroku apps `heroku create`, setting them to use your branch of the buildpack (must be available publicly) `heroku buildpacks:set https://github.com/your/buildpack-url#branch-name`, then deploying the app `git push heroku master`. It has built-in features such as API rate throttling (so your deploys slow down instead of fail) and internal retry mechanisms. Once deployed, it can `heroku run <command>` for you to allow you to assert behavior.
 
-Hatchet assumes a test framework doesn't exist. [This project](https://github.com/heroku/hatchet) uses `Test::Unit` to run it's own tests. While the [heroku-ruby-buildpack](https://github.com/heroku/heroku-buildpack-ruby) uses rspec.
+### Can I use Hatchet to test my buildpack locally?
 
-Running `focused: true` in rspec allows you to choose which test to run and to tag tests. Rspec has useful plugins, such as `gem 'rspec-retry'` which will re-run any failed tests a given number of times (I recommend setting this to at least 2) to decrease false negatives in your tests.
+Yes, but the workflow is less than ideal since Heroku (and by extension, Hatchet) need your work to be available at a public URL. Let's say you're doing TDD and have already written a single failing test. You are developing on a branch and have already committed the test to that branch. To test your new code, you'll need to commit what you've got, push it to your public source repository.
 
-Whatever testing framework you chose, we recommend using a parallel test runner when running the full suite. [Parallel_tests](https://github.com/grosser/parallel_tests) works with rspec and test::unit and is amazing.
+```
+$ git add -P
+$ git commit -m "[ci skip] WIP"
+$ git push origin <current-branchname>
+$ bundle exec rspec spec/path-to-your-test.rb:5 # This syntax focus runs a single test on line number 5
+```
+
+Now when the tests execute Hatchet will use your code on your public branch. If you don't like a bunch of ugly "wip" commits you can keep amending the same commit over and over while you're iterating, alternatively you can [rebase your commits when you're done](https://www.codetriage.com/rebase).
 
-If you're unfamiliar with the ruby testing eco-system or want some help, start by looking at existing projects.
+### Isn't deploying an app to Heroku overkill for testing? I want to go faster.
 
-*Spoilers*: There is a section below on getting Hatchet to work on CI
+Hatchet is for integration testing. You can also unit test your code if you want your tests to execute much quicker. If your buildpack is written in bash, there is [shUnit2](https://github.com/kward/shunit2/), for example. It is recommended that you use both integration and unit tests.
 
-## Testing a Buildpack
+But can't you integration test the buildpack by calling `bin/compile` directly without having to jump through deploying a Heroku app? It is possible to call your `bin/compile` script from your machine locally without Hatchet, but you'll not have access to config vars, addons, release phase, `heroku run`, and many more features. Also, calling `bin/compile` is very slow, and a medium to large buildpack can have upwards of 70 different integration test cases. If each were to take 1 minute optimistically, it would take over an hour to run your whole suite. Since Hatchet can be safely run via a parallel runner, it can execute most of these builds in parallel, and the whole suite would take roughly 5 minutes when running on CI.
 
-Hatchet was built for testing the Ruby buildpack, but Hatchet can test any buildpack provided your tests are written in Ruby.
+In addition to speed, Hatchet provides isolation. Suppose you're executing `bin/compile` locally. In that case, you need to be very careful not to pollute the environment or local disk between runs, or you'll end up with odd failures that are seemingly impossible to hunt down.
 
-You will need copies of applications that can be deployed by your buildpack. You can see the ones for the Hatchet unit tests (and the Ruby buildpack) https://github.com/sharpstone. Hatchet does not require that you keep these apps checked into your git repo which would make fetching your buildpack slow, instead declare them in a `hatchet.json` file (see below).
+## Quicklinks
 
-Hatchet will automate retrieving these files `$ hatchet install`, deploy these files using your local copy of the buildpack, retrieve the build output and run commands against deploying applications.
+- Concepts
+  - [Tell Hatchet how to find your buildpack](#specify-buildpack)
+  - [Give Hatchet some example apps to deploy](#example-apps)
+  - [Use Hatchet to deploy app](#deploying-apps)
+  - [Use Hatchet to test runtime behavior and environment](#build-versus-run-testing)
+  - [How to update or modify test app files safely in parallel](#modifying-apps-on-disk-before-deploy)
+  - [Understand how Hatchet (does and does not) clean up apps](#app-reaping)
+  - [How to re-deploy the same app](#deploying-multiple-times)
+  - [How to test your buildpack on Heroku CI](#testing-ci)
+  - [How to safely test locally without modifying disk or your environment](#testing-on-local-disk-without-deploying)
+  - [How to set up your buildpack on a Continuous Integration (CI) service](#running-your-buildpack-tests-on-a-ci-service)
 
+- Reference Docs:
+  - Method arguments to `Hatchet::Runner.new` [docs](#init-options)
+  - Method documentation for `Hatchet::Runner` and `TestRun` objects [docs](#app-methods)
+  - All ENV vars and what they do [docs](#env-vars)
 
-## Hatchet.json
+- Ruby language and ecosystem basics
+  - [Introduction to the Rspec testing framework for non-rubyists](#basic-rspec)
+  - [Introduction to Ruby for non-rubyists](#basic-ruby)
 
-Hatchet expects a json file in the root of your buildpack called `hatchet.json`. You can configure install options using the `"hatchet"` key. In this example, we're telling Hatchet to install the given repos to our `test/fixtures` directory instead of the default current directory.
+## Concepts
+
+### Specify buildpack
+
+Tell Hatchet what buildpack you want to use by default by setting environment variables, this is commonly done in the `spec_helper.rb` file:
+
+```ruby
+ENV["HATCHET_BUILDPACK_BASE"] = "https://github.com/path-to-your/buildpack"
+require 'hatchet'`
+```
+
+If you do not specify `HATCHET_BUILDPACK_URL` the default Ruby buildpack will be used. If you do not specify a `HATCHET_BUILDPACK_BRANCH` the current branch you are on will be used. This is how the Ruby buildpack runs tests on branches on CI (by leaving `HATCHET_BUILDPACK_BRANCH` blank).
+
+The workflow generally looks like this:
+
+1. Make a change to the codebase
+2. Commit it and push to GitHub so it's publicly available
+3. Execute your test suite or individual test
+4. Repeat until you're happy
+5. Be happy
+
+### Example apps:
+
+Hatchet works by deploying example apps to Heroku's production service first you'll need an app to deploy that works with the buildpack you want to test. This method is preferred if you've got a very small app that might only need one or two files. There are two ways to give Hatchet a test app, you can either specify a remote app or a local directory.
+
+- **Local directory use of Hatchet:**
+
+```ruby
+Hatchet::Runner.new("path/to/local/directory").deploy do |app|
+end
+```
+
+An example of this is the [heroku/nodejs buildpack tests](https://github.com/heroku/heroku-buildpack-nodejs/blob/9898b875f45639d9fe0fd6959f42aea5214504db/spec/ci/node_10_spec.rb#L6).
+
+You can either check in your apps to your source control or, you can use code to generate them, for example:
+
+- [Generating an example app](https://github.com/sharpstone/force_absolute_paths_buildpack/blob/53c3cffb039fd366b5abb4524fb32983c11f9344/spec/hatchet/buildpack_spec.rb#L5-L20)
+- [Source code for `generate_fixture_app`](https://github.com/sharpstone/force_absolute_paths_buildpack/blob/53c3cffb039fd366b5abb4524fb32983c11f9344/spec/spec_helper.rb#L34-L64)
+
+If you generate example apps programmatically, then add the folder you put them in to your `.gitignore`.
+
+> Note: If you're not using the `hatchet.json` you'll still need an empty one in your project with contents `{}`
+
+- **Github app use of Hatchet:**
+
+Instead of storing your apps locally or generating them, you can point Hatchet at a remote github repo. This method of storing apps on GitHub is preferred is you have an app that is large or has many files (for example, a Rails app).
+
+Hatchet expects a json file in the root of your buildpack called `hatchet.json`. You can configure the install options using the `"hatchet"` key. In this example, we're telling Hatchet to install the given repos to our `test/fixtures` directory instead of the current default directory.
 
 ```
 {
@@ -78,19 +156,7 @@ You can reference one of these applications in your test by using it's git name:
 Hatchet::Runner.new('no_lockfile')
 ```
 
-If you have conflicting names, use full paths.
-
-To test with fixtures that are checked in locally, add the fixture directory to the path and skip the `hatchet install`:
-
-```
-Hatchet::Runner.new("spec/fixtures/repos/node-10-metrics")
-```
-
-Be careful when including repos inside of your test directory. If you're using a runner that looks for patterns such as `*_test.rb` to run your hatchet tests, it may run the tests inside of the repos. To prevent this problem, move your repos directory out of `test/` or into specific directories such as `test/hatchet`. Then change your pattern. If you are using `Rake::TestTask`, it might look like this:
-
-    t.pattern = 'test/hatchet/**/*_test.rb'
-
-When basing tests on external repos, do not change the tests or they may spontaneously fail. We may create a hatchet.lockfile or something to declare the commit in the future.
+If you have conflicting names, use full paths like `Hatchet::RUnner.new("sharpstone/no_lockfile")`.
 
 When you run `hatchet install` it will lock all the Repos to a specific commit. This is done so that if a repo changes upstream that introduces an error the test suite won't automatically pick it up. For example in https://github.com/sharpstone/lock_fail/commit/e61ba47043fbae131abb74fd74added7e6e504df an error is added, but this will only cause a failure if your project intentionally locks to commit `e61ba47043fbae131abb74fd74added7e6e504df` or later.
 
@@ -108,298 +174,808 @@ You can re-lock your projects by running `hatchet lock`. This modifies the `hatc
   - 6e642963acec0ff64af51bd6fba8db3c4176ed6e
 - - test/fixtures/repos/lock/lock_fail
   - da748a59340be8b950e7bbbfb32077eb67d70c3c
-- - test/fixtures/repos/lock/lock_fail_master
-  - master
+- - test/fixtures/repos/lock/lock_fail_main
+  - main
 - - test/fixtures/repos/rails2/rails2blog
   - b37357a498ae5e8429f5601c5ab9524021dc2aaa
 - - test/fixtures/repos/rails3/rails3_mri_193
   - 88c5d0d067cfd11e4452633994a85b04627ae8c7
 ```
 
-If you don't want to lock a specific commit, you can instead specify `master` and it will always grab the latest commits.
+> Note: If you don't want to lock to a specific commit, you can always use the latest commit by specifying `main` manually as seen above. This will always give you the latest commit on the `main` branch. The `master` keyword is supported as well.
 
+### Deploying apps
 
-## Deploying apps
+Once you've got an app and have set up your buildpack, you can deploy an app and assert based on the output (all examples use rspec for testing framework).
 
+```ruby
+Hatchet::Runner.new("default_ruby").deploy do |app|
+  expect(app.output).to match("Installing dependencies using bundler")
+end
+```
 
-You can specify the location of your public buildpack url in an environment variable:
+By default, an error will be raised if the deploy doesn't work, which forces the test to fail. If you're trying to test failing behavior (for example you want to test that an app without a `Gemfile.lock` fails to build), you can manually allow failures:
 
-```sh
-HATCHET_BUILDPACK_BASE=https://github.com/heroku/heroku-buildpack-ruby.git
-HATCHET_BUILDPACK_BRANCH=master
+```ruby
+Hatchet::Runner.new("no_lockfile", allow_failure: true).deploy do |app|
+  expect(app).not_to be_deployed
+  expect(app.output).to include("Gemfile.lock required")
+end
 ```
 
-If you do not specify `HATCHET_BUILDPACK_URL` the default Ruby buildpack will be used. If you do not specify a `HATCHET_BUILDPACK_BRANCH` the current branch you are on will be used. This is how the Ruby buildpack runs tests on branches on CI (by leaving `HATCHET_BUILDPACK_BRANCH` blank).
+### Build versus run testing
 
-If the `ENV['HATCHET_RETRIES']` is set to a number, deploys are expected to work and automatically retry that number of times. Due to testing using a network and random failures, setting this value to `3` retries seems to work well. If an app cannot be deployed within its allotted number of retries, an error will be raised.
+In addition to testing what the build output was, the next most common thing to assert is that behavior at runtime produces expected results. Hatchet provides a helper for calling `heroku run <cmd>` and asserting against it. For example:
 
-If you are testing an app that is supposed to fail deployment, you can set the `allow_failure: true` flag when creating the app:
+```ruby
+Hatchet::Runner.new("minimal_webpacker", buildpacks: buildpacks).deploy do |app, heroku|
+  expect(app.run("which node")).to match("/app/bin/node")
+end
+```
+
+In this example, Hatchet is calling `heroku run which node` and passing the results back to the test so we can assert against it.
+
+- **Asserting exit status:**
+
+In ruby the way you assert a command you ran on the shell was successful or not is by using the `$?` "magic object". By default calling `app.run` will set this variable which can be used in your tests:
 
 ```ruby
-Hatchet::Runner.new("no_lockfile", allow_failure: true).deploy do |app|
+Hatchet::Runner.new("minimal_webpacker", buildpacks: buildpacks).deploy do |app, heroku|
+  expect(app.run("which node")).to match("/app/bin/node")
+  expect($?.exitstatus).to eq(0)
+  expect($?.success?).to be_truthy
+
+  # In Ruby all objects except `nil` and `false` are "truthy" in this case it could also be tested using `be_true` but
+  # it's best practice to use this test helper in rspec
+end
 ```
 
-After the block finishes, your app will be queued to be removed from heroku. If you are investigating a deploy, you can add the `debug: true` flag to your app:
+You can disable this behavior [see how to do it in the reference tests](https://github.com/heroku/hatchet/blob/master/spec/hatchet/app_spec.rb)
+
+- **Escaping and raw mode:**
+
+By default `app.run()` will escape the input so you can safely call `app.run("cmd && cmd")` and it works as expected. But if you want to do something custom, you can enable raw mode by passing in `raw: true` [see how to do it in the reference tests](https://github.com/heroku/hatchet/blob/master/spec/hatchet/app_spec.rb)
+
+- **Heroku options:**
+
+You can use all the options available to `heroku run bash` such as `heroku run bash --env FOO=bar` [see how to do it in the reference tests](https://github.com/heroku/hatchet/blob/master/spec/hatchet/app_spec.rb)
+
+### Modifying apps on disk before deploy
+
+Hatchet is designed to play nicely with running tests in parallel via threads or processes. To support this the code that is executed in the `deploy` block is being run in a new directory. This allows you to modify files on disk safely without having to worry about race conditions. Still, it introduces the unexpected behavior that changes might not work like you think they will.
+
+One typical pattern is to have a minimal example app, and then to modify it as needed before your tests. You can do this safely using the `before_deploy` block.
 
 ```ruby
-Hatchet::Runner.new("rails3_mri_193", debug: true).deploy do |app|
+Hatchet::Runner.new("default_ruby").tap do |app|
+  app.before_deploy do
+    out = `echo 'ruby "2.7.1"' >> Gemfile`
+    raise "Echo command failed: #{out}" unless $?.success?
+  end
+  app.deploy do |app|
+    expect(app.output).to include("Using Ruby version: ruby-2.6.6")
+  end
+end
 ```
 
-After Hatchet is done deploying your app, it will remain on Heroku. It will also output the name of the app into your test logs so that you can `heroku run bash` into it for detailed postmortem.
+This example will add the string `ruby "2.7.1"` to the end of the Gemfile on disk. It accomplishes this by shelling out to `echo`. If you prefer, you can directly use `File.open` to write contents to disk.
+
+> Note: The above [tap method in ruby](https://ruby-doc.org/core-2.4.0/Object.html#method-i-tap) returns itself in a block, it makes this example cleaner.
+
+> Note: that we're checking the status code of the shell command we're running (shell commands are executed via backticks in ruby), a common pattern is to write a simple helper function to automate this:
 
-If you are wanting to run a test against a specific app without deploying to it, you can specify the app name like this:
+```ruby
+# spec_helper.rb
+
+def run!(cmd)
+  out = `#{cmd}`
+  raise "Command #{cmd} failed with output #{out}" unless $?.success?
+  out
+end
+```
+
+Then you can use it in your tests:
 
 ```ruby
-app = Hatchet::Runner.new("rails3_mri_193", name: "testapp")
+Hatchet::Runner.new("default_ruby").tap do |app|
+  app.before_deploy do
+    run!(%Q{echo 'ruby "2.7.1"'})
+  end
+  app.deploy do |app|
+    expect(app.output).to include("Using Ruby version: ruby-2.6.6")
+  end
+end
 ```
 
-Deploying the app takes a few minutes. You may want to skip deployment to make debugging faster.
+> Note: that `%Q{}` is a method of creating a string in Ruby if we didn't use it here we could escape the quotes:
 
-If you need to deploy using a different buildpack you can specify one manually:
+```ruby
+run!("echo 'ruby \"2.7.1\"'")
+```
+
+In Ruby double quotes allow for the insert operator in strings, but single quotes do not:
 
 ```ruby
+name = "schneems"
+puts "Hello #{name}"     # => Hello schneems
+puts 'Hello #{name}'     # => Hello #{name}
+puts "Hello '#{name}'"   # => Hello 'schneems'
+puts %Q{Hello "#{name}"} # => Hello "schneems"
+```
 
-def test_deploy
-  Hatchet::Runner.new("rails3_mri_193", buildpack: "https://github.com/heroku/heroku-buildpack-ruby.git").deploy do |app|
-  # ...
+### App reaping
+
+When your tests are running you'll see hatchet output some details about what it's doing:
+
+```
+Hatchet setup: "hatchet-t-bed73940a6" for "rails51_webpacker"
 ```
 
-You can specify multiple buildpacks by passing in an array. When you do that you also need to tell hatchet where to place your buildpack. Since hatchet needs to build your buildpack from a branch you should not hardcode a path like `heroku/ruby` instead Hatchet has a replacement mechanism. Use the `:default` symbol where you want your buildpack to execute. For example:
+And later:
 
 ```
-Hatchet::Runner.new("default_ruby", buildpacks: [:default, "https://github.com/pgbouncer/pgbouncer"])
+Destroying "hatchet-t-fd25e3626b". Hatchet app limit: 80
 ```
 
-That will expand your buildpack and branch. For example if you're on the `update_readme` branch of the `heroku-buildpack-ruby` buildpack it would expand to:
+By default, Hatchet does not destroy your app at the end of the test run, that way if your test failed unexpectedly if it's not destroyed yet, you can:
 
 ```
-Hatchet::Runner.new("default_ruby", buildpacks: ["https://github.com/heroku/heroku-buildpack-ruby#update_readme", "https://github.com/pgbouncer/pgbouncer"])
+$ heroku run bash -a hatchet-t-bed73940a6
 ```
 
-You can also specify a stack:
+And use that to debug. Hatchet deletes old apps on demand. You tell it what your limits are and it will stay within those limits:
 
 ```
-Hatchet::Runner.new("rails3_mri_193", stack: "cedar-14").deploy do |app|
+HATCHET_APP_LIMIT=20
 ```
 
-## Getting Deploy Output
+With these env vars, Hatchet will "reap" older hatchet apps when it sees there are 20 or more hatchet apps. For CI, it's recommended that you increase the `HATCHET_APP_LIMIT` to 80-100. Hatchet will mark apps as safe for deletion once they've finished, and the `teardown!` method has been called on them (it tracks this by enabling maintenance mode on apps). Hatchet only tracks its apps. Hatchet uses a regex pattern on the name of apps to see which ones it can manage. If your account has reached the maximum number of global Heroku apps, you'll need to remove some manually.
+
+If an app is not marked as being in maintenance mode for some reason, it can be deleted, but only after it has been allowed to live for some time. This behavior is configured by the `HATCHET_ALIVE_TTL_MINUTES` env var. For example, if you set it for `7`, Hatchet will ensure that any apps that are not marked as being in maintenance mode are allowed to live for at least seven minutes. This should give the app time to finish the test's execution, so it is not deleted mid-deploy. When this deletion happens, you'll see a warning in your output. It could indicate you're not properly cleaning up and calling `teardown!` on some of your apps, or it could mean that you're attempting to execute more tests concurrently than your `HATCHET_APP_LIMIT` allows. This deletion-mid-test behavior might otherwise be triggered if you have multiple CI runs executing at the same time.
+
+It's recommended you don't use your personal Heroku API key for running tests on a CI server since the hatchet apps count against your account maximum limits. Running tests using your account locally is fine for debugging one or two tests.
+
+If you find your local account has hit your maximum app limit, one handy trick is to get rid of any old "default" Heroku apps you've created. This plugin (https://github.com/hunterloftis/heroku-destroy-temp) can help:
+
+```
+$ heroku plugins:install heroku-destroy-temp
+$ heroku apps:destroy-temp
+```
+
+> This won't detect hatchet apps, but it's still handy for cleaning up other unused apps.
+
+### Deploying multiple times
 
-After Hatchet deploys your app you can get the output by using `app.output`
+If your buildpack uses the cache, you'll likely want to deploy multiple times against the same app to assert the cache was used. Here's an example of how to do that:
 
 ```ruby
-Hatchet::Runner.new("rails3_mri_193").deploy do |app|
-  puts app.output
+Hatchet::Runner.new("python_default").deploy do |app|
+  expect(app.output).to match(/Installing pip/)
+
+  # Redeploy with changed requirements file
+  run!(%Q{echo "pygments" >> requirements.txt})
+  app.commit!
+
+  app.push! # <======= HERE
+
+  expect(app.output).to match("Requirements file has been changed, clearing cached dependencies")
 end
 ```
 
-If you told Hatchet to `allow_failure: true`, then the full output of the failed build will be in `app.output` even though the app was not deployed. It is a good idea to test against the output for text that should be present. Using a testing framework such as `Test::Unit` a failed test output may look like this:
+### Testing CI
+
+You can run an app against CI using the `run_ci` command (instead of `deploy`). You can re-run tests against the same app with the `run_again` command.
 
 ```ruby
-Hatchet::Runner.new("no_lockfile", allow_failure: true).deploy do |app|
-  assert_match "Gemfile.lock required", app.output
+Hatchet::Runner.new("python_default").run_ci do |test_run|
+  expect(test_run.output).to match("Downloading nose")
+  expect(test_run.status).to eq(:succeeded)
+
+  test_run.run_again
+
+  expect(test_run.output).to match("installing from cache")
+  expect(test_run.output).to_not match("Downloading nose")
 end
 ```
 
-Since an error will be raised on failed deploys you don't need to check for a deployed status (the error will automatically fail the test for you).
+> Note: That  thing returned by the `run_ci` command is not an "app" object but rather a `test_run` object.
 
-## Running Processes
+- `test_run.output` will have the setup and test output of your tests.
+- `test_run.app` has a reference to the "app" you're testing against, however currently no `heroku create` is run (as it's not needed to run tests, only a pipeline and a blob of code).
 
-Often times asserting output of a build can only get you so far, and you will need to actually run a task on the dyno. To run a non-interactive command such as `heroku run ls`, you can use the `app.run()` command without passing a block:
+An exception will be raised if either the test times out or a status of `:errored` or `:failed` is returned. If you expect your test to fail, you can pass in `allow_failure: true` when creating your hatchet runner. If you do that, you'll also get access to different statuses:
 
-```ruby
-Hatchet::Runner.new("rails3_mri_193").deploy do |app|
-  assert_match "applications.css", app.run("ls public/assets")
-```
+- `test_run.status` will return a symbol of the status of your test. Statuses include, but are not limited to `:pending`, `:building`, `:errored`, `:creating`, `:succeeded`, and `:failed`
+
+You can pass in a different timeout to the `run_ci` method `run_ci(timeout: 300)`.
 
-This is useful for checking the existence of generated files such as assets. To run an interactive session such as `heroku run bash` or `heroku run rails console`, run the command and pass a block:
+You probably need an `app.json` in the root directory of the app you're deploying. For example:
 
+```json
+{
+  "environments": {
+    "test": {
+      "addons":[
+         "heroku-postgresql"
+      ]
+    }
+  }
+}
 ```
-Hatchet::Runner.new("rails3_mri_193").deploy do |app|
-  app.run("cat Procfile")
+
+This is on [a Rails5 test app](https://github.com/sharpstone/rails5_ruby_schema_format/blob/master/app.json) that needs the database to run.
+
+Do **NOT** specify a `buildpacks` key in the `app.json` because Hatchet will automatically do this for you. If you need to set buildpacks, you can pass them into the `buildpacks:` keyword argument:
+
+```ruby
+buildpacks = [
+  "https://github.com/heroku/heroku-buildpack-pgbouncer.git",
+  :default
+]
+
+Hatchet::Runner.new("rails5_ruby_schema_format", buildpacks: buildpacks).run_ci do |test_run|
+  # ...
 end
 ```
 
-By default commands will be shell escaped (to prevent commands from escaping the `heroku run` command), if you want to manage your own quoting you can use the `raw: true` option:
+> Note that the `:default` symbol (like a singleton string object in Ruby) can be used for where you want your buildpack inserted, it will be replaced with your app's repo and git branch you're testing against.
 
+### Testing on local disk without deploying
+
+Sometimes you might want to assert something against a test app without deploying. This modification is tricky if you're modifying files or the environment in your test. To help out there's a helper `in_directory_fork`:
+
+```ruby
+Hatchet::App.new('rails6-basic').in_directory_fork do
+  require 'language_pack/rails5'
+  require 'language_pack/rails6'
+
+  expect(LanguagePack::Rails5.use?).to eq(false)
+  expect(LanguagePack::Rails6.use?).to eq(true)
+end
 ```
-app.run('echo \$HELLO \$NAME', raw: true)
+
+## Running your buildpack tests on a CI service
+
+Once you've got your tests working locally, you'll likely want to get them running on CI. For reference, see the [Circle CI config from this repo](https://github.com/heroku/hatchet/blob/master/.circleci/config.yml) and the [Heroku CI config from the ruby buildpack](https://github.com/heroku/heroku-buildpack-ruby/blob/master/app.json).
+
+To make running on CI easier, there is a setup script in Hatchet that can be run on your CI server each time before your tests are executed:
+
+```yml
+bundle exec hatchet ci:setup
 ```
 
-You can specify Heroku flags to the `heroku run` command by passing in the `heroku:` key along with a hash.
+If you're a Heroku employee, see [private instructions for setting up test users](https://github.com/heroku/languages-team/blob/master/guides/create_test_users_for_ci.md) to generate a user a grab the API token.
+
+Once you have an API token you'll want to set up these env vars with your CI provider:
 
 ```
-app.run("nproc", heroku: { "size" => "performance-l" })
-# => 8
+HATCHET_APP_LIMIT=100
+HATCHET_RETRIES=2
+HEROKU_API_KEY=<redacted>
+HEROKU_API_USER=<redacted@example.com>
 ```
 
-You can see a list of Heroku flags by running:
+You can reference this PR for getting a buildpack set up from scratch with tests to see what kinds of files you might need: https://github.com/sharpstone/force_absolute_paths_buildpack/pull/2.
+
+## Reference docs
+
+The `Hatchet::Runner.new` takes several arguments.
 
+### Init options
+
+- stack (String): The stack you want to deploy to on Heroku.
+
+```ruby
+Hatchet::Runner.new("default_ruby", stack: "heroku-16").deploy do |app|
+  # ...
+end
 ```
-$ heroku run --help
-run a one-off process inside a heroku dyno
 
-USAGE
-  $ heroku run
+- name (String): The name of an app you want to use. If you choose to provide your own app name, then Hatchet will not reap it, you'll have to delete it manually.
+- allow_failure (Boolean): If set to a truthy value then the test won't error if the deploy fails
+- labs (Array): Heroku has "labs" that are essentially features that are not enabled by default, one of the most popular ones is "preboot" https://devcenter.heroku.com/articles/preboot.
+- buildpacks (Array):  Pass in the buildpacks you want to use against your app
 
-OPTIONS
-  -a, --app=app        (required) app to run command against
-  -e, --env=env        environment variables to set (use ';' to split multiple vars)
-  -r, --remote=remote  git remote of app to use
-  -s, --size=size      dyno size
-  -x, --exit-code      passthrough the exit code of the remote command
-  --no-notify          disables notification when dyno is up (alternatively use HEROKU_NOTIFICATIONS=0)
-  --no-tty             force the command to not run in a tty
-  --type=type          process type
+```ruby
+Hatchet::Runner.new("default_ruby", buildpacks: ["heroku/nodejs", :default]).deploy do |app|
+  # ...
+end
 ```
 
-By default Hatchet will set the app name and the exit code
+In this example, the app would use the nodejs buildpack, and then `:default` gets replaced by your Git url and branch name.
 
+- before_deploy (Block): Instead of using the `tap` syntax you can provide a block directly to hatchet app initialization:
 
+```ruby
+Hatchet::Runner.new("default_ruby", before_deploy: ->{ FileUtils.touch("foo.txt")}).deploy do
+  # Assert stuff
+end
 ```
-app.run("exit 127")
-puts $?.exitcode
-# => 127
+
+A block in ruby is essentially an un-named method. Think of it as code to be executed later. See docs below for more info on blocks, procs and lambdas.
+
+- config (Hash): You can set config vars against your app:
+
+```ruby
+config = { "DEPLOY_TASKS" => "run:bloop", "FOO" => "bar" }
+Hatchet::Runner.new('default_ruby', config: config).deploy do |app|
+  expect(app.run("echo $DEPLOY_TASKS").to match("run:bloop")
+end
 ```
 
-To skip a value you can use the constant:
+> A hash in Ruby is like a dict in python. It is a set of key/value pairs. The syntax `=>` is called a "hashrocket" and is an alternative syntax to "json" syntax for hashes. It is used to allow for string keys instead of symbol keys.
 
+- `run_multi` (Boolean): Allows you to run more than a single "one-off" dyno at a time (the `HATCHET_EXPENSIVE_MODE` env var must be set to use this feature). By default, "free" Heroku apps are restricted to only allowing one dyno to run at a time. You can increase this limit by scaling an application to paid application, but it will incur charges against your application:
+
+```ruby
+Hatchet::Runner.new("default_ruby", run_multi: true).deploy do |app|
+  # This code runs in the background
+  app.run_multi("ls") do |out, status|
+    expect(status.success?).to be_truthy
+    expect(out).to include("Gemfile")
+  end
+
+  # This code runs in the background in parallel
+  app.run_multi("ruby -v") do |out, status|
+    expect(status.success?).to be_truthy
+    expect(out).to include("ruby")
+  end
+
+  # This line will be reached before either of the above blocks finish
+end
 ```
-app.run("exit 127", heroku: { "exit-code" => Hatchet::App::SkipDefaultOption})
-puts $?.exitcode
-# => 0
+
+In this example, the `heroku run ls` and `heroku run ruby -v` will be executed concurrently. The order that the `run_multi` blocks execute is not guaranteed. You can toggle this `run_multi` setting on globally by using `HATCHET_RUN_MULTI=1`. Without this setting enabled, you might need to add a `sleep` between multiple `app.run` invocations.
+
+WARNING: Enabling `run_multi` setting on an app will charge your Heroku account 🤑.
+WARNING: Do not use `run_multi` if you're not using the `deploy` block syntax or manually call `teardown!` inside the text context [more info about how behavior does not work with the `after` block syntax in rspec](https://github.com/heroku/hatchet/issues/110).
+WARNING: To work, `run_multi` requires your application to have a `web` process associated with it.
+
+### App methods:
+
+- `app.set_config()`: Updates the configuration on your app taking in a hash
+
+You can also update your config using the `set_config` method:
+
+```ruby
+app = Hatchet::Runner.new("default_ruby")
+app.set_config({"DEPLOY_TASKS" => "run:bloop", "FOO" => "bar"})
+app.deploy do
+  expect(app.run("echo $DEPLOY_TASKS").to match("run:bloop")
+end
 ```
 
-To specify a flag that has no value (such as `--no-notify`, `no-tty`, or `--exit-code`) pass a `nil` value:
+- `app.get_config()`: returns the Heroku value for a specific env var:
+
+```ruby
+app = Hatchet::Runner.new("default_ruby")
+app.set_config({"DEPLOY_TASKS" => "run:bloop", "FOO" => "bar"})
+app.get_config("DEPLOY_TASKS") # => "run:bloop"
+```
+
+- `app.set_lab()`: Enables the specified lab/feature on the app
+- `app.add_database()`: adds a database to the app, defaults to the "dev" database
+- `app.run()`: Runs a `heroku run bash` session with the arguments, covered above.
+- `app.run_multi()`: Runs a `heroku run bash` session in the background and yields the results. This requires the `run_multi` flag of the app to be set to `true`, which will charge your application (the `HATCHET_EXPENSIVE_MODE` env var must also be set to use this feature). Example above.
+- `app.create_app`: Can be used to manually create the app without deploying it (You probably want `setup!` though)
+- `app.setup!`: Gets the application in a state ready for deploy.
+  - Creates the Heroku app
+  - Sets up any specified labs (from initialization)
+  - Sets up any specified buildpacks
+  - Sets up specified config
+  - Calls the contents of the `before_deploy` block
+- `app.before_deploy`: Allows you to update the `before_deploy` block
 
+```ruby
+Hatchet::Runner.new("default_ruby").tap do |app|
+  app.before_deploy do
+    FileUtils.touch("foo.txt")
+  end
+  app.deploy do
+  end
+end
 ```
-app.run("echo 'foo'", heroku: { "no-notify" => nil })
-# This is the same as `heroku run echo 'foo' --no-notify`
+
+Has the same result as:
+
+```ruby
+before_deploy_proc = Proc.new do
+  FileUtils.touch("foo.txt")
+end
+
+Hatchet::Runner.new("default_ruby", before_deploy: before_deploy_proc).deploy do |app|
+end
+```
+
+- `app.commit!`: Will updates the contents of your local git dir if you've modified files on disk
+
+```ruby
+Hatchet::Runner.new("python_default").deploy do |app|
+  expect(app.output).to match(/Installing pip/)
+
+  # Redeploy with changed requirements file
+  run!(%Q{echo "" >> requirements.txt})
+  run!(%Q{echo "pygments" >> requirements.txt})
+
+  app.commit! # <=== Here
+
+  app.push!
+end
 ```
 
-## Modify Application Files on Disk
+> Note: Any changes to disk from a `before_deploy` block will be committed automatically after the block executes
 
-While template apps provided from your `hatchet.json` can provide different test cases, you may want to test minor varriations of an app. You can do this by using the `before_deploy` hook to modify files on disk inside of an app in a threadsafe way that will only affect the app's local instance:
+- `app.in_directory`: Runs the given block in a temp directory (but in the same process). One advanced debugging technique is to indefinitely pause test execution after outputting the directory so you can `cd` there and manually debug:
 
 ```ruby
-Hatchet::App.new("default_ruby", before_deploy: { FileUtils.touch("foo.txt")}).deploy do
-  # Assert stuff
+Hatchet::Runner.new("python_default").in_directory do |app|
+  puts "Temp dir is: #{Dir.pwd}"
+  STDIN.gets("foo") # <==== Pauses tests until stdin receives "foo"
 end
 ```
 
-After the `before_deploy` block fires, the results will be committed to git automatically before the app deploys.
+> Note: If you want to execute tests in this temp directory, you likely want to use `in_directory_fork` otherwise, you might accidentally contaminate the current environment's variables if you modify them.
 
-You can also manually call the `before_deploy` method:
+- `app.in_directory_fork`: Runs the given block in a temp directory and inside of a forked process, an example given above.
+- `app.directory`: Returns the directory of the example application on disk, this is NOT the temp directory that you're currently executing against. It's probably not what you want.
+- `app.deploy`: Your main method takes a block to execute after the deploy is successful. If no block is provided, you must manually call `app.teardown!` (see below for an example).
+- `app.output`: The output contents of the deploy
+- `app.platform_api`: Returns an instance of the [platform-api Heroku client](https://github.com/heroku/platform-api). If Hatchet doesn't give you access to a part of Heroku that you need, you can likely do it with the platform-api client.
+- `app.push!`: Push code to your Heroku app. It can be used inside of a `deploy` block to re-deploy.
+- `app.run_ci`: Runs Heroku CI against the app returns a TestRun object in the block
+- `app.teardown!`: This method is called automatically when using `app.deploy` in block mode after the deploy block finishes. When called it will clean up resources, mark the app as being finished (by setting `{"maintenance" => true}` on the app) so that the reaper knows it is safe to delete later. Here is an example of a test that creates and deploys an app manually, then later tears it down manually. If you deploy an application without calling `teardown!` then Hatchet will not know it is safe to delete and may keep it around for much longer than required for the test to finish.
 
 ```ruby
-app  = Hatchet::App.new("default_ruby")
-app.before_deploy do
-  FileUtils.touch("foo.txt")
+before(:each) do
+  @app = Hatchet::Runner.new("default_ruby")
+  @app.deploy
 end
-app.deploy do
-  # Assert stuff
+
+after(:each) do
+  @app.teardown! if @app
+end
+
+it "uses ruby" do
+  expect(@app.run("ruby -v")).to match("ruby")
 end
 ```
 
-Note: If you're going to shell out in this `before_deploy` section, you should check the success of your command, for example:
+- `test_run.run_again`: Runs the app again in Heroku CI
+- `test_run.status`: Returns the status of the CI run (possible values are `:pending`, `:building`, `:creating`, `:succeeded`, `:failed`, `:errored`)
+- `test_run.output`: The output of a given test run
+
+### ENV vars
+
+```sh
+HATCHET_BUILDPACK_BASE=https://github.com/heroku/heroku-buildpack-nodejs.git
+HATCHET_BUILDPACK_BRANCH=<branch name if you dont want Hatchet to set it for you>
+HATCHET_RETRIES=2
+HATCHET_APP_LIMIT=(set to something low like 20 locally, set higher like 80-100 on CI)
+HEROKU_API_KEY=<redacted>
+HEROKU_API_USER=<redacted@redacted.com>
+HATCHET_ALIVE_TTL_MINUTES=7
+
+# HATCHET_RUN_MULTI=1      # WARNING: Setting this env var will incur charges against your account. To use this env var you must also enable `HATCHET_EXPENSIVE_MODE`
+# HATCHET_EXPENSIVE_MODE=1 # WARNING: Do not set this environment variable unless you're okay with possibly large bills
+```
+
+> The syntax to set an env var in Ruby is `ENV["HATCHET_RETRIES"] = "2"` all env vars are strings.
+
+- `HATCHET_BUILDPACK_BASE`: This is the URL where Hatchet can find your buildpack. It must be public for Heroku to be able to use your buildpack.
+- `HATCHET_BUILDPACK_BRANCH`: By default, Hatchet will use your current git branch name. If, for some reason, git is not available or you want to manually specify it like `ENV["HATCHET_BUILDPACK_BRANCH'] = ENV[`MY_CI_BRANCH`]` then you can.
+- `HATCHET_RETRIES` If the `ENV['HATCHET_RETRIES']` is set to a number, deploys are expected to work and automatically retry that number of times. Due to testing using a network and random failures, setting this value to `3` retries seems to work well. If an app cannot be deployed within its allotted number of retries, an error will be raised. The downside of a larger number is that your suite will keep running for much longer when there are legitimate failures.
+- `HATCHET_APP_LIMIT`: The maximum number of **hatchet** apps that Hatchet will allow in the given account before running the reaper. For local execution, keep this low as you don't want your account dominated by hatchet apps. For CI, you want it to be much larger, 80-100 since it's not competing with non-hatchet apps. Your test runner account needs to be a dedicated account.
+- `HEROKU_API_KEY`: The API key of your test account user. If you run locally without this set, it will use your personal credentials.
+- `HEROKU_API_USER`: The email address of your user account. If you run locally without this set, it will use your personal credentials.
+- `HATCHET_RUN_MULTI`: If enabled, this will scale up deployed apps to "standard-1x" once deployed instead of running on the free tier. This enables the `run_multi` method capability, however scaling up is not free. WARNING: Setting this env var will incur charges to your Heroku account. We recommended never to enable this setting unless you work for Heroku. To use this you must also set `HATCHET_EXPENSIVE_MODE=1`
+- `HATCHET_EXPENSIVE_MODE`: This is intended to be a "safety" environment variable. If it is not set, Hatchet will prevent you from using the `run_multi: true` setting or the `HATCHET_RUN_MULTI` environment variables. There are still ways to incur charges without this feature, but unless you're absolutely confident your test setup will not leave "orphan" apps that are billing you, do not enable this setting. Even then, only set this value if you work for Heroku. To recap WARNING: setting this is expensive.
+
+## Basic
+
+### Basic rspec
+
+Hatchet needs to run inside of a test framework such as minitest or rspec. Here's an example of some existing test suites that use Hatchet: [This project](https://github.com/heroku/hatchet) uses rspec to run it's own tests you can use these as a reference as well as the [heroku-ruby-buildpack](https://github.com/heroku/heroku-buildpack-ruby). If you're new to Ruby, testing, or Hatchet, it is recommended to reference other project's tests heavily. If you can't pick between minitest and rspec, go with rspec since that's what most reference tests use.
+
+Whatever testing framework you chose, we recommend using a parallel test runner when running the full suite. [parallel_split_test](https://github.com/grosser/parallel_split_test).
+
+**rspec plugins**  - Rspec has useful plugins, such as `gem 'rspec-retry'` which will re-run any failed tests a given number of times (I recommend setting this to at least 2) to decrease false negatives in your tests when running on CI.
+
+Rspec is a testing framework for Ruby. It allows you to "describe" your tests using strings and blocks. This section is intended to be a brief introduction and includes a few pitfalls but is not comprehensive.
+
+In your directory rspec assumes a `spec/` folder. It's common to have a `spec_helper.rb` in the root of that folder:
+
+- **spec/spec_helper.rb**
+
+Here's an example of a `spec_helper.rb`: https://github.com/sharpstone/force_absolute_paths_buildpack/blob/master/spec/spec_helper.rb
+
+In this file, you'll require files you need to set up the project. You can also set environment variables like `ENV["HATCHET_BUILDPACK_BASE"]`. You can use it to configure your app. Any methods you define in this file will be available to your tests. For example:
 
 ```ruby
-before_deploy = Proc.new do
-  cmd = "bundle update"
-  output = `#{cmd}`
-  raise "Command #{cmd.inspect} failed unexpectedly with output: #{output}"
+def run!(cmd)
+  out = `#{cmd}`
+  raise "Error running #{cmd}, output: #{out}" unless $?.success?
+  out
 end
-Hatchet::App.new("default_ruby", before_deploy: before_deploy).deploy do
-  # Assert stuff
+```
+
+- **spec/hatchet/buildpack_spec.rb**
+
+Rspec knows a file is a test file or not by the name. It looks for files that end in `spec.rb` you can have as many as you want. I recommend putting them in a "spec/hatchet" sub-folder.
+
+- **File contents**
+
+In rspec you can group several tests under a "description" using `Rspec.describe`. Here's an example: https://github.com/sharpstone/force_absolute_paths_buildpack/blob/master/spec/hatchet/buildpack_spec.rb
+
+An empty example of `spec/hatchet/buildpack_spec.rb` would look like this:
+
+```ruby
+require_relative "../spec_helper.rb"
+
+RSpec.describe "This buildpack" do
+  it "accepts absolute paths at build and runtime" do
+    # expect(true).to eq(true)
+  end
 end
 ```
 
-It's helpful to make a helper function in your library if this pattern happens a lot in your app.
+Each `it` block represents a test case. If you ever get an error about no method `expect` it might be that you've forgotten to put your test case inside of a "describe" block.
 
-## Heroku CI
+- **expect syntax**
 
-Hatchet supports testing Heroku CI.
+Once inside of a test, you can assert an expected value against an actual value:
 
 ```ruby
-Hatchet::Runner.new("rails5_ruby_schema_format").run_ci do |test_run|
-  assert_match "Ruby buildpack tests completed successfully", test_run.output
+value = true
+expect(value).to eq(true)
+```
+
+This might look like a weird syntax, but it's valid ruby. It's shorthand for this:
+
+```ruby
+expect(value).to(eq(true))
+```
+
+Where `eq` is a method.
+
+If you want to assert the opposite, you can use `to_not`:
+
+```ruby
+expect(value).to_not eq(false)
+```
+
+- **matcher syntax**
+
+In the above example, the `eq` is called a "matcher". You're matching it against an object. In this case, you're looking for equality `==`.
+
+There are other matchers: https://relishapp.com/rspec/rspec-expectations/v/3-2/docs/built-in-matchers
+
+```ruby
+expect(value).to be_truthy
+
+value = "hello there"
+expect(value).to include("there")
+```
+
+Rspec uses some "magic" to convert anything you pass to
+
+Since most values in Hatchet are strings, the ones I use the most are:
+
+- Rspec matchers
+  - include https://relishapp.com/rspec/rspec-expectations/v/3-2/docs/built-in-matchers/include-matcher#string-usage
+  - match https://relishapp.com/rspec/rspec-expectations/v/3-2/docs/built-in-matchers/match-matcher
+
+Generally, I use the include when I know the exact value I want to assert against, I use match when there are dynamic values, and I want to be able to use a regular expression.
+
+For building regular expressions, I like to use the tool https://rubular.com/ for developing and testing regular expressions. Ruby's regular expression engine is mighty.
 
-  test_run.run_again # Runs tests again, for example to make sure the cache was used
 
-  assert_match "Using rake", test_run.output
+- **Keep it simple**
+
+Rspec is a massive library with a host of features. It's possible to quickly make your tests unmaintainable and unreadable in the efforts to keep your code [DRY](https://en.wikipedia.org/wiki/Don%27t_repeat_yourself). I recommend sticking to only the features mentioned here at first before trying to do anything fancy.
+
+- **What to test**
+
+Here's a PR with a description of several standard failure modes that lots of buildpacks should be aware of and reference implementations:
+
+https://github.com/heroku/heroku-buildpack-python/pull/969
+
+- **before(:all) gotcha**
+
+In rspec you can use `before` blocks to execute before a test, and `after` blocks to execute after a test. This might sound like you can deploy a hatchet app once and then write multiple tests against that app. However if `before(:all)` can be executed N times if you're running via parallel processes. Example:
+
+```ruby
+# Warning running `before(:all)` in a multi-process test runner context likely executes your
+# block N times where N is the number of tests in that context: https://github.com/grosser/parallel_split_test/pull/22/files
+before(:all) do
+  @app = Hatchet::Runner.new("default_ruby") # Warning: This is a gotcha
+  @app.deploy
+end
+
+after(:all) do
+  @app.teardown! if @app # Warning: This is a gotcha
+end
+
+it "tests app somehow" do
+  expect(@app.run("ruby -v")).to match("ruby") # Warning: This is a gotcha
+end
+
+
+it "tests app somehow 2" do
+  expect(@app.run("ls")).to match("Gemfile") # Warning: This is a gotcha
 end
 ```
 
-Call the `run_ci` method on the hatchet `Runner`. The object passed to the block is a `Hatchet::TestRun` object. You can call:
+Running this via the parallel_split_test gem will cause the `before(:all)` block to be invoked multiple times:
 
-- `test_run.output` will have the setup and test output of your tests.
-- `test_run.app` has a reference to the "app" you're testing against, however currently no `heroku create` is run (as it's not needed to run tests, only a pipeline and a blob of code).
+```
+$ PARALLEL_SPLIT_TEST_PROCESSES=3 bundle exec parallel_split_test spec/
+Hatchet setup: "hatchet-t-af7dffc006"
+Hatchet setup: "hatchet-t-bf7dffc006"
+```
 
-An exception will be raised if either the test times out or a status of `:errored` or `:failed` is returned. If you expect your test to fail, you can pass in `allow_failure: true` when creating your hatchet runner. If you do that, you'll also get access to different statuses:
+It would result in 2 apps being deployed. You can find more information [on the documentation](https://github.com/grosser/parallel_split_test#beforeall-rspec-hooks). For clarity of what will happen behind the scenes when running with multiple processes, it's recommended to use `before(:each)` instead of `before(:all)`.
 
-- `test_run.status` will return a symbol of the status of your test. Statuses include, but are not limited to `:pending`, `:building`, `:errored`, `:creating`, `:succeeded`, and `:failed`
+### Basic Ruby
 
-You can pass in a different timeout to the `run_ci` method `run_ci(timeout: 300)`.
+If you're not a Ruby specialist, not to worry. Here are a few things you might want to do:
 
-You probably need an `app.json` in the root directory of the app you're deploying. For example:
+- **Write a file and manipulate disk**
 
-```json
-{
-  "environments": {
-    "test": {
-      "addons":[
-         "heroku-postgresql"
-      ]
-    }
-  }
-}
+```ruby
+File.open("facts.txt", "w+") do |f|
+  f.write("equal does not mean equitable")
+end
 ```
 
-This is on [a Rails5 test app](https://github.com/sharpstone/rails5_ruby_schema_format/blob/master/app.json) that needs the database to run.
+The first argument is the file name, and the second is the object "mode", here `"w+"` means open for writing and create the file if it doesn't exist. If you want to append to a file instead you can use the mode `"a"`.
 
-Do **NOT** specify a `buildpacks` key in the `app.json` because Hatchet will automatically do this for you. If you need to set buildpacks you can pass them into the `buildpacks:` keword argument:
+The file name can be a relative or absolute path. My personal favorite though is using the Pathname class to represent files on disk [ruby Pathname api docs](https://ruby-doc.org/stdlib-2.7.1/libdoc/pathname/rdoc/Pathname.html). You can also use a pathname object to write and manipulate the disk directly:
 
+```ruby
+require 'pathname'
+Pathname.new("facts.txt").write("equal does not mean equitable")
 ```
-buildpacks = []
-buildpacks << "https://github.com/heroku/heroku-buildpack-pgbouncer.git"
-buildpacks << [HATCHET_BUILDPACK_BASE, HATCHET_BUILDPACK_BRANCH.call].join("#")
 
-Hatchet::Runner.new("rails5_ruby_schema_format", buildpacks: buildpacks).run_ci do |test_run|
-  # ...
+- API docs:
+  - [File](https://ruby-doc.org/core-2.7.0/File.html)
+  - [FileUtils](https://ruby-doc.org/stdlib-2.7.1/libdoc/fileutils/rdoc/FileUtils.html)
+  - [Pathname](https://ruby-doc.org/stdlib-2.7.1/libdoc/pathname/rdoc/Pathname.html)
+  - [Dir](https://ruby-doc.org/core-2.7.1/Dir.html)
+
+- **HEREDOC**
+
+You can define a multi-line string in Ruby using `<<~EOM` with a closing `EOM`. Technically, `EOM` can be any string, but you're not here for technicalities.
+
+```ruby
+File.open("bin/yarn", "w") do |f|
+  f.write <<~EOM
+    #! /usr/bin/env bash
+
+    echo "Called bin/yarn binstub"
+    `yarn install`
+  EOM
 end
 ```
 
-## Testing on CI
+This version of heredoc will strip out indentation:
 
-Once you've got your tests working locally, you'll likely want to get them running on CI. For reference see the [Circle CI config from this repo](https://github.com/heroku/hatchet/blob/master/.circleci/config.yml) and the [Heroku CI config from the ruby buildpack](https://github.com/heroku/heroku-buildpack-ruby/blob/master/app.json).
+```ruby
+puts <<~EOM
+           # Notice that the spaces are stripped out of the front of this string
+EOM
+# => "# Notice that the spaces are stripped out of the front of this string"
+```
 
-To make running on CI easier, there is a setup script in Hatchet that can be run before your tests are executed:
+The `~` Is usually the operator for a heredoc that you want, it's supported in Ruby 2.5+.
 
-```yml
-bundle exec hatchet ci:setup
+- **Hashes**
+
+A hash is like a dict in python. Docs: https://ruby-doc.org/core-2.7.1/Hash.html
+
+```ruby
+person_hash = { "name" => "schneems", "level" => 6 }
+puts person_hash["name"]
+# => "schneems"
 ```
 
-If you're a Heroku employee see [private instructions for setting up test users](https://github.com/heroku/languages-team/blob/master/guides/create_test_users_for_ci.md) to generate a user a grab the API token. Once you have an API token you'll want to set up these env vars with your CI provider:
+You can also mutate a hash:
 
+```ruby
+person_hash = { "name" => "schneems", "level" => 6 }
+person_hash["name"] = "Richard"
+puts person_hash["name"]
+# => "Richard"
 ```
-HATCHET_APP_LIMIT=100
-HATCHET_RETRIES=3
-HEROKU_API_KEY=<redacted>
-HEROKU_API_USER=<redacted@example.com>
+
+You can inspect full objects by calling `inspect` on them:
+
+```ruby
+puts person_hash.inspect
+# => {"name"=>"schneems", "level"=>6}
+```
+
+As an implementation detail note that hashes are ordered
+
+- **ENV**
+
+You can access the current processes' environment variables as a hash using the ENV object:
+
+```ruby
+ENV["MY_CUSTOM_ENV_VAR"] = "blm"
+puts `echo $MY_CUSTOM_ENV_VAR`.upcase
+# => BLM
+```
+
+All values in an env var must be a string. See the Hash docs for more information on manipulating hashes https://ruby-doc.org/core-2.7.1/Hash.html. Also see the current ENV docs https://ruby-doc.org/core-2.7.1/ENV.html.
+
+- **Strings versus symbols**
+
+In Ruby you can have a define a symbol `:thing` as well as a `"string"`. They look and behave very closely but are different. A symbol is a singleton object, while the string is unique object. One really confusing thing is you can have a hash with both string and symbol keys:
+
+```ruby
+my_hash = {}
+my_hash["dog"] = "cinco"
+my_hash[:dog] = "river"
+puts my_hash.inspect
+# => {"dog"=>"cinco", :dog=>"river"}
+```
+
+- **Blocks, procs, and lambdas**
+
+Blocks are a concept in Ruby for closure. Depending on how it's used it can be an anonymous method. It's always a method for passing around code. When you see `do |app|` that's the beginning of an implicit block. In addition to an implicit block you can create an explicit block using lambdas and procs. In Hatchet, these are most likely to be used to update the app `before_deploy`. Here's an example of some syntax for creating various blocks.
+
+```ruby
+before_deploy = -> { FileUtils.touch("foo.txt") } # This syntax is called a "stabby lambda"
+before_deploy = lambda { FileUtils.touch("foo.txt") } # This is a more verbose lambda
+before_deploy = lambda do
+  FileUtils.touch("foo.txt") # Multi-line lambda
+end
+before_deploy = Proc.new { FileUtils.touch("foo.txt") } # A proc and lambda are subtly different, it mostly won't matter to you though
+before_deploy = Proc.new do
+  FileUtils.touch("foo.txt") # Multi-line proc
+end
+```
+
+All of these things do the same thing more-or-less. You can execute a block/proc/lambda by running:
+
+```ruby
+before_deploy.call
+```
+
+- **Parens**
+
+You might have noticed that some ruby methods use parens and some don't. I.e. `puts "yo"` versus `puts("yo")`. If the parser can determine your intent then you don't have to use parens.
+
+- **Debugging**
+
+If you're not used to debugging Ruby you can reference the [Ruby debugging magic cheat sheet](https://www.schneems.com/2016/01/25/ruby-debugging-magic-cheat-sheet.html). The Ruby language is very powerful in it's ability to [reflect on itself](https://en.wikipedia.org/wiki/Reflection_%28computer_programming%29). Essentially the Ruby code is able to introspect itself to tell you what it's doing. If you're ever lost, ask your ruby code. It might confuse you, but it won't lie to you.
+
+Another good debugging tool is the [Pry debugger and repl](https://github.com/pry/pry).
+
+- **Common Ruby errors**
+
+```
+SyntaxError ((irb):14: syntax error, unexpected `end')
 ```
 
-## Extra App Commands
+If you see this, it likely means you forgot a `do` on a block, for example `.deploy |app|` instead of `.deploy do |app|`.
 
 ```
-app.add_database # adds a database to specified app
-app.heroku       # returns a Herou Api client https://github.com/heroku/heroku.rb
+NoMethodError (undefined method `upcase' for nil:NilClass)
 ```
 
+If you see this it means a variable you're using is `nil` unexpectedly. You'll need to use the [above debugging techniques](https://www.schneems.com/2016/01/25/ruby-debugging-magic-cheat-sheet.html) to figure out why.
+
+- **More**
+
+Ruby is full of multitudes, this isn't even close to being exhaustive, just enough to make you dangerous and write a few tests. It's infinitely useful for testing, writing CLIs and web apps.
+
 ## Hatchet CLI
 
 Hatchet has a CLI for installing and maintaining external repos you're
 using to test against. If you have Hatchet installed as a gem run
 
-    $ hatchet --help
+    $ Hatchet --help
 
 For more info on commands. If you're using the source code you can run
 the command by going to the source code directory and running:
@@ -407,8 +983,30 @@ the command by going to the source code directory and running:
     $ ./bin/hatchet --help
 
 
-## License
+## Developing Hatchet
 
-MIT
+If you want to add a feature to Hatchet (this library) you'll need to install it locally and be able to run the tests:
+
+
+## Install locally
+
+```
+$ git clone https://github.com/heroku/hatchet
+$ cd hatchet
+$ bundle install
+```
+
+### Run the Tests
 
+```
+$ PARALLEL_SPLIT_TEST_PROCESSES=10 bundle exec parallel_split_test spec/
+```
+This will execute all tests, you can also run a single test by specifying a file and line number:
 
+```
+$ bundle exec rspec spec/hatchet/app_spec.rb:4
+```
+
+## License
+
+MIT