quarantine 1.0.3 → 2.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: dce45c3ec38cb39613ea7e1e5dff68524fcbfd3a7005c879985b41ba62343688
-  data.tar.gz: ba098ba05cc35e0aedc90db91907b1767556718520e11bd8dcb2240b52872342
+  metadata.gz: 47aa8e8df1db802259f0107a7a06c2d05162899b803ffd224da8a53246c1e105
+  data.tar.gz: 8cb6c6c9c5b79666c10feaf38e0077eab27b57d494dc08e0c385925de1c89583
 SHA512:
-  metadata.gz: dda9843f287d437087882386da70ed78876b1592e9e803d6b62d33b609baf366e457b2e91a6ac817ea9617316679845e9fa793deafde01ddd4eb3f481dc5af15
-  data.tar.gz: 6708068fbac697316d51a07a4b1fcc58e48b832aa074afb8ee7794142206eccbb556f5ca7726d0694816011883f3a321304565e1289036aec2a5ac640e1cf4f6
+  metadata.gz: e3f9ba5314c425861f18b7451a7f28f8ced4af50479ad880f0be263a9104ba3c6dd85bcada8a0c89d194e9487d02702c9fe4930caceebf835413891fc31d6a37
+  data.tar.gz: 3177827c0ff075302fa8e62c0b3480728769a3cb6199cb8eb0e510237df9839d0efc10db0180d32016d1ced933d9d50f87336f5f2d6ff3f741622a5aed3364b2

data/CHANGELOG.md

@@ -1,9 +1,20 @@
+### 1.0.7
+Support `it_behaves_like` behavior
+
+### 1.0.6
+Update DynamoDB batch_write_item implementation to check for duplicates based on different keys before uploading
+
+### 1.0.5
+Add aws_credentials argument during dynamodb initialization to override the AWS SDK credential chain
+
+### 1.0.4
+Enable upstream callers to mark an example as flaky through the example's metadata
+
 ### 1.0.3
-  Only require dynamodb instead of full aws-sdk
+Only require dynamodb instead of full aws-sdk
 
 ### 1.0.2
-  Relax Aws gem version constraint as aws-sdk v3 is 100% backwards
-  compatible with v2
+Relax Aws gem version constraint as aws-sdk v3 is 100% backwards compatible with v2
 
 ### 1.0.1
-  Initial Release
+Initial Release

data/README.md

@@ -1,154 +1,152 @@
 # Quarantine
+
 [![Build Status](https://travis-ci.com/flexport/quarantine.svg?branch=master)](https://travis-ci.com/flexport/quarantine)
 
-Quarantine provides a run-time solution to diagnosing and disabling flaky tests and automates the workflow around test suite maintenance.
+Quarantine automatically detects flaky tests (i.e. those which fail non-deterministically) and disables them until they're proven reliable.
 
-The quarantine gem supports testing frameworks:
+Quarantine current supports the following testing frameworks. If you need an additional one, please file an issue or open a pull request.
 - [RSpec](http://rspec.info/)
 
-The quarantine gem supports CI pipelines:
-- [Buildkite](https://buildkite.com/docs/tutorials/getting-started)
-
-If you are interested in using quarantine but it does not support your CI or testing framework, feel free to reach out or create an issue and we can try to make it happen.
+Quarantine should provide the necessary hooks for compatibility with any CI solution. If it's insufficient for yours, please file an issue or open a pull request.
 
-## Purpose
-Flaky tests impact engineering velocity, reduce faith in test reliablity and give a false representation of code coverage. Managing flaky tests is a clunky process that involves constant build monitorization, difficult diagnosis and manual ticket creation. As a result, here at Flexport, we have created a Gem to automate the entire process to help improve the workflow and keep our massive test suites in pristine condition.
+## Getting started
 
-The workflow at Flexport involves:
+Quarantine works in tandem with [RSpec::Retry](https://github.com/NoRedInk/rspec-retry). Add this to your `Gemfile` and run `bundle install`:
 
-![ideal workflow](misc/flexport_workflow.png)
-
----
-## Installation and Setup
-
-Add these lines to your application's Gemfile:
 ```rb
 group :test do
   gem 'quarantine'
-  gem 'rspec-retry
+  gem 'rspec-retry'
 end
 ```
 
-And then execute:
-```sh
-bundle install
-```
+In your `spec_helper.rb`, set up Quarantine and RSpec::Retry. See [RSpec::Retry](https://github.com/NoRedInk/rspec-retry)'s documentation for details of its configuration.
 
-In your `spec_helper.rb` setup quarantine and rspec-retry gem. Click [rspec-retry](https://github.com/NoRedInk/rspec-retry) to get a more detailed explaination on rspec-retry configurations and how to setup.
 ```rb
 require 'quarantine'
-require 'rspec-retry'
+require 'rspec/retry'
 
-Quarantine.bind({database: :dynamodb, aws_region: 'us-west-1'})
+Quarantine::RSpecAdapter.bind
 
 RSpec.configure do |config|
+  # Also accepts `:credentials` to override the standard AWS credential chain
+  config.quarantine_database = {type: :dynamodb, region: 'us-west-1'}
+  # Prevent the list of flaky tests from being polluted by local development and PRs
+  config.quarantine_record_tests = ENV["CI"] && ENV["BRANCH"] == "master"
 
-  config.around(:each) do |ex|
-    ex.run_with_retry(retry: 3)
+  config.around(:each) do |example|
+    example.run_with_retry(retry: 3)
   end
-
 end
 ```
 
-Consider wrapping `Quarantine.bind` in if statements so local flaky tests don't pollute the list of quarantined tests
+Quarantine comes with a CLI tool for setting up the necessary table in DynamoDB, if you use that database.
 
-```rb
-if ENV[CI] && ENV[BRANCH] == "master"
-  Quarantine.bind({database: :dynamodb, aws_region: 'us-west-1'})
-end
-```
-
-Setup tables in AWS DynamoDB to support pulling and uploading quarantined tests
 ```sh
-bundle exec quarantine_dynamodb -h    # see all options
+bundle exec quarantine_dynamodb -h    # See all options
 
-bundle exec quarantine_dynamodb \     # create the tables in us-west-1 in aws dynamodb
-  --aws_region us-west-1              # with "quarantine_list" and "master_failed_tests"
-                                      # table names
+bundle exec quarantine_dynamodb \     # Create the "test_statuses" table in us-west-1 in AWS DynamoDB
+  --region us-west-1
 ```
 
-You are all set to start quarantining tests!
+## How It Works
+
+A flaky test fails on the first run, but passes after being retried via RSpec::Retry.
 
-## Try Quarantining Tests Locally
-Add a test that will flake
 ```rb
 require "spec_helper"
 
 describe Quarantine do
-  it "this test should flake 33% of the time" do
-    Random.rand(3).to eq(3)
+  it "fails on the first run" do
+    raise "error" if RSpec.current_example.attempts == 0
+    # otherwise, pass
   end
 end
 ```
 
-Run `rspec` on the test
 ```sh
-CI=1 BRANCH=master rspec <filename>
+$ CI=1 BRANCH=master bundle exec rspec <filename>
+[quarantine] Quarantined tests:
+  ./bar_spec.rb[1:1] Quarantine fails on the first run
 ```
 
-If the test fails and passes on the test run (rspec-retry re-ran the test), the test should be quarantined and uploaded to DynamoDB. Check the `quarantine_list` table in DynamoDB.
+When the build completes, all test statuses are written to the database. Flaky tests are marked `quarantined`, and will be executed in future builds, but any failures will be ignored.
+
+A test can be removed from quarantine by updating the database manually (outside this gem), or by configuring `quarantine_release_at_consecutive_passes` to remove it after it passes on a certain number of builds in a row.
 
 ## Configuration
 
-Go to `spec/spec_helper.rb` and set configuration variables through:
+In `spec_helper.rb`, you can set configuration variables by doing:
+
 ```rb
 RSpec.configure do |config|
-    config.VAR_NAME = VALUE
+  config.VAR_NAME = VALUE
 end
 ```
-- Table name for quarantined tests `:quarantine_list_table, default: "quarantine_list"`
 
-- Table name where failed test are uploaded `:quarantine_failed_tests_table, default: "master_failed_tests"`
+- `quarantine_database`: Database configuration (see below), default: `{ type: :dynamodb, region: 'us-west-1' }`
+- `test_statusus_table`: Table name for test statuses, default: `"test_statuses"`
+- `skip_quarantined_tests`: Skipping quarantined tests during test runs default: `true`
+- `quarantine_record_tests`: Recording test statuses, default: `true`
+- `quarantine_logging`: Outputting quarantined gem info, default: `true`
+- `quarantine_extra_attributes`: Storing custom per-example attributes in the table, default: `nil`
+- `quarantine_failsafe_limit`: A failsafe limit of quarantined tests in a single run, default: `10`
+- `quarantine_release_at_consecutive_passes`: Releasing a test from quarantine after it records enough consecutive passes (`nil` to disable this feature), default: `nil`
 
-- Quarantined tests are not skipped automatically `:skip_quarantined_tests, default: true`
+### Databases
 
-- Recording failed tests `:quarantine_record_failed_tests, default: true`
+Quarantine comes with built-in support for the following database types:
+- `:dynamodb`
 
-- Recording flaky tests `:quarantine_record_flaky_tests, default: true`
+To use `:dynamodb`, be sure to add `gem 'aws-sdk-dynamodb', '~> 1', group: :test` to your `Gemfile`.
 
-- Outputting quarantined gem info `:quarantine_logging, default: true`
+To use a custom database that's not provided, subclass `Quarantine::Databases::Base` and pass an instance of your class as the `quarantine_database` setting:
 
----
-## Setup Jira Workflow
+```rb
+class MyDatabase < Quarantine::Databases::Base
+  ...
+end
+
+RSpec.configure do |config|
+  config.quarantine_database = MyDatabase.new(...)
+end
+```
 
-To automatically create Jira tickets, take a look at: `examples/create_tickets.rb`
+### Extra attributes
 
-To automatically unquarantine tests on Jira ticket completion, take a look at: `examples/unquarantine.rb`
+Use `quarantine_extra_attributes` to store custom data with each test in the database, e.g. variables useful for your CI setup.
+
+```rb
+ config.quarantine_extra_attributes = Proc.new do |example|
+   {
+     build_url: ENV['BUILDKITE_BUILD_URL'],
+     job_id: ENV['BUILDKITE_JOB_ID'],
+   }
+ end
+```
 
----
-## Contributing
-1. Create an issue regarding a bug fix or feature request
-2. Fork the repository
-3. Create your feature branch, commit changes and create a pull request
-
-## Contributors
-- [Eric Zhu](https://github.com/eric-zhu-uw)
-- [Kevin Miller](https://github.com/Gasparila)
-- [Nicholas Silva](https://github.com/flexportnes)
-- [Ankur Dahiya](https://github.com/legalosLOTR)
 ---
 
 ## FAQs
 
 #### Why are quarantined tests not being skipped locally?
 
-The `quarantine` gem may be configured to only run on certain environments. Make sure you pass all these `ENV` variables to `rspec` when you call it locally
+Quarantine may be configured to only run in certain environments. Check your `spec_helper.rb`, and make sure you have all necessary environment variables set, e.g.:
 
 ```sh
-CI="1" BRANCH="master" rspec
+CI=1 BRANCH=master bundle exec rspec
 ```
 
-#### Why is dynamodb failing to connect?
+#### Why is Quarantine failing to connect to DynamoDB?
 
-The AWS client loads credentials from the following locations:
+The AWS client attempts to loads credentials from the following locations, in order:
+- The optional `credentials` field in `RSpec.configuration.quarantine_database`
 - `ENV['AWS_ACCESS_KEY_ID']` and `ENV['AWS_SECRET_ACCESS_KEY']`
 - `Aws.config[:credentials]`
 - The shared credentials ini file at `~/.aws/credentials`
 
-To get AWS credentials, please contact your AWS administrator to get access to dynamodb and create your credentials through IAM.
-
-More detailed information can be found: [AWS documentation](https://docs.aws.amazon.com/sdkforruby/api/Aws/S3/Client.html)
+More detailed information can be found in the [AWS SDK documentation](https://docs.aws.amazon.com/sdkforruby/api/Aws/S3/Client.html)
 
 #### Why is `example.clear_exception` failing locally?
- 
- `example.clear_exception` is an attribute added through `rspec_retry`. Make sure `rspec-retry` has been installed and configured.
+
+`Example#clear_exception` is an attribute added through [RSpec::Retry](https://github.com/NoRedInk/rspec-retry). Make sure it has been installed and configured.

data/lib/quarantine.rb

@@ -1,141 +1,156 @@
+# typed: strict
+
+require 'sorbet-runtime'
+
 require 'rspec/retry'
 require 'quarantine/rspec_adapter'
 require 'quarantine/test'
 require 'quarantine/databases/base'
 require 'quarantine/databases/dynamo_db'
 
-class Quarantine
-  extend RSpecAdapter
-
-  attr_accessor :database
-  attr_reader :quarantine_map
-  attr_reader :failed_tests
-  attr_reader :flaky_tests
-  attr_reader :duplicate_tests
-  attr_reader :buildkite_build_number
-  attr_reader :summary
-
-  def initialize(options = {})
-    case options[:database]
-    # default database option is dynamodb
-    when :dynamodb, nil
-      @database = Quarantine::Databases::DynamoDB.new(options)
-    else
-      raise Quarantine::UnsupportedDatabaseError.new("Quarantine does not support #{options[:database]}")
+module RSpec
+  module Core
+    class Example
+      extend T::Sig
+
+      # The implementation of clear_exception in rspec-retry doesn't work
+      # for examples that use `it_behaves_like`, so we implement our own version that
+      # clear the exception field recursively.
+      sig { void }
+      def clear_exception!
+        @exception = T.let(nil, T.untyped)
+        T.unsafe(self).example.clear_exception! if defined?(example)
+      end
     end
+  end
+end
+
+class Quarantine
+  extend T::Sig
+
+  sig { returns(T::Hash[String, Quarantine::Test]) }
+  attr_reader :tests
+
+  sig { returns(T::Hash[String, Quarantine::Test]) }
+  attr_reader :old_tests
 
-    @quarantine_map = {}
-    @failed_tests = []
-    @flaky_tests = []
-    @buildkite_build_number = ENV['BUILDKITE_BUILD_NUMBER'] || '-1'
-    @summary = { id: 'quarantine', quarantined_tests: [], flaky_tests: [], database_failures: [] }
+  sig { params(options: T::Hash[T.untyped, T.untyped]).void }
+  def initialize(options)
+    @options = options
+    @old_tests = T.let({}, T::Hash[String, Quarantine::Test])
+    @tests = T.let({}, T::Hash[String, Quarantine::Test])
+    @database_failures = T.let([], T::Array[String])
+    @database = T.let(nil, T.nilable(Quarantine::Databases::Base))
   end
 
-  # Scans the quarantine_list from the database and store the individual tests in quarantine_map
-  def fetch_quarantine_list
+  sig { returns(Quarantine::Databases::Base) }
+  def database
+    @database ||=
+      case @options[:database]
+      when Quarantine::Databases::Base
+        @options[:database]
+      else
+        database_options = @options[:database].dup
+        type = database_options.delete(:type)
+        case type
+        when :dynamodb
+          Quarantine::Databases::DynamoDB.new(database_options)
+        else
+          raise Quarantine::UnsupportedDatabaseError.new("Quarantine does not support database type: #{type.inspect}")
+        end
+      end
+  end
+
+  # Scans the test_statuses from the database and store their IDs in quarantined_ids
+  sig { void }
+  def fetch_test_statuses
     begin
-      quarantine_list = database.scan(RSpec.configuration.quarantine_list_table)
+      test_statuses = database.fetch_items(@options[:test_statuses_table_name])
     rescue Quarantine::DatabaseError => e
-      add_to_summary(:database_failures, "#{e&.cause&.class}: #{e&.cause&.message}")
+      @database_failures << "#{e.cause&.class}: #{e.cause&.message}"
       raise Quarantine::DatabaseError.new(
         <<~ERROR_MSG
-          Failed to pull the quarantine list from #{RSpec.configuration.quarantine_list_table}
-          because of #{e&.cause&.class}: #{e&.cause&.message}
+          Failed to pull the quarantine list from #{@options[:test_statuses_table_name]}
+          because of #{e.cause&.class}: #{e.cause&.message}
         ERROR_MSG
       )
     end
 
-    quarantine_list.each do |example|
-      # on the rare occassion there are duplicate tests ids in the quarantine_list,
-      # quarantine the most recent instance of the test (det. through build_number)
-      # and ignore the older instance of the test
-      next if
-        quarantine_map.key?(example['id']) &&
-        example['build_number'].to_i < quarantine_map[example['id']].build_number.to_i
-
-      quarantine_map.store(
-        example['id'],
-        Quarantine::Test.new(example['id'], example['full_description'], example['location'], example['build_number'])
-      )
-    end
+    pairs =
+      test_statuses
+      .group_by { |t| t['id'] }
+      .map { |_id, tests| tests.max_by { |t| t['created_at'] } }
+      .compact
+      .filter { |t| t['last_status'] == 'quarantined' }
+      .map do |t|
+        [
+          t['id'],
+          Quarantine::Test.new(
+            id: t['id'],
+            status: t['last_status'].to_sym,
+            consecutive_passes: t['consecutive_passes'].to_i,
+            full_description: t['full_description'],
+            location: t['location'],
+            extra_attributes: t['extra_attributes']
+          )
+        ]
+      end
+
+    @old_tests = Hash[pairs]
   end
 
-  # Based off the type, upload a list of tests to a particular database table
-  def upload_tests(type)
-    if type == :failed
-      tests = failed_tests
-      table_name = RSpec.configuration.quarantine_failed_tests_table
-    elsif type == :flaky
-      tests = flaky_tests
-      table_name = RSpec.configuration.quarantine_list_table
-    else
-      raise Quarantine::UnknownUploadError.new(
-        "Quarantine gem did not know how to handle #{type} upload of tests to dynamodb"
-      )
-    end
-
-    return unless tests.length < 10 && tests.length > 0
+  sig { void }
+  def upload_tests
+    return if @tests.empty? || @tests.values.count { |test| test.status == :quarantined } >= @options[:failsafe_limit]
 
     begin
       timestamp = Time.now.to_i / 1000 # Truncated millisecond from timestamp for reasons specific to Flexport
-      database.batch_write_item(
-        table_name,
-        tests,
-        {
-          build_job_id: ENV['BUILDKITE_JOB_ID'] || '-1',
-          created_at: timestamp,
-          updated_at: timestamp
-        }
+      database.write_items(
+        @options[:test_statuses_table_name],
+        @tests.values.map { |item| item.to_hash.merge(updated_at: timestamp) }
       )
     rescue Quarantine::DatabaseError => e
-      add_to_summary(:database_failures, "#{e&.cause&.class}: #{e&.cause&.message}")
+      @database_failures << "#{e.cause&.class}: #{e.cause&.message}"
     end
   end
 
   # Param: RSpec::Core::Example
-  # Add the example to the internal failed tests list
-  def record_failed_test(example)
-    failed_tests << Quarantine::Test.new(
-      example.id,
-      example.full_description,
-      example.location,
-      buildkite_build_number
-    )
-  end
-
-  # Param: RSpec::Core::Example
-  # Add the example to the internal flaky tests list
-  def record_flaky_test(example)
-    flaky_test = Quarantine::Test.new(
-      example.id,
-      example.full_description,
-      example.location,
-      buildkite_build_number
+  # Add the example to the internal tests list
+  sig { params(example: T.untyped, status: Symbol, passed: T::Boolean).void }
+  def record_test(example, status, passed:)
+    extra_attributes = @options[:extra_attributes] ? @options[:extra_attributes].call(example) : {}
+
+    new_consecutive_passes = passed ? (@old_tests[example.id]&.consecutive_passes || 0) + 1 : 0
+    release_at = @options[:release_at_consecutive_passes]
+    new_status = !release_at.nil? && new_consecutive_passes >= release_at ? :passing : status
+    test = Quarantine::Test.new(
+      id: example.id,
+      status: new_status,
+      consecutive_passes: new_consecutive_passes,
+      full_description: example.full_description,
+      location: example.location,
+      extra_attributes: extra_attributes
     )
 
-    flaky_tests << flaky_test
-    add_to_summary(:flaky_tests, flaky_test.id)
-  end
-
-  # Param: RSpec::Core::Example
-  # Clear exceptions on a flaky tests that has been quarantined
-  #
-  # example.clear_exception is tightly coupled with the rspec-retry gem and will only exist if
-  # the rspec-retry gem is enabled
-  def pass_flaky_test(example)
-    example.clear_exception
-    add_to_summary(:quarantined_tests, example.id)
+    @tests[test.id] = test
   end
 
   # Param: RSpec::Core::Example
-  # Check the internal quarantine_map to see if this test should be quarantined
+  # Check the internal old_tests to see if this test should be quarantined
+  sig { params(example: T.untyped).returns(T::Boolean) }
   def test_quarantined?(example)
-    quarantine_map.key?(example.id)
+    @old_tests[example.id]&.status == :quarantined
   end
 
-  # Param: Symbol, Any
-  # Adds the item to the specified attribute in summary
-  def add_to_summary(attribute, item)
-    summary[attribute] << item if summary.key?(attribute)
+  sig { returns(String) }
+  def summary
+    quarantined_tests = @tests.values.select { |test| test.status == :quarantined }.sort_by(&:id)
+    <<~MESSAGE
+      \n[quarantine] Quarantined tests:
+        #{quarantined_tests.map { |test| "#{test.id} #{test.full_description}" }.join("\n  ")}
+
+      [quarantine] Database errors:
+        #{@database_failures.join("\n  ")}
+    MESSAGE
   end
 end

data/lib/quarantine/cli.rb

@@ -1,43 +1,42 @@
+# typed: strict
+
 require 'optparse'
 require_relative 'databases/base'
 require_relative 'databases/dynamo_db'
 
 class Quarantine
   class CLI
-    attr_accessor :options
+    extend T::Sig
+
+    sig { returns(T::Hash[T.untyped, T.untyped]) }
+    attr_reader :options
 
+    sig { void }
     def initialize
       # default options
-      @options = {
-        quarantine_list_table_name: 'quarantine_list',
-        failed_test_table_name: 'master_failed_tests'
-      }
+      @options = T.let(
+        {
+          test_statuses_table_name: 'test_statuses'
+        }, T::Hash[Symbol, T.untyped]
+      )
     end
 
+    sig { void }
     def parse
       OptionParser.new do |parser|
         parser.banner = 'Usage: quarantine_dynamodb [options]'
 
-        parser.on('-rREGION', '--aws_region=REGION', String, 'Specify the aws region for DynamoDB') do |aws_region|
-          options[:aws_region] = aws_region
+        parser.on('-rREGION', '--region=REGION', String, 'Specify the aws region for DynamoDB') do |region|
+          @options[:region] = region
         end
 
         parser.on(
           '-qTABLE',
           '--quarantine_table=TABLE',
           String,
-          "Specify the table name for the quarantine list | Default: #{options[:quarantine_list_table_name]}"
-        ) do |table_name|
-          options[:quarantine_list_table_name] = table_name
-        end
-
-        parser.on(
-          '-fTABLE',
-          '--failed_table=TABLE',
-          String,
-          "Specify the table name for the failed test list | Default: #{options[:failed_test_table_name]}"
+          "Specify the table name for the quarantine list | Default: #{@options[:test_statuses_table_name]}"
         ) do |table_name|
-          options[:failed_test_table_name] = table_name
+          @options[:test_statuses_table_name] = table_name
         end
 
         parser.on('-h', '--help', 'Prints help page') do
@@ -46,7 +45,7 @@ class Quarantine
         end
       end.parse!
 
-      if options[:aws_region].nil?
+      if @options[:region].nil?
         error_msg = 'Failed to specify the required aws region with -r option'.freeze
         warn error_msg
         raise ArgumentError.new(error_msg)
@@ -54,12 +53,12 @@ class Quarantine
     end
 
     # TODO: eventually move to a separate file & create_table by db type when my db adapters
+    sig { void }
     def create_tables
-      dynamodb = Quarantine::Databases::DynamoDB.new(options)
+      dynamodb = Quarantine::Databases::DynamoDB.new(region: @options[:region])
 
       attributes = [
-        { attribute_name: 'id', attribute_type: 'S', key_type: 'HASH' },
-        { attribute_name: 'build_number', attribute_type: 'S', key_type: 'RANGE' }
+        { attribute_name: 'id', attribute_type: 'S', key_type: 'HASH' }
       ]
 
       additional_arguments = {
@@ -70,15 +69,9 @@ class Quarantine
       }
 
       begin
-        dynamodb.create_table(options[:quarantine_list_table_name], attributes, additional_arguments)
-      rescue Quarantine::DatabaseError => e
-        warn "#{e&.cause&.class}: #{e&.cause&.message}"
-      end
-
-      begin
-        dynamodb.create_table(options[:failed_test_table_name], attributes, additional_arguments)
+        dynamodb.create_table(@options[:test_statuses_table_name], attributes, additional_arguments)
       rescue Quarantine::DatabaseError => e
-        warn "#{e&.cause&.class}: #{e&.cause&.message}"
+        warn "#{e.cause&.class}: #{e.cause&.message}"
       end
     end
   end

data/lib/quarantine/databases/base.rb

@@ -1,17 +1,34 @@
+# typed: strict
+
 class Quarantine
   module Databases
     class Base
-      def initialize
-        raise NotImplementedError
-      end
+      extend T::Sig
+      extend T::Helpers
 
-      def scan
-        raise NotImplementedError
+      abstract!
+
+      Item = T.type_alias do
+        {
+          'id' => String,
+          'last_status' => String,
+          'consecutive_passes' => Integer,
+          'full_description' => String,
+          'location' => String,
+          'extra_attributes' => T.untyped
+        }
       end
 
-      def batch_write_item
-        raise NotImplementedError
+      sig { abstract.params(table_name: String).returns(T::Enumerable[Item]) }
+      def fetch_items(table_name); end
+
+      sig do
+        abstract.params(
+          table_name: String,
+          items: T::Array[Item]
+        ).void
       end
+      def write_items(table_name, items); end
     end
   end
 end

data/lib/quarantine/databases/dynamo_db.rb

@@ -1,57 +1,83 @@
-require 'aws-sdk-dynamodb'
+# typed: strict
+
+begin
+  require 'aws-sdk-dynamodb'
+rescue LoadError
+end
 require 'quarantine/databases/base'
 require 'quarantine/error'
 
 class Quarantine
   module Databases
     class DynamoDB < Base
+      extend T::Sig
+
+      Attribute = T.type_alias { { attribute_name: String, attribute_type: String, key_type: String } }
+
+      sig { returns(Aws::DynamoDB::Client) }
       attr_accessor :dynamodb
 
-      def initialize(aws_region: 'us-west-1', **_additional_arguments)
-        @dynamodb = Aws::DynamoDB::Client.new({ region: aws_region })
+      sig { params(options: T::Hash[T.untyped, T.untyped]).void }
+      def initialize(options)
+        super()
+
+        @dynamodb = T.let(Aws::DynamoDB::Client.new(options), Aws::DynamoDB::Client)
       end
 
-      def scan(table_name)
+      sig { override.params(table_name: String).returns(T::Enumerable[Item]) }
+      def fetch_items(table_name)
         begin
-          result = dynamodb.scan({ table_name: table_name })
+          result = @dynamodb.scan(table_name: table_name)
         rescue Aws::DynamoDB::Errors::ServiceError
           raise Quarantine::DatabaseError
         end
 
-        result&.items
+        result.items
       end
 
-      def batch_write_item(table_name, items, additional_attributes = {})
-        dynamodb.batch_write_item(
-          { request_items: {
+      sig do
+        override.params(
+          table_name: String,
+          items: T::Array[Item]
+        ).void
+      end
+      def write_items(table_name, items)
+        @dynamodb.batch_write_item(
+          request_items: {
             table_name => items.map do |item|
               {
                 put_request: {
-                  item: { **item.to_hash, **additional_attributes }
+                  item: item
                 }
               }
             end
-          } }
+          }
         )
       rescue Aws::DynamoDB::Errors::ServiceError
         raise Quarantine::DatabaseError
       end
 
-      def delete_item(table_name, keys)
-        dynamodb.delete_item(
-          {
-            table_name: table_name,
-            key: {
-              **keys
-            }
+      sig { params(table_name: String, keys: T::Hash[T.untyped, T.untyped]).void }
+      def delete_items(table_name, keys)
+        @dynamodb.delete_item(
+          table_name: table_name,
+          key: {
+            **keys
           }
         )
       rescue Aws::DynamoDB::Errors::ServiceError
         raise Quarantine::DatabaseError
       end
 
+      sig do
+        params(
+          table_name: String,
+          attributes: T::Array[Attribute],
+          additional_arguments: T::Hash[T.untyped, T.untyped]
+        ).void
+      end
       def create_table(table_name, attributes, additional_arguments = {})
-        dynamodb.create_table(
+        @dynamodb.create_table(
           {
             table_name: table_name,
             attribute_definitions: attributes.map do |attribute|

data/lib/quarantine/error.rb

@@ -1,3 +1,5 @@
+# typed: strict
+
 class Quarantine
   class Error < StandardError; end
 

data/lib/quarantine/rspec_adapter.rb

@@ -1,84 +1,105 @@
-module RSpecAdapter
+# typed: strict
+
+module Quarantine::RSpecAdapter # rubocop:disable Style/ClassAndModuleChildren
+  extend T::Sig
+
   # Purpose: create an instance of Quarantine which contains information
   #          about the test suite (ie. quarantined tests) and binds RSpec configurations
   #          and hooks onto the global RSpec class
-  def bind(options = {})
-    quarantine = Quarantine.new(options)
+  sig { void }
+  def self.bind
     bind_rspec_configurations
-    bind_quarantine_list(quarantine)
-    bind_quarantine_checker(quarantine)
-    bind_quarantine_record_tests(quarantine)
-    bind_logger(quarantine)
+    bind_fetch_test_statuses
+    bind_record_tests
+    bind_upload_tests
+    bind_logger
   end
 
-  private
+  sig { returns(Quarantine) }
+  def self.quarantine
+    @quarantine = T.let(@quarantine, T.nilable(Quarantine))
+    @quarantine ||= Quarantine.new(
+      database: RSpec.configuration.quarantine_database,
+      test_statuses_table_name: RSpec.configuration.quarantine_test_statuses,
+      extra_attributes: RSpec.configuration.quarantine_extra_attributes,
+      failsafe_limit: RSpec.configuration.quarantine_failsafe_limit
+    )
+  end
 
   # Purpose: binds rspec configuration variables
-  def bind_rspec_configurations
+  sig { void }
+  def self.bind_rspec_configurations
     ::RSpec.configure do |config|
-      config.add_setting(:quarantine_list_table, { default: 'quarantine_list' })
-      config.add_setting(:quarantine_failed_tests_table, { default: 'master_failed_tests' })
+      config.add_setting(:quarantine_database, default: { type: :dynamodb, region: 'us-west-1' })
+      config.add_setting(:quarantine_test_statuses, { default: 'test_statuses' })
       config.add_setting(:skip_quarantined_tests, { default: true })
-      config.add_setting(:quarantine_record_failed_tests, { default: true })
-      config.add_setting(:quarantine_record_flaky_tests, { default: true })
+      config.add_setting(:quarantine_record_tests, { default: true })
       config.add_setting(:quarantine_logging, { default: true })
+      config.add_setting(:quarantine_extra_attributes)
+      config.add_setting(:quarantine_failsafe_limit, default: 10)
+      config.add_setting(:quarantine_release_at_consecutive_passes)
     end
   end
 
-  # Purpose: binds quarantine to fetch the quarantine_list from dynamodb in the before suite
-  def bind_quarantine_list(quarantine)
+  # Purpose: binds quarantine to fetch the test_statuses from dynamodb in the before suite
+  sig { void }
+  def self.bind_fetch_test_statuses
     ::RSpec.configure do |config|
       config.before(:suite) do
-        quarantine.fetch_quarantine_list
+        Quarantine::RSpecAdapter.quarantine.fetch_test_statuses
       end
     end
   end
 
-  # Purpose: binds quarantine to skip and pass tests that have been quarantined in the after suite
-  def bind_quarantine_checker(quarantine)
-    ::RSpec.configure do |config|
-      config.after(:each) do |example|
-        if RSpec.configuration.skip_quarantined_tests && quarantine.test_quarantined?(example)
-          quarantine.pass_flaky_test(example)
-        end
-      end
-    end
-  end
-
-  # Purpose: binds quarantine to record failed and flaky tests
-  def bind_quarantine_record_tests(quarantine)
+  # Purpose: binds quarantine to record test statuses
+  sig { void }
+  def self.bind_record_tests
     ::RSpec.configure do |config|
       config.after(:each) do |example|
         metadata = example.metadata
 
-        # will record the failed test if is not quarantined and it is on it's final retry from the rspec-retry gem
-        quarantine.record_failed_test(example) if
-          RSpec.configuration.quarantine_record_failed_tests &&
-          !quarantine.test_quarantined?(example) &&
-          metadata[:retry_attempts] + 1 == metadata[:retry] && example.exception
-
-        # will record the flaky test if is not quarantined and it failed the first run but passed a subsequent run
-        quarantine.record_flaky_test(example) if
-          RSpec.configuration.quarantine_record_flaky_tests &&
-          !quarantine.test_quarantined?(example) &&
-          metadata[:retry_attempts] > 0 && example.exception.nil?
+        # optionally, the upstream RSpec configuration could define an after hook that marks an example as flaky in
+        # the example's metadata
+        quarantined = Quarantine::RSpecAdapter.quarantine.test_quarantined?(example) || metadata[:flaky]
+        if example.exception
+          if metadata[:retry_attempts] + 1 == metadata[:retry]
+            # will record the failed test if it's final retry from the rspec-retry gem
+            if RSpec.configuration.skip_quarantined_tests && quarantined
+              example.clear_exception!
+              Quarantine::RSpecAdapter.quarantine.record_test(example, :quarantined, passed: false)
+            else
+              Quarantine::RSpecAdapter.quarantine.record_test(example, :failing, passed: false)
+            end
+          end
+        elsif metadata[:retry_attempts] > 0
+          # will record the flaky test if it failed the first run but passed a subsequent run
+          Quarantine::RSpecAdapter.quarantine.record_test(example, :quarantined, passed: false)
+        elsif quarantined
+          Quarantine::RSpecAdapter.quarantine.record_test(example, :quarantined, passed: true)
+        else
+          Quarantine::RSpecAdapter.quarantine.record_test(example, :passing, passed: true)
+        end
       end
     end
+  end
 
+  sig { void }
+  def self.bind_upload_tests
     ::RSpec.configure do |config|
       config.after(:suite) do
-        quarantine.upload_tests(:failed) if RSpec.configuration.quarantine_record_failed_tests
-
-        quarantine.upload_tests(:flaky) if RSpec.configuration.quarantine_record_flaky_tests
+        Quarantine::RSpecAdapter.quarantine.upload_tests if RSpec.configuration.quarantine_record_tests
       end
     end
   end
 
   # Purpose: binds quarantine logger to output test to RSpec formatter messages
-  def bind_logger(quarantine)
+  sig { void }
+  def self.bind_logger
     ::RSpec.configure do |config|
       config.after(:suite) do
-        RSpec.configuration.reporter.message(quarantine.summary) if RSpec.configuration.quarantine_logging
+        if RSpec.configuration.quarantine_logging
+          RSpec.configuration.reporter.message(Quarantine::RSpecAdapter.quarantine.summary)
+        end
       end
     end
   end

data/lib/quarantine/test.rb

@@ -1,23 +1,25 @@
+# typed: strict
+
 class Quarantine
-  class Test
-    attr_accessor :id
-    attr_accessor :full_description
-    attr_accessor :location
-    attr_accessor :build_number
+  class Test < T::Struct
+    extend T::Sig
 
-    def initialize(id, full_description, location, build_number)
-      @id = id
-      @full_description = full_description
-      @location = location
-      @build_number = build_number
-    end
+    const :id, String
+    const :status, Symbol
+    const :consecutive_passes, Integer
+    const :full_description, String
+    const :location, String
+    const :extra_attributes, T::Hash[T.untyped, T.untyped]
 
+    sig { returns(Quarantine::Databases::Base::Item) }
     def to_hash
       {
-        id: id,
-        full_description: full_description,
-        location: location,
-        build_number: build_number
+        'id' => id,
+        'last_status' => status.to_s,
+        'consecutive_passes' => consecutive_passes,
+        'full_description' => full_description,
+        'location' => location,
+        'extra_attributes' => extra_attributes
       }
     end
   end

data/lib/quarantine/version.rb

@@ -1,3 +1,3 @@
 class Quarantine
-  VERSION = '1.0.3'.freeze
+  VERSION = '2.0.0'.freeze
 end

data/quarantine.gemspec

@@ -11,14 +11,14 @@ Gem::Specification.new do |s|
   s.version = Quarantine::VERSION
   s.authors = ['Flexport Engineering, Eric Zhu']
   s.email = ['ericzhu77@gmail.com']
-  s.summary = 'Quarantine flaky Ruby Rspec tests'
+  s.summary = 'Quarantine flaky RSpec tests'
   s.homepage = 'https://github.com/flexport/quarantine'
   s.license = 'MIT'
   s.files = Dir['{lib, bin}/**/*', '*.md', '*.gemspec']
   s.executables = ['quarantine_dynamodb']
   s.required_ruby_version = '>= 2.0'
 
-  s.add_dependency('aws-sdk-dynamodb', '~> 1')
   s.add_dependency('rspec', '~> 3.0')
   s.add_dependency('rspec-retry', '~> 0.6')
+  s.add_dependency('sorbet-runtime', '0.5.6338')
 end

metadata

@@ -1,58 +1,58 @@
 --- !ruby/object:Gem::Specification
 name: quarantine
 version: !ruby/object:Gem::Version
-  version: 1.0.3
+  version: 2.0.0
 platform: ruby
 authors:
 - Flexport Engineering, Eric Zhu
-autorequire: 
+autorequire:
 bindir: bin
 cert_chain: []
-date: 2019-05-14 00:00:00.000000000 Z
+date: 2021-04-11 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
-  name: aws-sdk-dynamodb
+  name: rspec
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '1'
+        version: '3.0'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '1'
+        version: '3.0'
 - !ruby/object:Gem::Dependency
-  name: rspec
+  name: rspec-retry
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '3.0'
+        version: '0.6'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '3.0'
+        version: '0.6'
 - !ruby/object:Gem::Dependency
-  name: rspec-retry
+  name: sorbet-runtime
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - '='
       - !ruby/object:Gem::Version
-        version: '0.6'
+        version: 0.5.6338
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - '='
       - !ruby/object:Gem::Version
-        version: '0.6'
-description: 
+        version: 0.5.6338
+description:
 email:
 - ericzhu77@gmail.com
 executables:
@@ -76,7 +76,7 @@ homepage: https://github.com/flexport/quarantine
 licenses:
 - MIT
 metadata: {}
-post_install_message: 
+post_install_message:
 rdoc_options: []
 require_paths:
 - lib
@@ -91,9 +91,8 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubyforge_project: 
-rubygems_version: 2.7.8
-signing_key: 
+rubygems_version: 3.0.8
+signing_key:
 specification_version: 4
-summary: Quarantine flaky Ruby Rspec tests
+summary: Quarantine flaky RSpec tests
 test_files: []