quarantine 1.0.7 → 2.1.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 4b012ecdd8fb1b5f4b8d014d294bdce9e02370d46afd79bf1f69577c4821f9dc
-  data.tar.gz: 6e9ac0c7edfb4d38db0487db389623b7ad75bdf342f74e97e471a6073d81357e
+  metadata.gz: 6ab8e4aa2ba3c2ab497cb3adfb08b6c14459eab525f26a00f4bc1fb9b27e1084
+  data.tar.gz: ffafe0bdf3508389ae614ab5b8946c4cb9b5303f7697ca4b378b6284f335375f
 SHA512:
-  metadata.gz: e63db8d4dfc55809f9099fde29350ad86d88d6fb5617bfff576263bbd9ebb155f76d0f38c6e5dc9c7b465c969a8b21714ebacde8c7f8534c207aada2337d632b
-  data.tar.gz: 06ee957f72bcdf07da8b3b67d0bb5cbf62aa970c81dd991eb2a28c962d69c232b7f06a37062cf288b74531099e4aa7f2d0935fa00e296961d095c814ede34faf
+  metadata.gz: c4d12718e41b13558f30add75b2a5d5e25f5d29d4e19b55704e7500c1980f7192ed1a8fa3e4fc83e45329486c94bae42bd56f99a718144f42c6884878b926faf
+  data.tar.gz: 283aa10d1dafe98d3e5ed08f1add97facdcfbe6f4a1761759e42677b7dd26fafbc323d958ab81965076a1d19e951e18c67e35e01679ad43e44d622876649641f

data/CHANGELOG.md

@@ -1,21 +1,20 @@
 ### 1.0.7
-  Support `it_behaves_like` behavior
+Support `it_behaves_like` behavior
 
 ### 1.0.6
-  Update DynamoDB batch_write_item implementation to check for duplicates based on different keys before uploading
+Update DynamoDB batch_write_item implementation to check for duplicates based on different keys before uploading
 
 ### 1.0.5
-  Add aws_credentials argument during dynamodb initialization to override the AWS SDK credential chain
+Add aws_credentials argument during dynamodb initialization to override the AWS SDK credential chain
 
 ### 1.0.4
-  Enable upstream callers to mark an example as flaky through the example's metadata
+Enable upstream callers to mark an example as flaky through the example's metadata
 
 ### 1.0.3
-  Only require dynamodb instead of full aws-sdk
+Only require dynamodb instead of full aws-sdk
 
 ### 1.0.2
-  Relax Aws gem version constraint as aws-sdk v3 is 100% backwards
-  compatible with v2
+Relax Aws gem version constraint as aws-sdk v3 is 100% backwards compatible with v2
 
 ### 1.0.1
-  Initial Release
+Initial Release

data/README.md

@@ -1,27 +1,18 @@
 # Quarantine
+
 [![Build Status](https://travis-ci.com/flexport/quarantine.svg?branch=master)](https://travis-ci.com/flexport/quarantine)
 
-Quarantine provides a run-time solution to diagnosing and disabling flaky tests and automates the workflow around test suite maintenance.
+Quarantine automatically detects flaky tests (i.e. those which fail non-deterministically) and disables them until they're proven reliable.
 
-The quarantine gem supports testing frameworks:
+Quarantine current supports the following testing frameworks. If you need an additional one, please file an issue or open a pull request.
 - [RSpec](http://rspec.info/)
 
-The quarantine gem supports CI pipelines:
-- [Buildkite](https://buildkite.com/docs/tutorials/getting-started)
-
-If you are interested in using quarantine but it does not support your CI or testing framework, feel free to reach out or create an issue and we can try to make it happen.
-
-## Purpose
-Flaky tests impact engineering velocity, reduce faith in test reliablity and give a false representation of code coverage. Managing flaky tests is a clunky process that involves constant build monitorization, difficult diagnosis and manual ticket creation. As a result, here at Flexport, we have created a Gem to automate the entire process to help improve the workflow and keep our massive test suites in pristine condition.
-
-The workflow at Flexport involves:
+Quarantine should provide the necessary hooks for compatibility with any CI solution. If it's insufficient for yours, please file an issue or open a pull request.
 
-![ideal workflow](misc/flexport_workflow.png)
+## Getting started
 
----
-## Installation and Setup
+Quarantine works in tandem with [RSpec::Retry](https://github.com/NoRedInk/rspec-retry). Add this to your `Gemfile` and run `bundle install`:
 
-Add these lines to your application's Gemfile:
 ```rb
 group :test do
   gem 'quarantine'
@@ -29,123 +20,144 @@ group :test do
 end
 ```
 
-And then execute:
-```sh
-bundle install
-```
+In your `spec_helper.rb`, set up Quarantine and RSpec::Retry. See [RSpec::Retry](https://github.com/NoRedInk/rspec-retry)'s documentation for details of its configuration.
 
-In your `spec_helper.rb` setup quarantine and rspec-retry gem. Click [rspec-retry](https://github.com/NoRedInk/rspec-retry) to get a more detailed explaination on rspec-retry configurations and how to setup.
 ```rb
 require 'quarantine'
-require 'rspec-retry'
+require 'rspec/retry'
 
-Quarantine.bind({database: :dynamodb, aws_region: 'us-west-1'}) # Also accepts aws_credentials to override the standard AWS credential chain
+Quarantine::RSpecAdapter.bind
 
 RSpec.configure do |config|
+  # Also accepts `:credentials` to override the standard AWS credential chain
+  config.quarantine_database = {type: :dynamodb, region: 'us-west-1'}
+  # Prevent the list of flaky tests from being polluted by local development and PRs
+  config.quarantine_record_tests = ENV["CI"] && ENV["BRANCH"] == "master"
 
-  config.around(:each) do |ex|
-    ex.run_with_retry(retry: 3)
+  config.around(:each) do |example|
+    example.run_with_retry(retry: 3)
   end
-
 end
 ```
 
-Consider wrapping `Quarantine.bind` in if statements so local flaky tests don't pollute the list of quarantined tests
+Quarantine comes with a CLI tool for setting up the necessary table in DynamoDB, if you use that database.
 
-```rb
-if ENV[CI] && ENV[BRANCH] == "master"
-  Quarantine.bind({database: :dynamodb, aws_region: 'us-west-1'})
-end
-```
-
-Setup tables in AWS DynamoDB to support pulling and uploading quarantined tests
 ```sh
-bundle exec quarantine_dynamodb -h    # see all options
+bundle exec quarantine_dynamodb -h    # See all options
 
-bundle exec quarantine_dynamodb \     # create the tables in us-west-1 in aws dynamodb
-  --aws_region us-west-1              # with "quarantine_list" and "master_failed_tests"
-                                      # table names
+bundle exec quarantine_dynamodb \     # Create the "test_statuses" table in us-west-1 in AWS DynamoDB
+  --region us-west-1
 ```
 
-You are all set to start quarantining tests!
+## How It Works
+
+A flaky test fails on the first run, but passes after being retried via RSpec::Retry.
 
-## Try Quarantining Tests Locally
-Add a test that will flake
 ```rb
 require "spec_helper"
 
 describe Quarantine do
-  it "this test should flake 33% of the time" do
-    Random.rand(3).to eq(3)
+  it "fails on the first run" do
+    raise "error" if RSpec.current_example.attempts == 0
+    # otherwise, pass
   end
 end
 ```
 
-Run `rspec` on the test
 ```sh
-CI=1 BRANCH=master rspec <filename>
+$ CI=1 BRANCH=master bundle exec rspec <filename>
+[quarantine] Quarantined tests:
+  ./bar_spec.rb[1:1] Quarantine fails on the first run
 ```
 
-If the test fails and passes on the test run (rspec-retry re-ran the test), the test should be quarantined and uploaded to DynamoDB. Check the `quarantine_list` table in DynamoDB.
+When the build completes, all test statuses are written to the database. Flaky tests are marked `quarantined`, and will be executed in future builds, but any failures will be ignored.
+
+A test can be removed from quarantine by updating the database manually (outside this gem), or by configuring `quarantine_release_at_consecutive_passes` to remove it after it passes on a certain number of builds in a row.
 
 ## Configuration
 
-Go to `spec/spec_helper.rb` and set configuration variables through:
+In `spec_helper.rb`, you can set configuration variables by doing:
+
 ```rb
 RSpec.configure do |config|
-    config.VAR_NAME = VALUE
+  config.VAR_NAME = VALUE
 end
 ```
-- Table name for quarantined tests `:quarantine_list_table, default: "quarantine_list"`
 
-- Table name where failed test are uploaded `:quarantine_failed_tests_table, default: "master_failed_tests"`
+- `quarantine_database`: Database configuration (see below), default: `{ type: :dynamodb, region: 'us-west-1' }`
+- `test_statusus_table`: Table name for test statuses, default: `"test_statuses"`
+- `skip_quarantined_tests`: Skipping quarantined tests during test runs default: `true`
+- `quarantine_record_tests`: Recording test statuses, default: `true`
+- `quarantine_logging`: Outputting quarantined gem info, default: `true`
+- `quarantine_extra_attributes`: Storing custom per-example attributes in the table, default: `nil`
+- `quarantine_failsafe_limit`: A failsafe limit of quarantined tests in a single run, default: `10`
+- `quarantine_release_at_consecutive_passes`: Releasing a test from quarantine after it records enough consecutive passes (`nil` to disable this feature), default: `nil`
 
-- Skipping quarantined tests during test runs `:skip_quarantined_tests, default: true`
+### Databases
 
-- Recording failed tests `:quarantine_record_failed_tests, default: true`
+Quarantine comes with built-in support for the following database types:
+- `:dynamodb`
+- `:google_sheets`
 
-- Recording flaky tests `:quarantine_record_flaky_tests, default: true`
+To use `:dynamodb`, be sure to add `gem 'aws-sdk-dynamodb', '~> 1', group: :test` to your `Gemfile`.
 
-- Outputting quarantined gem info `:quarantine_logging, default: true`
+To use `:google_sheets`, be sure to add `gem 'google_drive', '~> 2', group: :test` to your `Gemfile`. Here's an example:
 
----
-## Setup Jira Workflow
+```rb
+config.quarantine_database = {
+  type: :google_sheets,
+  authorization: {type: :service_account_key, file: "service_account.json"}, # also accepts `type: :config`
+  spreadsheet: {type: :by_key, "1Jb5fC6wSuIMnP85tUR5knuZ4f5fuu4nMzQF6-0l-EXAMPLE"}, # also accepts `type: :by_title` and `type: :by_url`
+}
+```
 
-To automatically create Jira tickets, take a look at: `examples/create_tickets.rb`
+To use a custom database that's not provided, subclass `Quarantine::Databases::Base` and pass an instance of your class as the `quarantine_database` setting:
 
-To automatically unquarantine tests on Jira ticket completion, take a look at: `examples/unquarantine.rb`
+```rb
+class MyDatabase < Quarantine::Databases::Base
+  ...
+end
+
+RSpec.configure do |config|
+  config.quarantine_database = MyDatabase.new(...)
+end
+```
+
+### Extra attributes
+
+Use `quarantine_extra_attributes` to store custom data with each test in the database, e.g. variables useful for your CI setup.
+
+```rb
+ config.quarantine_extra_attributes = Proc.new do |example|
+   {
+     build_url: ENV['BUILDKITE_BUILD_URL'],
+     job_id: ENV['BUILDKITE_JOB_ID'],
+   }
+ end
+```
 
----
-## Contributing
-1. Create an issue regarding a bug fix or feature request
-2. Fork the repository
-3. Create your feature branch, commit changes and create a pull request
-
-## Contributors
-- [Eric Zhu](https://github.com/eric-zhu-uw)
-- [Kevin Miller](https://github.com/Gasparila)
-- [Nicholas Silva](https://github.com/flexportnes)
-- [Ankur Dahiya](https://github.com/legalosLOTR)
 ---
 
 ## FAQs
 
 #### Why are quarantined tests not being skipped locally?
 
-The `quarantine` gem may be configured to only run on certain environments. Make sure you pass all these `ENV` variables to `rspec` when you call it locally
+Quarantine may be configured to only run in certain environments. Check your `spec_helper.rb`, and make sure you have all necessary environment variables set, e.g.:
 
 ```sh
-CI="1" BRANCH="master" rspec
+CI=1 BRANCH=master bundle exec rspec
 ```
 
-#### Why is dynamodb failing to connect?
+#### Why is Quarantine failing to connect to DynamoDB?
 
-The AWS client loads credentials from the following locations (in order of precedence):
-- The optional `aws_credentials` parameter passed into `Quarantine.bind`
+The AWS client attempts to loads credentials from the following locations, in order:
+- The optional `credentials` field in `RSpec.configuration.quarantine_database`
 - `ENV['AWS_ACCESS_KEY_ID']` and `ENV['AWS_SECRET_ACCESS_KEY']`
 - `Aws.config[:credentials]`
 - The shared credentials ini file at `~/.aws/credentials`
 
-To get AWS credentials, please contact your AWS administrator to get access to dynamodb and create your credentials through IAM.
+More detailed information can be found in the [AWS SDK documentation](https://docs.aws.amazon.com/sdkforruby/api/Aws/S3/Client.html)
+
+#### Why is `example.clear_exception` failing locally?
 
-More detailed information can be found: [AWS documentation](https://docs.aws.amazon.com/sdkforruby/api/Aws/S3/Client.html)
+`Example#clear_exception` is an attribute added through [RSpec::Retry](https://github.com/NoRedInk/rspec-retry). Make sure it has been installed and configured.

data/lib/quarantine.rb

@@ -1,152 +1,168 @@
+# typed: strict
+
+require 'sorbet-runtime'
+
 require 'rspec/retry'
 require 'quarantine/rspec_adapter'
 require 'quarantine/test'
 require 'quarantine/databases/base'
 require 'quarantine/databases/dynamo_db'
+require 'quarantine/databases/google_sheets'
 
 module RSpec
   module Core
     class Example
+      extend T::Sig
+
       # The implementation of clear_exception in rspec-retry doesn't work
       # for examples that use `it_behaves_like`, so we implement our own version that
       # clear the exception field recursively.
+      sig { void }
       def clear_exception!
-        @exception = nil
-        example.clear_exception! if defined?(example)
+        @exception = T.let(nil, T.untyped)
+        T.unsafe(self).example.clear_exception! if defined?(example)
       end
     end
   end
 end
 
 class Quarantine
-  extend RSpecAdapter
-
-  attr_accessor :database
-  attr_reader :quarantine_map
-  attr_reader :failed_tests
-  attr_reader :flaky_tests
-  attr_reader :duplicate_tests
-  attr_reader :buildkite_build_number
-  attr_reader :summary
-
-  def initialize(options = {})
-    case options[:database]
-    # default database option is dynamodb
-    when :dynamodb, nil
-      @database = Quarantine::Databases::DynamoDB.new(options)
-    else
-      raise Quarantine::UnsupportedDatabaseError.new("Quarantine does not support #{options[:database]}")
-    end
+  extend T::Sig
+
+  sig { returns(T::Hash[String, Quarantine::Test]) }
+  attr_reader :tests
+
+  sig { returns(T::Hash[String, Quarantine::Test]) }
+  attr_reader :old_tests
 
-    @quarantine_map = {}
-    @failed_tests = []
-    @flaky_tests = []
-    @buildkite_build_number = ENV['BUILDKITE_BUILD_NUMBER'] || '-1'
-    @summary = { id: 'quarantine', quarantined_tests: [], flaky_tests: [], database_failures: [] }
+  sig { params(options: T::Hash[T.untyped, T.untyped]).void }
+  def initialize(options)
+    @options = options
+    @old_tests = T.let({}, T::Hash[String, Quarantine::Test])
+    @tests = T.let({}, T::Hash[String, Quarantine::Test])
+    @database_failures = T.let([], T::Array[String])
+    @database = T.let(nil, T.nilable(Quarantine::Databases::Base))
   end
 
-  # Scans the quarantine_list from the database and store the individual tests in quarantine_map
-  def fetch_quarantine_list
+  sig { returns(Quarantine::Databases::Base) }
+  def database
+    @database ||=
+      case @options[:database]
+      when Quarantine::Databases::Base
+        @options[:database]
+      else
+        database_options = @options[:database].dup
+        type = database_options.delete(:type)
+        case type
+        when :dynamodb
+          Quarantine::Databases::DynamoDB.new(database_options)
+        when :google_sheets
+          Quarantine::Databases::GoogleSheets.new(database_options)
+        else
+          raise Quarantine::UnsupportedDatabaseError.new("Quarantine does not support database type: #{type.inspect}")
+        end
+      end
+  end
+
+  # Scans the test_statuses from the database and store their IDs in quarantined_ids
+  sig { void }
+  def on_start
     begin
-      quarantine_list = database.scan(RSpec.configuration.quarantine_list_table)
+      test_statuses = database.fetch_items(@options[:test_statuses_table_name])
     rescue Quarantine::DatabaseError => e
-      add_to_summary(:database_failures, "#{e&.cause&.class}: #{e&.cause&.message}")
+      @database_failures << "#{e.cause&.class}: #{e.cause&.message}"
       raise Quarantine::DatabaseError.new(
         <<~ERROR_MSG
-          Failed to pull the quarantine list from #{RSpec.configuration.quarantine_list_table}
-          because of #{e&.cause&.class}: #{e&.cause&.message}
+          Failed to pull the quarantine list from #{@options[:test_statuses_table_name]}
+          because of #{e.cause&.class}: #{e.cause&.message}
         ERROR_MSG
       )
     end
 
-    quarantine_list.each do |example|
-      # on the rare occassion there are duplicate tests ids in the quarantine_list,
-      # quarantine the most recent instance of the test (det. through build_number)
-      # and ignore the older instance of the test
-      next if
-        quarantine_map.key?(example['id']) &&
-        example['build_number'].to_i < quarantine_map[example['id']].build_number.to_i
-
-      quarantine_map.store(
-        example['id'],
-        Quarantine::Test.new(example['id'], example['full_description'], example['location'], example['build_number'])
-      )
-    end
+    pairs =
+      test_statuses
+      .group_by { |t| t['id'] }
+      .map { |_id, tests| tests.max_by { |t| t['created_at'] } }
+      .compact
+      .filter { |t| t['last_status'] == 'quarantined' }
+      .map do |t|
+        [
+          t['id'],
+          Quarantine::Test.new(
+            id: t['id'],
+            status: t['last_status'].to_sym,
+            consecutive_passes: t['consecutive_passes'].to_i,
+            full_description: t['full_description'],
+            location: t['location'],
+            extra_attributes: t['extra_attributes']
+          )
+        ]
+      end
+
+    @old_tests = Hash[pairs]
   end
 
-  # Based off the type, upload a list of tests to a particular database table
-  def upload_tests(type)
-    if type == :failed
-      tests = failed_tests
-      table_name = RSpec.configuration.quarantine_failed_tests_table
-    elsif type == :flaky
-      tests = flaky_tests
-      table_name = RSpec.configuration.quarantine_list_table
+  sig { void }
+  def on_complete
+    quarantined_tests = @tests.values.select { |test| test.status == :quarantined }.sort_by(&:id)
+
+    if !@options[:record_tests]
+      log('Recording tests disabled; skipping')
+    elsif @tests.empty?
+      log('No tests found; skipping recording')
+    elsif quarantined_tests.count { |test| old_tests[test.id]&.status != :quarantined } >= @options[:failsafe_limit]
+      log('Number of quarantined tests above failsafe limit; skipping recording')
     else
-      raise Quarantine::UnknownUploadError.new(
-        "Quarantine gem did not know how to handle #{type} upload of tests to dynamodb"
-      )
+      begin
+        timestamp = Time.now.to_i / 1000 # Truncated millisecond from timestamp for reasons specific to Flexport
+        database.write_items(
+          @options[:test_statuses_table_name],
+          @tests.values.map { |item| item.to_hash.merge('updated_at' => timestamp) }
+        )
+      rescue Quarantine::DatabaseError => e
+        @database_failures << "#{e.cause&.class}: #{e.cause&.message}"
+      end
     end
 
-    return unless tests.length < 10 && tests.length > 0
-
-    begin
-      timestamp = Time.now.to_i / 1000 # Truncated millisecond from timestamp for reasons specific to Flexport
-      database.batch_write_item(
-        table_name,
-        tests,
-        {
-          build_job_id: ENV['BUILDKITE_JOB_ID'] || '-1',
-          created_at: timestamp,
-          updated_at: timestamp
-        }
-      )
-    rescue Quarantine::DatabaseError => e
-      add_to_summary(:database_failures, "#{e&.cause&.class}: #{e&.cause&.message}")
-    end
-  end
+    log(<<~MESSAGE)
+      \n[quarantine] Quarantined tests:
+        #{quarantined_tests.map { |test| "#{test.id} #{test.full_description}" }.join("\n  ")}
 
-  # Param: RSpec::Core::Example
-  # Add the example to the internal failed tests list
-  def record_failed_test(example)
-    failed_tests << Quarantine::Test.new(
-      example.id,
-      example.full_description,
-      example.location,
-      buildkite_build_number
-    )
+      [quarantine] Database errors:
+        #{@database_failures.join("\n  ")}
+    MESSAGE
   end
 
   # Param: RSpec::Core::Example
-  # Add the example to the internal flaky tests list
-  def record_flaky_test(example)
-    flaky_test = Quarantine::Test.new(
-      example.id,
-      example.full_description,
-      example.location,
-      buildkite_build_number
+  # Add the example to the internal tests list
+  sig { params(example: T.untyped, status: Symbol, passed: T::Boolean).void }
+  def on_test(example, status, passed:)
+    extra_attributes = @options[:extra_attributes] ? @options[:extra_attributes].call(example) : {}
+
+    new_consecutive_passes = passed ? (@old_tests[example.id]&.consecutive_passes || 0) + 1 : 0
+    release_at = @options[:release_at_consecutive_passes]
+    new_status = !release_at.nil? && new_consecutive_passes >= release_at ? :passing : status
+    test = Quarantine::Test.new(
+      id: example.id,
+      status: new_status,
+      consecutive_passes: new_consecutive_passes,
+      full_description: example.full_description,
+      location: example.location,
+      extra_attributes: extra_attributes
     )
 
-    flaky_tests << flaky_test
-    add_to_summary(:flaky_tests, flaky_test.id)
-  end
-
-  # Param: RSpec::Core::Example
-  # Clear exceptions on a flaky tests that has been quarantined
-  def pass_flaky_test(example)
-    example.clear_exception!
-    add_to_summary(:quarantined_tests, example.id)
+    @tests[test.id] = test
   end
 
   # Param: RSpec::Core::Example
-  # Check the internal quarantine_map to see if this test should be quarantined
+  # Check the internal old_tests to see if this test should be quarantined
+  sig { params(example: T.untyped).returns(T::Boolean) }
   def test_quarantined?(example)
-    quarantine_map.key?(example.id)
+    @old_tests[example.id]&.status == :quarantined
   end
 
-  # Param: Symbol, Any
-  # Adds the item to the specified attribute in summary
-  def add_to_summary(attribute, item)
-    summary[attribute] << item if summary.key?(attribute)
+  sig { params(message: String).void }
+  def log(message)
+    @options[:log].call(message) if @options[:logging]
   end
 end

data/lib/quarantine/cli.rb

@@ -1,43 +1,42 @@
+# typed: strict
+
 require 'optparse'
 require_relative 'databases/base'
 require_relative 'databases/dynamo_db'
 
 class Quarantine
   class CLI
-    attr_accessor :options
+    extend T::Sig
+
+    sig { returns(T::Hash[T.untyped, T.untyped]) }
+    attr_reader :options
 
+    sig { void }
     def initialize
       # default options
-      @options = {
-        quarantine_list_table_name: 'quarantine_list',
-        failed_test_table_name: 'master_failed_tests'
-      }
+      @options = T.let(
+        {
+          test_statuses_table_name: 'test_statuses'
+        }, T::Hash[Symbol, T.untyped]
+      )
     end
 
+    sig { void }
     def parse
       OptionParser.new do |parser|
         parser.banner = 'Usage: quarantine_dynamodb [options]'
 
-        parser.on('-rREGION', '--aws_region=REGION', String, 'Specify the aws region for DynamoDB') do |aws_region|
-          options[:aws_region] = aws_region
+        parser.on('-rREGION', '--region=REGION', String, 'Specify the aws region for DynamoDB') do |region|
+          @options[:region] = region
         end
 
         parser.on(
           '-qTABLE',
           '--quarantine_table=TABLE',
           String,
-          "Specify the table name for the quarantine list | Default: #{options[:quarantine_list_table_name]}"
-        ) do |table_name|
-          options[:quarantine_list_table_name] = table_name
-        end
-
-        parser.on(
-          '-fTABLE',
-          '--failed_table=TABLE',
-          String,
-          "Specify the table name for the failed test list | Default: #{options[:failed_test_table_name]}"
+          "Specify the table name for the quarantine list | Default: #{@options[:test_statuses_table_name]}"
         ) do |table_name|
-          options[:failed_test_table_name] = table_name
+          @options[:test_statuses_table_name] = table_name
         end
 
         parser.on('-h', '--help', 'Prints help page') do
@@ -46,7 +45,7 @@ class Quarantine
         end
       end.parse!
 
-      if options[:aws_region].nil?
+      if @options[:region].nil?
         error_msg = 'Failed to specify the required aws region with -r option'.freeze
         warn error_msg
         raise ArgumentError.new(error_msg)
@@ -54,12 +53,12 @@ class Quarantine
     end
 
     # TODO: eventually move to a separate file & create_table by db type when my db adapters
+    sig { void }
     def create_tables
-      dynamodb = Quarantine::Databases::DynamoDB.new(options)
+      dynamodb = Quarantine::Databases::DynamoDB.new(region: @options[:region])
 
       attributes = [
-        { attribute_name: 'id', attribute_type: 'S', key_type: 'HASH' },
-        { attribute_name: 'build_number', attribute_type: 'S', key_type: 'RANGE' }
+        { attribute_name: 'id', attribute_type: 'S', key_type: 'HASH' }
       ]
 
       additional_arguments = {
@@ -70,15 +69,9 @@ class Quarantine
       }
 
       begin
-        dynamodb.create_table(options[:quarantine_list_table_name], attributes, additional_arguments)
-      rescue Quarantine::DatabaseError => e
-        warn "#{e&.cause&.class}: #{e&.cause&.message}"
-      end
-
-      begin
-        dynamodb.create_table(options[:failed_test_table_name], attributes, additional_arguments)
+        dynamodb.create_table(@options[:test_statuses_table_name], attributes, additional_arguments)
       rescue Quarantine::DatabaseError => e
-        warn "#{e&.cause&.class}: #{e&.cause&.message}"
+        warn "#{e.cause&.class}: #{e.cause&.message}"
       end
     end
   end

data/lib/quarantine/databases/base.rb

@@ -1,17 +1,25 @@
+# typed: strict
+
 class Quarantine
   module Databases
     class Base
-      def initialize
-        raise NotImplementedError
-      end
+      extend T::Sig
+      extend T::Helpers
 
-      def scan
-        raise NotImplementedError
-      end
+      abstract!
+
+      Item = T.type_alias { T::Hash[String, T.untyped] } # TODO: must have `id` key
+
+      sig { abstract.params(table_name: String).returns(T::Enumerable[Item]) }
+      def fetch_items(table_name); end
 
-      def batch_write_item
-        raise NotImplementedError
+      sig do
+        abstract.params(
+          table_name: String,
+          items: T::Array[Item]
+        ).void
       end
+      def write_items(table_name, items); end
     end
   end
 end

data/lib/quarantine/databases/dynamo_db.rb

@@ -1,51 +1,53 @@
-require 'aws-sdk-dynamodb'
+# typed: strict
+
+begin
+  require 'aws-sdk-dynamodb'
+rescue LoadError
+end
 require 'quarantine/databases/base'
 require 'quarantine/error'
 
 class Quarantine
   module Databases
     class DynamoDB < Base
+      extend T::Sig
+
+      Attribute = T.type_alias { { attribute_name: String, attribute_type: String, key_type: String } }
+
+      sig { returns(Aws::DynamoDB::Client) }
       attr_accessor :dynamodb
 
-      def initialize(aws_region: 'us-west-1', aws_credentials: nil, **_additional_arguments)
-        options = { region: aws_region }
-        options[:credentials] = aws_credentials if aws_credentials
+      sig { params(options: T::Hash[T.untyped, T.untyped]).void }
+      def initialize(options)
+        super()
 
-        @dynamodb = Aws::DynamoDB::Client.new(options)
+        @dynamodb = T.let(Aws::DynamoDB::Client.new(options), Aws::DynamoDB::Client)
       end
 
-      def scan(table_name)
+      sig { override.params(table_name: String).returns(T::Enumerable[Item]) }
+      def fetch_items(table_name)
         begin
-          result = dynamodb.scan(table_name: table_name)
+          result = @dynamodb.scan(table_name: table_name)
         rescue Aws::DynamoDB::Errors::ServiceError
           raise Quarantine::DatabaseError
         end
 
-        result&.items
+        result.items
       end
 
-      def batch_write_item(table_name, items, additional_attributes = {}, dedup_keys = %w[id full_description])
-        return if items.empty?
-
-        # item_a is a duplicate of item_b if all values for each dedup_key in both item_a and item_b match
-        is_a_duplicate = ->(item_a, item_b) { dedup_keys.all? { |key| item_a[key] == item_b[key] } }
-
-        scanned_items = scan(table_name)
-
-        deduped_items = items.reject do |item|
-          scanned_items.any? do |scanned_item|
-            is_a_duplicate.call(item.to_string_hash, scanned_item)
-          end
-        end
-
-        return if deduped_items.empty?
-
-        dynamodb.batch_write_item(
+      sig do
+        override.params(
+          table_name: String,
+          items: T::Array[Item]
+        ).void
+      end
+      def write_items(table_name, items)
+        @dynamodb.batch_write_item(
           request_items: {
-            table_name => deduped_items.map do |item|
+            table_name => items.map do |item|
               {
                 put_request: {
-                  item: { **item.to_hash, **additional_attributes }
+                  item: item
                 }
               }
             end
@@ -55,19 +57,15 @@ class Quarantine
         raise Quarantine::DatabaseError
       end
 
-      def delete_item(table_name, keys)
-        dynamodb.delete_item(
-          table_name: table_name,
-          key: {
-            **keys
-          }
-        )
-      rescue Aws::DynamoDB::Errors::ServiceError
-        raise Quarantine::DatabaseError
+      sig do
+        params(
+          table_name: String,
+          attributes: T::Array[Attribute],
+          additional_arguments: T::Hash[T.untyped, T.untyped]
+        ).void
       end
-
       def create_table(table_name, attributes, additional_arguments = {})
-        dynamodb.create_table(
+        @dynamodb.create_table(
           {
             table_name: table_name,
             attribute_definitions: attributes.map do |attribute|

data/lib/quarantine/databases/google_sheets.rb

@@ -0,0 +1,114 @@
+# typed: strict
+
+begin
+  require 'google_drive'
+rescue LoadError
+end
+require 'quarantine/databases/base'
+
+class Quarantine
+  module Databases
+    class GoogleSheets < Base
+      extend T::Sig
+
+      sig { params(options: T::Hash[T.untyped, T.untyped]).void }
+      def initialize(options)
+        super()
+
+        @options = options
+      end
+
+      sig { override.params(table_name: String).returns(T::Enumerable[Item]) }
+      def fetch_items(table_name)
+        parse_rows(spreadsheet.worksheet_by_title(table_name))
+      rescue GoogleDrive::Error, Google::Apis::Error
+        raise Quarantine::DatabaseError
+      end
+
+      sig do
+        override.params(
+          table_name: String,
+          items: T::Array[Item]
+        ).void
+      end
+      def write_items(table_name, items)
+        worksheet = spreadsheet.worksheet_by_title(table_name)
+        headers = worksheet.rows.first.reject(&:empty?)
+        new_rows = []
+
+        # Map existing ID to row index
+        parsed_rows = parse_rows(worksheet)
+        indexes = Hash[parsed_rows.each_with_index.map { |item, idx| [item['id'], idx] }]
+
+        items.each do |item|
+          cells = headers.map { |header| item[header].to_s }
+          row_idx = indexes[item['id']]
+          if row_idx
+            # Overwrite existing row
+            headers.each_with_index do |_header, col_idx|
+              worksheet[row_idx + 2, col_idx + 1] = cells[col_idx]
+            end
+          else
+            new_rows << cells
+          end
+        end
+
+        # Insert any items whose IDs weren't found in existing rows at the end
+        worksheet.insert_rows(parsed_rows.count + 2, new_rows)
+        worksheet.save
+      rescue GoogleDrive::Error, Google::Apis::Error
+        raise Quarantine::DatabaseError
+      end
+
+      private
+
+      sig { returns(GoogleDrive::Session) }
+      def session
+        @session = T.let(@session, T.nilable(GoogleDrive::Session))
+        @session ||= begin
+          authorization = @options[:authorization]
+          case authorization[:type]
+          when :service_account_key
+            GoogleDrive::Session.from_service_account_key(authorization[:file])
+          when :config
+            GoogleDrive::Session.from_config(authorization[:file])
+          else
+            raise "Invalid authorization type: #{authorization[:type]}"
+          end
+        end
+      end
+
+      sig { returns(GoogleDrive::Spreadsheet) }
+      def spreadsheet
+        @spreadsheet = T.let(@spreadsheet, T.nilable(GoogleDrive::Spreadsheet))
+        @spreadsheet ||= begin
+          spreadsheet = @options[:spreadsheet]
+          case spreadsheet[:type]
+          when :by_key
+            session.spreadsheet_by_key(spreadsheet[:key])
+          when :by_title
+            session.spreadsheet_by_title(spreadsheet[:title])
+          when :by_url
+            session.spreadsheet_by_url(spreadsheet[:url])
+          else
+            raise "Invalid spreadsheet type: #{spreadsheet[:type]}"
+          end
+        end
+      end
+
+      sig { params(worksheet: GoogleDrive::Worksheet).returns(T::Enumerable[Item]) }
+      def parse_rows(worksheet)
+        headers, *rows = worksheet.rows
+
+        rows.map do |row|
+          hash_row = Hash[headers.zip(row)]
+          # TODO: use Google Sheets developer metadata to store type information
+          unless hash_row['id'].empty?
+            hash_row['extra_attributes'] = JSON.parse(hash_row['extra_attributes']) if hash_row['extra_attributes']
+            hash_row
+          end
+        end.compact
+      end
+    end
+  end
+end

data/lib/quarantine/error.rb

@@ -1,3 +1,5 @@
+# typed: strict
+
 class Quarantine
   class Error < StandardError; end
 

data/lib/quarantine/rspec_adapter.rb

@@ -1,87 +1,101 @@
-module RSpecAdapter
-  # Purpose: create an instance of Quarantine which contains information
-  #          about the test suite (ie. quarantined tests) and binds RSpec configurations
-  #          and hooks onto the global RSpec class
-  def bind(options = {})
-    quarantine = Quarantine.new(options)
-    bind_rspec_configurations
-    bind_quarantine_list(quarantine)
-    bind_quarantine_checker(quarantine)
-    bind_quarantine_record_tests(quarantine)
-    bind_logger(quarantine)
-  end
+# typed: strict
 
-  private
+class Quarantine
+  module RSpecAdapter
+    extend T::Sig
 
-  # Purpose: binds rspec configuration variables
-  def bind_rspec_configurations
-    ::RSpec.configure do |config|
-      config.add_setting(:quarantine_list_table, { default: 'quarantine_list' })
-      config.add_setting(:quarantine_failed_tests_table, { default: 'master_failed_tests' })
-      config.add_setting(:skip_quarantined_tests, { default: true })
-      config.add_setting(:quarantine_record_failed_tests, { default: true })
-      config.add_setting(:quarantine_record_flaky_tests, { default: true })
-      config.add_setting(:quarantine_logging, { default: true })
+    sig { void }
+    def self.bind
+      register_rspec_configurations
+      bind_on_start
+      bind_on_test
+      bind_on_complete
+    end
+
+    sig { returns(Quarantine) }
+    def self.quarantine
+      @quarantine = T.let(@quarantine, T.nilable(Quarantine))
+      @quarantine ||= Quarantine.new(
+        database: RSpec.configuration.quarantine_database,
+        test_statuses_table_name: RSpec.configuration.quarantine_test_statuses,
+        extra_attributes: RSpec.configuration.quarantine_extra_attributes,
+        failsafe_limit: RSpec.configuration.quarantine_failsafe_limit,
+        release_at_consecutive_passes: RSpec.configuration.quarantine_release_at_consecutive_passes,
+        logging: RSpec.configuration.quarantine_logging,
+        log: method(:log),
+        record_tests: RSpec.configuration.quarantine_record_tests
+      )
     end
-  end
 
-  # Purpose: binds quarantine to fetch the quarantine_list from dynamodb in the before suite
-  def bind_quarantine_list(quarantine)
-    ::RSpec.configure do |config|
-      config.before(:suite) do
-        quarantine.fetch_quarantine_list
+    # Purpose: binds rspec configuration variables
+    sig { void }
+    def self.register_rspec_configurations
+      ::RSpec.configure do |config|
+        config.add_setting(:quarantine_database, default: { type: :dynamodb, region: 'us-west-1' })
+        config.add_setting(:quarantine_test_statuses, { default: 'test_statuses' })
+        config.add_setting(:skip_quarantined_tests, { default: true })
+        config.add_setting(:quarantine_record_tests, { default: true })
+        config.add_setting(:quarantine_logging, { default: true })
+        config.add_setting(:quarantine_extra_attributes)
+        config.add_setting(:quarantine_failsafe_limit, default: 10)
+        config.add_setting(:quarantine_release_at_consecutive_passes)
       end
     end
-  end
 
-  # Purpose: binds quarantine to skip and pass tests that have been quarantined in the after suite
-  def bind_quarantine_checker(quarantine)
-    ::RSpec.configure do |config|
-      config.after(:each) do |example|
-        if RSpec.configuration.skip_quarantined_tests && quarantine.test_quarantined?(example)
-          quarantine.pass_flaky_test(example)
+    # Purpose: binds quarantine to fetch the test_statuses from dynamodb in the before suite
+    sig { void }
+    def self.bind_on_start
+      ::RSpec.configure do |config|
+        config.before(:suite) do
+          Quarantine::RSpecAdapter.quarantine.on_start
         end
       end
     end
-  end
 
-  # Purpose: binds quarantine to record failed and flaky tests
-  def bind_quarantine_record_tests(quarantine)
-    ::RSpec.configure do |config|
-      config.after(:each) do |example|
-        metadata = example.metadata
+    # Purpose: binds quarantine to record test statuses
+    sig { void }
+    def self.bind_on_test
+      ::RSpec.configure do |config|
+        config.after(:each) do |example|
+          metadata = example.metadata
 
-        # will record the failed test if is not quarantined and it is on it's final retry from the rspec-retry gem
-        quarantine.record_failed_test(example) if
-          RSpec.configuration.quarantine_record_failed_tests &&
-          !quarantine.test_quarantined?(example) &&
-          metadata[:retry_attempts] + 1 == metadata[:retry] && example.exception
-
-        # will record the flaky test if is not quarantined and it failed the first run but passed a subsequent run;
-        # optionally, the upstream RSpec configuration could define an after hook that marks an example as flaky in
-        # the example's metadata
-        quarantine.record_flaky_test(example) if
-          RSpec.configuration.quarantine_record_flaky_tests &&
-          !quarantine.test_quarantined?(example) &&
-          (metadata[:retry_attempts] > 0 && example.exception.nil?) || metadata[:flaky]
+          # optionally, the upstream RSpec configuration could define an after hook that marks an example as flaky in
+          # the example's metadata
+          quarantined = Quarantine::RSpecAdapter.quarantine.test_quarantined?(example) || metadata[:flaky]
+          if example.exception
+            if metadata[:retry_attempts] + 1 == metadata[:retry]
+              # will record the failed test if it's final retry from the rspec-retry gem
+              if RSpec.configuration.skip_quarantined_tests && quarantined
+                example.clear_exception!
+                Quarantine::RSpecAdapter.quarantine.on_test(example, :quarantined, passed: false)
+              else
+                Quarantine::RSpecAdapter.quarantine.on_test(example, :failing, passed: false)
+              end
+            end
+          elsif metadata[:retry_attempts] > 0
+            # will record the flaky test if it failed the first run but passed a subsequent run
+            Quarantine::RSpecAdapter.quarantine.on_test(example, :quarantined, passed: false)
+          elsif quarantined
+            Quarantine::RSpecAdapter.quarantine.on_test(example, :quarantined, passed: true)
+          else
+            Quarantine::RSpecAdapter.quarantine.on_test(example, :passing, passed: true)
+          end
+        end
       end
     end
 
-    ::RSpec.configure do |config|
-      config.after(:suite) do
-        quarantine.upload_tests(:failed) if RSpec.configuration.quarantine_record_failed_tests
-
-        quarantine.upload_tests(:flaky) if RSpec.configuration.quarantine_record_flaky_tests
+    sig { void }
+    def self.bind_on_complete
+      ::RSpec.configure do |config|
+        config.after(:suite) do
+          Quarantine::RSpecAdapter.quarantine.on_complete
+        end
       end
     end
-  end
 
-  # Purpose: binds quarantine logger to output test to RSpec formatter messages
-  def bind_logger(quarantine)
-    ::RSpec.configure do |config|
-      config.after(:suite) do
-        RSpec.configuration.reporter.message(quarantine.summary) if RSpec.configuration.quarantine_logging
-      end
+    sig { params(message: String).void }
+    def self.log(message)
+      RSpec.configuration.reporter.message(message)
     end
   end
 end

data/lib/quarantine/test.rb

@@ -1,32 +1,25 @@
+# typed: strict
+
 class Quarantine
-  class Test
-    attr_accessor :id
-    attr_accessor :full_description
-    attr_accessor :location
-    attr_accessor :build_number
+  class Test < T::Struct
+    extend T::Sig
 
-    def initialize(id, full_description, location, build_number)
-      @id = id
-      @full_description = full_description
-      @location = location
-      @build_number = build_number
-    end
+    const :id, String
+    const :status, Symbol
+    const :consecutive_passes, Integer
+    const :full_description, String
+    const :location, String
+    const :extra_attributes, T::Hash[T.untyped, T.untyped]
 
+    sig { returns(Quarantine::Databases::Base::Item) }
     def to_hash
-      {
-        id: id,
-        full_description: full_description,
-        location: location,
-        build_number: build_number
-      }
-    end
-
-    def to_string_hash
       {
         'id' => id,
+        'last_status' => status.to_s,
+        'consecutive_passes' => consecutive_passes,
         'full_description' => full_description,
         'location' => location,
-        'build_number' => build_number
+        'extra_attributes' => extra_attributes
       }
     end
   end

data/lib/quarantine/version.rb

@@ -1,3 +1,3 @@
 class Quarantine
-  VERSION = '1.0.7'.freeze
+  VERSION = '2.1.3'.freeze
 end

data/quarantine.gemspec

@@ -11,14 +11,14 @@ Gem::Specification.new do |s|
   s.version = Quarantine::VERSION
   s.authors = ['Flexport Engineering, Eric Zhu']
   s.email = ['ericzhu77@gmail.com']
-  s.summary = 'Quarantine flaky Ruby Rspec tests'
+  s.summary = 'Quarantine flaky RSpec tests'
   s.homepage = 'https://github.com/flexport/quarantine'
   s.license = 'MIT'
   s.files = Dir['{lib, bin}/**/*', '*.md', '*.gemspec']
   s.executables = ['quarantine_dynamodb']
   s.required_ruby_version = '>= 2.0'
 
-  s.add_dependency('aws-sdk-dynamodb', '~> 1')
   s.add_dependency('rspec', '~> 3.0')
   s.add_dependency('rspec-retry', '~> 0.6')
+  s.add_dependency('sorbet-runtime', '0.5.6338')
 end

metadata

@@ -1,57 +1,57 @@
 --- !ruby/object:Gem::Specification
 name: quarantine
 version: !ruby/object:Gem::Version
-  version: 1.0.7
+  version: 2.1.3
 platform: ruby
 authors:
 - Flexport Engineering, Eric Zhu
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2020-12-01 00:00:00.000000000 Z
+date: 2021-04-15 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
-  name: aws-sdk-dynamodb
+  name: rspec
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '1'
+        version: '3.0'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '1'
+        version: '3.0'
 - !ruby/object:Gem::Dependency
-  name: rspec
+  name: rspec-retry
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '3.0'
+        version: '0.6'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
-        version: '3.0'
+        version: '0.6'
 - !ruby/object:Gem::Dependency
-  name: rspec-retry
+  name: sorbet-runtime
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - '='
       - !ruby/object:Gem::Version
-        version: '0.6'
+        version: 0.5.6338
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - "~>"
+    - - '='
       - !ruby/object:Gem::Version
-        version: '0.6'
+        version: 0.5.6338
 description:
 email:
 - ericzhu77@gmail.com
@@ -67,6 +67,7 @@ files:
 - lib/quarantine/cli.rb
 - lib/quarantine/databases/base.rb
 - lib/quarantine/databases/dynamo_db.rb
+- lib/quarantine/databases/google_sheets.rb
 - lib/quarantine/error.rb
 - lib/quarantine/rspec_adapter.rb
 - lib/quarantine/test.rb
@@ -91,8 +92,8 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.0.2
+rubygems_version: 3.0.8
 signing_key:
 specification_version: 4
-summary: Quarantine flaky Ruby Rspec tests
+summary: Quarantine flaky RSpec tests
 test_files: []