pact_broker 2.109.0 → 2.110.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: a915f7f2cb962cfc2d315338305f170cc875746ce0567826b54a8245dc06afa9
-  data.tar.gz: 3f2db5d2e4c1a5ece6b6b5fef712f3bbcaf85eae49856b9309632ff525d23173
+  metadata.gz: 0747ca988604f56bec96bb0511242e8b3e9e2c18c823ec53742a1c58f80eee16
+  data.tar.gz: fbdc8b9be85cf443ccd8ddcb58704a1aece12759e267aea1b4f79ba0d05fbf9c
 SHA512:
-  metadata.gz: 4cc5845f6611967e5a33865b05da2a214d8fa2d28762943ea4f112bab8a02163830acde551342486cb2b03690564b8a33eb3e5242afd93bc5e70249c1416ca56
-  data.tar.gz: 12f531f2a74c399bc48a26b7afb82e68453b35764c3f5aca5434a1fc55c9f377590508ea6ee41840547652ce240cce2dc4d244d88e4fc2b4074b8c8a8f441dfb
+  metadata.gz: 416b29a9c6fe749e990ce9a051e526568d1a453c78db704d68e63002bb0ba1c17f8117b1d54ec540bea609168e26d2e7583bf087394aa249d9bacd6f0ad58ea5
+  data.tar.gz: b729576d06caac277699def3e48bfd1c75400467018da3b740fad3ff1ba84e628388369dddb60f964cf4b948d66d27096feca800f8e80314be762d327cdd1e54

data/CHANGELOG.md

@@ -1,3 +1,28 @@
+<a name="v2.110.0"></a>
+### v2.110.0 (2024-04-02)
+
+#### Features
+
+* reduce contention when updating the contract_data_updated_at field for integrations (#671)	 ([ff72d03c](/../../commit/ff72d03c))
+* support consumer version selector for all branches (#667)	 ([34334ca8](/../../commit/34334ca8))
+
+* **clean**
+  * use postgres advisory locks to ensure only one process can run a clean at a time (#672)	 ([637c25fa](/../../commit/637c25fa))
+
+#### Bug Fixes
+
+* use for_all_tag_heads instead of latest_by_consumer_tag when fetching wip by branch	 ([14148a34](/../../commit/14148a34))
+* optimise WIP pacts by using branch/tag heads (#668)	 ([871209e1](/../../commit/871209e1))
+* improve performance of WIP pacts by using branch heads instead of calculating latest pact for branch	 ([f9705583](/../../commit/f9705583))
+
+<a name="v2.109.1"></a>
+### v2.109.1 (2024-02-21)
+
+#### Bug Fixes
+
+* improve performance for 'pacts for verification' queries	 ([299a6abe](/../../commit/299a6abe))
+* correct spelling in message when pact is modified	 ([ae62ae7a](/../../commit/ae62ae7a))
+
 <a name="v2.109.0"></a>
 ### v2.109.0 (2024-02-01)
 

data/README.md

@@ -2,7 +2,6 @@
 [![Gem Version](https://badge.fury.io/rb/pact_broker.svg)](http://badge.fury.io/rb/pact_broker)
  ![Build status](https://github.com/pact-foundation/pact_broker/workflows/Test/badge.svg)
  [![Join the chat at https://pact-foundation.slack.com/](https://img.shields.io/badge/chat-on%20slack-blue.svg?logo=slack)](https://slack.pact.io)
- [![security](https://hakiri.io/github/pact-foundation/pact_broker/master.svg)](https://hakiri.io/github/pact-foundation/pact_broker/master)
 
 The Pact Broker is an application for sharing of consumer driven contracts and verification results. It is optimised for use with "pacts" (contracts created by the [Pact][pact-docs] framework), but can be used for any type of contract that can be serialized to JSON.
 
@@ -151,7 +150,7 @@ You can use the [Pact Broker Docker image][docker] or [Terraform on AWS][terrafo
 
 * Are you sure you don't just want to use the [Pact Broker Docker image][docker]? No Docker at your company yet? Ah well, keep reading.
 * Create a PostgreSQL database.
-  * To ensure you're on a supported version of the database that you choose, check the [travis.yml][travisyml] file to see which versions we're currently running our tests against.
+  * To ensure you're on a supported version of the database that you choose, check the [.github/workflows/test.yml](.github/workflows/test.yml) file to see which versions we're currently running our tests against.
   * MySQL was supported for the native Ruby application until around 2021, but the official `pactfoundation/pact-broker` Docker image does not support it. New features will not be optimised for MySQL, and some new features may not even be supported on it (eg. the database clean feature).
 * You'll find a sample database creation script in the [example/config.ru](https://github.com/pact-foundation/pact_broker/blob/master/example/config.ru).
 * Install ruby 2.7 and the latest version of bundler (if you've come this far, I'm assuming you know how to do both of these. Did I mention there was a [Docker][docker] image?)

data/docs/developer/design_pattern_for_eager_loading_collections.md

@@ -0,0 +1,23 @@
+# Design pattern for eager loading collections
+
+For collection resources (eg. `/versions` ), associations included in the items (eg. branch versions) must be eager loaded for performance reasons.
+
+The responsiblities of each class used to render a collection resource are as follows:
+
+* collection decorator (eg. `VersionsDecorator`) - delegate each item in the collection to be rendered by the decorator for the individual item, render pagination links
+* item decorator (eg. `VersionDecorator`) - render the JSON for each item
+* resource (eg. `PactBroker::Api::Resources::Versions`) - coordinate between, and delegate to, the service and the decorator
+* service (eg. `PactBroker::Versions::Service`) - just delegate to repository, as there is no business logic required
+* repository (eg. `PactBroker::Versions::Repository`) - load the domain objects from the database
+
+If the associations for a model are not eager loaded, then each individual association will be lazy loaded when the decorator for the item calls the association method to render it. This results in at least `<number of items in the collection> * <number of associations to render>` calls to the database, and potentially more if any of the associations have their own associations that are required to render the item. This can cause significant performance issues.
+
+To efficiently render a collection resource, associations must be eager loaded when the collection items are loaded from the database in the repository. Since the repository method for loading the collection may be used in multiple places, and the eager loaded associations required for each of those places may be different (some may not require any associations to be eager loaded), we do not want to hard code the repository to load a fixed set of associations. The list of associations to eager load is therefore passed in to the repository finder method as an argument `eager_load_associations`.
+
+The decorator is the class that knows what associations are going to be called on the model to render the JSON, so following the design guideline of "put things together that change together", the best place for the declaration of "what associations should be eager loaded for this decorator" is in the decorator itself. The `PactBroker::Api::Decorators::BaseDecorator` has a default implementation of this method called `eager_load_associations` which attempts to automatically identify the required associations, but this can be overridden when necessary.
+
+We can therefore add the following responsiblities to our previous list:
+
+* item decorator - return a list of all the associations (including nested associations) that should be eager loaded in order to render its item
+* repository - eager load the associations that have been passed into it
+* resource - pass in the eager load associations to the repository from the decorator

data/docs/developer/matrix.md

@@ -0,0 +1,95 @@
+# The Matrix
+
+Read [these docs](https://docs.pact.io/pact_broker/advanced_topics/matrix_selectors) first for a introduction to matrix selectors and options from a user's perspective.
+
+## Terminology
+
+### Selectors
+
+#### Selector classes
+
+* `Unresolved selectors` - `PactBroker::Matrix::UnresolvedSelector` the class of the specified and ignore selectors, as they are created directly from the HTTP query. At this point, they are "unresolved" in that we do not know the IDs of the pacticipants/versions that they represent, and whether or not they even exist.
+
+* `Resolved selectors` - `PactBroker::Matrix::ResolvedSelector` the class of the object created for each selector, after its pacticipant and version IDs have been identified. If a selector represents multiple pacticipant versions (eg `{ branch: "main" }`) then one `ResolvedSelector` object will be returned for each pacticipant version found. The resolved selector keeps a reference to its original unresolved selector, as well as the pacticipant ID and the version ID. If the version or pacticipant specified does not actually exist, then a resolved selector is returned that indicates this.
+
+#### Selector collections
+
+* `Specified selectors` - the collection of objects that describe the application version(s) specified explicitly in the matrix query. eg. in `can-i-deploy --pacticipant Foo --version 1` the specified selector is `PactBroker::Matrix::UnresolvedSelector.new(pacticipant_name: "Foo", pacticipant_version_number: "1")`. There may be multiple specified selectors, for example, in the Matrix page for an integration Foo/Bar, the specified selectors would be `[ UnresolvedSelector.new(pacticipant_name: "Foo"), UnresolvedSelector.new(pacticipant_name: "Bar")]`. The selectors may use various combinations of pacticipant name, version number, branch, tag or environment to identify the pacticipant versions.
+
+* `Ignore selectors` - the collection of objects that describe the applications (or application versions) to ignore, if there are any missing or failed verifications between the versions for the specified selectors and the ignore selectors. eg. If we know that provider Dog is not ready and the verifications are failing, but the integration is feature toggled off in the Foo code, we would use the command `can-i-deploy --pacticipant Foo --version 1 --to-environment production --ignore Dog` to allow `can-i-deploy` to pass. An ignore selector can have a version number, but it's more common to just provide the name.
+
+* `Inferred selectors` - the collection of objects that describe the application versions(s) that already exist in the target environment/with the target tag/on the target branches. These are identified during the matrix query when the `can-i-deploy` query has a `--to TAG`/`--to-environment`/`--with-main-branches` option specified, and they allow us to find the full set of matrix rows that tell us whether or the application version we're about to deploy is compatible with its integrated applications. For example, given Foo v1 is a consumer of Bar, and Bar v8 is in production, and the `can-i-deploy` command is `can-i-deploy --pacticipant Foo --version 1 --to-environment production`, then an inferred unresolved selector is created for Bar in the production environment (`UnresolvedSelector.new(pacticipant_name: "Bar", environment_name: "production")`) which is then resolved to Bar v8.
+
+#### Notes on selector classes/collections
+
+Specified, ignore, and inferred selectors all start life as `UnresolvedSelector` objects, which then get resolved into `ResolvedSelector` objects.
+
+### Options
+
+TBC.
+
+## How the matrix query works
+
+1. Given that Foo v1 is a consumer of Bar, and Bar v8 is currently in production, take the pact-broker CLI command `can-i-deploy --pacticipant Foo --version 1 --to-environment production --ignore Dog`
+
+1. Turn that into a query string and make a request to the `/matrix` endpoint.
+
+1. Parse the HTTP query into the specified selectors, the ignore selectors and options.
+
+  * specified selectors - `PactBroker::Matrix::UnresolvedSelector.new(pacticipant_name: "Foo", pacticipant_version_number: "1")`
+  * ignore selectors - `PactBroker::Matrix::UnresolvedSelector.new(pacticipant_name: "Dog")`
+  * options: `{ to_environment: "production" }`
+
+1. Validate the selectors.
+
+  * Ensure conflicting fields are not used.
+  * Return an error if any of the pacticipants in the specified selectors or the environment do not exist.
+  * Do not check for the existance of version numbers or tags, or pacticipants in the ignore selectors.
+
+1. "Resolve" the ignore selectors (find the pacticipant and version IDs for the selectors).
+
+1. "Resolve" the specified selectors, passing in the ignore selectors to identify whether or not the resolved selector should be "ignored".
+
+1. Identify the integrations that the versions in the specified selectors are involved in.
+
+  * Identify the providers of the consumers: for the versions of each specified selector, find any pacts created for that version. If any exist, that means
+    the provider for each pact MUST be deployed to the target environment, and the deployed version must have a successful verification with the specified consumer version for the consumer to be safe to deploy. The providers must be identified for the specific consumer *version* of the selector, not just *consumer*, as different versions of a consumer may have different providers as integrations are created and removed over time. This is why we cannot just query the integrations table.
+
+    Represent the integration by creating a `PactBroker::Matrix::Integration` object for each consumer/provider pair, and mark it as `required: true`.
+
+  * Identify the consumers of the providers: for the pacticipant of each specified selector, find any integrations in the integrations table where
+    the specified pacticipant is the provider. Because specific provider versions don't have a dependency on any specific consumer versions being already present in an environment, we do not need to run this query at the pacticipant version level - we can just check the presence of any consumers at the integration level.
+
+    Represent the integration by creating a `PactBroker::Matrix::Integration` object for each consumer/provider pair, and mark it as `required: false`.
+
+1. Identify the inferred unresolved selectors (the selectors for the pacticipant versions that are already in the target scope)
+
+  * Collect all the pacticipant names from the integrations identified in the previous step. For every pacticipant name that does NOT already have a specified selector, create a new unresolved "inferred" selector for that pacticipant, and set the version selection attributes to be the target scope from the original `can-i-deploy` query. eg. Given we have identifed the required `Integration` for consumer Foo and provider Bar, and we are determining if we can deploy Foo to production, create an unresolved selector for Bar in the production environment (`UnresolvedSelector.new(pacticipant_name: "Bar", environment_name: "production"`).
+
+1. "Resolve" the inferred selectors (find the pacticipant and version IDs).
+
+1. Add the specified and inferred resolved selectors together to make the final collection of selectors that will be used to query the matrix.
+
+1. Peform the matrix query.
+
+  * When there are 2 or more total resolved selectors (the usual usecase):
+
+    * Create a collection of all the pacticipants in the selectors (let's call it `all_pacticipants_in_selectors`).
+
+    * Create the pact/consumer version dataset
+
+      * For each selector, find the pact publications where the consumer version is one of the versions described by the selector, and the provider is one of `all_pacticipants_in_selectors`.
+
+    * Create the verification/provider version dataset.
+
+      * For each selector, find the verifications where the provider version is one of the versions described by the selector, and the consumer is one of `all_pacticipants_in_selectors`.
+
+    * Join the pact/consumer verison dataset to the verification/provider version dataset.
+
+      * The two datasets are joined on the `pact_version_id` - this is the ID of the unique pact content that every pact publication and provider verification has a reference to. A left outer join is used, so that if there is a pact that doesn't have a verification, a row is present for the pact, with empty provider version columns. This allows us to identify that there is a missing verification.
+
+
+  * When there is only 1 total resolved selector: this is an unusual usecase, and cannot be done via the UI or the CLI, so I'm not going to spend time documenting it. Just know it is supported and theoretically possible.
+
+
+

data/docs/developer/rack.md

@@ -0,0 +1,11 @@
+# Rack
+
+https://medium.com/quick-code/rack-middleware-vs-rack-application-vs-rack-the-gem-vs-rack-the-architecture-912cd583ed24
+https://github.com/rack/rack/blob/main/SPEC.rdoc
+https://www.rubyguides.com/2018/09/rack-middleware/
+
+
+* Responds to `call`
+* Accepts a hash of parameters
+* Returns an array where the first item is the http status, the second is a hash of headers, and the third is an object that responds to `each` (or `call`) that provides the body (99% of the time it's an array of length 1 with a string)
+

data/lib/pact_broker/api/contracts/consumer_version_selector_contract.rb

@@ -13,7 +13,7 @@ module PactBroker
         json do
           optional(:mainBranch).filled(included_in?: [true])
           optional(:tag).filled(:str?)
-          optional(:branch).filled(:str?)
+          optional(:branch).filled { str? | eql?(true) }
           optional(:matchingBranch).filled(included_in?: [true])
           optional(:latest).filled(included_in?: [true, false])
           optional(:fallbackTag).filled(:str?)

data/lib/pact_broker/api/decorators/base_decorator.rb

@@ -34,7 +34,16 @@ module PactBroker
           end
         end
 
-        # Returns the names of the model associations to eager load for use with this decorator
+        # Returns the names of the model associations to eager load for use with this decorator.
+        # The default implementation attempts to do an "auto detect" of the associations.
+        # For single item decorators, it attempts to identify the attributes that are themselves models.
+        # For collection decorators, it delegates to the eager_load_associations
+        # method of the single item decorator used to decorate the collection.
+        #
+        # The "auto detect" logic can only go so far. It cannot identify when a child object needs its own
+        # child object(s) to render the attribute.
+        # This method should be overridden when the "auto detect" logic cannot identify the correct associations
+        # to load. eg VersionDecorator
         # @return [Array<Symbol>]
         def self.eager_load_associations
           if is_collection_resource?

data/lib/pact_broker/api/decorators/version_decorator.rb

@@ -17,6 +17,20 @@ module PactBroker
 
         include Timestamps
 
+        # Returns the list of associations that must be eager loaded to efficiently render a version
+        # when this decorator is used in a collection (eg. VersionsDecorator)
+        # The associations that need to be eager loaded for the VersionDecorator
+        # are hand coded
+        # @return <Array>
+        def self.eager_load_associations
+          [
+            :pacticipant,
+            :pact_publications,
+            { branch_versions: [:version, :branch_head, { branch: :pacticipant }] },
+            { tags: :head_tag }
+          ]
+        end
+
         link :self do | options |
           {
             title: "Version",

data/lib/pact_broker/api/resources/branch_versions.rb

@@ -31,7 +31,7 @@ module PactBroker
         end
 
         def versions
-          @versions ||= version_service.find_pacticipant_versions_in_reverse_order(pacticipant_name, { branch_name: identifier_from_path[:branch_name] }, pagination_options)
+          @versions ||= version_service.find_pacticipant_versions_in_reverse_order(pacticipant_name, { branch_name: identifier_from_path[:branch_name] }, pagination_options, decorator_class(:versions_decorator).eager_load_associations)
         end
 
         def policy_name

data/lib/pact_broker/api/resources/provider_pacts_for_verification.rb

@@ -15,6 +15,10 @@ module PactBroker
           [["application/hal+json", :to_json]]
         end
 
+        # TODO drop support for GET in next major version.
+        # GET was only used by the very first Ruby Pact clients that supported the 'pacts for verification'
+        # feature, until it became clear that the parameters for the request were going to get nested and complex,
+        # at which point the POST was added.
         def allowed_methods
           ["GET", "POST", "OPTIONS"]
         end
@@ -32,6 +36,7 @@ module PactBroker
           end
         end
 
+        # For this endoint, the POST is a "read" action (used for Pactflow)
         def read_methods
           super + %w{POST}
         end

data/lib/pact_broker/api/resources/versions.rb

@@ -31,7 +31,7 @@ module PactBroker
         end
 
         def versions
-          @versions ||= version_service.find_all_pacticipant_versions_in_reverse_order(pacticipant_name, pagination_options)
+          @versions ||= version_service.find_pacticipant_versions_in_reverse_order(pacticipant_name, {}, pagination_options, decorator_class(:versions_decorator).eager_load_associations)
         end
 
         def policy_name

data/lib/pact_broker/app.rb

@@ -27,6 +27,7 @@ require "pact_broker/config/basic_auth_configuration"
 require "pact_broker/api/authorization/resource_access_policy"
 require "pact_broker/api/middleware/http_debug_logs"
 require "pact_broker/application_context"
+require "pact_broker/db/advisory_lock"
 
 module PactBroker
 
@@ -101,22 +102,19 @@ module PactBroker
 
     def prepare_database
       logger.info "Database schema version is #{PactBroker::DB.version(configuration.database_connection)}"
+      lock = PactBroker::DB::AdvisoryLock.new(configuration.database_connection, :migrate, :pg_advisory_lock)
       if configuration.auto_migrate_db
-        migration_options = { allow_missing_migration_files: configuration.allow_missing_migration_files }
-        if PactBroker::DB.is_current?(configuration.database_connection, migration_options)
-          logger.info "Skipping database migrations as the latest migration has already been applied"
-        else
-          logger.info "Migrating database schema"
-          PactBroker::DB.run_migrations configuration.database_connection, migration_options
-          logger.info "Database schema version is now #{PactBroker::DB.version(configuration.database_connection)}"
+        lock.with_lock do
+          ensure_all_database_migrations_are_applied
         end
       else
         logger.info "Skipping database schema migrations as database auto migrate is disabled"
       end
 
       if configuration.auto_migrate_db_data
-        logger.info "Migrating data"
-        PactBroker::DB.run_data_migrations configuration.database_connection
+        lock.with_lock do
+          run_data_migrations
+        end
       else
         logger.info "Skipping data migrations"
       end
@@ -125,6 +123,23 @@ module PactBroker
       PactBroker::Webhooks::Service.fail_retrying_triggered_webhooks
     end
 
+    def ensure_all_database_migrations_are_applied
+      migration_options = { allow_missing_migration_files: configuration.allow_missing_migration_files }
+
+      if PactBroker::DB.is_current?(configuration.database_connection, migration_options)
+        logger.info "Skipping database migrations as the latest migration has already been applied"
+      else
+        logger.info "Migrating database schema"
+        PactBroker::DB.run_migrations(configuration.database_connection, migration_options)
+        logger.info "Database schema version is now #{PactBroker::DB.version(configuration.database_connection)}"
+      end
+    end
+
+    def run_data_migrations
+      logger.info "Migrating data"
+      PactBroker::DB.run_data_migrations(configuration.database_connection)
+    end
+
     def load_configuration_from_database
       configuration.load_from_database!
     end

data/lib/pact_broker/contracts/service.rb

@@ -35,7 +35,7 @@ module PactBroker
         version, version_notices = create_version(parsed_contracts)
         tags = create_tags(parsed_contracts, version)
         pacts, pact_notices = create_pacts(parsed_contracts, base_url)
-        update_integrations(pacts)
+        create_or_update_integrations(pacts)
         notices = version_notices + pact_notices
         ContractsPublicationResults.from_hash(
           pacticipant: version.pacticipant,
@@ -304,7 +304,11 @@ module PactBroker
         PactBroker::Api::PactBrokerUrls.triggered_webhook_logs_url(triggered_webhook, base_url)
       end
 
-      def update_integrations(pacts)
+      # Creating/updating the integrations all at once at the end of the transaction instead
+      # of one by one, as each pact is created, reduces the amount of time that
+      # a lock is held on the integrations table, therefore reducing contention
+      # and potential for deadlocks when there are many pacts being published at once.
+      def create_or_update_integrations(pacts)
         integration_service.handle_bulk_contract_data_published(pacts)
       end
 

data/lib/pact_broker/db/advisory_lock.rb

@@ -0,0 +1,58 @@
+require "pact_broker/logging"
+
+# Uses a Postgres advisory lock to ensure that a given block of code can only have ONE
+# thread in excution at a time against a given database.
+# When the database is not Postgres, the block will yield without any locks, allowing
+# this class to be used safely with other database types, but without the locking
+# functionality.
+#
+# This is a wrapper around the actual implementation code in the Sequel extension from https://github.com/yuryroot/sequel-pg_advisory_lock
+# which was copied into this codebase and modified for usage in this codebase.
+#
+# See https://www.postgresql.org/docs/16/functions-admin.html#FUNCTIONS-ADVISORY-LOCKS for docs on lock types
+#
+
+module PactBroker
+  module DB
+    class AdvisoryLock
+      include PactBroker::Logging
+
+      def initialize(database_connection, name, type = :pg_try_advisory_lock)
+        @database_connection = database_connection
+        @name = name
+        @type = type
+        @lock_obtained = false
+        register_advisory_lock if postgres?
+      end
+
+      def with_lock
+        if postgres?
+          @database_connection.with_advisory_lock(@name) do
+            logger.debug("Lock #{@name} obtained")
+            @lock_obtained = true
+            yield
+          end
+        else
+          logger.warn("Executing block without lock as this is not a Postgres database")
+          @lock_obtained = true
+          yield
+        end
+      end
+
+      def lock_obtained?
+        @lock_obtained
+      end
+
+      private
+
+      def postgres?
+        @database_connection.adapter_scheme.to_s =~ /postgres/
+      end
+
+      def register_advisory_lock
+        @database_connection.extension :pg_advisory_lock
+        @database_connection.register_advisory_lock(@name, @type)
+      end
+    end
+  end
+end

data/lib/pact_broker/domain/version.rb

@@ -4,9 +4,10 @@ require "pact_broker/versions/eager_loaders"
 
 module PactBroker
   module Domain
-    VERSION_COLUMNS = [:id, :number, :repository_ref, :pacticipant_id, :order, :created_at, :updated_at, :build_url]
+    class Version < Sequel::Model
+      VERSION_COLUMNS = Sequel::Model.db.schema(:versions).collect(&:first) - [:branch] # do not include the branch column, as we now have a branches table
+      set_dataset(Sequel::Model.db[:versions].select(*VERSION_COLUMNS.collect{ | column | Sequel.qualify(:versions, column) }))
 
-    class Version < Sequel::Model(Sequel::Model.db[:versions].select(*VERSION_COLUMNS.collect{ | column | Sequel.qualify(:versions, column) }))
       set_primary_key :id
 
       plugin :timestamps, update_on_create: true

data/lib/pact_broker/integrations/integration.rb

@@ -7,7 +7,17 @@ require "pact_broker/verifications/latest_verification_for_consumer_and_provider
 
 module PactBroker
   module Integrations
-    class Integration < Sequel::Model(Sequel::Model.db[:integrations].select(:id, :consumer_id, :provider_id, :contract_data_updated_at))
+    # The columns are explicitly specified for the Integration object so that the consumer_name and provider_name columns aren't included
+    # in the model.
+    # Those columns exist in the integrations table because the integrations table used to be an integrations view based on the
+    # pact_publications table, and those columns existed in the view.
+    # When the view was migrated to be a table (in db/migrations/20211102_create_table_temp_integrations.rb and the following migrations)
+    # the columns had to be maintained for backwards compatiblity.
+    # They are not used by the current code, however.
+    class Integration < Sequel::Model
+      INTEGRATION_COLUMNS = Sequel::Model.db.schema(:integrations).collect(&:first) - [:consumer_name, :provider_name]
+      set_dataset(Sequel::Model.db[:integrations].select(*INTEGRATION_COLUMNS))
+
       set_primary_key :id
       plugin :insert_ignore, identifying_columns: [:consumer_id, :provider_id]
       associate(:many_to_one, :consumer, :class => "PactBroker::Domain::Pacticipant", :key => :consumer_id, :primary_key => :id)

data/lib/pact_broker/integrations/repository.rb

@@ -28,6 +28,20 @@ module PactBroker
         nil
       end
 
+      # Ensure an Integration exists for each consumer/provider pair.
+      # Using SELECT ... INSERT IGNORE rather than just INSERT IGNORE so that we do not
+      # need to lock the table at all when the integrations already exist, which will
+      # be the most common use case. New integrations get created incredibly rarely.
+      # The INSERT IGNORE is used rather than just INSERT to handle race conditions
+      # when requests come in parallel.
+      # @param [Array<Object>] where each object has a consumer and a provider
+      def create_for_pacts(objects_with_consumer_and_provider)
+        published_integrations = objects_with_consumer_and_provider.collect{ |i| { consumer_id: i.consumer.id, provider_id: i.provider.id } }
+        existing_integrations = Sequel::Model.db[:integrations].select(:consumer_id, :provider_id).where(Sequel.|(*published_integrations) ).all
+        new_integrations = (published_integrations - existing_integrations).collect{ |i| i.merge(created_at: Sequel.datetime_class.now, contract_data_updated_at: Sequel.datetime_class.now) }
+        Integration.dataset.insert_ignore.multi_insert(new_integrations)
+      end
+
       def delete(consumer_id, provider_id)
         Integration.where(consumer_id: consumer_id, provider_id: provider_id).delete
       end
@@ -36,18 +50,43 @@ module PactBroker
       # @param [PactBroker::Domain::Pacticipant, nil] consumer the consumer for the integration, or nil if for a provider-only event (eg. Pactflow provider contract published)
       # @param [PactBroker::Domain::Pacticipant] provider the provider for the integration
       def set_contract_data_updated_at(consumer, provider)
-        Integration
-          .where({ consumer_id: consumer&.id, provider_id: provider.id }.compact )
-          .update(contract_data_updated_at: Sequel.datetime_class.now)
+        set_contract_data_updated_at_for_multiple_integrations([OpenStruct.new(consumer: consumer, provider: provider)])
       end
 
 
-      # Sets the contract_data_updated_at for the integrations as specified by an array of objects which each have a consumer and provider
-      # @param [Array<Object>] where each object has a consumer and a provider
+      # Sets the contract_data_updated_at for the integrations as specified by an array of objects which each have a consumer and provider.
+      #
+      # The contract_data_updated_at attribute is only ever used for ordering the list of integrations on the index page of the *Pact Broker* UI,
+      # so that the most recently updated integrations (the ones you're likely working on) are showed at the top of the first page.
+      # There is often contention around updating it however, which can cause deadlocks, and slow down API responses.
+      # Because it's not a critical field (eg. it won't change any can-i-deploy results), the easiest way to reduce this contention
+      # is to just not update it if the row is locked, because if it is locked, the value of contract_data_updated_at is already
+      # going to be a date from a few seconds ago, which is perfectly fine for the purposes for which we are using the value.
+      #
+      # Notes on SKIP LOCKED:
+      # SKIP LOCKED is only supported by Postgres.
+      # When executing SELECT ... FOR UPDATE SKIP LOCKED, the SELECT will run immediately, not waiting for any other transactions,
+      # and only return rows that are not already locked by another transaction.
+      # The FOR UPDATE is required to make it work this way - SKIP LOCKED on its own does not work.
+      #
+      # @param [Array<Object>] where each object MAY have a consumer and does have a provider (for Pactflow provider contract published there is no consumer)
       def set_contract_data_updated_at_for_multiple_integrations(objects_with_consumer_and_provider)
-        consumer_and_provider_ids = objects_with_consumer_and_provider.collect{ | object | [object.consumer.id, object.provider.id] }.uniq
+        consumer_and_provider_ids = objects_with_consumer_and_provider.collect{ | object | { consumer_id: object.consumer&.id, provider_id: object.provider.id }.compact }.uniq
+
+        # MySQL doesn't support an UPDATE with a subquery. FFS. Really need to do a major version release and delete the support code.
+        criteria =  if Integration.dataset.supports_skip_locked?
+                      integration_ids_to_update = Integration
+                                                  .select(:id)
+                                                  .where(Sequel.|(*consumer_and_provider_ids))
+                                                  .for_update
+                                                  .skip_locked
+                      { id: integration_ids_to_update }
+                    else
+                      Sequel.|(*consumer_and_provider_ids)
+                    end
+
         Integration
-          .where([:consumer_id, :provider_id] => consumer_and_provider_ids)
+          .where(criteria)
           .update(contract_data_updated_at: Sequel.datetime_class.now)
       end
     end

data/lib/pact_broker/integrations/service.rb

@@ -21,6 +21,7 @@ module PactBroker
       # @param [PactBroker::Domain::Pacticipant] consumer or nil
       # @param [PactBroker::Domain::Pacticipant] provider
       def self.handle_contract_data_published(consumer, provider)
+        integration_repository.create_for_pact(consumer.id, provider.id)
         integration_repository.set_contract_data_updated_at(consumer, provider)
       end
 
@@ -28,6 +29,7 @@ module PactBroker
       # Callback to invoke when a batch of contract data is published (eg. the publish contracts endpoint)
       # @param [Array<Object>] where each object has a consumer and a provider
       def self.handle_bulk_contract_data_published(objects_with_consumer_and_provider)
+        integration_repository.create_for_pacts(objects_with_consumer_and_provider)
         integration_repository.set_contract_data_updated_at_for_multiple_integrations(objects_with_consumer_and_provider)
       end
 

data/lib/pact_broker/locale/en.yml

@@ -50,7 +50,7 @@ en:
       contract:
         pact_published: Pact successfully published for %{consumer_name} version %{consumer_version_number} and provider %{provider_name}.
         same_pact_content_published: Pact successfully republished for %{consumer_name} version %{consumer_version_number} and provider %{provider_name} with no content changes.
-        pact_modified_for_same_version: Pact with changed content published over existing content for %{consumer_name} version %{consumer_version_number} and provider %{provider_name}. This is not recommended in normal cicumstances and may indicate that you have not configured your Pact pipeline correctly. For more information see https://docs.pact.io/go/versioning
+        pact_modified_for_same_version: Pact with changed content published over existing content for %{consumer_name} version %{consumer_version_number} and provider %{provider_name}. This is not recommended in normal circumstances and may indicate that you have not configured your Pact pipeline correctly. For more information see https://docs.pact.io/go/versioning
         pact_merged: Pact content merged with existing content for %{consumer_name} version %{consumer_version_number} and provider %{provider_name}.
       events:
         pact_published_unchanged_with_single_tag: pact content is the same as previous version with tag %{tag_name} and no new tags were applied

data/lib/pact_broker/matrix/matrix_row.rb

@@ -55,7 +55,6 @@ module PactBroker
         Sequel[:v][:verification_id],
         Sequel[:v][:provider_version_id],
         Sequel[:v][:provider_version_created_at]
-
       ]
 
       # Must be kept in sync with PactBroker::Matrix::EveryRow

data/lib/pact_broker/matrix/resolved_selector.rb

@@ -11,7 +11,6 @@ require "pact_broker/hash_refinements"
 module PactBroker
   module Matrix
     class ResolvedSelector < Hash
-
       using PactBroker::HashRefinements
 
       # A version ID of -1 will not match any rows, which is what we want to ensure that