pact_broker 2.24.0 → 2.25.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 221e35de8afb505b63a3bd3eb76ccc8a332e9c99
-  data.tar.gz: 985e0d73ff17ad8d6a7d9cd044863576a53b1ce9
+  metadata.gz: ee46917fc8d40d1f2621e54b29820617690837c3
+  data.tar.gz: e8b2a5f1c826f21d9c8e974a4cee4d3a9fcca951
 SHA512:
-  metadata.gz: 21b9b6aee402ff56fdaf1149867f2ff729a5ca350c3a60c82b0b2c2df9b3788ac2db15b185113fc9108f80e52cb0430340c04f66a473bdef536b5323c6e09275
-  data.tar.gz: 22b9cc5f105543a9a7fd359cf7616c78c6e4e168f9f315ed90944133c97b4a9bd6dc7cf6c4a7c1346a888b4243e468cfe6361edfbbf2c9ffedf902662353ad0f
+  metadata.gz: 625e5ef7a725cd4f62a814f5387dcd2ad78692b1859c61f99cf577b1a325069b8d50b066c233eb5c48c0662a751409fe34a3d1e9b8672b71d29fde071700c359
+  data.tar.gz: 525863990061dcd5f96541e49ed063b277d3b9e33e39e617d21ca88a9dfcd33e701e9ea0d61333c08791f4e1e0375b80cb664768db8410b1f8725823ba0b7be8

data/CHANGELOG.md

@@ -1,3 +1,41 @@
+<a name="v2.25.0"></a>
+### v2.25.0 (2018-08-09)
+
+
+#### Features
+
+* make contract_content_changed tag aware	 ([8e7d0e47](/../../commit/8e7d0e47))
+* move "latest id" upsert logic into own class	 ([360d236f](/../../commit/360d236f))
+* replace latest_verifications view with latest_verifications_for_provider_versions	 ([ee4fef99](/../../commit/ee4fef99))
+* sort tags in text/plain version of /dashboard	 ([0a821490](/../../commit/0a821490))
+* optimise data migrations that set pacticipant ids	 ([9a148b80](/../../commit/9a148b80))
+* add pact_version_id to latest_pact_publication_ids table	 ([627ce6db](/../../commit/627ce6db))
+* optimise query for loading latest verification for the latest pacts for each tag on index page	 ([8bc58442](/../../commit/8bc58442))
+* add consumer_id to pact_publications table	 ([d520e228](/../../commit/d520e228))
+* add consumer id and provider id to verifications table to speed up queries	 ([d5698901](/../../commit/d5698901))
+* run data migrations on startup to ensure data created by old nodes is migrated when performing a rolling update	 ([ec59ba5b](/../../commit/ec59ba5b))
+* move migrations on feature branch to run last	 ([78d31c49](/../../commit/78d31c49))
+* remove refresh of materialized_matrix and materialized_head_matrix	 ([262af756](/../../commit/262af756))
+* keep track of the latest verification for a pact version for each provider version, rather than calculating it	 ([110495a9](/../../commit/110495a9))
+* store latest pact publication ids for each consumer version	 ([d239bfdf](/../../commit/d239bfdf))
+* keep track of latest pact revision in table rather than calculating it	 ([e9cd9e1b](/../../commit/e9cd9e1b))
+* move pb:wip-provider-pacts to beta:wip-provider-pacts	 ([3008140a](/../../commit/3008140a))
+* make feature toggles case insensitive and space aware	 ([f4a03c0c](/../../commit/f4a03c0c))
+* add feature toggle via environment variable PACT_BROKER_FEATURES	 ([f0737b97](/../../commit/f0737b97))
+* keep track of the latest verification for a pact version for each provider version, rather than calculating it	 ([b4f18770](/../../commit/b4f18770))
+* store latest pact publication ids for each consumer version	 ([c8fa5545](/../../commit/c8fa5545))
+* keep track of latest pact revision in table rather than calculating it	 ([1db5b7ba](/../../commit/1db5b7ba))
+
+* **index**
+  * eager load latest verification for consumer and provider	 ([1bf61afc](/../../commit/1bf61afc))
+
+
+#### Bug Fixes
+
+* **index page**
+  * correct pact URL when showing index with tags	 ([a5bda98f](/../../commit/a5bda98f))
+
+
 <a name="v2.24.0"></a>
 ### v2.24.0 (2018-07-08)
 

data/CONTRIBUTING.md

@@ -15,8 +15,8 @@ Please provide the following information with your issue to enable us to respond
 * Write tests for any changes
 * Follow existing code style and conventions
 * Separate unrelated changes into multiple pull requests
-* [Squash your commits](squash) into one commit before submitting your PR.
-* Use [semantic commit messages](semantic-commit-messages) so that your changes will show up correctly in the generated change log.
+* [Squash your commits][squash] into one commit before submitting your PR.
+* Use [semantic commit messages][semantic-commit-messages] so that your changes will show up correctly in the generated change log.
 
 * For bigger changes, make sure you start a discussion first by creating an issue and explaining the intended change
 

data/DEVELOPER_DOCUMENTATION.md

@@ -1,5 +1,22 @@
 # Developer Documentation
 
+## File structure
+
+* Application code - [lib](lib)
+  * List of API endpoints - [lib/pact_broker/api.rb](lib/pact_broker/api.rb)
+  * API - [lib/pact_broker/api](lib/pact_broker/api)
+    * HTTP Resources - [lib/pact_broker/api/resources](lib/pact_broker/api/resources) These handle the HTTP requests.
+    * Decorators - [lib/pact_broker/api/decorators](lib/pact_broker/api/decorators) These render the response bodies.
+    * Contracts - [lib/pact_broker/api/contracts](lib/pact_broker/api/contracts) These validate incoming API requests.
+  * Domain - Domain classes were intially created in [lib/pact_broker/domain](lib/pact_broker/domain) but are now put in their own modules. The ones left here just haven't been migrated yet.
+* Database migrations - [db/migrations](db/migrations)
+
+* Tests - `spec`
+  * Isolated tests (mostly) - `spec/lib`
+  * Contract tests - `spec/service_consumers`
+  * High level API functional tests - `spec/features`
+  * Migration tests - `spec/migrations`
+
 ## Domain and database design
 
 ### Domain
@@ -54,38 +71,25 @@ Domain classes are found in `lib/pact_broker/domain`. Many of these classes are
 
 * `latest_tagged_pact_publications` - This view has the same columns as `all_pact_publications`, plus a `tag_name` column. It is used to return the pact for the latest tagged version of a consumer.
 
-* `latest_verifications` - The most recent verification for each pact version.
+* `latest_verifications_for_pact_versions` - The most recent verification for each pact version.
 
 * `matrix` - The matrix of every pact publication and verification. Includes every pact revision (eg. publishing to the same consumer version twice, or using PATCH) and every verification (including 'overwritten' ones. eg. when the same provider build runs twice.)
 
-* `latest_matrix_for_consumer_version_and_provider_version` - This view is a subset of, and has the same columns as, the `matrix`. It removes 'overwritten' pacts and verifications from the matrix (ie. only show latest pact revision for each consumer version and latest verification for each provider version)
-
-### Materialized Views
-
-We can't use proper materialized views because we have to support MySQL :|
-
-So as a hacky solution, there are two tables which act as materialized views to speed up the performance of the matrix and index queries. These tables are updated after any resource is published with a `consumer_name`, `provider_name` or `pacticipant_name` in the URL (see lib/pact_broker/api/resources/base_resource.rb#finish_request).
-
-* `materialized_matrix` table - is populated from the `matrix` view.
-
-* `materialized_head_matrix` table - is populated from `head_matrix` view, and is based on `materialized_matrix`.
-
 ### Dependencies
 
-materialized_head_matrix table (is populated from...)
-  = head_matrix view
-    -> latest_matrix_for_consumer_version_and_provider_version view
-      -> materialized_matrix table (is populated from...)
-        = matrix view
-          -> verifications table
-          -> versions table
-          -> all_pact_publications view
-            -> pact_versions table
-            -> pact_publications table
-            -> pacticipants table
-            -> versions table
-      -> latest_verification_id_for_consumer_version_and_provider_version view
-      -> latest_pact_publication_revision_numbers view
+```
+= head_matrix view
+  -> latest_pact_publications view
+    -> latest_pact_publications_by_consumer_versions view
+      -> latest_pact_publication_ids_by_consumer_versions
+      -> all_pact_publications
+        -> versions, pacticipants, pact_publications, pact_versions
+  -> latest_verifications_for_pact_versions
+    -> latest_verification_ids_for_pact_versions
+    -> versions
+  -> latest_tagged_pact_consumer_version_orders
+  -> latest_pact_publications_by_consumer_versions
+```
 
 ### Useful to know stuff
 

data/DEVELOPER_SETUP.md

@@ -2,8 +2,9 @@
 
 * You will need to install Ruby 2.4, and preferably a ruby version manager. I recommend using [chruby][chruby] and [ruby-install][ruby-install].
 * Install bundler (the Ruby gem dependency manager) `gem install bundler`
-* Check out the pact_broker repository.
-* Run `bundle exec pact_broker:dev:setup`. This will create an example application that you can run locally, that uses the local source code.
+* Check out the pact_broker repository and cd into it.
+* Run `bundle install`. If you have not got mysql or postgres installed locally, comment out the `mysql2` and `pg` development dependency lines in `pact_broker.gemspec`, as these are only really required on Travis.
+* Run `bundle exec rake pact_broker:dev:setup`. This will create an example application that you can run locally, that uses the local source code.
 * To run the example:
 
       cd dev

data/config/database.yml

@@ -11,6 +11,11 @@ test:
   postgres:
     <<: *default
     adapter: postgres
+  docker_postgres:
+    <<: *default
+    adapter: postgres
+    host: "127.0.0.1"
+    port: "5433"
   mysql:
     <<: *default
     adapter: mysql2
@@ -22,6 +27,11 @@ development:
   postgres:
     <<: *default
     adapter: postgres
+  docker_postgres:
+    <<: *default
+    adapter: postgres
+    host: "127.0.0.1"
+    port: "5433"
   mysql:
     <<: *default
     adapter: mysql2

data/db/ddl_statements.rb

@@ -0,0 +1,3 @@
+Dir.glob(File.expand_path(File.join(__FILE__, "..", "ddl_statements", "*.rb"))).sort.each do | path |
+  require path
+end

data/db/ddl_statements/head_matrix_v001.rb

@@ -0,0 +1,31 @@
+HEAD_MATRIX_V1 = "
+select
+  p.consumer_id, p.consumer_name, p.consumer_version_id, p.consumer_version_number, p.consumer_version_order,
+  p.id as pact_publication_id, p.pact_version_id, p.pact_version_sha, p.revision_number as pact_revision_number,
+  p.created_at as pact_created_at,
+  p.provider_id, p.provider_name, lv.provider_version_id, lv.provider_version_number, lv.provider_version_order,
+  lv.id as verification_id, lv.success, lv.number as verification_number, lv.execution_date as verification_executed_at,
+  lv.build_url as verification_build_url,
+  null as consumer_version_tag_name
+from latest_pact_publications p
+left outer join latest_verifications lv
+  on p.pact_version_id = lv.pact_version_id
+
+union all
+
+select
+  p.consumer_id, p.consumer_name, p.consumer_version_id, p.consumer_version_number, p.consumer_version_order,
+  p.id as pact_publication_id, p.pact_version_id, p.pact_version_sha, p.revision_number as pact_revision_number,
+  p.created_at as pact_created_at,
+  p.provider_id, p.provider_name, lv.provider_version_id, lv.provider_version_number, lv.provider_version_order,
+  lv.id as verification_id, lv.success, lv.number as verification_number, lv.execution_date as verification_executed_at,
+  lv.build_url as verification_build_url,
+  lt.tag_name as consumer_version_tag_name
+from latest_tagged_pact_consumer_version_orders lt
+inner join latest_pact_publications_by_consumer_versions p
+  on lt.consumer_id = p.consumer_id
+  and lt.provider_id = p.provider_id
+  and lt.latest_consumer_version_order = p.consumer_version_order
+left outer join latest_verifications lv
+  on p.pact_version_id = lv.pact_version_id
+"

data/db/ddl_statements/head_matrix_v002.rb

@@ -0,0 +1,31 @@
+HEAD_MATRIX_V2 = "
+select
+  p.consumer_id, p.consumer_name, p.consumer_version_id, p.consumer_version_number, p.consumer_version_order,
+  p.id as pact_publication_id, p.pact_version_id, p.pact_version_sha, p.revision_number as pact_revision_number,
+  p.created_at as pact_created_at,
+  p.provider_id, p.provider_name, lv.provider_version_id, lv.provider_version_number, lv.provider_version_order,
+  lv.id as verification_id, lv.success, lv.number as verification_number, lv.execution_date as verification_executed_at,
+  lv.build_url as verification_build_url,
+  null as consumer_version_tag_name
+from latest_pact_publications p
+left outer join latest_verifications_for_pact_versions lv
+  on p.pact_version_id = lv.pact_version_id
+
+union all
+
+select
+  p.consumer_id, p.consumer_name, p.consumer_version_id, p.consumer_version_number, p.consumer_version_order,
+  p.id as pact_publication_id, p.pact_version_id, p.pact_version_sha, p.revision_number as pact_revision_number,
+  p.created_at as pact_created_at,
+  p.provider_id, p.provider_name, lv.provider_version_id, lv.provider_version_number, lv.provider_version_order,
+  lv.id as verification_id, lv.success, lv.number as verification_number, lv.execution_date as verification_executed_at,
+  lv.build_url as verification_build_url,
+  lt.tag_name as consumer_version_tag_name
+from latest_tagged_pact_consumer_version_orders lt
+inner join latest_pact_publications_by_consumer_versions p
+  on lt.consumer_id = p.consumer_id
+  and lt.provider_id = p.provider_id
+  and lt.latest_consumer_version_order = p.consumer_version_order
+left outer join latest_verifications_for_pact_versions lv
+  on p.pact_version_id = lv.pact_version_id
+"

data/db/ddl_statements/latest_verification_ids_for_pact_versions_v001.rb

@@ -0,0 +1,4 @@
+LATEST_VERIFICATION_IDS_FOR_PACT_VERSIONS_V1 =
+  "select pact_version_id, MAX(verification_id) latest_verification_id
+    FROM latest_verification_id_for_pact_version_and_provider_version
+    GROUP BY pact_version_id"

data/db/migrations/000028_create_all_pact_publications.rb

@@ -21,7 +21,8 @@ Sequel.migration do
       group by provider_id, consumer_id, consumer_version_order"
     )
 
-    # Latest pact_publication details for each consumer version
+    # Latest pact_publication (revision) for each provider/consumer version
+    # updated in 20180519_recreate_views.rb
     create_view(:latest_pact_publications_by_consumer_versions,
       "select app.*
       from all_pact_publications app
@@ -33,6 +34,8 @@ Sequel.migration do
         )
 
 
+    # updated in 20180519_recreate_views.rb
+    # This view tells us the latest consumer version with a pact for a consumer/provider pair
     create_or_replace_view(:latest_pact_consumer_version_orders,
       "select provider_id, consumer_id, max(consumer_version_order) as latest_consumer_version_order
       from all_pact_publications

data/db/migrations/20180311_optimise_head_matrix.rb

@@ -1,4 +1,5 @@
 require_relative 'migration_helper'
+require_relative '../ddl_statements'
 
 Sequel.migration do
   up do
@@ -16,7 +17,9 @@ Sequel.migration do
     )
 
     # Add provider_version_order to original definition
-    # The most recent verification for each pact version
+    # The most recent verification for each pact_version
+    # provider_version column is DEPRECATED, use provider_version_number
+    # Think this can be replaced by latest_verification_id_for_pact_version_and_provider_version?
     v = :verifications
     create_or_replace_view(:latest_verifications,
       from(v)
@@ -44,38 +47,6 @@ Sequel.migration do
     )
 
 
-    create_or_replace_view(:head_matrix,
-      "
-      select
-        p.consumer_id, p.consumer_name, p.consumer_version_id, p.consumer_version_number, p.consumer_version_order,
-        p.id as pact_publication_id, p.pact_version_id, p.pact_version_sha, p.revision_number as pact_revision_number,
-        p.created_at as pact_created_at,
-        p.provider_id, p.provider_name, lv.provider_version_id, lv.provider_version_number, lv.provider_version_order,
-        lv.id as verification_id, lv.success, lv.number as verification_number, lv.execution_date as verification_executed_at,
-        lv.build_url as verification_build_url,
-        null as consumer_version_tag_name
-      from latest_pact_publications p
-      left outer join latest_verifications lv
-        on p.pact_version_id = lv.pact_version_id
-
-      union all
-
-      select
-        p.consumer_id, p.consumer_name, p.consumer_version_id, p.consumer_version_number, p.consumer_version_order,
-        p.id as pact_publication_id, p.pact_version_id, p.pact_version_sha, p.revision_number as pact_revision_number,
-        p.created_at as pact_created_at,
-        p.provider_id, p.provider_name, lv.provider_version_id, lv.provider_version_number, lv.provider_version_order,
-        lv.id as verification_id, lv.success, lv.number as verification_number, lv.execution_date as verification_executed_at,
-        lv.build_url as verification_build_url,
-        lt.tag_name as consumer_version_tag_name
-      from latest_tagged_pact_consumer_version_orders lt
-      inner join latest_pact_publications_by_consumer_versions p
-        on lt.consumer_id = p.consumer_id
-        and lt.provider_id = p.provider_id
-        and lt.latest_consumer_version_order = p.consumer_version_order
-      left outer join latest_verifications lv
-        on p.pact_version_id = lv.pact_version_id
-      "
-    )
+    create_or_replace_view(:head_matrix, HEAD_MATRIX_V1)
   end
 end

data/db/migrations/20180523_create_latest_verifications_for_consumer_version_tags.rb

@@ -1,6 +1,8 @@
 Sequel.migration do
   up do
     # The latest verification id for each consumer version tag
+    # This is not the latest verification for the latest pact with a given tag,
+    # this is the latest verification for any pact with the tag
     create_view(:latest_verification_ids_for_consumer_version_tags,
       "select
         pv.pacticipant_id as provider_id,

data/db/migrations/20180612_add_pacticipant_ids_to_verifications.rb

@@ -0,0 +1,17 @@
+Sequel.migration do
+  change do
+    alter_table(:verifications) do
+      add_foreign_key(:consumer_id, :pacticipants)
+      add_foreign_key(:provider_id, :pacticipants)
+      add_index(:consumer_id, name: "verifications_consumer_id_index")
+      add_index(:provider_id, name: "verifications_provider_id_index")
+      add_index([:provider_id, :consumer_id], name: "verifications_provider_id_consumer_id_index")
+    end
+
+    # TODO
+    # alter_table(:verifications) do
+    #   set_column_not_null(:consumer_id)
+    #   set_column_not_null(:provider_id)
+    # end
+  end
+end

data/db/migrations/20180613_migrate_pacticipant_ids_for_verifications.rb

@@ -0,0 +1,7 @@
+require 'pact_broker/db/data_migrations/set_pacticipant_ids_for_verifications'
+
+Sequel.migration do
+  up do
+    PactBroker::DB::DataMigrations::SetPacticipantIdsForVerifications.call(self)
+  end
+end

data/db/migrations/20180614_update_latest_verification_ids_for_consumer_and_provider.rb

@@ -0,0 +1,27 @@
+Sequel.migration do
+  up do
+    # The latest verification id for each consumer version tag
+    create_or_replace_view(:latest_verification_ids_for_consumer_and_provider,
+      "select
+        provider_id,
+        consumer_id,
+        max(id) as latest_verification_id
+        from verifications v
+        group by provider_id, consumer_id")
+  end
+
+  down do
+    # The latest verification id for each consumer version tag
+    create_or_replace_view(:latest_verification_ids_for_consumer_and_provider,
+      "select
+        pv.pacticipant_id as provider_id,
+        lpp.consumer_id,
+        max(v.id) as latest_verification_id
+      from verifications v
+      join latest_pact_publications_by_consumer_versions lpp
+        on v.pact_version_id = lpp.pact_version_id
+      join versions pv
+        on v.provider_version_id = pv.id
+      group by pv.pacticipant_id, lpp.consumer_id")
+  end
+end

data/db/migrations/20180615_add_consumer_id_to_pact_publications.rb

@@ -0,0 +1,13 @@
+Sequel.migration do
+  change do
+    alter_table(:pact_publications) do
+      add_foreign_key(:consumer_id, :pacticipants)
+      add_index(:consumer_id, name: "pact_publications_consumer_id_index")
+    end
+
+    # TODO
+    # alter_table(:pact_publications) do
+    #   set_column_not_null(:consumer_id)
+    # end
+  end
+end

data/db/migrations/20180616_migrate_consumer_ids_for_pact_publications.rb

@@ -0,0 +1,11 @@
+require 'pact_broker/db/data_migrations/set_consumer_ids_for_pact_publications'
+
+Sequel.migration do
+  up do
+    PactBroker::DB::DataMigrations::SetConsumerIdsForPactPublications.call(self)
+  end
+
+  down do
+    from(:pact_publications).update(consumer_id: nil)
+  end
+end

data/db/migrations/20180720_create_latest_pact_publication_ids.rb

@@ -0,0 +1,25 @@
+Sequel.migration do
+  up do
+    # Latest pact_publication (revision) for each provider/consumer version.
+    # Keeping track of this in a table rather than having to calculate the
+    # latest revision speeds things up.
+    # We don't have to worry about updating it if a revision is deleted, because
+    # you can't delete a single pact revision through the API - all the revisions
+    # for a pact are deleted together when you delete the pact resource for that
+    # consumer version, and when that happens, this row will cascade delete.
+
+    create_table(:latest_pact_publication_ids_for_consumer_versions, charset: 'utf8') do
+      foreign_key :consumer_id, :pacticipants, null: false, on_delete: :cascade # redundant, but speeds up queries by removing need for extra join
+      foreign_key :consumer_version_id, :versions, null: false, on_delete: :cascade
+      foreign_key :provider_id, :pacticipants, null: false, on_delete: :cascade
+      foreign_key :pact_publication_id, :pact_publications, null: false, on_delete: :cascade, unique: true
+      foreign_key :pact_version_id, :pact_versions, null: false, on_delete: :cascade
+      index [:provider_id, :consumer_version_id], unique: true, name: "unq_latest_ppid_prov_conver"
+      index [:provider_id, :consumer_id], name: "lpp_provider_id_consumer_id_index"
+    end
+  end
+
+  down do
+    drop_table(:latest_pact_publication_ids_for_consumer_versions)
+  end
+end