davinci_dtr_test_kit 0.9.0

data/lib/davinci_dtr_test_kit/docs/dtr_payer_server_suite_description_v201.md

@@ -0,0 +1,127 @@
+The Da Vinci DTR Test Kit Payer Server Suite validates the conformance of payer server
+systems to the STU 2 version of the HL7® FHIR®
+[Da Vinci Documentation Templates and Rules (DTR) Implementation Guide](https://hl7.org/fhir/us/davinci-dtr/STU2/).
+
+## Scope
+
+These tests are a **DRAFT** intended to allow payer implementers to perform
+preliminary checks of their systems against DTR IG requirements and [provide
+feedback](https://github.com/inferno-framework/davinci-dtr-test-kit/issues)
+on the tests. Future versions of these tests may validate other
+requirements and may change the test validation logic.
+
+## Test Methodology
+
+Inferno will simulate a DTR client application (SMART client or Full EHR) that
+is making requests for questionnaires for the server under test to interact with. 
+The server will be expected to respond to these requests made by Inferno. Over the
+course of these interactions, Inferno will seek to observe conformant handling of 
+DTR [requirements](https://hl7.org/fhir/us/davinci-dtr/STU2/specification.html#defining-questionnaires) 
+around standard questionnaires, adaptive questionnaires, or both. 
+
+In terms of the validation performed on the returned questionnaires these
+tests assume that payers are not required to support specific features, but only those
+that they need in order to implement their forms. The DTR IG does [require that](https://hl7.org/fhir/us/davinci-dtr/STU2/specification.html#population) 
+"Questionnaires SHALL include logic that supports population from the EHR where possible"
+and puts some restrictions on how Questionnaire features can be used. Thus, these
+tests require demonstration of pre-population features and validates conformant
+use of Questionnaire features, but don't currently require demonstration of all
+Questionnnaire Must Support elements.
+
+Because the business logic that determines which questionnaires are returned
+is outside of the DTR specification and will vary between implementers, testers
+are required to provide the requests that Inferno will make to the server, either
+by providing the requests to make up-front, or by sending them to Inferno during
+test execution using a tester-controlled client.
+
+All responses returned by the server, as well as tester-provided requests, will be checked 
+for conformance to the DTR IG requirements individually and used in aggregate to determine
+whether required features and functionality are present. HL7® FHIR® resources are
+validated with the Java validator using `tx.fhir.org` as the terminology server.
+
+## Running the Tests
+
+### Quick Start
+
+Execution of these tests require a significant amount of tester input in the
+form of requests that Inferno will make against the server under test. If
+you don't have a server or know specific requests that will elicit representative
+questionnaires from your server, see the *Sample Execution* section below.
+
+Otherwise, the requests for Inferno to make can be provided in one of two ways:
+1. Statically within the Inferno inputs: provide the json request bodies when running
+   the tests. This approach works very well for testing standard (static) questionnaires
+   where there is only one request for Inferno to make (input = *Initial Static Questionnaire Request*). It is less ideal for adaptive
+   questionnaires as a sequence of `$next-question` requests (inputs = *Initial Adaptive Questionnaire Request* and *Next Question Requests*) is required, which is provided as a json list of
+   request body objects.
+2. Dynamically during test execution: use a tester-controlled client to provide requests to
+   Inferno while the tests are running. At points that Inferno needs to make a request, execution
+   will pause while the request is provided from the client. Inferno uses a bearer token
+   provided in the test inputs (input = *Access Token* for the DTR Client Flow (not the
+   one under the *OAuth Credentials* input)) to identify these requests. The provided token 
+   must be sent as a part of the string `Bearer <token>` within the `Authentication` header of
+   all requests. When Inferno receives a request from the tester-controlled client, it will 
+   send that to the server under test and then return the response back to the client so that
+   the client's workflow, e.g., around an adaptive questionnaire can continue.
+
+In addition to the above configuration needed for identification of tests, the following additional
+inputs are required
+- *Questionnaire Retrieval Method*: indicate whether only static, only adaptive, or both types
+  of questionnaires will be tested.
+- *FHIR Server Base Url*: the location of the server to test
+- *OAuth Credentials*: if the server under test requires authentication, provide those details
+  here.
+
+For more details on additional inputs that may be needed, see the *Additional Configuration Details*
+section below.
+
+### Sample Execution
+
+If you would like to try out the tests but don't have a DTR payer server implementation,
+you can run these tests against the DTR SMART Client test suite included in this test kit
+using the following steps:
+1. Start an Inferno session of the Da Vinci DTR SMART App Test Suite.
+1. Select test 2.1.1 *Static Questionnaire Workflow* from the menu on the left.
+1. Click the "Run All Tests" button in the upper right.
+1. In the "access_token" input, put `cnVuIHRvZ2V0aGVy`.
+1. Click the "submit" button in the dialog that appears. The client tests will now be waiting for requests.
+1. Start an Inferno session of the DTR Payer Server test suite.
+1. Select test 1 *Static Questionnaire Package Retrieval* from the menu on the left.
+1. Select the *Run Against DTR SMART App Tests* option from the Preset dropdown in the
+   upper left.
+1. Click the "Run All Tests" button in the upper right.
+1. Click the "submit" button in the dialog that appears. The server tests will now make requests
+   against the client test session, which will respond with a static questionnaire that the 
+   these server tests can validate.
+
+At this time, only the standard questionnaire functionality can be tested using this approach as
+the client tests have not implemented an adaptive questionnaire set of tests.
+
+## Additional Configuration Details
+
+The details provided here supplement the documentation of individual fields in the input dialog
+that appears when initiating a test run.
+
+### Custom Endpoint for Accessing a Particular Resource
+
+If the `$questionnaire-package` requests should be made to a location other than 
+`/Questionnaire/$questionnaire-package` under the base server URL, enter the
+location the requests should be made relative to the base server URL.
+
+## Limitations
+
+These tests currently require the server under test to demonstrate a single example of
+a conformant standard (static) and / or an adaptive questionnaire. This is based
+on the interpretation of the DTR IG as allowing payers to choose the features that
+they want to support. If this interpretation turns out to be inconsistent with the
+intention of the IG authors then future versions of the tests may require the payer
+to provide additional examples.
+
+The payer responses are also tested to ensure that appropriate libraries and expressions are
+ included to faciliate pre-population of questionnaires. The following is not tested:
+- CQL is version 1.5
+- CQL is valid and executed to populate the questionnaire
+- CQL has a context of “Patient”
+- CQL definitions and variables defined on ancestor elements or preceding expression extensions within the same
+Questionnaire item are in scope for referencing in descendant/following expressions.
+- Within Expression elements, the base expression CQL SHALL be accompanied by a US Public Health Alternative Expression Extension containing the compiled JSON ELM for the expression.

data/lib/davinci_dtr_test_kit/docs/dtr_smart_app_suite_description_v201.md

@@ -0,0 +1,118 @@
+The Da Vinci DTR Test Kit SMART App Suite validates the conformance of SMART apps
+to the STU 2 version of the HL7® FHIR®
+[Da Vinci Documentation Templates and Rules (DTR) Implementation Guide](https://hl7.org/fhir/us/davinci-dtr/STU2/).
+
+## Scope
+
+These tests are a **DRAFT** intended to allow app implementers to perform
+preliminary checks of their systems against DTR IG requirements and [provide
+feedback](https://github.com/inferno-framework/davinci-dtr-test-kit/issues)
+on the tests. Future versions of these tests may validate other
+requirements and may change the test validation logic.
+
+## Test Methodology
+
+Inferno will simulate a DTR payer server and light EHR that will response to
+requests for questionnaires and clinical data for the app under test to interact with. 
+The app will be expected to initiate requests to Inferno to elicit responses. Over the
+course of these interactions, Inferno will seek to observe conformant handling of 
+DTR workflows and requirements around the retrieval, completion, and storage of
+questionnaires.
+
+Tests within this suite are associated with specific questionnaires that the app will
+demonstrate completion of. In each case, the app under test will initiate a request to
+the payer server simulated by Inferno for a questionnaire using the 
+`$questionnaire-package` operation. Inferno will always return the specific questionnaire
+for the test being executed regardless of the input provided by the app, though it must
+be conformant. The app will then be asked to complete the questionnaire, including
+- Pre-populating answers based on directives in the questionnaire, which includes the
+  fetching of associated data from the light EHR simulated by Inferno
+- Rendering the questionnaire for users and allowing them to make additional updates.
+  These tests can include specific directions on details to include in the completed
+  questionnaire.
+- Storing the completed questionnaire back to the light EHR simulated by Inferno. Inferno
+  will validate the stored questionnaire, including pre-populated values (Inferno knows
+  the pre-population logic and the data used in calculation) and other conformance details.
+
+Apps will be required to complete all questionnaires in the suite, which in aggregate
+contain all questionnaire features that apps must support. Currently, the suite includes
+two questionnaires:
+1. A fictious "dinner" questionnaire created for these tests. It tests basic
+   item rendering and pre-population.
+2. A Respiratory Assist Device questionnaire pulled from the DTR reference implementation.
+   It tests additional features and represents a more realistic questionnaire.
+Additional questionnaires will be added in the future.
+
+All requests sent by the app will be checked 
+for conformance to the DTR IG requirements individually and used in aggregate to determine
+whether required features and functionality are present. HL7® FHIR® resources are
+validated with the Java validator using `tx.fhir.org` as the terminology server.
+
+## Running the Tests
+
+### Quick Start
+
+Inferno does not currently include the ability to launch the client. Therefore, clients
+must be manually configured to point to Inferno's simulated server endpoints. The endpoints
+can be inferred from the URL of the test session which will be of the form `[URL prefix]/dtr_smart_app/[session id]`: (NOTE: both currently use the same URL)
+- Payer Server Base FHIR URL: `[URL prefix]/custom/dtr_smart_app/fhir`
+- Light EHR Base FHIR URL: `[URL prefix]/custom/dtr_smart_app/fhir`
+
+In order for Inferno to associate requests sent to locations under these base URLs with this session,
+it needs to know the bearer token that the app will send on requests, for which 
+there are two options.
+
+1. If you want to choose your own bearer token, then
+    1. Select the "2. Basic Workflows" test from the list on the left (or other target test). 
+    2. Click the '*Run All Tests*' button on the right.
+    3. In the "access_token" field, enter the bearer token that will be sent by the client 
+       under test (as part of the Authorization header - `Bearer <provided value>`).
+    4. Click the '*Submit*' button at the bottom of the dialog.
+2. If you want to use a client_id to obtain an access token, then
+    1. Click the '*Run All Tests*' button on the right.
+    2. Provide the client's registered id "client_id" field of the input (NOTE, Inferno doesn't support the
+        registration API, so this must be obtained from another system or configured manually).
+    3. Click the '*Submit*' button at the bottom of the dialog.
+    4. Make a token request that includes the specified client id to the
+        `[URL prefix]/custom/dtr_smart_app/mock_auth/token` endpoint to get
+        an access token to use on the request of the requests.
+
+In either case, the tests will continue from that point. Further executions of tests under
+this session will also use the selected bearer token.
+
+Note: authentication options for these tests have not been finalized and are subject to change.
+
+### Postman-based Demo
+
+If you do not have a DTR SMART app but would like to try the tests out, you can use
+[this Postman collection](https://github.com/inferno-framework/davinci-dtr-test-kit/blob/main/config/DTR%20Test%20Kit.postman_collection.json)
+to make requests against Inferno. This does not include the capability to render the complete the
+questionnaires, but does have samples of correctly and incorrectly completed QuestionnaireResponses.
+The following is a list of tests with the Postman requests that can be used with them:
+
+- **2.1** *Static Questionnaire Workflow*: use requests in the `Static Dinner` folder
+  - **2.1.1.01** *Invoke the DTR Questionnaire Package operation*: submit request `Questionnaire Package for Dinner (Static)` while this test is waiting.
+  - **2.1.3.01** *Save the QuestionnaireResponse after completing it*: submit request `Save QuestionnaireResponse for Dinner (Static)` while this test is waiting. If you want to see a failure, submit request `Save QuestionnaireResponse for Dinner (Static) - missing origin extension` instead.
+- **3.1** *Respiratory Assist Device Questionnaire Workflow*: use requests in the `Respiratory Assist Device` folder
+  - **3.1.1.01** *Invoke the DTR Questionnaire Package operation*: submit request `Questionnaire Package for Resp Assist Device` while this test is waiting.
+  - **3.1.3.01** *Save the QuestionnaireResponse after completing it*: submit request `Save Questionnaire Response for Resp Assist Device` while this test is waiting. If you want to see a failure, submit request `Save Questionnaire Response for Resp Assist Device - unexpected override` instead.
+
+## Limitations
+
+The DTR IG is a complex specification and these tests currently validate SMART app
+configuration to only part of it. Future versions of the test suite will test further
+features. A few specific features of interest are listed below.
+
+### Launching and security
+
+The primary limitation on this test suite is that it requires the client under test
+to be manually configured to point to the Inferno endpoints and send a bearer token. 
+In the future, the tests will provide a mechanism for launching the application using
+the SMART app launch mechanism. To provide feedback and input on the design of this feature,
+submit a ticket [here](https://github.com/inferno-framework/davinci-pas-test-kit/issues).
+
+### Questionnaire Feature Coverage
+
+Not all questionnaire features that are must support within the DTR IG are currently represented
+in questionnaires tested by the IG. Adaptive questionnaires are a notable omission.
+Additional questionnaires testing additional features will be added in the future.

data/lib/davinci_dtr_test_kit/dtr_full_ehr_suite.rb

@@ -0,0 +1,55 @@
+require_relative 'ext/inferno_core/runnable'
+require_relative 'ext/inferno_core/record_response_route'
+require_relative 'ext/inferno_core/request'
+require_relative 'client_groups/resp_assist_device/dtr_full_ehr_questionnaire_workflow_group'
+require_relative 'auth_groups/oauth2_authentication_group'
+require_relative 'mock_payer'
+require_relative 'version'
+
+module DaVinciDTRTestKit
+  class DTRFullEHRSuite < Inferno::TestSuite
+    extend MockPayer
+
+    id :dtr_full_ehr
+    title 'Da Vinci DTR Full EHR Test Suite'
+    description %(
+        # Da Vinci DTR Full EHR Test Suite
+
+        This suite validates that an EHR or other application can act
+        as a full DTR application requesting questionnaires from a
+        payer server and using local data to complete and store them.
+        Inferno will act as payer server returning questionnaires
+        in response to queries from the system under test and validating
+        that they can be completed as expected.
+      )
+
+    version VERSION
+
+    # All FHIR validation requsets will use this FHIR validator
+    validator do
+      url ENV.fetch('VALIDATOR_URL')
+    end
+
+    allow_cors QUESTIONNAIRE_PACKAGE_PATH
+
+    record_response_route :post, TOKEN_PATH, 'dtr_auth', method(:token_response) do |request|
+      DTRFullEHRSuite.extract_client_id(request)
+    end
+
+    record_response_route :post, '/fhir/Questionnaire/$questionnaire-package', QUESTIONNAIRE_PACKAGE_TAG,
+                          method(:questionnaire_package_response) do |request|
+      DTRFullEHRSuite.extract_bearer_token(request)
+    end
+
+    resume_test_route :get, RESUME_PASS_PATH do |request|
+      DTRFullEHRSuite.extract_token_from_query_params(request)
+    end
+
+    resume_test_route :get, RESUME_FAIL_PATH, result: 'fail' do |request|
+      DTRFullEHRSuite.extract_token_from_query_params(request)
+    end
+
+    group from: :oauth2_authentication
+    group from: :dtr_full_ehr_questionnaire_workflow
+  end
+end

data/lib/davinci_dtr_test_kit/dtr_light_ehr_suite.rb

@@ -0,0 +1,39 @@
+require_relative 'version'
+
+module DaVinciDTRTestKit
+  class DTRLightEHRSuite < Inferno::TestSuite
+    id :dtr_light_ehr
+    title 'Da Vinci DTR Light EHR Test Suite'
+    description %(
+        # Da Vinci DTR Light EHR Test Suite
+
+        This suite validates that an EMR or other application
+        can act as a data source for a DTR SMART App. Inferno
+        will act as a DTR SMART App making requests for data
+        against the system under test and storing completed
+        questionnaire responses.
+      )
+
+    version VERSION
+
+    # These inputs will be available to all tests in this suite
+    input :url,
+          title: 'FHIR Server Base Url'
+
+    input :credentials,
+          title: 'OAuth Credentials',
+          type: :oauth_credentials,
+          optional: true
+
+    # All FHIR requests in this suite will use this FHIR client
+    fhir_client do
+      url :url
+      oauth_credentials :credentials
+    end
+
+    # All FHIR validation requsets will use this FHIR validator
+    validator do
+      url ENV.fetch('VALIDATOR_URL')
+    end
+  end
+end

data/lib/davinci_dtr_test_kit/dtr_payer_server_suite.rb

@@ -0,0 +1,104 @@
+require_relative 'ext/inferno_core/runnable'
+require_relative 'ext/inferno_core/record_response_route'
+require_relative 'ext/inferno_core/request'
+require_relative 'payer_server_groups/payer_server_static_group'
+require_relative 'payer_server_groups/payer_server_adaptive_group'
+require_relative 'tags'
+require_relative 'mock_payer'
+require_relative 'version'
+
+module DaVinciDTRTestKit
+  class DTRPayerServerSuite < Inferno::TestSuite
+    extend MockPayer
+
+    id :dtr_payer_server
+    title 'Da Vinci DTR Payer Server Test Suite'
+    description File.read(File.join(__dir__, 'docs', 'dtr_payer_server_suite_description_v201.md'))
+
+    version VERSION
+    # These inputs will be available to all tests in this suite
+
+    input :retrieval_method,
+          title: 'Questionnaire Retrieval Method',
+          type: 'radio',
+          default: 'Both',
+          options: {
+            list_options: [
+              {
+                label: 'Static',
+                value: 'Static'
+              },
+              {
+                label: 'Adaptive',
+                value: 'Adaptive'
+              },
+              {
+                label: 'Both',
+                value: 'Both'
+              }
+            ]
+          }
+
+    input :url,
+          title: 'FHIR Server Base Url',
+          description: 'Required for All Flows'
+
+    input :access_token,
+          optional: true,
+          title: 'Access Token',
+          description: 'DTR Client Flow'
+
+    input :custom_endpoint,
+          optional: true,
+          title: 'Custom Endpoint for Accessing a Particular Resource',
+          description: 'Either Flow (optional)'
+
+    input :credentials,
+          title: 'OAuth Credentials',
+          type: :oauth_credentials,
+          optional: true
+
+    input_order :retrieval_method,
+                :url,
+                :access_token,
+                :custom_endpoint,
+                :initial_static_questionnaire_request,
+                :initial_adaptive_questionnaire_request,
+                :next_question_requests,
+                :credentials
+
+    # All FHIR requests in this suite will use this FHIR client
+    fhir_client do
+      url :url
+      oauth_credentials :credentials
+    end
+
+    # All FHIR validation requsets will use this FHIR validator
+    validator do
+      url ENV.fetch('VALIDATOR_URL')
+    end
+
+    allow_cors QUESTIONNAIRE_PACKAGE_PATH, NEXT_PATH
+
+    record_response_route :post, TOKEN_PATH, 'dtr_auth', method(:token_response) do |request|
+      DTRPayerServerSuite.extract_client_id(request)
+    end
+
+    record_response_route :post, '/fhir/Questionnaire/$questionnaire-package', QUESTIONNAIRE_TAG,
+                          method(:payer_questionnaire_response), resumes: method(:test_resumes?) do |request|
+      DTRPayerServerSuite.extract_bearer_token(request)
+    end
+
+    record_response_route :post, '/fhir/Questionnaire/$next-question', NEXT_TAG,
+                          method(:questionnaire_next_response), resumes: method(:test_resumes?) do |request|
+      DTRPayerServerSuite.extract_bearer_token(request)
+    end
+
+    resume_test_route :get, RESUME_PASS_PATH do |request|
+      DTRPayerServerSuite.extract_token_from_query_params(request)
+    end
+
+    group from: :payer_server_static_package
+    group from: :payer_server_adaptive_questionnaire
+  end
+end

data/lib/davinci_dtr_test_kit/dtr_questionnaire_response_validation.rb

@@ -0,0 +1,180 @@
+require_relative 'fixtures'
+
+module DaVinciDTRTestKit
+  module DTRQuestionnaireResponseValidation
+    include Fixtures
+
+    CQL_EXPRESSION_EXTENSIONS = [
+      'http://hl7.org/fhir/uv/sdc/StructureDefinition/sdc-questionnaire-initialExpression',
+      'http://hl7.org/fhir/uv/sdc/StructureDefinition/sdc-questionnaire-calculatedExpression',
+      'http://hl7.org/fhir/uv/sdc/StructureDefinition/sdc-questionnaire-candidateExpression'
+    ].freeze
+
+    def validate_questionnaire_pre_population(questionnaire_response, test_id)
+      # Requirements:
+      #  - Prior to exposing the draft QuestionnaireResponse to the user for completion and/or review, the DTR client
+      #    SHALL execute all CQL necessary to resolve the initialExpression, candidateExpression and
+      #    calculatedExpression extensions found in the Questionnaire for any enabled elements.
+      #  - All items that are pre-populated (whether by the payer in the initial QuestionnaireResponse provided in the
+      #    questionnaire package, or from data retrieved from the EHR) SHALL have their origin.source set to ‘auto’.
+      #
+      # Note that in the questionnaire fixture, all cql expression elements are enabled, so we don't filter
+      template_questionnaire_response = find_questionnaire_response_for_test_id(test_id)
+      raise "missing QuestionnaireResponse template for test #{test_id}" unless template_questionnaire_response.present?
+
+      questionnaire = find_questionnaire_instance_for_test_id(test_id)
+      raise "missing Questionnaire for test #{test_id}" unless questionnaire.present?
+
+      questionnaire_cql_expression_link_ids = collect_questionnaire_cql_expression_link_ids(questionnaire.item)
+      template_prepopulation_expectations = {}
+      template_override_expectations = {}
+      extract_expected_answers_from_template(template_questionnaire_response,
+                                             questionnaire_cql_expression_link_ids,
+                                             template_prepopulation_expectations,
+                                             template_override_expectations)
+      validation_errors = []
+      validate_cql_executed(questionnaire_response.item, questionnaire_cql_expression_link_ids,
+                            template_prepopulation_expectations, template_override_expectations, validation_errors)
+
+      validation_errors.each { |msg| messages << { type: 'error', message: msg } }
+      assert validation_errors.blank?, 'QuestionnaireResponse is not conformant. Check messages for issues found.'
+    end
+
+    def validate_cql_executed(actual_items, questionnaire_cql_expression_link_ids, template_prepopulation_expectations,
+                              template_override_expectations, error_messages)
+
+      actual_items&.each do |item_to_validate|
+        link_id = item_to_validate.linkId
+        if questionnaire_cql_expression_link_ids.include?(link_id)
+          if template_prepopulation_expectations.key?(link_id)
+            check_item_prepopulation(item_to_validate, template_prepopulation_expectations[link_id], error_messages,
+                                     false)
+          elsif template_override_expectations.include?(link_id)
+            check_item_prepopulation(item_to_validate, template_override_expectations[link_id], error_messages, true)
+          else
+            raise "template missing expectation for question `#{link_id}`"
+          end
+        end
+
+        validate_cql_executed(item_to_validate.item, questionnaire_cql_expression_link_ids,
+                              template_prepopulation_expectations, template_override_expectations, error_messages)
+      end
+      error_messages
+    end
+
+    def extract_expected_answers_from_template(template_questionnaire_response,
+                                               questionnaire_cql_expression_link_ids,
+                                               expected_prepopulated = {},
+                                               expected_overrides = {})
+
+      questionnaire_cql_expression_link_ids.each do |target_link_id|
+        target_item = find_item_by_link_id(template_questionnaire_response.item, target_link_id)
+        raise "Template QuestionnaireResponse missing item with link id `#{target_link_id}`" unless target_item.present?
+
+        target_item_answer = target_item.answer.first
+        unless target_item_answer.present?
+          raise "Template QuestionnaireResponse missing an answer for item with link id `#{target_link_id}`"
+        end
+
+        origin_extension = find_extension(target_item_answer,
+                                          'http://hl7.org/fhir/us/davinci-dtr/StructureDefinition/information-origin')
+        source_extension = find_extension(origin_extension, 'source')
+
+        unless source_extension.present?
+          raise "Template QuestionnaireResponse item `#{target_link_id}` missing the `origin.source` extension"
+        end
+
+        # TODO: handle other data types
+        if source_extension.value == 'auto'
+          expected_prepopulated[target_link_id] = target_item_answer.value
+        elsif source_extension.value == 'override'
+          expected_overrides[target_link_id] = target_item_answer.value
+        else
+          raise "`origin.source` extension for item `#{target_link_id}` has unexpected value: #{source_extension.value}"
+        end
+      end
+    end
+
+    def validate_data_requirements_retrieved(expected_questionnaire_response, questionnaire_response)
+      error_messages = []
+
+      DATA_REQUIREMENT_ANSWERS.each do |library_name, link_id|
+        expected = find_item_by_link_id(expected_questionnaire_response.item, link_id).answer.first.value
+        actual = find_item_by_link_id(questionnaire_response.item, link_id)&.answer&.first&.value
+        next if coding_equal?(expected, actual)
+
+        error_messages << "dataRequirement not satisfied for Library '#{library_name}'. Expected answer to " \
+                          "question with linkId `#{link_id}` to have coding with system: '`#{expected.system}`' " \
+                          "and value: '`#{expected.code}`'"
+      end
+      error_messages
+    end
+
+    def check_item_prepopulation(item, expected_answer, error_list, override)
+      answer = item.answer.first
+      if answer&.value&.present?
+        # check answer
+        if override && answer.value == expected_answer
+          error_list << "Answer to item `#{item.linkId}` was not overriden from the pre-populated value. " \
+                        "Found #{expected_answer}, but should be different"
+        elsif !override && answer.value != expected_answer
+          error_list << "answer to item `#{item.linkId}` contains unexpected value. Expected: #{expected_answer}. " \
+                        "Found #{answer.value}"
+        end
+
+        # check origin.source extension
+        origin_extension = find_extension(answer,
+                                          'http://hl7.org/fhir/us/davinci-dtr/StructureDefinition/information-origin')
+        source_extension = find_extension(origin_extension, 'source')
+
+        if source_extension.present?
+          expected_source_value = override ? 'override' : 'auto'
+          if source_extension.value != expected_source_value
+            error_list << "`origin.source` extension on item `#{item.linkId}` contains unexpected value. Expected: " \
+                          "#{expected_source_value}. Found #{source_extension.value}"
+          end
+        else
+          error_list << "Required `origin.source` extension not present on answer to item `#{item.linkId}`"
+        end
+      else
+        error_list << "No answer for item `#{item.linkId}`"
+      end
+    end
+
+    def find_item_by_link_id(items, link_id)
+      items.each do |item|
+        return item if item.linkId == link_id
+
+        match = find_item_by_link_id(item.item, link_id)
+        return match if match
+      end
+      nil
+    end
+
+    def find_extension(element, url)
+      element&.extension&.find { |e| e.url == url }
+    end
+
+    def collect_questionnaire_cql_expression_link_ids(items, link_ids = [])
+      items.each do |item|
+        link_ids << item.linkId if item_is_cql_expression?(item)
+        collect_questionnaire_cql_expression_link_ids(item.item, link_ids) if item&.item&.any?
+      end
+      link_ids
+    end
+
+    def item_is_cql_expression?(item)
+      item.extension&.any? { |ext| CQL_EXPRESSION_EXTENSIONS.include?(ext.url) }
+    end
+
+    def answer_value_equal?(expected, actual)
+      return coding_equal?(expected.value, actual.value) if expected.valueCoding.present?
+
+      expected.value == actual.value
+    end
+
+    def coding_equal?(expected, actual)
+      expected.system == actual&.system && expected.code == actual&.code
+    end
+  end
+end