elasticsearch-test-runner 0.1.0 → 0.20.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 139cdbecff8a7f080c63ce718ceee51b4718629d604b2a70f49e456fb898abee
-  data.tar.gz: f5917106b5f1a17c1e74b664c1a16c485e4d0e623e1e2e5293f20954fb5d0ea5
+  metadata.gz: 6f353417226574d9f9f31aad08493921ff64ddc0442eb3b1c78ca810dd6d5f05
+  data.tar.gz: 8c665f93673ae59cefc2c621b37dd270acda75f1ce15e0ddacdf7b3ba8fdd0fd
 SHA512:
-  metadata.gz: b592e11ccbeae18bb9fbd0fe5760caea1bda3ed891895073854dfc11861f6d7c386dd6b973b3807598565d6da81332be6839bcae4ad64d75b8693645bea08a2d
-  data.tar.gz: db573c22f278bef0e57a1fc3a79fd7c9ab296504d108b2b2cded3648057c2e3df5f59230ef41a14c7fe61a92e3c10cfee247734e8f45125560d4adb1a7a9c659
+  metadata.gz: 51a23ebbdd4550e88083a38d9ae953a87cbc6555175f72e3cfb8798531e980772d6ff19d6aa075e52336355779e041dddfc87b713ddb8941816022e6d0a32340
+  data.tar.gz: a5dd823bd406475e67ddf632051d1102f981ad6b9813134d452c6eb3df8da1fec71fb0752d5ab9f24801754b00b1f57a7345fbbc7bc1c6ce1c0d2dedfca786a0

data/CHANGELOG.md

@@ -1,5 +1,130 @@
-## [Unreleased]
+# Changelog
 
-## [0.1.0] - 2024-02-22
+## [0.20.0] - 2026-05-20
 
-- Initial release
+- Replaces variables in arrays, update for the latest tests update.
+
+## [0.19.0] - 2026-05-12
+
+- Adds exception to print_failure if there is one.
+- Better formatting when printing match failures.
+- Raise error if the requires section is missing from a test.
+
+## [0.18.2] - 2026-03-06
+
+- Fixes skipping multiple tests.
+- Fixes running tests in a directory.
+
+## [0.18.1] - 2026-03-06
+
+- Fixes skipping tests.
+
+## [0.18.0] - 2026-03-05
+
+- Lots of improvements when displaying errors. Fixes missing exceptions during particular failures. The display for errors and failures is now more detailed and comfortable to read.
+- Fixes printing a success when there's no response object, for instance when catching an expected exception.
+- The code for catching exceptions was refactored and made more reliable, including showing better information as previously mentioned.
+
+## [0.17.0] - 2026-03-04
+
+- Better debugging detection, check for `ENV['DEBUG']` to be equal to `true`, not just if it's set.
+
+## [0.16.1] - 2026-02-26
+
+- Updates debug display message for String response bodies.
+
+## [0.16.0] - 2026-02-25
+
+- Lets Ruby handle successful exit code at the end of the test run, only uses `exit 1` if there are errors.
+- Display of debug messages has been improved when `DEBUG=true` is used - [Pull Request](https://github.com/elastic/es-test-runner-ruby/pull/24).
+
+## [0.15.1] - 2026-01-30
+
+- Rescues invalid byte sequence for search_mvt test.
+
+## [0.15.0] - 2026-01-29
+
+- Bugfix: refactors response display for boolean responses (ping, exists) when in debug mode.
+- Sets boxes screen size to whole screen.
+
+## [0.14.0] - 2026-01-29
+
+- Introduces Ruby TTY tools for better debugging when `ENV['DEBUG']` is set and general display improvements.
+- Better information for actions in passing/erroring tests, showing actions instead of the test title.
+
+## [0.13.0] - 2025-06-05
+
+- Moves download tests file to Rakefile (namespace is unchanged).
+- Updates Elasticsearch Clients Tests repo URL.
+
+## [0.12.0] - 2025-01-14
+
+- Check TEST_SUITE variable for `serverless`.
+
+## [0.11.0] - 2024-12-12
+
+- Adds skipping tests.
+
+## [0.10.1] - 2024-11-13
+
+- Creates directory for tests if it doesn't exist.
+
+## [0.10.0] - 2024-11-13
+
+- Using tar.gz file for downloaded tests instead of zip file.
+- Updates logging and debug information. More information being logged for different levels of Logger.
+
+## [0.9.0] - 2024-08-15
+
+- Better display of action and response when `ENV['DEBUG']` is true.
+- Checks for more error types when catching exceptions.
+- Better matching for regular expressions.
+- Refactors `set_param_variable` for better substitution.
+
+## [0.8.0] - 2024-08-14
+
+- Adds support for variable keys (`$`).
+
+## [0.7.0] - 2024-08-08
+
+- Improves test (file) name display
+- Updates QUIET environment variable check
+
+## [0.6.0] - 2024-08-07
+
+- Adds `header` support, headers can be specified for an action.
+- Show response in errors.
+
+## [0.5.0] - 2024-07-30
+
+- Clears `@response` before running a new action
+- Fixes `expected_exception?` for failures
+
+## [0.4.0] - 2024-07-08
+
+- Refactors display of errors/passing tests. Adds `QUIET` environment variable parsing. If `true`, display for passing tests will be more compact (not showing file/test names).
+- Updates count for better accuracy in test results.
+- Rescues SystemExit, Interrupt to actually exit the script
+- Adds support for catch in tests: If the arguments to `do` include `catch`, then we are expecting an error, which should be caught and tested.
+
+## [0.3.1] - 2024-06-27
+
+- Fixes in error handling.
+
+## [0.3.0] - 2024-06-27
+
+- Fixes exit code.
+- Refactors error handling for more detailed information.
+
+## [0.2.0] - 2024-06-25
+
+- Renames gemspec from elasticsearch-tests to elasticsearch-test-runner.
+- Adds ability to run individual test files or directories.
+
+## [0.1.1] - 2024-06-20
+
+- Require 'fileutils' in Downloader.
+
+## [0.1.0] - 2024-06-20
+
+- Initial release.

data/README.md

@@ -29,15 +29,53 @@ Elasticsearch::Tests::TestRunner.new(client, tests_path).run
 
 [The tests](https://github.com/elastic/elasticsearch-clients-tests) are designed for the Elasticsearch REST API and the Elasticsearch Serverless REST API. If you pass in an `ElasticsearchServerless::Client`, it will only run the tests that have the `requires.serverless: true` statement. Otherwise, it will only run the ones with `requires.stack: true`.
 
+
+### Logging
+
 You can optionally pass in an object that implements Ruby's Logger to the `TestRunner` initializer. This will log more information, particularly useful in the case of errors where it'll log stacktraces for exceptions and more:
 
 ```ruby
 logger = Logger.new($stdout)
-logger.level = Logger::WARN unless ENV['DEBUG']
+logger.level = Logger::WARN unless ENV['DEBUG'] == 'true'
+
+runner = Elasticsearch::Tests::TestRunner.new(client, tests_path, logger)
+runner.run
+```
+
+### Running particular tests
+
+When you run the tests, you can pass in the name of a particular test or a whole test folder, to run only those tests. Tests in the clients project are located [in the `tests` directory](https://github.com/elastic/elasticsearch-clients-tests/tree/main/tests) either as single yaml files or inside a specific directory, referring to a specific namespace. For example [`tests/get.yml`](https://github.com/elastic/elasticsearch-clients-tests/blob/main/tests/get.yml) and [`tests/bulk/10_basic.yml`](https://github.com/elastic/elasticsearch-clients-tests/blob/main/tests/bulk/10_basic.yml). If you want to run the `get.yml` test, you can pass in the file name to `run`:
+
+```ruby
+runner.run('get.yml')
+```
 
-Elasticsearch::Tests::TestRunner.new(client, tests_path, logger).run
+If you want to run the basic bulk tests, you can run:
+
+```ruby
+runner.run('bulk/10_basic.yml')
 ```
 
+If you want to run all the tests in a directory, you can pass in the directory:
+
+```ruby
+runner.run('indices')
+```
+
+This will run all the tests in [`tests/indices`](https://github.com/elastic/elasticsearch-clients-tests/tree/main/tests/indices) such as `alias.yml`, `analyze.yml`, and so on.
+
+### Skipping tests
+
+If you want to skip any given tests, you can do it using `add_tests_to_skip` before calling `run` like this:
+
+```ruby
+runner.add_tests_to_skip(['bulk/10_basic.yml', 'get.yml'])
+```
+
+You need to pass in an Array of file or folder names, or a single test file as a String.
+
+### Downloading the test suite
+
 You can **download the YAML test files** from [the clients tests project](https://github.com/elastic/elasticsearch-clients-tests) with the following code:
 
 ```ruby
@@ -47,6 +85,40 @@ Elasticsearch::Tests::Downloader::run(tests_path)
 
 Additionally, you can run the rake task `rake es_tests:download` included in `lib/elasticsearch/tasks`.
 
+### Environment variables
+
+You can set the following environment variables when using the test runner:
+
+#### `DEBUG`
+
+If you set `DEBUG` to `true`, you'll see debug messages for each tests, including the response status, body and headers sent from Elasticsearch:
+
+```
+🟢 cluster/put_settings.yml - is_true: acknowledged passed [200]
+┌[DEBUG]──────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┐
+│File: cluster/remote_info.yml | Action: remote_info                                                                              │
+│Parameters: {}                                                                                                                   │
+│                                                                                                                                 │
+│Response status: 200                                                                                                             │
+│Response body:                                                                                                                   │
+│Response headers:                                                                                                                │
+│  x-elastic-product: Elasticsearch                                                                                               │
+│  content-type: application/vnd.elasticsearch+json;compatible-with=9                                                             │
+│  content-length: 2                                                                                                              │
+└─────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────────┘
+```
+
+#### `QUIET`
+
+If you set `QUIET` to anything other than `false` or the String `"false"`, tests will run in quiet mode, where you'll only see the green (success) and/or red (failure) output for the tests:
+
+```
+$ QUIET=1 be rake test:yaml
+🟢 🟢 🟢 🟢 🟢 🔴 🟢 🟢 🟢 🟢 🟢 🟢 🟢 🟢 🟢 🟢 🟢 🟢 🟢 🟢 🟢 🟢 🟢 🟢 🟢 🟢 🟢 🟢 🟢 🟢 🟢 🟢 🟢 🟢 🟢
+ ```
+
+It will still print out the summary and information about failures and/or errors at the bottom.
+
 ## Development
 
 See [CONTRIBUTING](./CONTRIBUTING.md).

data/Rakefile

@@ -16,5 +16,18 @@
 # under the License.
 
 require 'bundler/gem_tasks'
+require 'elasticsearch/tests/downloader'
 task default: %i[]
-Dir.glob('lib/elasticsearch/tasks/**/*.rake').each { |f| import f }
+
+namespace :es_tests do
+  desc 'Download YAML test files'
+  task :download do |_, args|
+    tests_path = args[:path] || File.expand_path('./tmp', __dir__)
+    Elasticsearch::Tests::Downloader::run(tests_path)
+  end
+end
+
+desc 'Run unit tests'
+task :test do
+  sh 'bundle exec rspec'
+end

data/lib/elasticsearch/tests/code_runner.rb

@@ -1,3 +1,5 @@
+# frozen_string_literal: true
+
 # Licensed to Elasticsearch B.V. under one or more contributor
 # license agreements. See the NOTICE file distributed with
 # this work for additional information regarding copyright
@@ -35,7 +37,15 @@ module Elasticsearch
       # specifications. These are function calls to the Elasticsearch clients.
       #
       def do_action(action)
+        @response = nil
+        catchable = action.delete('catch')
         client = @client
+        if action['headers']
+          client.transport.options[:transport_options][:headers].merge(
+            { headers: action.delete('headers') }
+          )
+        end
+
         action = action.first if action.is_a?(Array)
         method, params = action.is_a?(String) ? [action, {}] : action.first
 
@@ -45,18 +55,70 @@ module Elasticsearch
           client = @client.send(arrayed_method.first)
           method = arrayed_method.last
         end
-
-        @response = client.send(method.to_sym, process_params(params))
+        params = process_params(params)
+        @response = client.send(method.to_sym, params)
+        print_debug_message(method.to_sym, params) if debug?
         @response
       rescue StandardError => e
-        raise e
+        # Raise if it's an actual error:
+        unless expected_exception?(catchable, e)
+          print_failure(action, @response, e)
+          raise e
+        end
+        # Show success if we caught an expected exception:
+        print_success
+        print_debug_catchable(e) if debug?
+      end
+
+      # The keyword `catch` is used in the yaml tests to catch exceptions. This code looks at the
+      # expected error and the response error to compare them and check that we got the expected
+      # exception.
+      def expected_exception?(error_type, e)
+        return false if error_type.nil?
+
+        case error_type
+        when 'request_timeout'
+          e.is_a?(Elastic::Transport::Transport::Errors::RequestTimeout)
+        when 'missing', /resource_not_found_exception/
+          e.is_a?(Elastic::Transport::Transport::Errors::NotFound)
+        when 'conflict'
+          e.is_a?(Elastic::Transport::Transport::Errors::Conflict)
+        when 'request'
+          e.is_a?(Elastic::Transport::Transport::Errors::InternalServerError)
+        when 'bad_request'
+          e.is_a?(Elastic::Transport::Transport::Errors::BadRequest)
+        when 'param'
+          actual_error.is_a?(ArgumentError)
+        when 'unauthorized'
+          e.is_a?(Elastic::Transport::Transport::Errors::Unauthorized)
+        when 'forbidden'
+          e.is_a?(Elastic::Transport::Transport::Errors::Forbidden)
+        when /error parsing field/, /illegal_argument_exception/
+          e.message.match?(/\[400\]/) ||
+            e.is_a?(Elastic::Transport::Transport::Errors::BadRequest)
+        when /NullPointerException/
+          e.message.match?(/\[400\]/)
+        when /status_exception/
+          e.message.match?(/\[409\]/)
+        else
+          e.message.match?(/#{error_type}/)
+        end
       end
 
       # Code for matching expectations and response
       #
       def do_match(action)
         k, v = action['match'].first
-        v = instance_variable_get(v.gsub('$', '@')) if v.is_a?(String) && v.include?('$')
+
+        v = if v.is_a?(String) && v.include?('$')
+              instance_variable_get(v.gsub('$', '@'))
+            elsif v.is_a?(Array)
+              v.map do |a|
+                a.is_a?(String) && a.start_with?('$') ? instance_variable_get(a.gsub('$', '@')) : a
+              end
+            else
+              v
+            end
         result = search_in_response(k)
 
         if !result.nil? && (
@@ -67,20 +129,27 @@ module Elasticsearch
           print_success
         else
           print_match_failure(action)
+          message = "Match failure\nExpected #{k}: #{v}.\nResult: #{result}"
+          raise ActionError.new(message, $test_file, action)
         end
       end
 
       def match_regexp(expected, result)
-        expected.is_a?(String) && expected.match?(/^\//) && result.match?(Regexp.new(expected.gsub('/', '')))
+        expected.is_a?(String) &&
+          expected.match?(%r{^/}) &&
+          result.match?(Regexp.new(expected.gsub('/', '').strip))
       end
 
       def do_length(action)
         k, v = action['length'].first
-        result = search_in_response(k).count
+        result = search_in_response(k)
+        result = result.count
         if result && result == v
           print_success
         else
           print_failure(action, @response)
+          message = "do_length failure\nExpected #{k}: #{v}.\nResult: #{result}"
+          raise ActionError.new(message, $test_file, action)
         end
       end
 
@@ -99,6 +168,8 @@ module Elasticsearch
           print_success
         else
           print_failure(action, @response)
+          message = "is_true failure\nResponse: #{response_value || @response}."
+          raise ActionError.new(message, $test_file, action)
         end
       end
 
@@ -108,6 +179,8 @@ module Elasticsearch
           print_success
         else
           print_failure(action, @response)
+          message = "is_false failure\nResponse: #{response_value || @response}."
+          raise ActionError.new(message, $test_file, action)
         end
       end
 
@@ -118,11 +191,14 @@ module Elasticsearch
       #
       def compare(action)
         operator, value = action.first
+
         result = search_in_response(value.keys.first)
         if result&.send(COMPARATORS[operator], value[value.keys.first])
           print_success
         else
           print_failure(action, @response)
+          message = "comparision failure\n#{result} #{COMPARATORS[operator]} #{value[value.keys.first]}"
+          raise ActionError.new(message, $test_file, action)
         end
       end
 
@@ -135,16 +211,18 @@ module Elasticsearch
 
       private
 
+      def debug?
+        ENV['DEBUG'] == 'true'
+      end
+
       # Given a key coming from a test definition, search the response body for a matching value.
       #
       def search_in_response(keys)
-        if (match = /\$([a-z]+)/.match(keys))
-          return @response.send(match[1])
-        end
-
         if keys.include?('.')
           key = split_and_parse_key(keys)
           return find_value_in_document(key, @response.body)
+        elsif (match = /\$([a-z]+)/.match(keys))
+          return @response.send(match[1])
         end
 
         @response[keys]
@@ -164,7 +242,13 @@ module Elasticsearch
       end
 
       def set_param_variable(params, key, param)
-        params[key] = instance_variable_get(param.gsub('$', '@')) if param.is_a?(String) && param.include?('$')
+        return unless param.is_a?(String) && param.include?('$')
+
+        # Param can be a single '$value' string or '{ something: $value }'
+        replaceable = param.match(/(\$[0-9a-z_-]+)/)[0]
+        value = instance_variable_get(replaceable.gsub('$', '@'))
+        content = param.gsub(replaceable, value)
+        params[key] = content
       end
 
       # Given a list of keys, find the value in a recursively nested document.
@@ -179,7 +263,9 @@ module Elasticsearch
         return document[chain[0]] unless chain.size > 1
 
         # a number can be a string key in a Hash or indicate an element in a list
-        if document.is_a?(Hash)
+        if chain[0].is_a?(String) && chain[0].match?(/^\$/)
+          find_value_in_document(chain[1..], instance_variable_get("@#{chain[0].gsub('$', '')}"))
+        elsif document.is_a?(Hash)
           find_value_in_document(chain[1..], document[chain[0].to_s]) if document[chain[0].to_s]
         elsif document[chain[0]]
           find_value_in_document(chain[1..], document[chain[0]]) if document[chain[0]]

data/lib/elasticsearch/tests/downloader.rb

@@ -15,24 +15,25 @@
 # specific language governing permissions and limitations
 # under the License.
 require 'open-uri'
+require 'fileutils'
 
 module Elasticsearch
   module Tests
     # Module for downloading the test files
     module Downloader
       class << self
-        FILENAME = 'tests.zip'.freeze
+        FILENAME = 'tests.tar.gz'.freeze
 
-        def run(path, branch = 'main')
-          delete_files(path)
-          url = "https://api.github.com/repos/elastic/serverless-clients-tests/zipball/#{branch}"
+        def run(path, branch = 'main', delete: false)
+          delete_files(path) if delete
+          url = "https://api.github.com/repos/elastic/elasticsearch-clients-tests/tarball/#{branch}"
           if download_tests(url)
             puts "Successfully downloaded #{FILENAME}"
           else
             warn "[!] Couldn't download #{FILENAME}"
             return
           end
-          unzip_file(path)
+          untar_file(path)
           File.delete(FILENAME)
         end
 
@@ -48,11 +49,12 @@ module Elasticsearch
 
         private
 
-        def unzip_file(path)
-          puts 'Unzipping files'
+        def untar_file(path)
+          puts 'Extracting tar files'
           puts path
-          `unzip #{FILENAME} -d #{path}/`
-          puts 'Removing zip file'
+          FileUtils.mkdir_p(path) unless File.directory?(path)
+          `tar -zxf #{FILENAME} --strip-components=1 -C #{path}/`
+          puts 'Removing tar file'
         end
 
         def delete_files(path)

data/lib/elasticsearch/tests/errors.rb

@@ -19,5 +19,16 @@ module Elasticsearch
   module Tests
     class TestFailure < StandardError; end
     class TestError < StandardError; end
+
+    # Class to track exactly which action errored
+    class ActionError < StandardError
+      attr_reader :test_file, :action
+
+      def initialize(message, test_file, action)
+        super(message)
+        @test_file = test_file
+        @action = action
+      end
+    end
   end
 end

data/lib/elasticsearch/tests/printer.rb

@@ -14,6 +14,8 @@
 # KIND, either express or implied.  See the License for the
 # specific language governing permissions and limitations
 # under the License.
+require 'tty-box'
+require 'tty-screen'
 require_relative 'errors'
 
 module Elasticsearch
@@ -22,58 +24,136 @@ module Elasticsearch
     # Functions to print out test results, errors, summary, etc.
     #
     module Printer
+      BOX_WIDTH = TTY::Screen.width
+
       def print_success
-        response = if [true, false].include? @response
-                     @response
-                   else
-                     @response.status
-                   end
-        puts "🟢 #{@file} #{@title} passed. Response: #{response}"
+        if quiet?
+          print '🟢 '
+        else
+          msg = "🟢 \e[33m#{@short_name}\e[0m - #{@action} \e[32mpassed\e[0m"
+          if @response.nil?
+            puts msg
+          else
+            status = boolean_response? ? @response : @response.status
+            puts "#{msg} [#{status}]"
+          end
+        end
       end
 
-      def print_failure(action, response)
-        puts "🔴 #{@file} #{@title} failed"
-        puts "Expected result: #{action}" # TODO: Show match/length differently
-        if response.is_a?(ElasticsearchServerless::API::Response)
-          puts 'Response:'
-          pp response.body
+      def quiet?
+        !ENV['QUIET'].nil? && ![false, 'false'].include?(ENV['QUIET'])
+      end
+
+      def print_failure(action, response, exception = nil)
+        if quiet?
+          print '🔴 '
         else
-          pp response
+          puts "🔴 \e[33m#{@short_name}\e[0m - #{@action}  \e[31mfailed\e[0m"
         end
-        raise Elasticsearch::Tests::TestFailure
+        message = ["Expected result: #{action}"]
+        if response&.body
+          message << response.body
+        else
+          message << response
+        end
+        message << "Exception: #{exception}" if exception
       end
 
       def print_match_failure(action)
         keys = action['match'].keys.first
         value = action['match'].values.first
+
+        if quiet?
+          print '🔴 '
+        else
+          puts "🔴 \e[33m#{@short_name} (match)\e[0m - \e[32;9m#{keys}: #{value}\e[0m  \e[95m#{keys}: #{search_in_response(action['match'].keys.first)}\e[0m \e[31mfailed\e[0m"
+        end
         message = <<~MSG
-          🔴 #{@file} #{@title} failed
-          Expected: { #{keys}: #{value} }
-          Actual  : { #{keys}: #{search_in_response(action['match'].keys.first)} }
+          #{@short_name} #{@title} failed
+          \e[31m\e[1mExpected: { #{keys}: #{value} }\e[0m
+          \e[32m\e[1mGot     : { #{keys}: #{search_in_response(action['match'].keys.first)} }\e[0m
+          \e[1mResponse\e[0m:\n\e[36m #{@response}\e[0m
         MSG
         raise Elasticsearch::Tests::TestFailure.new(message)
       end
 
       def print_error(error)
-        puts "❌ ERROR: #{@file} #{@title} failed\n"
+        print TTY::Box.error("❌ ERROR: #{@short_name} #{@title} failed", width: BOX_WIDTH)
         logger.error error.display
         backtrace = error.backtrace.join("\n")
         logger.error "#{backtrace}\n"
         raise error
       end
 
-      def self.display_errors(errors)
-        puts "+++ ❌ Errors/Failures: #{errors.count}"
-        errors.map do |error|
-          puts "* Test: #{error[:file]}\n #{error[:error].message}"
-          pp error[:error].backtrace.join("$/\n") if ENV['DEBUG']
-          puts
+      def self.display_errors(errors, logger)
+        puts
+        print TTY::Box.frame("❌ Errors/Failures: #{errors.count}", width: BOX_WIDTH, style: { border: { fg: :red } })
+
+        errors.each do |error|
+          print TTY::Box.frame("🧪 Test: #{error[:file]}", width: BOX_WIDTH * 0.6, style: { border: { fg: :red } })
+          message = []
+          message << "▶ Action: #{error[:action].first}" if error[:action]
+          message << "🔬 #{error[:error].class} - #{error[:error].message}"
+          message << error[:error].backtrace.join("$/\n") if ENV['DEBUG'] == 'true'
+          puts message.join("\n")
+          logger.error(message.join("\n"))
         end
       end
 
-      def self.display_summary(tests_count, errors_count, start_time)
-        puts "--- 🧪 Tests: #{tests_count} | Passed: #{tests_count - errors_count} | Failed: #{errors_count}"
-        puts "--- ⏲  Elapsed time: #{Time.at(Time.now - start_time).utc.strftime("%H:%M:%S")}"
+      def self.display_summary(tests_count, errors_count, start_time, logger)
+        summary = "🧪 Tests: #{tests_count} | Passed: #{tests_count - errors_count} | Failed: #{errors_count}"
+        logger.info summary
+        duration = "⏲  Elapsed time: #{Time.at(Time.now - start_time).utc.strftime('%H:%M:%S')}"
+        message = <<~MSG
+                     #{summary}
+                     #{duration}
+        MSG
+        puts
+        print TTY::Box.frame(message, width: BOX_WIDTH, title: { top_left: '[SUMMARY]' }, style: { border: { fg: :cyan } })
+        logger.info duration
+      end
+
+      def print_debug_message(method, params)
+        begin
+          message = [
+            "File: #{$test_file} | Action: #{method}",
+            "Parameters: #{params}",
+            ''
+          ]
+          if boolean_response?
+            message << "Response: #{@response}"
+          else
+            message << "Response status: #{@response.status}"
+            message << 'Response body:'
+            if @response.body.is_a?(String)
+              message.push(@response.body.empty? ? '  ""' : @response.body)
+            elsif @response.body.is_a?(Hash)
+              message.push(*@response.body.map { |k, v| "  #{k}: #{v}" })
+            end
+            message << 'Response headers:'
+            message.push(*@response.headers.map { |k, v| "  #{k}: #{v}" })
+          end
+          puts TTY::Box.frame(message.join("\n"), width: BOX_WIDTH, title: { top_left: '[DEBUG]' })
+        rescue ArgumentError => e
+          if e.message == 'invalid byte sequence in UTF-8'
+            @response.body.encode!('UTF-8', 'binary', invalid: :replace, undef: :replace, replace: '')
+            print_debug_message(method, params)
+          end
+        end
+      end
+
+      def print_debug_catchable(exception)
+        puts TTY::Box.frame(
+               "Catchable: #{exception}\nResponse: #{@response}\n",
+               width: BOX_WIDTH,
+               title: { top_left: '[DEBUG]' }
+             )
+      end
+
+      private
+
+      def boolean_response?
+        [true, false].include? @response
       end
     end
   end

data/lib/elasticsearch/tests/test.rb

@@ -16,6 +16,7 @@
 # under the License.
 
 require_relative 'code_runner'
+require_relative 'errors'
 
 module Elasticsearch
   module Tests
@@ -32,7 +33,9 @@ module Elasticsearch
         @teardown = extract_teardown!(yaml)
         @title = yaml.first.keys.first
         @actions = yaml.first[@title]
-        @file = test_filename(file)
+        @file = file
+        name = file.split('/')
+        @short_name = "#{name[-2]}/#{name[-1]}"
         @client = client
       end
 
@@ -56,20 +59,28 @@ module Elasticsearch
 
         case definition
         when 'do'
+          @action = action['do'].keys.first
           do_action(action['do'])
         when 'set'
           set_variable(action)
         when 'match'
+          @action = "#{action.keys.first} #{format_action(action[action.keys.first])}"
           do_match(action)
         when 'length'
+          @action = format_action(action)
           do_length(action)
         when 'is_true'
+          @action = format_action(action)
           is_true(action)
         when 'is_false'
+          @action = format_action(action)
           is_false(action)
         when 'gt', 'gte', 'lt', 'lte'
+          @action = format_action(action)
           compare(action)
         end
+      rescue StandardError => e
+        raise ActionError.new(e.message, @file, action)
       end
 
       def run_setup
@@ -87,12 +98,7 @@ module Elasticsearch
       end
 
       def count
-        @actions.length
-      end
-
-      def test_filename(file)
-        name = file.split('/')
-        "#{name[-2]}/#{name[-1]}"
+        @actions.select { |a| a.keys.first == 'do' }.count
       end
 
       def extract_setup!(yaml)
@@ -106,6 +112,17 @@ module Elasticsearch
           yaml.delete_at(i) if a.keys.first == 'teardown'
         end.compact.first
       end
+
+      private
+
+      def format_action(action)
+        action.to_s
+              .gsub(/^{/, '')
+              .gsub(' =>', ':')
+              .gsub('"', '')
+              .gsub('}', '')
+              .gsub('{', '|')
+      end
     end
   end
 end

data/lib/elasticsearch/tests/test_runner.rb

@@ -25,13 +25,23 @@ module Elasticsearch
     # Main YAML test runner
     class TestRunner
       LOGGER = Logger.new($stdout)
-      LOGGER.level = Logger::WARN unless ENV['DEBUG']
+      LOGGER.level = ENV['DEBUG'] == 'true' ? Logger::DEBUG : Logger::WARN
 
       def initialize(client, path = nil, logger = nil)
         @client = client
-        @serverless = (defined?(::ElasticsearchServerless) && client.is_a?(::ElasticsearchServerless::Client))
+        @serverless = defined?(::ElasticsearchServerless) && client.is_a?(::ElasticsearchServerless::Client) ||
+                      ENV['TEST_SUITE'] == 'serverless'
         @path = path || File.expand_path('./tmp', __dir__)
         @logger = logger || LOGGER
+        @tests_to_skip = []
+      end
+
+      def add_tests_to_skip(tests)
+        if tests.is_a? String
+          @tests_to_skip << tests
+        else
+          @tests_to_skip += tests
+        end
       end
 
       def run(test_files = [])
@@ -39,8 +49,9 @@ module Elasticsearch
 
         @test_files = select_test_files(test_files)
         run_the_tests
-        Elasticsearch::Tests::Printer::display_errors(@errors) unless @errors.empty?
-        Elasticsearch::Tests::Printer::display_summary(@tests_count, @errors.count, @start_time)
+        Elasticsearch::Tests::Printer::display_errors(@errors, @logger) unless @errors.empty?
+        Elasticsearch::Tests::Printer::display_summary(@tests_count, @errors.count, @start_time, @logger)
+        exit 1 unless @errors.empty?
       end
 
       def run_the_tests
@@ -48,39 +59,60 @@ module Elasticsearch
         @tests_count = 0
         @errors = []
 
-        @test_files.map do |test_file|
-          build_and_run_tests(test_file)
+        @test_files.map do |test_path|
+          $test_file = test_filename(test_path)
+          build_and_run_tests(test_path)
         rescue Psych::SyntaxError => e
-          @errors << { error: e, file: test_file }
-          @logger.warn("YAML error in #{test_file}")
+          @errors << { error: e, file: $test_file }
+          @logger.warn("YAML error in #{$test_file}")
           @logger.warn e
+        rescue ActionError => e
+          @errors << { error: e, file: $test_file, action: e.action }
+          @logger.debug e
         rescue StandardError => e
-          @errors << { error: e, file: test_file }
+          @errors << { error: e, file: $test_file }
           @logger.debug e
+        rescue SystemExit, Interrupt
+          exit
         end
       end
 
-      def build_and_run_tests(test_file)
-        yaml = YAML.load_stream(File.read(test_file))
-        requires = extract_requires!(yaml).compact.first['requires']
+      def build_and_run_tests(test_path)
+        yaml = YAML.load_stream(File.read(test_path))
+        begin
+          requires = extract_requires!(yaml).compact.first['requires']
+        rescue StandardError => e
+          raise ArgumentError, "Could not find requires key in test file:\n #{test_path}\n#{e}"
+        end
         return unless (requires['serverless'] == true && @serverless) ||
                       (requires['stack'] == true && !@serverless)
 
-        test = Elasticsearch::Tests::Test.new(yaml, test_file, @client)
+        test = Elasticsearch::Tests::Test.new(yaml, test_path, @client)
         test.execute
         @tests_count += test.count
-      rescue StandardError => e
-        raise e
+      end
+
+      def test_filename(file)
+        name = file.split('/')
+        "#{name[-2]}/#{name[-1]}"
       end
 
       def select_test_files(test_files)
-        if test_files.empty?
-          Dir.glob("#{@path}/**/*.yml")
-        else
-          test_files.map do |test_file|
-            "#{@path}/tests/#{test_file}"
-          end
+        tests_path = if test_files.empty?
+                       "#{@path}/**/*.yml"
+                     elsif test_files.include?('yml')
+                       return ["#{@path}/tests/#{test_files}"]
+                     else
+                       "#{@path}/tests/#{test_files}/*.yml"
+                     end
+        tests = Dir.glob(tests_path)
+        # Find the full paths of tests to be skipped and delete them from the tests to run:
+        full_paths = @tests_to_skip.map do |skip|
+          tests.find { |t| t.match?(skip) }
         end
+        full_paths.each { |a| tests.delete(a) }
+
+        tests
       end
 
       def extract_requires!(yaml)

data/lib/elasticsearch/tests/version.rb

@@ -14,10 +14,11 @@
 # KIND, either express or implied.  See the License for the
 # specific language governing permissions and limitations
 # under the License.
+
 # frozen_string_literal: true
 
 module Elasticsearch
   module Tests
-    VERSION = "0.1.0"
+    VERSION = '0.20.0'
   end
 end

metadata

@@ -1,15 +1,42 @@
 --- !ruby/object:Gem::Specification
 name: elasticsearch-test-runner
 version: !ruby/object:Gem::Version
-  version: 0.1.0
+  version: 0.20.0
 platform: ruby
 authors:
 - Elastic Client Library Maintainers
-autorequire: 
 bindir: exe
 cert_chain: []
-date: 2024-06-20 00:00:00.000000000 Z
+date: 1980-01-02 00:00:00.000000000 Z
 dependencies:
+- !ruby/object:Gem::Dependency
+  name: tty-box
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: tty-screen
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 - !ruby/object:Gem::Dependency
   name: elasticsearch
   requirement: !ruby/object:Gem::Requirement
@@ -38,6 +65,34 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: rspec-mocks
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
 description: A test runner for the Elasticsearch clients YAML test suite, used in
   the elasticsearch and elasticsearch-serverless gems.
 email:
@@ -53,8 +108,6 @@ files:
 - NOTICE
 - README.md
 - Rakefile
-- elasticsearch-tests.gemspec
-- lib/elasticsearch/tasks/download_test_suite.rake
 - lib/elasticsearch/tests.rb
 - lib/elasticsearch/tests/code_runner.rb
 - lib/elasticsearch/tests/downloader.rb
@@ -70,7 +123,6 @@ metadata:
   homepage_uri: https://www.elastic.co/guide/en/elasticsearch/client/ruby-api/current/index.html
   changelog_uri: https://github.com/elastic/es-test-runner-ruby/blob/main/CHANGELOG.md
   source_code_uri: https://github.com/elastic/es-test-runner-ruby/tree/main
-post_install_message: 
 rdoc_options: []
 require_paths:
 - lib
@@ -85,8 +137,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.5.13
-signing_key: 
+rubygems_version: 4.0.10
 specification_version: 4
 summary: Tool to test Elasticsearch clients against the YAML clients test suite.
 test_files: []

data/elasticsearch-tests.gemspec

@@ -1,51 +0,0 @@
-# Licensed to Elasticsearch B.V. under one or more contributor
-# license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright
-# ownership. Elasticsearch B.V. licenses this file to you under
-# the Apache License, Version 2.0 (the "License"); you may
-# not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#   http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied.  See the License for the
-# specific language governing permissions and limitations
-# under the License.
-
-# frozen_string_literal: true
-
-require_relative 'lib/elasticsearch/tests/version'
-
-Gem::Specification.new do |spec|
-  spec.name = 'elasticsearch-test-runner'
-  spec.email = ['client-libs@elastic.co']
-  spec.version = Elasticsearch::Tests::VERSION
-  spec.authors = ['Elastic Client Library Maintainers']
-  spec.licenses = ['Apache-2.0']
-  spec.summary = 'Tool to test Elasticsearch clients against the YAML clients test suite.'
-  spec.homepage = 'https://www.elastic.co/guide/en/elasticsearch/client/ruby-api/current/index.html'
-  spec.description = 'A test runner for the Elasticsearch clients YAML test suite, used in the elasticsearch and elasticsearch-serverless gems.'
-  spec.metadata['homepage_uri'] = spec.homepage
-  spec.metadata['changelog_uri'] = 'https://github.com/elastic/es-test-runner-ruby/blob/main/CHANGELOG.md'
-  spec.metadata['source_code_uri'] = 'https://github.com/elastic/es-test-runner-ruby/tree/main'
-
-  spec.required_ruby_version = '>= 3.0'
-
-  # Specify which files should be added to the gem when it is released.
-  # The `git ls-files -z` loads the files in the RubyGem that have been added into git.
-  spec.files = Dir.chdir(__dir__) do
-    `git ls-files -z`.split("\x0").reject do |f|
-      (File.expand_path(f) == __FILE__) ||
-        f.start_with?(*%w[bin/ test/ spec/ features/ .git appveyor Gemfile])
-    end
-  end
-  spec.bindir = 'exe'
-  spec.executables = spec.files.grep(%r{\Aexe/}) { |f| File.basename(f) }
-  spec.require_paths = ['lib']
-
-  spec.add_development_dependency 'elasticsearch'
-  spec.add_development_dependency 'elasticsearch-serverless'
-end

data/lib/elasticsearch/tasks/download_test_suite.rake

@@ -1,25 +0,0 @@
-# Licensed to Elasticsearch B.V. under one or more contributor
-# license agreements. See the NOTICE file distributed with
-# this work for additional information regarding copyright
-# ownership. Elasticsearch B.V. licenses this file to you under
-# the Apache License, Version 2.0 (the "License"); you may
-# not use this file except in compliance with the License.
-# You may obtain a copy of the License at
-#
-#   http://www.apache.org/licenses/LICENSE-2.0
-#
-# Unless required by applicable law or agreed to in writing,
-# software distributed under the License is distributed on an
-# "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY
-# KIND, either express or implied.  See the License for the
-# specific language governing permissions and limitations
-# under the License.
-require 'elasticsearch/tests/downloader'
-
-namespace :es_tests do
-  desc 'Download YAML test files'
-  task :download do |_, args|
-    tests_path = args[:path] || File.expand_path('../../tmp', __dir__)
-    Elasticsearch::Tests::Downloader::run(tests_path)
-  end
-end