logstash-filter-elasticsearch 4.1.1 → 4.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 1ca5815c2b1127152e31b4ca9e0192021bfb0b0d9d2807ca090c2123feea2bb8
-  data.tar.gz: b3983e01f4e73076e48f3d793f2061cb3210490b09394352146b65377f946137
+  metadata.gz: deef535881ed311e6241449028bf6b54b65e0a42e6513b1ab7286de724f5445f
+  data.tar.gz: 103f17c0b6d6530c90afc412f69c05dc29ec4f76aecb7f82e22d7efd4f497647
 SHA512:
-  metadata.gz: 65822b043de7055e2422810c4b03662c45fb15d58f45edadd677a220287d37a8f8c7d1382e3b1dc04e00ac51e261a1e5f7e11c06b4d57481218be9da2f498534
-  data.tar.gz: 8738742640bf8729297f556154d6ef11878e99b55bfe1193da459f2f83cf255d5e5973537bc6e97e5d32021d1976f6aa4c284081a98f5b7a84c1073dfa039cc2
+  metadata.gz: 54b3f3b78874934793ac7d7dca148236578b7756a20bddb70d020e14f9b222fbc70b4c8f2b7f132b238ed9d9690d4ac8e684749e959aa64d3ccd0b58aded7790
+  data.tar.gz: 98e18b162c4dd25827fb1b74ec62d75d152458950b8ee157e635a6c772247774aeaa82ec2ca5031e808361a02b454360438e4fd717deb07987d3d2519e00a327

data/CHANGELOG.md

@@ -1,3 +1,9 @@
+## 4.3.0
+  - ES|QL support [#194](https://github.com/logstash-plugins/logstash-filter-elasticsearch/pull/194)
+
+## 4.2.0
+  - Add `target` configuration option to store the result into it [#196](https://github.com/logstash-plugins/logstash-filter-elasticsearch/pull/196)
+
 ## 4.1.1
   - Add elastic-transport client support used in elasticsearch-ruby 8.x [#191](https://github.com/logstash-plugins/logstash-filter-elasticsearch/pull/191)
 

data/docs/index.asciidoc

@@ -54,7 +54,7 @@ if [type] == "end" {
 
 The example below reproduces the above example but utilises the query_template.
 This query_template represents a full Elasticsearch query DSL and supports the
-standard Logstash field substitution syntax.  The example below issues
+standard {ls} field substitution syntax. The example below issues
 the same query as the first example but uses the template shown.
 
 [source,ruby]
@@ -118,6 +118,110 @@ Authentication to a secure Elasticsearch cluster is possible using _one_ of the
 Authorization to a secure Elasticsearch cluster requires `read` permission at index level and `monitoring` permissions at cluster level.
 The `monitoring` permission at cluster level is necessary to perform periodic connectivity checks.
 
+[id="plugins-{type}s-{plugin}-esql"]
+==== {esql} support
+
+.Technical Preview
+****
+The {esql} feature that allows using ES|QL queries with this plugin is in Technical Preview.
+Configuration options and implementation details are subject to change in minor releases without being preceded by deprecation warnings.
+****
+
+{es} Query Language ({esql}) provides a SQL-like interface for querying your {es} data.
+
+To use {esql}, this plugin needs to be installed in {ls} 8.17.4 or newer, and must be connected to {es} 8.11 or newer.
+
+To configure {esql} query in the plugin, set your {esql} query in the `query` parameter.
+
+IMPORTANT: We recommend understanding {ref}/esql-limitations.html[{esql} current limitations] before using it in production environments.
+
+The following is a basic {esql} query that sets the food name to transaction event based on upstream event's food ID:
+[source, ruby]
+    filter {
+      elasticsearch {
+        hosts => [ 'https://..']
+        api_key => '....'
+        query => '
+          FROM food-index
+            | WHERE id == ?food_id
+        '
+        query_params => {
+          "food_id" => "[food][id]"
+        }
+      }
+    }
+
+Set `config.support_escapes: true` in `logstash.yml` if you need to escape special chars in the query.
+
+In the result event, the plugin sets total result size in `[@metadata][total_values]` field.
+
+[id="plugins-{type}s-{plugin}-esql-event-mapping"]
+===== Mapping {esql} result to {ls} event
+{esql} returns query results in a structured tabular format, where data is organized into _columns_ (fields) and _values_ (entries).
+The plugin maps each value entry to an event, populating corresponding fields.
+For example, a query might produce a table like:
+
+[cols="2,1,1,1,2",options="header"]
+|===
+|`timestamp` |`user_id` | `action` | `status.code` | `status.desc`
+
+|2025-04-10T12:00:00 |123 |login |200 | Success
+|2025-04-10T12:05:00 |456 |purchase |403 | Forbidden (unauthorized user)
+|===
+
+For this case, the plugin creates two JSON look like objects as below and places them into the `target` field of the event if `target` is defined.
+If `target` is not defined, the plugin places the _only_ first result at the root of the event.
+[source, json]
+[
+  {
+    "timestamp": "2025-04-10T12:00:00",
+    "user_id": 123,
+    "action": "login",
+    "status": {
+        "code": 200,
+        "desc": "Success"
+    }
+  },
+  {
+    "timestamp": "2025-04-10T12:05:00",
+    "user_id": 456,
+    "action": "purchase",
+    "status": {
+        "code": 403,
+        "desc": "Forbidden (unauthorized user)"
+    }
+  }
+]
+
+NOTE: If your index has a mapping with sub-objects where `status.code` and `status.desc` actually dotted fields, they appear in {ls} events as a nested structure.
+
+[id="plugins-{type}s-{plugin}-esql-multifields"]
+===== Conflict on multi-fields
+
+{esql} query fetches all parent and sub-fields fields if your {es} index has https://www.elastic.co/docs/reference/elasticsearch/mapping-reference/multi-fields[multi-fields] or https://www.elastic.co/docs/reference/elasticsearch/mapping-reference/subobjects[subobjects].
+Since {ls} events cannot contain parent field's concrete value and sub-field values together, the plugin ignores sub-fields with warning and includes parent.
+We recommend using the `RENAME` (or `DROP` to avoid warning) keyword in your {esql} query explicitly rename the fields to include sub-fields into the event.
+
+This is a common occurrence if your template or mapping follows the pattern of always indexing strings as "text" (`field`) + " keyword" (`field.keyword`) multi-field.
+In this case it's recommended to do `KEEP field` if the string is identical and there is only one subfield as the engine will optimize and retrieve the keyword, otherwise you can do `KEEP field.keyword | RENAME field.keyword as field`.
+
+To illustrate the situation with example, assuming your mapping has a time `time` field with `time.min` and `time.max` sub-fields as following:
+[source, ruby]
+    "properties": {
+        "time": { "type": "long" },
+        "time.min": { "type": "long" },
+        "time.max": { "type": "long" }
+    }
+
+The {esql} result will contain all three fields but the plugin cannot map them into {ls} event.
+To avoid this, you can use the `RENAME` keyword to rename the `time` parent field to get all three fields with unique fields.
+[source, ruby]
+    ...
+    query => 'FROM my-index | RENAME time AS time.current'
+    ...
+
+For comprehensive ES|QL syntax reference and best practices, see the https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-syntax.html[{esql} documentation].
+
 [id="plugins-{type}s-{plugin}-options"]
 ==== Elasticsearch Filter Configuration Options
 
@@ -143,6 +247,8 @@ NOTE: As of version `4.0.0` of this plugin, a number of previously deprecated se
 | <<plugins-{type}s-{plugin}-password>> |<<password,password>>|No
 | <<plugins-{type}s-{plugin}-proxy>> |<<uri,uri>>|No
 | <<plugins-{type}s-{plugin}-query>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-query_type>> |<<string,string>>, one of `["dsl", "esql"]`|No
+| <<plugins-{type}s-{plugin}-query_params>> |<<hash,hash>> or <<hash,hash>>|No
 | <<plugins-{type}s-{plugin}-query_template>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-result_size>> |<<number,number>>|No
 | <<plugins-{type}s-{plugin}-retry_on_failure>> |<<number,number>>|No
@@ -162,6 +268,7 @@ NOTE: As of version `4.0.0` of this plugin, a number of previously deprecated se
 | <<plugins-{type}s-{plugin}-ssl_truststore_type>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-ssl_verification_mode>> |<<string,string>>, one of `["full", "none"]`|No
 | <<plugins-{type}s-{plugin}-tag_on_failure>> |<<array,array>>|No
+| <<plugins-{type}s-{plugin}-target>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-user>> |<<string,string>>|No
 |=======================================================================
 
@@ -175,8 +282,11 @@ filter plugins.
 
   * Value type is <<hash,hash>>
   * Default value is `{}`
+  * Format: `"aggregation_name" => "[path][on][event]"`:
+  ** `aggregation_name`: aggregation name in result from {es}
+  ** `[path][on][event]`: path for where to place the value on the current event, using field-reference notation
 
-Hash of aggregation names to copy from elasticsearch response into Logstash event fields
+A mapping of aggregations to copy into the <<plugins-{type}s-{plugin}-target>> of the current event.
 
 Example:
 [source,ruby]
@@ -246,8 +356,11 @@ These custom headers will override any headers previously set by the plugin such
 
   * Value type is <<hash,hash>>
   * Default value is `{}`
+  * Format: `"path.in.source" => "[path][on][event]"`:
+  ** `path.in.source`: field path in document source of result from {es}, using dot-notation
+  ** `[path][on][event]`: path for where to place the value on the current event, using field-reference notation
 
-Hash of docinfo fields to copy from old event (found via elasticsearch) into new event
+A mapping of docinfo (`_source`) fields to copy into the <<plugins-{type}s-{plugin}-target>> of the current event.
 
 Example:
 [source,ruby]
@@ -273,9 +386,11 @@ Whether results should be sorted or not
 
   * Value type is <<array,array>>
   * Default value is `{}`
+  * Format: `"path.in.result" => "[path][on][event]"`:
+  ** `path.in.result`: field path in indexed result from {es}, using dot-notation
+  ** `[path][on][event]`: path for where to place the value on the current event, using field-reference notation
 
-An array of fields to copy from the old event (found via elasticsearch) into the
-new event, currently being processed.
+A mapping of indexed fields to copy into the <<plugins-{type}s-{plugin}-target>> of the current event.
 
 In the following example, the values of `@timestamp` and `event_id` on the event
 found via elasticsearch are copied to the current event's
@@ -330,11 +445,30 @@ environment variables e.g. `proxy => '${LS_PROXY:}'`.
   * Value type is <<string,string>>
   * There is no default value for this setting.
 
-Elasticsearch query string. More information is available in the
-{ref}/query-dsl-query-string-query.html#query-string-syntax[Elasticsearch query
-string documentation].
-Use either `query` or `query_template`.
+The query to be executed.
+The accepted query shape is DSL query string or ES|QL.
+For the DSL query string, use either `query` or `query_template`.
+Read the {ref}/query-dsl-query-string-query.html[{es} query
+string documentation] or {ref}/esql.html[{es} ES|QL documentation] for more information.
 
+[id="plugins-{type}s-{plugin}-query_type"]
+===== `query_type`
+
+* Value can be `dsl` or `esql`
+* Default value is `dsl`
+
+Defines the <<plugins-{type}s-{plugin}-query>> shape.
+When `dsl`, the query shape must be valid {es} JSON-style string.
+When `esql`, the query shape must be a valid {esql} string and `index`, `query_template` and `sort` parameters are not allowed.
+
+[id="plugins-{type}s-{plugin}-query_params"]
+===== `query_params`
+
+* The value type is <<hash,hash>> or <<array,array>>. When an array provided, the array elements are pairs of `key` and `value`.
+* There is no default value for this setting
+
+Named parameters in {esql} to send to {es} together with <<plugins-{type}s-{plugin}-query>>.
+Visit {ref}/esql-rest.html#esql-rest-params[passing parameters to query page] for more information.
 
 [id="plugins-{type}s-{plugin}-query_template"]
 ===== `query_template` 
@@ -523,6 +657,44 @@ WARNING: Setting certificate verification to `none` disables many security benef
 
 Tags the event on failure to look up previous log event information. This can be used in later analysis.
 
+[id="plugins-{type}s-{plugin}-target"]
+===== `target`
+
+* Value type is <<string,string>>
+* There is no default value for this setting.
+
+Define the target field for placing the result data.
+If this setting is omitted, the target will be the root (top level) of the event.
+It is highly recommended to set when using `query_type=>'esql'` to set all query results into the event.
+
+When `query_type=>'dsl'`, the destination fields specified in <<plugins-{type}s-{plugin}-fields>>, <<plugins-{type}s-{plugin}-aggregation_fields>>, and <<plugins-{type}s-{plugin}-docinfo_fields>> are relative to this target.
+
+For example, if you want the data to be put in the `operation` field:
+[source,ruby]
+if [type] == "end" {
+    filter {
+      query => "type:start AND transaction:%{[transactionId]}"
+      elasticsearch {
+        target => "transaction"
+        fields => {
+          "@timestamp" => "started"
+          "transaction_id" => "id"
+        }
+      }
+    }
+}
+
+`fields` fields will be expanded into a data structure in the `target` field, overall shape looks like this:
+[source,ruby]
+    {
+      "transaction" => {
+        "started" => "2025-04-29T12:01:46.263Z"
+        "id" => "1234567890"
+      }
+    }
+
+NOTE: when writing to a field that already exists on the event, the previous value will be overwritten.
+
 [id="plugins-{type}s-{plugin}-user"]
 ===== `user` 
 

data/lib/logstash/filters/elasticsearch/client.rb

@@ -58,11 +58,19 @@ module LogStash
       def search(params={})
         @client.search(params)
       end
+      
+      def esql_query(params={})
+        @client.esql.query(params)
+      end
 
       def info
         @client.info
       end
 
+      def es_version
+        info&.dig('version', 'number')
+      end
+
       def build_flavor
         @build_flavor ||= info&.dig('version', 'build_flavor')
       end

data/lib/logstash/filters/elasticsearch/dsl_executor.rb

@@ -0,0 +1,140 @@
+# encoding: utf-8
+
+module LogStash
+  module Filters
+    class Elasticsearch
+      class DslExecutor
+        def initialize(plugin, logger)
+          @index = plugin.params["index"]
+          @query = plugin.params["query"]
+          @query_dsl = plugin.query_dsl
+          @fields = plugin.params["fields"]
+          @result_size = plugin.params["result_size"]
+          @docinfo_fields = plugin.params["docinfo_fields"]
+          @tag_on_failure = plugin.params["tag_on_failure"]
+          @enable_sort = plugin.params["enable_sort"]
+          @sort = plugin.params["sort"]
+          @aggregation_fields = plugin.params["aggregation_fields"]
+          @logger = logger
+          @event_decorator = plugin.method(:decorate)
+          @target_field = plugin.params["target"]
+          if @target_field
+            def self.apply_target(path); "[#{@target_field}][#{path}]"; end
+          else
+            def self.apply_target(path); path; end
+          end
+        end
+
+        def process(client, event)
+          matched = false
+          begin
+            params = { :index => event.sprintf(@index) }
+
+            if @query_dsl
+              query = LogStash::Json.load(event.sprintf(@query_dsl))
+              params[:body] = query
+            else
+              query = event.sprintf(@query)
+              params[:q] = query
+              params[:size] = @result_size
+              params[:sort] = @sort if @enable_sort
+            end
+
+            @logger.debug("Querying elasticsearch for lookup", :params => params)
+
+            results = client.search(params)
+            raise "Elasticsearch query error: #{results["_shards"]["failures"]}" if results["_shards"].include? "failures"
+
+            event.set("[@metadata][total_hits]", extract_total_from_hits(results['hits']))
+
+            result_hits = results["hits"]["hits"]
+            if !result_hits.nil? && !result_hits.empty?
+              matched = true
+              @fields.each do |old_key, new_key|
+                old_key_path = extract_path(old_key)
+                extracted_hit_values = result_hits.map do |doc|
+                  extract_value(doc["_source"], old_key_path)
+                end
+                value_to_set = extracted_hit_values.count > 1 ? extracted_hit_values : extracted_hit_values.first
+                set_to_event_target(event, new_key, value_to_set)
+              end
+              @docinfo_fields.each do |old_key, new_key|
+                old_key_path = extract_path(old_key)
+                extracted_docs_info = result_hits.map do |doc|
+                  extract_value(doc, old_key_path)
+                end
+                value_to_set = extracted_docs_info.count > 1 ? extracted_docs_info : extracted_docs_info.first
+                set_to_event_target(event, new_key, value_to_set)
+              end
+            end
+
+            result_aggregations = results["aggregations"]
+            if !result_aggregations.nil? && !result_aggregations.empty?
+              matched = true
+              @aggregation_fields.each do |agg_name, ls_field|
+                set_to_event_target(event, ls_field, result_aggregations[agg_name])
+              end
+            end
+
+          rescue => e
+            if @logger.trace?
+              @logger.warn("Failed to query elasticsearch for previous event", :index => @index, :query => @query, :event => event.to_hash, :error => e.message, :backtrace => e.backtrace)
+            elsif @logger.debug?
+              @logger.warn("Failed to query elasticsearch for previous event", :index => @index, :error => e.message, :backtrace => e.backtrace)
+            else
+              @logger.warn("Failed to query elasticsearch for previous event", :index => @index, :error => e.message)
+            end
+            @tag_on_failure.each { |tag| event.tag(tag) }
+          else
+            @event_decorator.call(event) if matched
+          end
+        end
+
+        private
+
+        # Given a "hits" object from an Elasticsearch response, return the total number of hits in
+        # the result set.
+        # @param hits [Hash{String=>Object}]
+        # @return [Integer]
+        def extract_total_from_hits(hits)
+          total = hits['total']
+
+          # Elasticsearch 7.x produces an object containing `value` and `relation` in order
+          # to enable unambiguous reporting when the total is only a lower bound; if we get
+          # an object back, return its `value`.
+          return total['value'] if total.kind_of?(Hash)
+          total
+        end
+
+        # get an array of path elements from a path reference
+        def extract_path(path_reference)
+          return [path_reference] unless path_reference.start_with?('[') && path_reference.end_with?(']')
+
+          path_reference[1...-1].split('][')
+        end
+
+        # given a Hash and an array of path fragments, returns the value at the path
+        # @param source [Hash{String=>Object}]
+        # @param path [Array{String}]
+        # @return [Object]
+        def extract_value(source, path)
+          path.reduce(source) do |memo, old_key_fragment|
+            break unless memo.include?(old_key_fragment)
+            memo[old_key_fragment]
+          end
+        end
+
+        # if @target is defined, creates a nested structure to inject result into target field
+        # if not defined, directly sets to the top-level event field
+        # @param event [LogStash::Event]
+        # @param new_key [String] name of the field to set
+        # @param value_to_set [Array] values to set
+        # @return [void]
+        def set_to_event_target(event, new_key, value_to_set)
+          key_to_set = self.apply_target(new_key)
+          event.set(key_to_set, value_to_set)
+        end
+      end
+    end
+  end
+end

data/lib/logstash/filters/elasticsearch/esql_executor.rb

@@ -0,0 +1,178 @@
+# encoding: utf-8
+
+module LogStash
+  module Filters
+    class Elasticsearch
+      class EsqlExecutor
+
+        ESQL_PARSERS_BY_TYPE = Hash.new(lambda { |x| x }).merge(
+          'date' => ->(value) { value && LogStash::Timestamp.new(value) },
+          )
+
+        def initialize(plugin, logger)
+          @logger = logger
+
+          @event_decorator = plugin.method(:decorate)
+          @query = plugin.params["query"]
+
+          query_params = plugin.query_params || {}
+          reference_valued_params, static_valued_params = query_params.partition { |_, v| v.kind_of?(String) && v.match?(/^\[.*\]$/) }
+          @referenced_params = reference_valued_params&.to_h
+          # keep static params as an array of hashes to attach to the ES|QL api param easily
+          @static_params = static_valued_params.map { |k, v| { k => v } }
+          @tag_on_failure = plugin.params["tag_on_failure"]
+          @logger.debug("ES|QL query executor initialized with ", query: @query, query_params: query_params)
+
+          # if the target is specified, all result entries will be copied to the target field
+          # otherwise, the first value of the result will be copied to the event
+          @target_field = plugin.params["target"]
+          @logger.warn("Only first query result will be copied to the event. Please specify `target` in plugin config to include all") if @target_field.nil?
+        end
+
+        def process(client, event)
+          resolved_params = @referenced_params&.any? ? resolve_parameters(event) : []
+          resolved_params.concat(@static_params) if @static_params&.any?
+          response = execute_query(client, resolved_params)
+          inform_warning(response)
+          process_response(event, response)
+          @event_decorator.call(event)
+        rescue => e
+          @logger.error("Failed to process ES|QL filter", exception: e)
+          @tag_on_failure.each { |tag| event.tag(tag) }
+        end
+
+        private
+
+        def resolve_parameters(event)
+          @referenced_params.map do |key, value|
+            begin
+              resolved_value = event.get(value)
+              @logger.debug("Resolved value for #{key}: #{resolved_value}, its class: #{resolved_value.class}")
+              { key => resolved_value }
+            rescue => e
+              # catches invalid field reference
+              raise "Failed to resolve parameter `#{key}` with `#{value}`. Error: #{e.message}"
+            end
+          end
+        end
+
+        def execute_query(client, params)
+          # debug logs may help to check what query shape the plugin is sending to ES
+          @logger.debug("Executing ES|QL query", query: @query, params: params)
+          client.esql_query({ body: { query: @query, params: params }, format: 'json', drop_null_columns: true })
+        end
+
+        def process_response(event, response)
+          columns = response['columns']&.freeze || []
+          values = response['values']&.freeze || []
+          if values.nil? || values.size == 0
+            @logger.debug("Empty ES|QL query result", columns: columns, values: values)
+            return
+          end
+
+          # this shouldn't happen but just in case to avoid crashes the plugin
+          if columns.nil? || columns.size == 0
+            @logger.error("No columns exist but received values", columns: columns, values: values)
+            return
+          end
+
+          event.set("[@metadata][total_values]", values.size)
+          @logger.debug("ES|QL query result values size ", size: values.size)
+
+          column_specs = columns.map { |column| ColumnSpec.new(column) }
+          sub_element_mark_map = mark_sub_elements(column_specs)
+          multi_fields = sub_element_mark_map.filter_map { |key, val| key.name if val == true }
+
+          @logger.debug("Multi-fields found in ES|QL result and they will not be available in the event. Please use `RENAME` command if you want to include them.", { :detected_multi_fields => multi_fields }) if multi_fields.any?
+
+          if @target_field
+            values_to_set = values.map do |row|
+              mapped_data = column_specs.each_with_index.with_object({}) do |(column, index), mapped_data|
+                # `unless value.nil?` is a part of `drop_null_columns` that if some of the columns' values are not `nil`, `nil` values appear,
+                # we should continuously filter them out to achieve full `drop_null_columns` on each individual row (ideal `LIMIT 1` result)
+                # we also exclude sub-elements of the base field
+                if row[index] && sub_element_mark_map[column] == false
+                  value_to_set = ESQL_PARSERS_BY_TYPE[column.type].call(row[index])
+                  mapped_data[column.name] = value_to_set
+                end
+              end
+              generate_nested_structure(mapped_data) unless mapped_data.empty?
+            end
+            event.set("[#{@target_field}]", values_to_set)
+          else
+            column_specs.zip(values.first).each do |(column, value) |
+              if value && sub_element_mark_map[column] == false
+                value_to_set = ESQL_PARSERS_BY_TYPE[column.type].call(value)
+                event.set(column.field_reference, value_to_set)
+              end
+            end
+          end
+        end
+
+        def inform_warning(response)
+          return unless (warning = response&.headers&.dig('warning'))
+          @logger.warn("ES|QL executor received warning", { message: warning })
+        end
+
+        # Transforms dotted keys to nested JSON shape
+        # @param dot_keyed_hash [Hash] whose keys are dotted (example 'a.b.c.d': 'val')
+        # @return [Hash] whose keys are nested with value mapped ({'a':{'b':{'c':{'d':'val'}}}})
+        def generate_nested_structure(dot_keyed_hash)
+          dot_keyed_hash.each_with_object({}) do |(key, value), result|
+            key_parts = key.to_s.split('.')
+            *path, leaf = key_parts
+            leaf_scope = path.inject(result) { |scope, part| scope[part] ||= {} }
+            leaf_scope[leaf] = value
+          end
+        end
+
+        # Determines whether each column in a collection is a nested sub-element (e.g "user.age")
+        # of another column in the same collection (e.g "user").
+        #
+        # @param columns [Array<ColumnSpec>] An array of objects with a `name` attribute representing field paths.
+        # @return [Hash<ColumnSpec, Boolean>] A hash mapping each column to `true` if it is a sub-element of another field, `false` otherwise.
+        # Time complexity: (O(NlogN+N*K)) where K is the number of conflict depth
+        #   without (`prefix_set`) memoization, it would be O(N^2)
+        def mark_sub_elements(columns)
+          # Sort columns by name length (ascending)
+          sorted_columns = columns.sort_by { |c| c.name.length }
+          prefix_set = Set.new # memoization set
+
+          sorted_columns.each_with_object({}) do |column, memo|
+            # Split the column name into parts (e.g., "user.profile.age" → ["user", "profile", "age"])
+            parts = column.name.split('.')
+
+            # Generate all possible parent prefixes (e.g., "user", "user.profile")
+            # and check if any parent prefix exists in the set
+            parent_prefixes = (0...parts.size - 1).map { |i| parts[0..i].join('.') }
+            memo[column] = parent_prefixes.any? { |prefix| prefix_set.include?(prefix) }
+            prefix_set.add(column.name)
+          end
+        end
+      end
+
+      # Class representing a column specification in the ESQL response['columns']
+      # The class's main purpose is to provide a structure for the event key
+      # columns is an array with `name` and `type` pair (example: `{"name"=>"@timestamp", "type"=>"date"}`)
+      # @attr_reader :name [String] The name of the column
+      # @attr_reader :type [String] The type of the column
+      class ColumnSpec
+        attr_reader :name, :type
+
+        def initialize(spec)
+          @name = isolate(spec.fetch('name'))
+          @type = isolate(spec.fetch('type'))
+        end
+
+        def field_reference
+          @_field_reference ||= '[' + name.gsub('.', '][') + ']'
+        end
+
+        private
+        def isolate(value)
+          value.frozen? ? value : value.clone.freeze
+        end
+      end
+    end
+  end
+end