logstash-filter-elasticsearch 3.18.0 → 3.19.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: f93bd5f45ce46abe35c9fb54f03ce2f2241a6a7e5c530671a2954324f7965b18
-  data.tar.gz: 3bd3ae0babbeb6e46a1e6001888aa85c8fc47d7fb5c794a0063ad7688f3d3bd0
+  metadata.gz: 28c0544d8e61fe99078cabecc2ce4cf898aefc35775e80a71a5bb34221d534d5
+  data.tar.gz: 6f42fd12be09e9aa4b619c95ffff8cc9dba427f5f71f915d2e7c30a583f83fec
 SHA512:
-  metadata.gz: a59b18ddbb4706e50c94c7af206886b430063370376adee33b5795635d813142e662d55830b8401cd17b2e4ecfc69bace939f2ffd16daf2ecedfc4e1f16e5717
-  data.tar.gz: eb3c6a279badcc7b9fdaea8831cfeff276cde141773d28f9ead912606c816f30320e706efd34799060349bb09df09c8a4e5f05bf5f6ed2ccb9b0d5b19207eaff
+  metadata.gz: 3aaa9a9b8b31e1782e927fe4bccb26fd28063206dfeaf00c574657e0d91e3c7c7544565c657a65d37a5e164c0e2477b8fdab3f4dfb2c7feb79971962e94cbb87
+  data.tar.gz: 339a881e589fe44d5657e4a4417f9e72f38921a08ff99b2cbc1534785fcddb00ae484960311a70d653f3ca058f38ceed0a290d789bb0d728d7db1ff8de8a6d46

data/CHANGELOG.md

@@ -1,3 +1,6 @@
+## 3.19.0
+  - ES|QL support [#199](https://github.com/logstash-plugins/logstash-filter-elasticsearch/pull/199)
+
 ## 3.18.0
   - Add `target` configuration option to store the result into it [#197](https://github.com/logstash-plugins/logstash-filter-elasticsearch/pull/197)
 

data/docs/index.asciidoc

@@ -54,7 +54,7 @@ if [type] == "end" {
 
 The example below reproduces the above example but utilises the query_template.
 This query_template represents a full Elasticsearch query DSL and supports the
-standard Logstash field substitution syntax.  The example below issues
+standard {ls} field substitution syntax. The example below issues
 the same query as the first example but uses the template shown.
 
 [source,ruby]
@@ -118,6 +118,110 @@ Authentication to a secure Elasticsearch cluster is possible using _one_ of the
 Authorization to a secure Elasticsearch cluster requires `read` permission at index level and `monitoring` permissions at cluster level.
 The `monitoring` permission at cluster level is necessary to perform periodic connectivity checks.
 
+[id="plugins-{type}s-{plugin}-esql"]
+==== {esql} support
+
+.Technical Preview
+****
+The {esql} feature that allows using ES|QL queries with this plugin is in Technical Preview.
+Configuration options and implementation details are subject to change in minor releases without being preceded by deprecation warnings.
+****
+
+{es} Query Language ({esql}) provides a SQL-like interface for querying your {es} data.
+
+To use {esql}, this plugin needs to be installed in {ls} 8.17.4 or newer, and must be connected to {es} 8.11 or newer.
+
+To configure {esql} query in the plugin, set your {esql} query in the `query` parameter.
+
+IMPORTANT: We recommend understanding {ref}/esql-limitations.html[{esql} current limitations] before using it in production environments.
+
+The following is a basic {esql} query that sets the food name to transaction event based on upstream event's food ID:
+[source, ruby]
+    filter {
+      elasticsearch {
+        hosts => [ 'https://..']
+        api_key => '....'
+        query => '
+          FROM food-index
+            | WHERE id == ?food_id
+        '
+        query_params => {
+          "food_id" => "[food][id]"
+        }
+      }
+    }
+
+Set `config.support_escapes: true` in `logstash.yml` if you need to escape special chars in the query.
+
+In the result event, the plugin sets total result size in `[@metadata][total_values]` field.
+
+[id="plugins-{type}s-{plugin}-esql-event-mapping"]
+===== Mapping {esql} result to {ls} event
+{esql} returns query results in a structured tabular format, where data is organized into _columns_ (fields) and _values_ (entries).
+The plugin maps each value entry to an event, populating corresponding fields.
+For example, a query might produce a table like:
+
+[cols="2,1,1,1,2",options="header"]
+|===
+|`timestamp` |`user_id` | `action` | `status.code` | `status.desc`
+
+|2025-04-10T12:00:00 |123 |login |200 | Success
+|2025-04-10T12:05:00 |456 |purchase |403 | Forbidden (unauthorized user)
+|===
+
+For this case, the plugin creates two JSON look like objects as below and places them into the `target` field of the event if `target` is defined.
+If `target` is not defined, the plugin places the _only_ first result at the root of the event.
+[source, json]
+[
+  {
+    "timestamp": "2025-04-10T12:00:00",
+    "user_id": 123,
+    "action": "login",
+    "status": {
+        "code": 200,
+        "desc": "Success"
+    }
+  },
+  {
+    "timestamp": "2025-04-10T12:05:00",
+    "user_id": 456,
+    "action": "purchase",
+    "status": {
+        "code": 403,
+        "desc": "Forbidden (unauthorized user)"
+    }
+  }
+]
+
+NOTE: If your index has a mapping with sub-objects where `status.code` and `status.desc` actually dotted fields, they appear in {ls} events as a nested structure.
+
+[id="plugins-{type}s-{plugin}-esql-multifields"]
+===== Conflict on multi-fields
+
+{esql} query fetches all parent and sub-fields fields if your {es} index has https://www.elastic.co/docs/reference/elasticsearch/mapping-reference/multi-fields[multi-fields] or https://www.elastic.co/docs/reference/elasticsearch/mapping-reference/subobjects[subobjects].
+Since {ls} events cannot contain parent field's concrete value and sub-field values together, the plugin ignores sub-fields with warning and includes parent.
+We recommend using the `RENAME` (or `DROP` to avoid warning) keyword in your {esql} query explicitly rename the fields to include sub-fields into the event.
+
+This is a common occurrence if your template or mapping follows the pattern of always indexing strings as "text" (`field`) + " keyword" (`field.keyword`) multi-field.
+In this case it's recommended to do `KEEP field` if the string is identical and there is only one subfield as the engine will optimize and retrieve the keyword, otherwise you can do `KEEP field.keyword | RENAME field.keyword as field`.
+
+To illustrate the situation with example, assuming your mapping has a time `time` field with `time.min` and `time.max` sub-fields as following:
+[source, ruby]
+    "properties": {
+        "time": { "type": "long" },
+        "time.min": { "type": "long" },
+        "time.max": { "type": "long" }
+    }
+
+The {esql} result will contain all three fields but the plugin cannot map them into {ls} event.
+To avoid this, you can use the `RENAME` keyword to rename the `time` parent field to get all three fields with unique fields.
+[source, ruby]
+    ...
+    query => 'FROM my-index | RENAME time AS time.current'
+    ...
+
+For comprehensive ES|QL syntax reference and best practices, see the https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-syntax.html[{esql} documentation].
+
 [id="plugins-{type}s-{plugin}-options"]
 ==== Elasticsearch Filter Configuration Options
 
@@ -140,6 +244,8 @@ This plugin supports the following configuration options plus the <<plugins-{typ
 | <<plugins-{type}s-{plugin}-password>> |<<password,password>>|No
 | <<plugins-{type}s-{plugin}-proxy>> |<<uri,uri>>|No
 | <<plugins-{type}s-{plugin}-query>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-query_type>> |<<string,string>>, one of `["dsl", "esql"]`|No
+| <<plugins-{type}s-{plugin}-query_params>> |<<hash,hash>> or <<hash,hash>>|No
 | <<plugins-{type}s-{plugin}-query_template>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-result_size>> |<<number,number>>|No
 | <<plugins-{type}s-{plugin}-retry_on_failure>> |<<number,number>>|No
@@ -337,11 +443,30 @@ environment variables e.g. `proxy => '${LS_PROXY:}'`.
   * Value type is <<string,string>>
   * There is no default value for this setting.
 
-Elasticsearch query string. More information is available in the
-{ref}/query-dsl-query-string-query.html#query-string-syntax[Elasticsearch query
-string documentation].
-Use either `query` or `query_template`.
+The query to be executed.
+The accepted query shape is DSL query string or ES|QL.
+For the DSL query string, use either `query` or `query_template`.
+Read the {ref}/query-dsl-query-string-query.html[{es} query
+string documentation] or {ref}/esql.html[{es} ES|QL documentation] for more information.
+
+[id="plugins-{type}s-{plugin}-query_type"]
+===== `query_type`
+
+* Value can be `dsl` or `esql`
+* Default value is `dsl`
+
+Defines the <<plugins-{type}s-{plugin}-query>> shape.
+When `dsl`, the query shape must be valid {es} JSON-style string.
+When `esql`, the query shape must be a valid {esql} string and `index`, `query_template` and `sort` parameters are not allowed.
+
+[id="plugins-{type}s-{plugin}-query_params"]
+===== `query_params`
+
+* The value type is <<hash,hash>> or <<array,array>>. When an array provided, the array elements are pairs of `key` and `value`.
+* There is no default value for this setting
 
+Named parameters in {esql} to send to {es} together with <<plugins-{type}s-{plugin}-query>>.
+Visit {ref}/esql-rest.html#esql-rest-params[passing parameters to query page] for more information.
 
 [id="plugins-{type}s-{plugin}-query_template"]
 ===== `query_template` 
@@ -538,8 +663,9 @@ Tags the event on failure to look up previous log event information. This can be
 
 Define the target field for placing the result data.
 If this setting is omitted, the target will be the root (top level) of the event.
+It is highly recommended to set when using `query_type=>'esql'` to set all query results into the event.
 
-The destination fields specified in <<plugins-{type}s-{plugin}-fields>>, <<plugins-{type}s-{plugin}-aggregation_fields>>, and <<plugins-{type}s-{plugin}-docinfo_fields>> are relative to this target.
+When `query_type=>'dsl'`, the destination fields specified in <<plugins-{type}s-{plugin}-fields>>, <<plugins-{type}s-{plugin}-aggregation_fields>>, and <<plugins-{type}s-{plugin}-docinfo_fields>> are relative to this target.
 
 For example, if you want the data to be put in the `operation` field:
 [source,ruby]

data/lib/logstash/filters/elasticsearch/client.rb

@@ -58,11 +58,19 @@ module LogStash
       def search(params={})
         @client.search(params)
       end
+      
+      def esql_query(params={})
+        @client.esql.query(params)
+      end
 
       def info
         @client.info
       end
 
+      def es_version
+        info&.dig('version', 'number')
+      end
+
       def build_flavor
         @build_flavor ||= info&.dig('version', 'build_flavor')
       end

data/lib/logstash/filters/elasticsearch/dsl_executor.rb

@@ -0,0 +1,140 @@
+# encoding: utf-8
+
+module LogStash
+  module Filters
+    class Elasticsearch
+      class DslExecutor
+        def initialize(plugin, logger)
+          @index = plugin.params["index"]
+          @query = plugin.params["query"]
+          @query_dsl = plugin.query_dsl
+          @fields = plugin.params["fields"]
+          @result_size = plugin.params["result_size"]
+          @docinfo_fields = plugin.params["docinfo_fields"]
+          @tag_on_failure = plugin.params["tag_on_failure"]
+          @enable_sort = plugin.params["enable_sort"]
+          @sort = plugin.params["sort"]
+          @aggregation_fields = plugin.params["aggregation_fields"]
+          @logger = logger
+          @event_decorator = plugin.method(:decorate)
+          @target_field = plugin.params["target"]
+          if @target_field
+            def self.apply_target(path); "[#{@target_field}][#{path}]"; end
+          else
+            def self.apply_target(path); path; end
+          end
+        end
+
+        def process(client, event)
+          matched = false
+          begin
+            params = { :index => event.sprintf(@index) }
+
+            if @query_dsl
+              query = LogStash::Json.load(event.sprintf(@query_dsl))
+              params[:body] = query
+            else
+              query = event.sprintf(@query)
+              params[:q] = query
+              params[:size] = @result_size
+              params[:sort] = @sort if @enable_sort
+            end
+
+            @logger.debug("Querying elasticsearch for lookup", :params => params)
+
+            results = client.search(params)
+            raise "Elasticsearch query error: #{results["_shards"]["failures"]}" if results["_shards"].include? "failures"
+
+            event.set("[@metadata][total_hits]", extract_total_from_hits(results['hits']))
+
+            result_hits = results["hits"]["hits"]
+            if !result_hits.nil? && !result_hits.empty?
+              matched = true
+              @fields.each do |old_key, new_key|
+                old_key_path = extract_path(old_key)
+                extracted_hit_values = result_hits.map do |doc|
+                  extract_value(doc["_source"], old_key_path)
+                end
+                value_to_set = extracted_hit_values.count > 1 ? extracted_hit_values : extracted_hit_values.first
+                set_to_event_target(event, new_key, value_to_set)
+              end
+              @docinfo_fields.each do |old_key, new_key|
+                old_key_path = extract_path(old_key)
+                extracted_docs_info = result_hits.map do |doc|
+                  extract_value(doc, old_key_path)
+                end
+                value_to_set = extracted_docs_info.count > 1 ? extracted_docs_info : extracted_docs_info.first
+                set_to_event_target(event, new_key, value_to_set)
+              end
+            end
+
+            result_aggregations = results["aggregations"]
+            if !result_aggregations.nil? && !result_aggregations.empty?
+              matched = true
+              @aggregation_fields.each do |agg_name, ls_field|
+                set_to_event_target(event, ls_field, result_aggregations[agg_name])
+              end
+            end
+
+          rescue => e
+            if @logger.trace?
+              @logger.warn("Failed to query elasticsearch for previous event", :index => @index, :query => @query, :event => event.to_hash, :error => e.message, :backtrace => e.backtrace)
+            elsif @logger.debug?
+              @logger.warn("Failed to query elasticsearch for previous event", :index => @index, :error => e.message, :backtrace => e.backtrace)
+            else
+              @logger.warn("Failed to query elasticsearch for previous event", :index => @index, :error => e.message)
+            end
+            @tag_on_failure.each { |tag| event.tag(tag) }
+          else
+            @event_decorator.call(event) if matched
+          end
+        end
+
+        private
+
+        # Given a "hits" object from an Elasticsearch response, return the total number of hits in
+        # the result set.
+        # @param hits [Hash{String=>Object}]
+        # @return [Integer]
+        def extract_total_from_hits(hits)
+          total = hits['total']
+
+          # Elasticsearch 7.x produces an object containing `value` and `relation` in order
+          # to enable unambiguous reporting when the total is only a lower bound; if we get
+          # an object back, return its `value`.
+          return total['value'] if total.kind_of?(Hash)
+          total
+        end
+
+        # get an array of path elements from a path reference
+        def extract_path(path_reference)
+          return [path_reference] unless path_reference.start_with?('[') && path_reference.end_with?(']')
+
+          path_reference[1...-1].split('][')
+        end
+
+        # given a Hash and an array of path fragments, returns the value at the path
+        # @param source [Hash{String=>Object}]
+        # @param path [Array{String}]
+        # @return [Object]
+        def extract_value(source, path)
+          path.reduce(source) do |memo, old_key_fragment|
+            break unless memo.include?(old_key_fragment)
+            memo[old_key_fragment]
+          end
+        end
+
+        # if @target is defined, creates a nested structure to inject result into target field
+        # if not defined, directly sets to the top-level event field
+        # @param event [LogStash::Event]
+        # @param new_key [String] name of the field to set
+        # @param value_to_set [Array] values to set
+        # @return [void]
+        def set_to_event_target(event, new_key, value_to_set)
+          key_to_set = self.apply_target(new_key)
+          event.set(key_to_set, value_to_set)
+        end
+      end
+    end
+  end
+end

data/lib/logstash/filters/elasticsearch/esql_executor.rb

@@ -0,0 +1,178 @@
+# encoding: utf-8
+
+module LogStash
+  module Filters
+    class Elasticsearch
+      class EsqlExecutor
+
+        ESQL_PARSERS_BY_TYPE = Hash.new(lambda { |x| x }).merge(
+          'date' => ->(value) { value && LogStash::Timestamp.new(value) },
+          )
+
+        def initialize(plugin, logger)
+          @logger = logger
+
+          @event_decorator = plugin.method(:decorate)
+          @query = plugin.params["query"]
+
+          query_params = plugin.query_params || {}
+          reference_valued_params, static_valued_params = query_params.partition { |_, v| v.kind_of?(String) && v.match?(/^\[.*\]$/) }
+          @referenced_params = reference_valued_params&.to_h
+          # keep static params as an array of hashes to attach to the ES|QL api param easily
+          @static_params = static_valued_params.map { |k, v| { k => v } }
+          @tag_on_failure = plugin.params["tag_on_failure"]
+          @logger.debug("ES|QL query executor initialized with ", query: @query, query_params: query_params)
+
+          # if the target is specified, all result entries will be copied to the target field
+          # otherwise, the first value of the result will be copied to the event
+          @target_field = plugin.params["target"]
+          @logger.warn("Only first query result will be copied to the event. Please specify `target` in plugin config to include all") if @target_field.nil?
+        end
+
+        def process(client, event)
+          resolved_params = @referenced_params&.any? ? resolve_parameters(event) : []
+          resolved_params.concat(@static_params) if @static_params&.any?
+          response = execute_query(client, resolved_params)
+          inform_warning(response)
+          process_response(event, response)
+          @event_decorator.call(event)
+        rescue => e
+          @logger.error("Failed to process ES|QL filter", exception: e)
+          @tag_on_failure.each { |tag| event.tag(tag) }
+        end
+
+        private
+
+        def resolve_parameters(event)
+          @referenced_params.map do |key, value|
+            begin
+              resolved_value = event.get(value)
+              @logger.debug("Resolved value for #{key}: #{resolved_value}, its class: #{resolved_value.class}")
+              { key => resolved_value }
+            rescue => e
+              # catches invalid field reference
+              raise "Failed to resolve parameter `#{key}` with `#{value}`. Error: #{e.message}"
+            end
+          end
+        end
+
+        def execute_query(client, params)
+          # debug logs may help to check what query shape the plugin is sending to ES
+          @logger.debug("Executing ES|QL query", query: @query, params: params)
+          client.esql_query({ body: { query: @query, params: params }, format: 'json', drop_null_columns: true })
+        end
+
+        def process_response(event, response)
+          columns = response['columns']&.freeze || []
+          values = response['values']&.freeze || []
+          if values.nil? || values.size == 0
+            @logger.debug("Empty ES|QL query result", columns: columns, values: values)
+            return
+          end
+
+          # this shouldn't happen but just in case to avoid crashes the plugin
+          if columns.nil? || columns.size == 0
+            @logger.error("No columns exist but received values", columns: columns, values: values)
+            return
+          end
+
+          event.set("[@metadata][total_values]", values.size)
+          @logger.debug("ES|QL query result values size ", size: values.size)
+
+          column_specs = columns.map { |column| ColumnSpec.new(column) }
+          sub_element_mark_map = mark_sub_elements(column_specs)
+          multi_fields = sub_element_mark_map.filter_map { |key, val| key.name if val == true }
+
+          @logger.debug("Multi-fields found in ES|QL result and they will not be available in the event. Please use `RENAME` command if you want to include them.", { :detected_multi_fields => multi_fields }) if multi_fields.any?
+
+          if @target_field
+            values_to_set = values.map do |row|
+              mapped_data = column_specs.each_with_index.with_object({}) do |(column, index), mapped_data|
+                # `unless value.nil?` is a part of `drop_null_columns` that if some of the columns' values are not `nil`, `nil` values appear,
+                # we should continuously filter them out to achieve full `drop_null_columns` on each individual row (ideal `LIMIT 1` result)
+                # we also exclude sub-elements of the base field
+                if row[index] && sub_element_mark_map[column] == false
+                  value_to_set = ESQL_PARSERS_BY_TYPE[column.type].call(row[index])
+                  mapped_data[column.name] = value_to_set
+                end
+              end
+              generate_nested_structure(mapped_data) unless mapped_data.empty?
+            end
+            event.set("[#{@target_field}]", values_to_set)
+          else
+            column_specs.zip(values.first).each do |(column, value) |
+              if value && sub_element_mark_map[column] == false
+                value_to_set = ESQL_PARSERS_BY_TYPE[column.type].call(value)
+                event.set(column.field_reference, value_to_set)
+              end
+            end
+          end
+        end
+
+        def inform_warning(response)
+          return unless (warning = response&.headers&.dig('warning'))
+          @logger.warn("ES|QL executor received warning", { message: warning })
+        end
+
+        # Transforms dotted keys to nested JSON shape
+        # @param dot_keyed_hash [Hash] whose keys are dotted (example 'a.b.c.d': 'val')
+        # @return [Hash] whose keys are nested with value mapped ({'a':{'b':{'c':{'d':'val'}}}})
+        def generate_nested_structure(dot_keyed_hash)
+          dot_keyed_hash.each_with_object({}) do |(key, value), result|
+            key_parts = key.to_s.split('.')
+            *path, leaf = key_parts
+            leaf_scope = path.inject(result) { |scope, part| scope[part] ||= {} }
+            leaf_scope[leaf] = value
+          end
+        end
+
+        # Determines whether each column in a collection is a nested sub-element (e.g "user.age")
+        # of another column in the same collection (e.g "user").
+        #
+        # @param columns [Array<ColumnSpec>] An array of objects with a `name` attribute representing field paths.
+        # @return [Hash<ColumnSpec, Boolean>] A hash mapping each column to `true` if it is a sub-element of another field, `false` otherwise.
+        # Time complexity: (O(NlogN+N*K)) where K is the number of conflict depth
+        #   without (`prefix_set`) memoization, it would be O(N^2)
+        def mark_sub_elements(columns)
+          # Sort columns by name length (ascending)
+          sorted_columns = columns.sort_by { |c| c.name.length }
+          prefix_set = Set.new # memoization set
+
+          sorted_columns.each_with_object({}) do |column, memo|
+            # Split the column name into parts (e.g., "user.profile.age" → ["user", "profile", "age"])
+            parts = column.name.split('.')
+
+            # Generate all possible parent prefixes (e.g., "user", "user.profile")
+            # and check if any parent prefix exists in the set
+            parent_prefixes = (0...parts.size - 1).map { |i| parts[0..i].join('.') }
+            memo[column] = parent_prefixes.any? { |prefix| prefix_set.include?(prefix) }
+            prefix_set.add(column.name)
+          end
+        end
+      end
+
+      # Class representing a column specification in the ESQL response['columns']
+      # The class's main purpose is to provide a structure for the event key
+      # columns is an array with `name` and `type` pair (example: `{"name"=>"@timestamp", "type"=>"date"}`)
+      # @attr_reader :name [String] The name of the column
+      # @attr_reader :type [String] The type of the column
+      class ColumnSpec
+        attr_reader :name, :type
+
+        def initialize(spec)
+          @name = isolate(spec.fetch('name'))
+          @type = isolate(spec.fetch('type'))
+        end
+
+        def field_reference
+          @_field_reference ||= '[' + name.gsub('.', '][') + ']'
+        end
+
+        private
+        def isolate(value)
+          value.frozen? ? value : value.clone.freeze
+        end
+      end
+    end
+  end
+end