logstash-filter-elasticsearch 3.19.0 → 4.0.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 28c0544d8e61fe99078cabecc2ce4cf898aefc35775e80a71a5bb34221d534d5
-  data.tar.gz: 6f42fd12be09e9aa4b619c95ffff8cc9dba427f5f71f915d2e7c30a583f83fec
+  metadata.gz: 015a98dbd36122dd3fc4c74da5744a31f6182f67023f802189cc73e837d5ba7a
+  data.tar.gz: 8f1c3a79c0af3fc4154501d16bf775e01b6aa7575627852cb18954a0dd952d91
 SHA512:
-  metadata.gz: 3aaa9a9b8b31e1782e927fe4bccb26fd28063206dfeaf00c574657e0d91e3c7c7544565c657a65d37a5e164c0e2477b8fdab3f4dfb2c7feb79971962e94cbb87
-  data.tar.gz: 339a881e589fe44d5657e4a4417f9e72f38921a08ff99b2cbc1534785fcddb00ae484960311a70d653f3ca058f38ceed0a290d789bb0d728d7db1ff8de8a6d46
+  metadata.gz: fc3971541568e34c0ef644c250243fefcb48c778b2471f81f5e87a1b59535a6f5ccc2a2c13d57f5123e0577efb80962b85f8096d3d6f70e0721df4f716d705e1
+  data.tar.gz: 5fe74aa6d8179e6dd3d9c69a6d0c81f284035bd82343543132f129be7bfb3355e01678eddbd051d9d4c360e9704f8a234737096e757f710b395b5d366f57936f

data/CHANGELOG.md

@@ -1,14 +1,12 @@
-## 3.19.0
-  - ES|QL support [#199](https://github.com/logstash-plugins/logstash-filter-elasticsearch/pull/199)
-
-## 3.18.0
-  - Add `target` configuration option to store the result into it [#197](https://github.com/logstash-plugins/logstash-filter-elasticsearch/pull/197)
-
-## 3.17.1
-  - Add elastic-transport client support used in elasticsearch-ruby 8.x [#193](https://github.com/logstash-plugins/logstash-filter-elasticsearch/pull/193)
-
-## 3.17.0
-  - Added support for custom headers [#190](https://github.com/logstash-plugins/logstash-filter-elasticsearch/pull/190)
+## 4.0.0
+  - SSL settings that were marked deprecated in version `3.15.0` are now marked obsolete, and will prevent the plugin from starting.
+  - These settings are:
+    - `ca_file`, which should be replaced by `ssl_certificate_authorities`
+    - `keystore`, which should be replaced by `ssl_keystore_path`
+    - `keystore_password`, which should be replaced by `ssl_keystore_password`
+    - `keystore_type`, which should be replaced by `ssl_keystore_password`
+    -  `ssl`, which should be replaced by `ssl_enabled`
+    - [#183](https://github.com/logstash-plugins/logstash-filter-elasticsearch/pull/183)
 
 ## 3.16.2
   - Add `x-elastic-product-origin` header to Elasticsearch requests [#185](https://github.com/logstash-plugins/logstash-filter-elasticsearch/pull/185)

data/docs/index.asciidoc

@@ -54,7 +54,7 @@ if [type] == "end" {
 
 The example below reproduces the above example but utilises the query_template.
 This query_template represents a full Elasticsearch query DSL and supports the
-standard {ls} field substitution syntax. The example below issues
+standard Logstash field substitution syntax.  The example below issues
 the same query as the first example but uses the template shown.
 
 [source,ruby]
@@ -110,7 +110,7 @@ Authentication to a secure Elasticsearch cluster is possible using _one_ of the
 * <<plugins-{type}s-{plugin}-user>> AND <<plugins-{type}s-{plugin}-password>>
 * <<plugins-{type}s-{plugin}-cloud_auth>>
 * <<plugins-{type}s-{plugin}-api_key>>
-* <<plugins-{type}s-{plugin}-keystore>> and/or <<plugins-{type}s-{plugin}-keystore_password>>
+* <<plugins-{type}s-{plugin}-ssl_keystore_path>> and/or <<plugins-{type}s-{plugin}-ssl_keystore_password>>
 
 [id="plugins-{type}s-{plugin}-autz"]
 ==== Authorization
@@ -118,114 +118,13 @@ Authentication to a secure Elasticsearch cluster is possible using _one_ of the
 Authorization to a secure Elasticsearch cluster requires `read` permission at index level and `monitoring` permissions at cluster level.
 The `monitoring` permission at cluster level is necessary to perform periodic connectivity checks.
 
-[id="plugins-{type}s-{plugin}-esql"]
-==== {esql} support
-
-.Technical Preview
-****
-The {esql} feature that allows using ES|QL queries with this plugin is in Technical Preview.
-Configuration options and implementation details are subject to change in minor releases without being preceded by deprecation warnings.
-****
-
-{es} Query Language ({esql}) provides a SQL-like interface for querying your {es} data.
-
-To use {esql}, this plugin needs to be installed in {ls} 8.17.4 or newer, and must be connected to {es} 8.11 or newer.
-
-To configure {esql} query in the plugin, set your {esql} query in the `query` parameter.
-
-IMPORTANT: We recommend understanding {ref}/esql-limitations.html[{esql} current limitations] before using it in production environments.
-
-The following is a basic {esql} query that sets the food name to transaction event based on upstream event's food ID:
-[source, ruby]
-    filter {
-      elasticsearch {
-        hosts => [ 'https://..']
-        api_key => '....'
-        query => '
-          FROM food-index
-            | WHERE id == ?food_id
-        '
-        query_params => {
-          "food_id" => "[food][id]"
-        }
-      }
-    }
-
-Set `config.support_escapes: true` in `logstash.yml` if you need to escape special chars in the query.
-
-In the result event, the plugin sets total result size in `[@metadata][total_values]` field.
-
-[id="plugins-{type}s-{plugin}-esql-event-mapping"]
-===== Mapping {esql} result to {ls} event
-{esql} returns query results in a structured tabular format, where data is organized into _columns_ (fields) and _values_ (entries).
-The plugin maps each value entry to an event, populating corresponding fields.
-For example, a query might produce a table like:
-
-[cols="2,1,1,1,2",options="header"]
-|===
-|`timestamp` |`user_id` | `action` | `status.code` | `status.desc`
-
-|2025-04-10T12:00:00 |123 |login |200 | Success
-|2025-04-10T12:05:00 |456 |purchase |403 | Forbidden (unauthorized user)
-|===
-
-For this case, the plugin creates two JSON look like objects as below and places them into the `target` field of the event if `target` is defined.
-If `target` is not defined, the plugin places the _only_ first result at the root of the event.
-[source, json]
-[
-  {
-    "timestamp": "2025-04-10T12:00:00",
-    "user_id": 123,
-    "action": "login",
-    "status": {
-        "code": 200,
-        "desc": "Success"
-    }
-  },
-  {
-    "timestamp": "2025-04-10T12:05:00",
-    "user_id": 456,
-    "action": "purchase",
-    "status": {
-        "code": 403,
-        "desc": "Forbidden (unauthorized user)"
-    }
-  }
-]
-
-NOTE: If your index has a mapping with sub-objects where `status.code` and `status.desc` actually dotted fields, they appear in {ls} events as a nested structure.
-
-[id="plugins-{type}s-{plugin}-esql-multifields"]
-===== Conflict on multi-fields
-
-{esql} query fetches all parent and sub-fields fields if your {es} index has https://www.elastic.co/docs/reference/elasticsearch/mapping-reference/multi-fields[multi-fields] or https://www.elastic.co/docs/reference/elasticsearch/mapping-reference/subobjects[subobjects].
-Since {ls} events cannot contain parent field's concrete value and sub-field values together, the plugin ignores sub-fields with warning and includes parent.
-We recommend using the `RENAME` (or `DROP` to avoid warning) keyword in your {esql} query explicitly rename the fields to include sub-fields into the event.
-
-This is a common occurrence if your template or mapping follows the pattern of always indexing strings as "text" (`field`) + " keyword" (`field.keyword`) multi-field.
-In this case it's recommended to do `KEEP field` if the string is identical and there is only one subfield as the engine will optimize and retrieve the keyword, otherwise you can do `KEEP field.keyword | RENAME field.keyword as field`.
-
-To illustrate the situation with example, assuming your mapping has a time `time` field with `time.min` and `time.max` sub-fields as following:
-[source, ruby]
-    "properties": {
-        "time": { "type": "long" },
-        "time.min": { "type": "long" },
-        "time.max": { "type": "long" }
-    }
-
-The {esql} result will contain all three fields but the plugin cannot map them into {ls} event.
-To avoid this, you can use the `RENAME` keyword to rename the `time` parent field to get all three fields with unique fields.
-[source, ruby]
-    ...
-    query => 'FROM my-index | RENAME time AS time.current'
-    ...
-
-For comprehensive ES|QL syntax reference and best practices, see the https://www.elastic.co/guide/en/elasticsearch/reference/current/esql-syntax.html[{esql} documentation].
-
 [id="plugins-{type}s-{plugin}-options"]
 ==== Elasticsearch Filter Configuration Options
 
-This plugin supports the following configuration options plus the <<plugins-{type}s-{plugin}-common-options>> and the <<plugins-{type}s-{plugin}-deprecated-options>> described later.
+This plugin supports the following configuration options plus the <<plugins-{type}s-{plugin}-common-options>> described later.
+
+NOTE: As of version `4.0.0` of this plugin, a number of previously deprecated settings related to SSL have been removed. Please see the
+<<plugins-{type}s-{plugin}-obsolete-options>> for more details.
 
 [cols="<,<,<",options="header",]
 |=======================================================================
@@ -235,7 +134,6 @@ This plugin supports the following configuration options plus the <<plugins-{typ
 | <<plugins-{type}s-{plugin}-ca_trusted_fingerprint>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-cloud_auth>> |<<password,password>>|No
 | <<plugins-{type}s-{plugin}-cloud_id>> |<<string,string>>|No
-| <<plugins-{type}s-{plugin}-custom_headers>> |<<hash,hash>>|No
 | <<plugins-{type}s-{plugin}-docinfo_fields>> |<<hash,hash>>|No
 | <<plugins-{type}s-{plugin}-enable_sort>> |<<boolean,boolean>>|No
 | <<plugins-{type}s-{plugin}-fields>> |<<array,array>>|No
@@ -244,14 +142,11 @@ This plugin supports the following configuration options plus the <<plugins-{typ
 | <<plugins-{type}s-{plugin}-password>> |<<password,password>>|No
 | <<plugins-{type}s-{plugin}-proxy>> |<<uri,uri>>|No
 | <<plugins-{type}s-{plugin}-query>> |<<string,string>>|No
-| <<plugins-{type}s-{plugin}-query_type>> |<<string,string>>, one of `["dsl", "esql"]`|No
-| <<plugins-{type}s-{plugin}-query_params>> |<<hash,hash>> or <<hash,hash>>|No
 | <<plugins-{type}s-{plugin}-query_template>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-result_size>> |<<number,number>>|No
 | <<plugins-{type}s-{plugin}-retry_on_failure>> |<<number,number>>|No
 | <<plugins-{type}s-{plugin}-retry_on_status>> |<<array,array>>|No
 | <<plugins-{type}s-{plugin}-sort>> |<<string,string>>|No
-| <<plugins-{type}s-{plugin}-ssl>> |<<boolean,boolean>>|__Deprecated__
 | <<plugins-{type}s-{plugin}-ssl_certificate>> |<<path,path>>|No
 | <<plugins-{type}s-{plugin}-ssl_certificate_authorities>> |list of <<path,path>>|No
 | <<plugins-{type}s-{plugin}-ssl_cipher_suites>> |list of <<string,string>>|No
@@ -266,7 +161,6 @@ This plugin supports the following configuration options plus the <<plugins-{typ
 | <<plugins-{type}s-{plugin}-ssl_truststore_type>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-ssl_verification_mode>> |<<string,string>>, one of `["full", "none"]`|No
 | <<plugins-{type}s-{plugin}-tag_on_failure>> |<<array,array>>|No
-| <<plugins-{type}s-{plugin}-target>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-user>> |<<string,string>>|No
 |=======================================================================
 
@@ -280,11 +174,8 @@ filter plugins.
 
   * Value type is <<hash,hash>>
   * Default value is `{}`
-  * Format: `"aggregation_name" => "[path][on][event]"`:
-  ** `aggregation_name`: aggregation name in result from {es}
-  ** `[path][on][event]`: path for where to place the value on the current event, using field-reference notation
 
-A mapping of aggregations to copy into the <<plugins-{type}s-{plugin}-target>> of the current event.
+Hash of aggregation names to copy from elasticsearch response into Logstash event fields
 
 Example:
 [source,ruby]
@@ -339,26 +230,13 @@ Cloud ID, from the Elastic Cloud web console. If set `hosts` should not be used.
 For more info, check out the
 {logstash-ref}/connecting-to-cloud.html[Logstash-to-Cloud documentation].
 
-
-[id="plugins-{type}s-{plugin}-custom_headers"]
-===== `custom_headers`
-
-  * Value type is <<hash,hash>>
-  * Default value is empty
-
-Pass a set of key value pairs as the headers sent in each request to Elasticsearch.
-These custom headers will override any headers previously set by the plugin such as the User Agent or Authorization headers.
-
 [id="plugins-{type}s-{plugin}-docinfo_fields"]
 ===== `docinfo_fields` 
 
   * Value type is <<hash,hash>>
   * Default value is `{}`
-  * Format: `"path.in.source" => "[path][on][event]"`:
-  ** `path.in.source`: field path in document source of result from {es}, using dot-notation
-  ** `[path][on][event]`: path for where to place the value on the current event, using field-reference notation
 
-A mapping of docinfo (`_source`) fields to copy into the <<plugins-{type}s-{plugin}-target>> of the current event.
+Hash of docinfo fields to copy from old event (found via elasticsearch) into new event
 
 Example:
 [source,ruby]
@@ -384,11 +262,9 @@ Whether results should be sorted or not
 
   * Value type is <<array,array>>
   * Default value is `{}`
-  * Format: `"path.in.result" => "[path][on][event]"`:
-  ** `path.in.result`: field path in indexed result from {es}, using dot-notation
-  ** `[path][on][event]`: path for where to place the value on the current event, using field-reference notation
 
-A mapping of indexed fields to copy into the <<plugins-{type}s-{plugin}-target>> of the current event.
+An array of fields to copy from the old event (found via elasticsearch) into the
+new event, currently being processed.
 
 In the following example, the values of `@timestamp` and `event_id` on the event
 found via elasticsearch are copied to the current event's
@@ -443,30 +319,11 @@ environment variables e.g. `proxy => '${LS_PROXY:}'`.
   * Value type is <<string,string>>
   * There is no default value for this setting.
 
-The query to be executed.
-The accepted query shape is DSL query string or ES|QL.
-For the DSL query string, use either `query` or `query_template`.
-Read the {ref}/query-dsl-query-string-query.html[{es} query
-string documentation] or {ref}/esql.html[{es} ES|QL documentation] for more information.
-
-[id="plugins-{type}s-{plugin}-query_type"]
-===== `query_type`
-
-* Value can be `dsl` or `esql`
-* Default value is `dsl`
-
-Defines the <<plugins-{type}s-{plugin}-query>> shape.
-When `dsl`, the query shape must be valid {es} JSON-style string.
-When `esql`, the query shape must be a valid {esql} string and `index`, `query_template` and `sort` parameters are not allowed.
-
-[id="plugins-{type}s-{plugin}-query_params"]
-===== `query_params`
-
-* The value type is <<hash,hash>> or <<array,array>>. When an array provided, the array elements are pairs of `key` and `value`.
-* There is no default value for this setting
+Elasticsearch query string. More information is available in the
+{ref}/query-dsl-query-string-query.html#query-string-syntax[Elasticsearch query
+string documentation].
+Use either `query` or `query_template`.
 
-Named parameters in {esql} to send to {es} together with <<plugins-{type}s-{plugin}-query>>.
-Visit {ref}/esql-rest.html#esql-rest-params[passing parameters to query page] for more information.
 
 [id="plugins-{type}s-{plugin}-query_template"]
 ===== `query_template` 
@@ -655,44 +512,6 @@ WARNING: Setting certificate verification to `none` disables many security benef
 
 Tags the event on failure to look up previous log event information. This can be used in later analysis.
 
-[id="plugins-{type}s-{plugin}-target"]
-===== `target`
-
-* Value type is <<string,string>>
-* There is no default value for this setting.
-
-Define the target field for placing the result data.
-If this setting is omitted, the target will be the root (top level) of the event.
-It is highly recommended to set when using `query_type=>'esql'` to set all query results into the event.
-
-When `query_type=>'dsl'`, the destination fields specified in <<plugins-{type}s-{plugin}-fields>>, <<plugins-{type}s-{plugin}-aggregation_fields>>, and <<plugins-{type}s-{plugin}-docinfo_fields>> are relative to this target.
-
-For example, if you want the data to be put in the `operation` field:
-[source,ruby]
-if [type] == "end" {
-    filter {
-      query => "type:start AND transaction:%{[transactionId]}"
-      elasticsearch {
-        target => "transaction"
-        fields => {
-          "@timestamp" => "started"
-          "transaction_id" => "id"
-        }
-      }
-    }
-}
-
-`fields` fields will be expanded into a data structure in the `target` field, overall shape looks like this:
-[source,ruby]
-    {
-      "transaction" => {
-        "started" => "2025-04-29T12:01:46.263Z"
-        "id" => "1234567890"
-      }
-    }
-
-NOTE: when writing to a field that already exists on the event, the previous value will be overwritten.
-
 [id="plugins-{type}s-{plugin}-user"]
 ===== `user` 
 
@@ -702,57 +521,21 @@ NOTE: when writing to a field that already exists on the event, the previous val
 Basic Auth - username
 
 
-[id="plugins-{type}s-{plugin}-deprecated-options"]
-==== Elasticsearch Filter Deprecated Configuration Options
-
-This plugin supports the following deprecated configurations.
+[id="plugins-{type}s-{plugin}-obsolete-options"]
+==== Elasticsearch Filter Obsolete Configuration Options
 
-WARNING: Deprecated options are subject to removal in future releases.
+WARNING: As of version `4.0.0` of this plugin, some configuration options have been replaced.
+The plugin will fail to start if it contains any of these obsolete options.
 
 [cols="<,<,<",options="header",]
 |=======================================================================
-|Setting|Input type|Replaced by
-| <<plugins-{type}s-{plugin}-ca_file>> |a valid filesystem path|<<plugins-{type}s-{plugin}-ssl_certificate_authorities>>
-| <<plugins-{type}s-{plugin}-keystore>> |a valid filesystem path|<<plugins-{type}s-{plugin}-ssl_keystore_path>>
-| <<plugins-{type}s-{plugin}-keystore_password>> |<<password,password>>|<<plugins-{type}s-{plugin}-ssl_keystore_password>>
+|Setting|Replaced by
+| ca_file |<<plugins-{type}s-{plugin}-ssl_certificate_authorities>>
+| keystore |<<plugins-{type}s-{plugin}-ssl_keystore_path>>
+| keystore_password |<<plugins-{type}s-{plugin}-ssl_keystore_password>>
+| ssl |<<plugins-{type}s-{plugin}-ssl_enabled>>
 |=======================================================================
 
-[id="plugins-{type}s-{plugin}-ca_file"]
-===== `ca_file`
-deprecated[3.15.0, Replaced by <<plugins-{type}s-{plugin}-ssl_certificate_authorities>>]
-
-* Value type is <<path,path>>
-* There is no default value for this setting.
-
-SSL Certificate Authority file
-
-[id="plugins-{type}s-{plugin}-ssl"]
-===== `ssl`
-deprecated[3.15.0, Replaced by <<plugins-{type}s-{plugin}-ssl_enabled>>]
-
-* Value type is <<boolean,boolean>>
-* Default value is `false`
-
-SSL
-
-[id="plugins-{type}s-{plugin}-keystore"]
-===== `keystore`
-deprecated[3.15.0, Replaced by <<plugins-{type}s-{plugin}-ssl_keystore_path>>]
-
-* Value type is <<path,path>>
-* There is no default value for this setting.
-
-The keystore used to present a certificate to the server. It can be either .jks or .p12
-
-[id="plugins-{type}s-{plugin}-keystore_password"]
-===== `keystore_password`
-deprecated[3.15.0, Replaced by <<plugins-{type}s-{plugin}-ssl_keystore_password>>]
-
-* Value type is <<password,password>>
-* There is no default value for this setting.
-
-Set the keystore password
-
 
 [id="plugins-{type}s-{plugin}-common-options"]
 include::{include_path}/{type}.asciidoc[]

data/lib/logstash/filters/elasticsearch/client.rb

@@ -1,6 +1,7 @@
 # encoding: utf-8
 require "elasticsearch"
 require "base64"
+require "elasticsearch/transport/transport/http/manticore"
 
 
 module LogStash
@@ -8,7 +9,6 @@ module LogStash
     class ElasticsearchClient
 
       attr_reader :client
-      attr_reader :es_transport_client_type
 
       BUILD_FLAVOR_SERVERLESS = 'serverless'.freeze
       DEFAULT_EAV_HEADER = { "Elastic-Api-Version" => "2023-10-31" }.freeze
@@ -20,8 +20,6 @@ module LogStash
         api_key = options.fetch(:api_key, nil)
         proxy = options.fetch(:proxy, nil)
         user_agent = options[:user_agent]
-        custom_headers = options[:custom_headers]
-
 
         transport_options = { }
         transport_options[:headers] = options.fetch(:serverless, false) ?  DEFAULT_EAV_HEADER.dup : {}
@@ -29,7 +27,6 @@ module LogStash
         transport_options[:headers].merge!(setup_api_key(api_key))
         transport_options[:headers].merge!({ 'user-agent' => "#{user_agent}" })
         transport_options[:headers].merge!(INTERNAL_ORIGIN_HEADER)
-        transport_options[:headers].merge!(custom_headers) unless custom_headers.empty?
 
         transport_options[:pool_max] = 1000
         transport_options[:pool_max_per_route] = 100
@@ -44,7 +41,7 @@ module LogStash
 
         client_options = {
                        hosts: hosts,
-             transport_class: get_transport_client_class,
+             transport_class: ::Elasticsearch::Transport::Transport::HTTP::Manticore,
            transport_options: transport_options,
                          ssl: ssl_options,
             retry_on_failure: options[:retry_on_failure],
@@ -58,19 +55,11 @@ module LogStash
       def search(params={})
         @client.search(params)
       end
-      
-      def esql_query(params={})
-        @client.esql.query(params)
-      end
 
       def info
         @client.info
       end
 
-      def es_version
-        info&.dig('version', 'number')
-      end
-
       def build_flavor
         @build_flavor ||= info&.dig('version', 'build_flavor')
       end
@@ -106,20 +95,6 @@ module LogStash
         token = ::Base64.strict_encode64(api_key.value)
         { 'Authorization' => "ApiKey #{token}" }
       end
-
-      def get_transport_client_class
-        # LS-core includes `elasticsearch` gem. The gem is composed of two separate gems: `elasticsearch-api` and `elasticsearch-transport`
-        # And now `elasticsearch-transport` is old, instead we have `elastic-transport`.
-        # LS-core updated `elasticsearch` > 8: https://github.com/elastic/logstash/pull/17161
-        # Following source bits are for the compatibility to support both `elasticsearch-transport` and `elastic-transport` gems
-        require "elasticsearch/transport/transport/http/manticore"
-        es_transport_client_type = "elasticsearch_transport"
-        ::Elasticsearch::Transport::Transport::HTTP::Manticore
-      rescue ::LoadError
-        require "elastic/transport/transport/http/manticore"
-        es_transport_client_type = "elastic_transport"
-        ::Elastic::Transport::Transport::HTTP::Manticore
-      end
     end
   end
 end