logstash-input-jms 3.1.0-java → 3.2.1-java

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 0c67e7b1c2a2cdef01628e322b3d7e7f2b0d569354c4284b667b30ed9ca874d5
-  data.tar.gz: 10f7a99686215d53cd59145a898c0b931763eca6ff20d5a4cd82ababde5b65ba
+  metadata.gz: 5dca4ef438036be47078b2db808c88cfd09206a952c793a235cb46ef655067c8
+  data.tar.gz: 1f8e6bd65539c095a4d78488121a08eaee5d5bd24c6eb1f67e35b070550638ff
 SHA512:
-  metadata.gz: 9fb991ca333e90e0900d36ddac2c299e46805f911d8a3f357a3794bcbab74638d7e63496bb35bb18f4ea7940a266336ab88082193a8f554b87b552c3ea6df17a
-  data.tar.gz: 0e86a615ad62ddc0a9a59d228debdc62cbb2e31f0994146754fc015e9c6cafa828873b462fcd3eaf970902fa2760ec193f9fa92ee5279fdf3d5bb5f7f1d7844e
+  metadata.gz: 49c794f7452a575a8ce57e168f862e945bb2d57193daca9ff5ac4767b2f9eed39045d5fbed3a8b735fb3c050848d48a8fe2b57febf7a34c19baf9820664c71f3
+  data.tar.gz: 707f05788271a99e6e4eff3555285fa4120e3f6e2b1251ed63ddfbce1e38a2e1a821e59df549f3ac912bc4e221fa6776a320f286e6f8ef8b4f070aa4fed59669

data/CHANGELOG.md

@@ -1,3 +1,19 @@
+## 3.2.1
+ - Fix: improve compatibility with MessageConsumer implementations [#51](https://github.com/logstash-plugins/logstash-input-jms/pull/51),
+   such as IBM MQ.
+ - Test: Fix test failures due to ECS compatibility default changes in `8.x` of logstash [#53](https://github.com/logstash-plugins/logstash-input-jms/pull/53)
+
+## 3.2.0
+ - Feat: event_factory support + targets to aid ECS [#49](https://github.com/logstash-plugins/logstash-input-jms/pull/49)
+ - Fix: when configured to add JMS headers to the event, headers whose value is not set no longer result in nil entries on the event
+ - Fix: when adding the `jms_reply_to` header to an event, a string representation is set instead of an opaque object.
+
+## 3.1.2
+ - Docs: Added additional troubleshooting information [#38](https://github.com/logstash-plugins/logstash-input-jms/pull/38)
+
+## 3.1.1
+ - Added documentation for `factory_settings` configuration setting [#36](https://github.com/logstash-plugins/logstash-input-jms/pull/36)
+
 ## 3.1.0
  - Added many improvements to plugin [#35](https://github.com/logstash-plugins/logstash-input-jms/pull/35), including:  
    - Added support for TLS
@@ -39,7 +55,7 @@
  - New dependency requirements for logstash-core for the 5.0 release
 
 ## 2.0.0
- - Plugins were updated to follow the new shutdown semantic, this mainly allows Logstash to instruct input plugins to terminate gracefully, 
+ - Plugins were updated to follow the new shutdown semantic, this mainly allows Logstash to instruct input plugins to terminate gracefully,
    instead of using Thread.raise on the plugins' threads. Ref: https://github.com/elastic/logstash/pull/3895
  - Dependency on logstash-core update to 2.0
 

data/README.md

@@ -1,6 +1,6 @@
 # Logstash Plugin
 
-[![Travis Build Status](https://travis-ci.org/logstash-plugins/logstash-input-jms.svg)](https://travis-ci.org/logstash-plugins/logstash-input-jms)
+[![Travis Build Status](https://travis-ci.com/logstash-plugins/logstash-input-jms.svg)](https://travis-ci.com/logstash-plugins/logstash-input-jms)
 
 This is a plugin for [Logstash](https://github.com/elastic/logstash).
 

data/docs/index.asciidoc

@@ -33,9 +33,16 @@ JMS configurations can be done either entirely in the Logstash configuration fil
   configurations, should also use the combination of yaml file and Logstash configuration.
 
 
+[id="plugins-{type}s-{plugin}-ecs"]
+==== Compatibility with the Elastic Common Schema (ECS)
 
-==== Sample Configuration using Logstash Configuration Only
+JMS data is application specific. ECS compliance for fields depends on the use case.
+The plugin includes sensible defaults that change based on <<plugins-{type}s-{plugin}-ecs_compatibility,ECS compatibility mode>>.
+When ECS compatibility is disabled, headers, properties, and payload are targeted at the root level of the event to maintain compatibility with legacy usage of this plugin.
+When targeting an ECS version, headers and properties target `@metadata` sub-fields unless configured otherwise in order to avoid conflict with ECS fields.
+See <<plugins-{type}s-{plugin}-headers_target>>, <<plugins-{type}s-{plugin}-properties_target>> and <<plugins-{type}s-{plugin}-target>>.
 
+==== Sample Configuration using Logstash Configuration Only
 
 Configurations can be configured either entirely in Logstash configuration, or via a combination of Logstash configuration
 and yaml file, which can be useful for sharing similar configurations across multiple inputs and outputs.
@@ -61,7 +68,7 @@ The JMS plugin can also be configured using JNDI if desired.
         truststore => '/Users/logstash-user/security/truststore.jks'
         truststore_password => 'yet_another_secret'
         # Parts of the JMS message to be included <8>
-        include_header => false
+        include_headers => false
         include_properties => false
         include_body => true
         # Message selector
@@ -86,7 +93,6 @@ The JMS plugin can also be configured using JNDI if desired.
 <7> Keystore and Truststore to use when connecting to the JMS provider, if required.
 <8> Parts of the JMS Message to include in the event - headers, properties and the message body can be included or
     excluded from the event.
-
 <9> Message selector: Use this to filter messages to be processed. The whole selector query should be double-quoted,
     string property values should be single quoted, and numeric property vaues should not be quoted.
     See JMS provider documentation for exact syntax.
@@ -100,7 +106,7 @@ The JMS plugin can also be configured using JNDI if desired.
  input {
     jms {
         # Logstash Configuration Settings. <1>
-        include_header => false
+        include_headers => false
         include_properties => false
         include_body => true
         use_jms_timestamp => false
@@ -146,7 +152,7 @@ This section contains sample configurations for connecting to a JMS provider tha
  input {
     jms {
         # Logstash Configuration File Settings <1>
-        include_header => false
+        include_headers => false
         include_properties => false
         include_body => true
         use_jms_timestamp => false
@@ -207,7 +213,7 @@ This section contains sample configurations for connecting to a JMS provider tha
  input {
     jms {
         # Logstash specific configuration settings <1>
-        include_header => false
+        include_headers => false
         include_properties => false
         include_body => true
         use_jms_timestamp => false
@@ -337,6 +343,37 @@ selector definitions must be double quoted in the Logstash configuration file, a
 single quoted, and numeric property values not quoted at all.
 
 
+===== Failed to create Event with MissingConverterException
+
+Messages from certain JMS providers may contain headers or properties that Logstash cannot interpret, which can lead to
+error messages such as:
+
+
+[source,txt]
+-----
+[2019-11-25T08:04:28,769][ERROR][logstash.inputs.jms      ] Failed to create event {:message=>Java::ComSolacesystemsJmsMessage::SolTextMessage: ...
+Attributes: {:jms_correlation_id=>"xxxx", :jms_delivery_mode_sym=>:non_persistent, :jms_destination=>"destination", :jms_expiration=>0, :jms_message_id=>"xxxxxx", :jms_priority=>0, :jms_redelivered=>false, :jms_reply_to=>#<Java::ComSolacesystemsJmsImpl::SolTopicImpl:0xdeadbeef>, :jms_timestamp=>1574669008862, :jms_type=>nil}
+Properties: nil, :exception=>org.logstash.MissingConverterException: Missing Converter handling for full class name=com.solacesystems.jms.impl.SolTopicImpl, simple name=SolTopicImpl, :backtrace=>["org.logstash.Valuefier.fallbackConvert(Valuefier.java:98)..."]}
+:exception=>org.logstash.MissingConverterException: Missing Converter handling for full class name=com.solacesystems.jms.impl.SolTopicImpl
+-----
+
+To get around this, use the `skip_headers` or `skip_properties` configuration setting to avoid attempting to process
+the offending header or property in the message.
+
+In the example shown above, this attribute is causing the
+`MissingConverterException`:
+
+`jms_reply_to=>#<Java::ComSolacesystemsJmsImpl::SolTopicImpl:0xdeadbeef>`
+
+
+To avoid this error, the configuration should include the following line:
+
+[source,ruby]
+-----
+skip_headers => ["jms_reply_to"]
+-----
+
+
 [id="plugins-{type}s-{plugin}-options"]
 ==== Jms Input Configuration Options
 
@@ -350,9 +387,13 @@ This plugin supports the following configuration options plus the <<plugins-{typ
 | <<plugins-{type}s-{plugin}-durable_subscriber>> |<<boolean,boolean>>|No
 | <<plugins-{type}s-{plugin}-durable_subscriber_client_id>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-durable_subscriber_name>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-ecs_compatibility>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-factory>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-factory_settings>> |<<hash,hash>>|No
+| <<plugins-{type}s-{plugin}-headers_target>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-include_body>> |<<boolean,boolean>>|No
 | <<plugins-{type}s-{plugin}-include_header>> |<<boolean,boolean>>|No
+| <<plugins-{type}s-{plugin}-include_headers>> |<<boolean,boolean>>|No
 | <<plugins-{type}s-{plugin}-include_properties>> |<<boolean,boolean>>|No
 | <<plugins-{type}s-{plugin}-interval>> |<<number,number>>|No
 | <<plugins-{type}s-{plugin}-jndi_context>> |<<hash,hash>>|No
@@ -361,6 +402,7 @@ This plugin supports the following configuration options plus the <<plugins-{typ
 | <<plugins-{type}s-{plugin}-keystore_password>> |<<password,password>>|No
 | <<plugins-{type}s-{plugin}-oracle_aq_buffered_messages>> |<<boolean,boolean>>|No
 | <<plugins-{type}s-{plugin}-password>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-properties_target>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-pub_sub>> |<<boolean,boolean>>|No
 | <<plugins-{type}s-{plugin}-require_jars>> |<<array,array>>|No
 | <<plugins-{type}s-{plugin}-runner>> |<<string,string>>|__Deprecated__
@@ -368,6 +410,7 @@ This plugin supports the following configuration options plus the <<plugins-{typ
 | <<plugins-{type}s-{plugin}-skip_headers>> |<<array,array>>|No
 | <<plugins-{type}s-{plugin}-skip_properties>> |<<array,array>>|No
 | <<plugins-{type}s-{plugin}-system_properties>> |<<hash,hash>>|No
+| <<plugins-{type}s-{plugin}-target>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-threads>> |<<number,number>>|No
 | <<plugins-{type}s-{plugin}-timeout>> |<<number,number>>|No
 | <<plugins-{type}s-{plugin}-truststore>> |a valid filesystem path|No
@@ -400,6 +443,8 @@ Url to use when connecting to the JMS provider. This is only relevant for non-JN
 
 Name of the destination queue or topic to use.
 
+TIP: If the destination setting doesn't appear to be working properly, try this format: `<JMS module name>!<JMS queue name>`.
+
 [id="plugins-{type}s-{plugin}-durable_subscriber"]
 ===== `durable_subscriber`
 
@@ -436,6 +481,21 @@ This represents the value of the client ID for a durable subscribtion, and is on
 This represents the value of the subscriber name for a durable subscribtion, and is only used if `durable_subscriber`
 is set to `true`. Please consult your JMS Provider documentation for constraints and requirements for this setting.
 
+[id="plugins-{type}s-{plugin}-ecs_compatibility"]
+===== `ecs_compatibility`
+
+  * Value type is <<string,string>>
+  * Supported values are:
+    ** `disabled`: does not use ECS-compatible field names (fields might be set at the root of the event)
+    ** `v1`, `v8`: avoids field names that might conflict with Elastic Common Schema (for example, JMS specific properties)
+  * Default value depends on which version of Logstash is running:
+    ** When Logstash provides a `pipeline.ecs_compatibility` setting, its value is used as the default
+    ** Otherwise, the default value is `disabled`.
+
+Controls this plugin's compatibility with the {ecs-ref}[Elastic Common Schema (ECS)].
+The value of this setting affects the _default_ value of <<plugins-{type}s-{plugin}-headers_target>> and
+<<plugins-{type}s-{plugin}-properties_target>>.
+
 [id="plugins-{type}s-{plugin}-factory"]
 ===== `factory`
 
@@ -444,6 +504,28 @@ is set to `true`. Please consult your JMS Provider documentation for constraints
 
 Full name (including package name) of Java connection factory used to create a connection with your JMS provider.
 
+[id="plugins-{type}s-{plugin}-factory_settings"]
+===== `factory_settings`
+
+  * Value type is <<hash,hash>>
+  * There is no default value for this setting.
+
+Hash of implementation specific configuration values to set on the connection factory of the JMS provider. Each JMS
+ Provider will have its own set of parameters that can be used here. These parameters are mapped to `set` methods on
+ the provided connection factory, and can be supplied in either 'snake' or 'camel' case. For example, a hash including
+ `exclusive_consumer => true` would call `setExclusiveConsumer(true)` on the supplied connection factory.
+ See your JMS provider documentation for implementation specific details.
+
+[id="plugins-{type}s-{plugin}-headers_target"]
+===== `headers_target`
+
+  * Value type is <<string,string>>
+  * Default value depends on whether <<plugins-{type}s-{plugin}-ecs_compatibility>> is enabled:
+    ** ECS Compatibility disabled: no default value for this setting
+    ** ECS Compatibility enabled: `"[@metadata][input][jms][headers]"
+
+The name of the field under which JMS headers will be added, if <<plugins-{type}s-{plugin}-include_headers>> is set.
+
 [id="plugins-{type}s-{plugin}-include_body"]
 ===== `include_body`
 
@@ -451,29 +533,54 @@ Full name (including package name) of Java connection factory used to create a c
   * Default value is `true`
 
 Include JMS Message Body in the event.
-Supports TextMessage, MapMessage and ByteMessage.
+Supports TextMessage, MapMessage and BytesMessage.
 
-If the JMS Message is a TextMessage or ByteMessage, then the value will be in
+If the JMS Message is a TextMessage or BytesMessage, then the value will be in
 the "message" field of the event. If the JMS Message is a MapMessage, then all
-the key/value pairs will be added in the Hashmap of the event.
+the key/value pairs will be added at the top-level of the event by default.
+To avoid pollution of the top-level namespace, when receiving a MapMessage, use the <<plugins-{type}s-{plugin}-target>>.
 
 StreamMessage and ObjectMessage are not supported.
 
 [id="plugins-{type}s-{plugin}-include_header"]
 ===== `include_header`
 
+  * Value type is <<boolean,boolean>>
+  * This option is deprecated
+
+Note: This option is deprecated and it will be removed in the next major version of Logstash.
+Use `include_headers` instead.
+
+[id="plugins-{type}s-{plugin}-include_headers"]
+===== `include_headers`
+
   * Value type is <<boolean,boolean>>
   * Default value is `true`
 
 A JMS message has three parts:
 
-* Message Headers (required)
-* Message Properties (optional)
-* Message Bodies (optional)
+  * Message Headers (required)
+  * Message Properties (optional)
+  * Message Body (optional)
 
 You can tell the input plugin which parts should be included in the event produced by Logstash.
 
-Include JMS Message Header Field values in the event.
+Include standard JMS message header field values in the event.
+Example headers:
+[source,ruby]
+-----
+    {
+        "jms_message_id" => "ID:amqhost-39547-1636977297920-71:1:1:1:1",
+        "jms_timestamp" => 1636977329102,
+        "jms_expiration" => 0,
+        "jms_delivery_mode" => "persistent",
+        "jms_redelivered" => false,
+        "jms_destination" => "topic://41ad5342149901ad",
+        "jms_priority" => 4,
+        "jms_type" => "sample",
+        "jms_correlation_id" => "28d975cb-14ff-4285-841e-05ef1e0a7ab2"
+    }
+-----
 
 [id="plugins-{type}s-{plugin}-include_properties"]
 ===== `include_properties`
@@ -546,6 +653,16 @@ Only for use with Oracle AQ
 
 Password to use when connecting to the JMS provider.
 
+[id="plugins-{type}s-{plugin}-properties_target"]
+===== `properties_target`
+
+  * Value type is <<string,string>>
+  * Default value depends on whether <<plugins-{type}s-{plugin}-ecs_compatibility>> is enabled:
+    ** ECS Compatibility disabled: no default value for this setting
+    ** ECS Compatibility enabled: `"[@metadata][input][jms][properties]"
+
+The name of the field under which JMS properties will be added, if <<plugins-{type}s-{plugin}-include_properties>> is set.
+
 [id="plugins-{type}s-{plugin}-pub_sub"]
 ===== `pub_sub`
 
@@ -606,6 +723,17 @@ If `include_properties` is set, a list of properties to skip processing on can b
 
 Any System properties that the JMS provider requires can be set either in a Hash here, or in `jvm.options`
 
+[id="plugins-{type}s-{plugin}-target"]
+===== `target`
+
+  * Value type is <<string,string>>
+  * There is no default value for this setting.
+
+The name of the field to assign MapMessage data into.
+If not specified data will be stored in the root of the event.
+
+NOTE: For TextMessage and BytesMessage the `target` has no effect. Use the codec setting instead
+e.g. `codec => json { target => "[jms]" }`.
 
 [id="plugins-{type}s-{plugin}-threads"]
 ===== `threads`
@@ -613,7 +741,7 @@ Any System properties that the JMS provider requires can be set either in a Hash
   * Value type is <<number,number>>
   * Default value is `1`
 
-* Note that if pub_sub is set to true, this value *must* be 1. A configuration error will be thrown otherwise
+NOTE: If pub_sub is set to `true`, this value *must* be `1`. A configuration error will be thrown otherwise!
 
 [id="plugins-{type}s-{plugin}-timeout"]
 ===== `timeout`
@@ -680,4 +808,4 @@ For some known examples, see https://github.com/reidmorrison/jruby-jms/blob/mast
 [id="plugins-{type}s-{plugin}-common-options"]
 include::{include_path}/{type}.asciidoc[]
 
-:default_codec!:
+:default_codec!: