logstash-input-jms 3.1.0-java → 3.2.1-java
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/CHANGELOG.md +17 -1
- data/README.md +1 -1
- data/docs/index.asciidoc +143 -15
- data/lib/logstash/inputs/jms.rb +219 -28
- data/logstash-input-jms.gemspec +8 -5
- data/spec/inputs/integration/jms_spec.rb +153 -63
- data/spec/inputs/spec_helper.rb +37 -1
- data/spec/inputs/unit/jms_spec.rb +66 -10
- metadata +50 -9
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 5dca4ef438036be47078b2db808c88cfd09206a952c793a235cb46ef655067c8
|
4
|
+
data.tar.gz: 1f8e6bd65539c095a4d78488121a08eaee5d5bd24c6eb1f67e35b070550638ff
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 49c794f7452a575a8ce57e168f862e945bb2d57193daca9ff5ac4767b2f9eed39045d5fbed3a8b735fb3c050848d48a8fe2b57febf7a34c19baf9820664c71f3
|
7
|
+
data.tar.gz: 707f05788271a99e6e4eff3555285fa4120e3f6e2b1251ed63ddfbce1e38a2e1a821e59df549f3ac912bc4e221fa6776a320f286e6f8ef8b4f070aa4fed59669
|
data/CHANGELOG.md
CHANGED
@@ -1,3 +1,19 @@
|
|
1
|
+
## 3.2.1
|
2
|
+
- Fix: improve compatibility with MessageConsumer implementations [#51](https://github.com/logstash-plugins/logstash-input-jms/pull/51),
|
3
|
+
such as IBM MQ.
|
4
|
+
- Test: Fix test failures due to ECS compatibility default changes in `8.x` of logstash [#53](https://github.com/logstash-plugins/logstash-input-jms/pull/53)
|
5
|
+
|
6
|
+
## 3.2.0
|
7
|
+
- Feat: event_factory support + targets to aid ECS [#49](https://github.com/logstash-plugins/logstash-input-jms/pull/49)
|
8
|
+
- Fix: when configured to add JMS headers to the event, headers whose value is not set no longer result in nil entries on the event
|
9
|
+
- Fix: when adding the `jms_reply_to` header to an event, a string representation is set instead of an opaque object.
|
10
|
+
|
11
|
+
## 3.1.2
|
12
|
+
- Docs: Added additional troubleshooting information [#38](https://github.com/logstash-plugins/logstash-input-jms/pull/38)
|
13
|
+
|
14
|
+
## 3.1.1
|
15
|
+
- Added documentation for `factory_settings` configuration setting [#36](https://github.com/logstash-plugins/logstash-input-jms/pull/36)
|
16
|
+
|
1
17
|
## 3.1.0
|
2
18
|
- Added many improvements to plugin [#35](https://github.com/logstash-plugins/logstash-input-jms/pull/35), including:
|
3
19
|
- Added support for TLS
|
@@ -39,7 +55,7 @@
|
|
39
55
|
- New dependency requirements for logstash-core for the 5.0 release
|
40
56
|
|
41
57
|
## 2.0.0
|
42
|
-
- Plugins were updated to follow the new shutdown semantic, this mainly allows Logstash to instruct input plugins to terminate gracefully,
|
58
|
+
- Plugins were updated to follow the new shutdown semantic, this mainly allows Logstash to instruct input plugins to terminate gracefully,
|
43
59
|
instead of using Thread.raise on the plugins' threads. Ref: https://github.com/elastic/logstash/pull/3895
|
44
60
|
- Dependency on logstash-core update to 2.0
|
45
61
|
|
data/README.md
CHANGED
@@ -1,6 +1,6 @@
|
|
1
1
|
# Logstash Plugin
|
2
2
|
|
3
|
-
[![Travis Build Status](https://travis-ci.
|
3
|
+
[![Travis Build Status](https://travis-ci.com/logstash-plugins/logstash-input-jms.svg)](https://travis-ci.com/logstash-plugins/logstash-input-jms)
|
4
4
|
|
5
5
|
This is a plugin for [Logstash](https://github.com/elastic/logstash).
|
6
6
|
|
data/docs/index.asciidoc
CHANGED
@@ -33,9 +33,16 @@ JMS configurations can be done either entirely in the Logstash configuration fil
|
|
33
33
|
configurations, should also use the combination of yaml file and Logstash configuration.
|
34
34
|
|
35
35
|
|
36
|
+
[id="plugins-{type}s-{plugin}-ecs"]
|
37
|
+
==== Compatibility with the Elastic Common Schema (ECS)
|
36
38
|
|
37
|
-
|
39
|
+
JMS data is application specific. ECS compliance for fields depends on the use case.
|
40
|
+
The plugin includes sensible defaults that change based on <<plugins-{type}s-{plugin}-ecs_compatibility,ECS compatibility mode>>.
|
41
|
+
When ECS compatibility is disabled, headers, properties, and payload are targeted at the root level of the event to maintain compatibility with legacy usage of this plugin.
|
42
|
+
When targeting an ECS version, headers and properties target `@metadata` sub-fields unless configured otherwise in order to avoid conflict with ECS fields.
|
43
|
+
See <<plugins-{type}s-{plugin}-headers_target>>, <<plugins-{type}s-{plugin}-properties_target>> and <<plugins-{type}s-{plugin}-target>>.
|
38
44
|
|
45
|
+
==== Sample Configuration using Logstash Configuration Only
|
39
46
|
|
40
47
|
Configurations can be configured either entirely in Logstash configuration, or via a combination of Logstash configuration
|
41
48
|
and yaml file, which can be useful for sharing similar configurations across multiple inputs and outputs.
|
@@ -61,7 +68,7 @@ The JMS plugin can also be configured using JNDI if desired.
|
|
61
68
|
truststore => '/Users/logstash-user/security/truststore.jks'
|
62
69
|
truststore_password => 'yet_another_secret'
|
63
70
|
# Parts of the JMS message to be included <8>
|
64
|
-
|
71
|
+
include_headers => false
|
65
72
|
include_properties => false
|
66
73
|
include_body => true
|
67
74
|
# Message selector
|
@@ -86,7 +93,6 @@ The JMS plugin can also be configured using JNDI if desired.
|
|
86
93
|
<7> Keystore and Truststore to use when connecting to the JMS provider, if required.
|
87
94
|
<8> Parts of the JMS Message to include in the event - headers, properties and the message body can be included or
|
88
95
|
excluded from the event.
|
89
|
-
|
90
96
|
<9> Message selector: Use this to filter messages to be processed. The whole selector query should be double-quoted,
|
91
97
|
string property values should be single quoted, and numeric property vaues should not be quoted.
|
92
98
|
See JMS provider documentation for exact syntax.
|
@@ -100,7 +106,7 @@ The JMS plugin can also be configured using JNDI if desired.
|
|
100
106
|
input {
|
101
107
|
jms {
|
102
108
|
# Logstash Configuration Settings. <1>
|
103
|
-
|
109
|
+
include_headers => false
|
104
110
|
include_properties => false
|
105
111
|
include_body => true
|
106
112
|
use_jms_timestamp => false
|
@@ -146,7 +152,7 @@ This section contains sample configurations for connecting to a JMS provider tha
|
|
146
152
|
input {
|
147
153
|
jms {
|
148
154
|
# Logstash Configuration File Settings <1>
|
149
|
-
|
155
|
+
include_headers => false
|
150
156
|
include_properties => false
|
151
157
|
include_body => true
|
152
158
|
use_jms_timestamp => false
|
@@ -207,7 +213,7 @@ This section contains sample configurations for connecting to a JMS provider tha
|
|
207
213
|
input {
|
208
214
|
jms {
|
209
215
|
# Logstash specific configuration settings <1>
|
210
|
-
|
216
|
+
include_headers => false
|
211
217
|
include_properties => false
|
212
218
|
include_body => true
|
213
219
|
use_jms_timestamp => false
|
@@ -337,6 +343,37 @@ selector definitions must be double quoted in the Logstash configuration file, a
|
|
337
343
|
single quoted, and numeric property values not quoted at all.
|
338
344
|
|
339
345
|
|
346
|
+
===== Failed to create Event with MissingConverterException
|
347
|
+
|
348
|
+
Messages from certain JMS providers may contain headers or properties that Logstash cannot interpret, which can lead to
|
349
|
+
error messages such as:
|
350
|
+
|
351
|
+
|
352
|
+
[source,txt]
|
353
|
+
-----
|
354
|
+
[2019-11-25T08:04:28,769][ERROR][logstash.inputs.jms ] Failed to create event {:message=>Java::ComSolacesystemsJmsMessage::SolTextMessage: ...
|
355
|
+
Attributes: {:jms_correlation_id=>"xxxx", :jms_delivery_mode_sym=>:non_persistent, :jms_destination=>"destination", :jms_expiration=>0, :jms_message_id=>"xxxxxx", :jms_priority=>0, :jms_redelivered=>false, :jms_reply_to=>#<Java::ComSolacesystemsJmsImpl::SolTopicImpl:0xdeadbeef>, :jms_timestamp=>1574669008862, :jms_type=>nil}
|
356
|
+
Properties: nil, :exception=>org.logstash.MissingConverterException: Missing Converter handling for full class name=com.solacesystems.jms.impl.SolTopicImpl, simple name=SolTopicImpl, :backtrace=>["org.logstash.Valuefier.fallbackConvert(Valuefier.java:98)..."]}
|
357
|
+
:exception=>org.logstash.MissingConverterException: Missing Converter handling for full class name=com.solacesystems.jms.impl.SolTopicImpl
|
358
|
+
-----
|
359
|
+
|
360
|
+
To get around this, use the `skip_headers` or `skip_properties` configuration setting to avoid attempting to process
|
361
|
+
the offending header or property in the message.
|
362
|
+
|
363
|
+
In the example shown above, this attribute is causing the
|
364
|
+
`MissingConverterException`:
|
365
|
+
|
366
|
+
`jms_reply_to=>#<Java::ComSolacesystemsJmsImpl::SolTopicImpl:0xdeadbeef>`
|
367
|
+
|
368
|
+
|
369
|
+
To avoid this error, the configuration should include the following line:
|
370
|
+
|
371
|
+
[source,ruby]
|
372
|
+
-----
|
373
|
+
skip_headers => ["jms_reply_to"]
|
374
|
+
-----
|
375
|
+
|
376
|
+
|
340
377
|
[id="plugins-{type}s-{plugin}-options"]
|
341
378
|
==== Jms Input Configuration Options
|
342
379
|
|
@@ -350,9 +387,13 @@ This plugin supports the following configuration options plus the <<plugins-{typ
|
|
350
387
|
| <<plugins-{type}s-{plugin}-durable_subscriber>> |<<boolean,boolean>>|No
|
351
388
|
| <<plugins-{type}s-{plugin}-durable_subscriber_client_id>> |<<string,string>>|No
|
352
389
|
| <<plugins-{type}s-{plugin}-durable_subscriber_name>> |<<string,string>>|No
|
390
|
+
| <<plugins-{type}s-{plugin}-ecs_compatibility>> |<<string,string>>|No
|
353
391
|
| <<plugins-{type}s-{plugin}-factory>> |<<string,string>>|No
|
392
|
+
| <<plugins-{type}s-{plugin}-factory_settings>> |<<hash,hash>>|No
|
393
|
+
| <<plugins-{type}s-{plugin}-headers_target>> |<<string,string>>|No
|
354
394
|
| <<plugins-{type}s-{plugin}-include_body>> |<<boolean,boolean>>|No
|
355
395
|
| <<plugins-{type}s-{plugin}-include_header>> |<<boolean,boolean>>|No
|
396
|
+
| <<plugins-{type}s-{plugin}-include_headers>> |<<boolean,boolean>>|No
|
356
397
|
| <<plugins-{type}s-{plugin}-include_properties>> |<<boolean,boolean>>|No
|
357
398
|
| <<plugins-{type}s-{plugin}-interval>> |<<number,number>>|No
|
358
399
|
| <<plugins-{type}s-{plugin}-jndi_context>> |<<hash,hash>>|No
|
@@ -361,6 +402,7 @@ This plugin supports the following configuration options plus the <<plugins-{typ
|
|
361
402
|
| <<plugins-{type}s-{plugin}-keystore_password>> |<<password,password>>|No
|
362
403
|
| <<plugins-{type}s-{plugin}-oracle_aq_buffered_messages>> |<<boolean,boolean>>|No
|
363
404
|
| <<plugins-{type}s-{plugin}-password>> |<<string,string>>|No
|
405
|
+
| <<plugins-{type}s-{plugin}-properties_target>> |<<string,string>>|No
|
364
406
|
| <<plugins-{type}s-{plugin}-pub_sub>> |<<boolean,boolean>>|No
|
365
407
|
| <<plugins-{type}s-{plugin}-require_jars>> |<<array,array>>|No
|
366
408
|
| <<plugins-{type}s-{plugin}-runner>> |<<string,string>>|__Deprecated__
|
@@ -368,6 +410,7 @@ This plugin supports the following configuration options plus the <<plugins-{typ
|
|
368
410
|
| <<plugins-{type}s-{plugin}-skip_headers>> |<<array,array>>|No
|
369
411
|
| <<plugins-{type}s-{plugin}-skip_properties>> |<<array,array>>|No
|
370
412
|
| <<plugins-{type}s-{plugin}-system_properties>> |<<hash,hash>>|No
|
413
|
+
| <<plugins-{type}s-{plugin}-target>> |<<string,string>>|No
|
371
414
|
| <<plugins-{type}s-{plugin}-threads>> |<<number,number>>|No
|
372
415
|
| <<plugins-{type}s-{plugin}-timeout>> |<<number,number>>|No
|
373
416
|
| <<plugins-{type}s-{plugin}-truststore>> |a valid filesystem path|No
|
@@ -400,6 +443,8 @@ Url to use when connecting to the JMS provider. This is only relevant for non-JN
|
|
400
443
|
|
401
444
|
Name of the destination queue or topic to use.
|
402
445
|
|
446
|
+
TIP: If the destination setting doesn't appear to be working properly, try this format: `<JMS module name>!<JMS queue name>`.
|
447
|
+
|
403
448
|
[id="plugins-{type}s-{plugin}-durable_subscriber"]
|
404
449
|
===== `durable_subscriber`
|
405
450
|
|
@@ -436,6 +481,21 @@ This represents the value of the client ID for a durable subscribtion, and is on
|
|
436
481
|
This represents the value of the subscriber name for a durable subscribtion, and is only used if `durable_subscriber`
|
437
482
|
is set to `true`. Please consult your JMS Provider documentation for constraints and requirements for this setting.
|
438
483
|
|
484
|
+
[id="plugins-{type}s-{plugin}-ecs_compatibility"]
|
485
|
+
===== `ecs_compatibility`
|
486
|
+
|
487
|
+
* Value type is <<string,string>>
|
488
|
+
* Supported values are:
|
489
|
+
** `disabled`: does not use ECS-compatible field names (fields might be set at the root of the event)
|
490
|
+
** `v1`, `v8`: avoids field names that might conflict with Elastic Common Schema (for example, JMS specific properties)
|
491
|
+
* Default value depends on which version of Logstash is running:
|
492
|
+
** When Logstash provides a `pipeline.ecs_compatibility` setting, its value is used as the default
|
493
|
+
** Otherwise, the default value is `disabled`.
|
494
|
+
|
495
|
+
Controls this plugin's compatibility with the {ecs-ref}[Elastic Common Schema (ECS)].
|
496
|
+
The value of this setting affects the _default_ value of <<plugins-{type}s-{plugin}-headers_target>> and
|
497
|
+
<<plugins-{type}s-{plugin}-properties_target>>.
|
498
|
+
|
439
499
|
[id="plugins-{type}s-{plugin}-factory"]
|
440
500
|
===== `factory`
|
441
501
|
|
@@ -444,6 +504,28 @@ is set to `true`. Please consult your JMS Provider documentation for constraints
|
|
444
504
|
|
445
505
|
Full name (including package name) of Java connection factory used to create a connection with your JMS provider.
|
446
506
|
|
507
|
+
[id="plugins-{type}s-{plugin}-factory_settings"]
|
508
|
+
===== `factory_settings`
|
509
|
+
|
510
|
+
* Value type is <<hash,hash>>
|
511
|
+
* There is no default value for this setting.
|
512
|
+
|
513
|
+
Hash of implementation specific configuration values to set on the connection factory of the JMS provider. Each JMS
|
514
|
+
Provider will have its own set of parameters that can be used here. These parameters are mapped to `set` methods on
|
515
|
+
the provided connection factory, and can be supplied in either 'snake' or 'camel' case. For example, a hash including
|
516
|
+
`exclusive_consumer => true` would call `setExclusiveConsumer(true)` on the supplied connection factory.
|
517
|
+
See your JMS provider documentation for implementation specific details.
|
518
|
+
|
519
|
+
[id="plugins-{type}s-{plugin}-headers_target"]
|
520
|
+
===== `headers_target`
|
521
|
+
|
522
|
+
* Value type is <<string,string>>
|
523
|
+
* Default value depends on whether <<plugins-{type}s-{plugin}-ecs_compatibility>> is enabled:
|
524
|
+
** ECS Compatibility disabled: no default value for this setting
|
525
|
+
** ECS Compatibility enabled: `"[@metadata][input][jms][headers]"
|
526
|
+
|
527
|
+
The name of the field under which JMS headers will be added, if <<plugins-{type}s-{plugin}-include_headers>> is set.
|
528
|
+
|
447
529
|
[id="plugins-{type}s-{plugin}-include_body"]
|
448
530
|
===== `include_body`
|
449
531
|
|
@@ -451,29 +533,54 @@ Full name (including package name) of Java connection factory used to create a c
|
|
451
533
|
* Default value is `true`
|
452
534
|
|
453
535
|
Include JMS Message Body in the event.
|
454
|
-
Supports TextMessage, MapMessage and
|
536
|
+
Supports TextMessage, MapMessage and BytesMessage.
|
455
537
|
|
456
|
-
If the JMS Message is a TextMessage or
|
538
|
+
If the JMS Message is a TextMessage or BytesMessage, then the value will be in
|
457
539
|
the "message" field of the event. If the JMS Message is a MapMessage, then all
|
458
|
-
the key/value pairs will be added
|
540
|
+
the key/value pairs will be added at the top-level of the event by default.
|
541
|
+
To avoid pollution of the top-level namespace, when receiving a MapMessage, use the <<plugins-{type}s-{plugin}-target>>.
|
459
542
|
|
460
543
|
StreamMessage and ObjectMessage are not supported.
|
461
544
|
|
462
545
|
[id="plugins-{type}s-{plugin}-include_header"]
|
463
546
|
===== `include_header`
|
464
547
|
|
548
|
+
* Value type is <<boolean,boolean>>
|
549
|
+
* This option is deprecated
|
550
|
+
|
551
|
+
Note: This option is deprecated and it will be removed in the next major version of Logstash.
|
552
|
+
Use `include_headers` instead.
|
553
|
+
|
554
|
+
[id="plugins-{type}s-{plugin}-include_headers"]
|
555
|
+
===== `include_headers`
|
556
|
+
|
465
557
|
* Value type is <<boolean,boolean>>
|
466
558
|
* Default value is `true`
|
467
559
|
|
468
560
|
A JMS message has three parts:
|
469
561
|
|
470
|
-
* Message Headers (required)
|
471
|
-
* Message Properties (optional)
|
472
|
-
* Message
|
562
|
+
* Message Headers (required)
|
563
|
+
* Message Properties (optional)
|
564
|
+
* Message Body (optional)
|
473
565
|
|
474
566
|
You can tell the input plugin which parts should be included in the event produced by Logstash.
|
475
567
|
|
476
|
-
Include JMS
|
568
|
+
Include standard JMS message header field values in the event.
|
569
|
+
Example headers:
|
570
|
+
[source,ruby]
|
571
|
+
-----
|
572
|
+
{
|
573
|
+
"jms_message_id" => "ID:amqhost-39547-1636977297920-71:1:1:1:1",
|
574
|
+
"jms_timestamp" => 1636977329102,
|
575
|
+
"jms_expiration" => 0,
|
576
|
+
"jms_delivery_mode" => "persistent",
|
577
|
+
"jms_redelivered" => false,
|
578
|
+
"jms_destination" => "topic://41ad5342149901ad",
|
579
|
+
"jms_priority" => 4,
|
580
|
+
"jms_type" => "sample",
|
581
|
+
"jms_correlation_id" => "28d975cb-14ff-4285-841e-05ef1e0a7ab2"
|
582
|
+
}
|
583
|
+
-----
|
477
584
|
|
478
585
|
[id="plugins-{type}s-{plugin}-include_properties"]
|
479
586
|
===== `include_properties`
|
@@ -546,6 +653,16 @@ Only for use with Oracle AQ
|
|
546
653
|
|
547
654
|
Password to use when connecting to the JMS provider.
|
548
655
|
|
656
|
+
[id="plugins-{type}s-{plugin}-properties_target"]
|
657
|
+
===== `properties_target`
|
658
|
+
|
659
|
+
* Value type is <<string,string>>
|
660
|
+
* Default value depends on whether <<plugins-{type}s-{plugin}-ecs_compatibility>> is enabled:
|
661
|
+
** ECS Compatibility disabled: no default value for this setting
|
662
|
+
** ECS Compatibility enabled: `"[@metadata][input][jms][properties]"
|
663
|
+
|
664
|
+
The name of the field under which JMS properties will be added, if <<plugins-{type}s-{plugin}-include_properties>> is set.
|
665
|
+
|
549
666
|
[id="plugins-{type}s-{plugin}-pub_sub"]
|
550
667
|
===== `pub_sub`
|
551
668
|
|
@@ -606,6 +723,17 @@ If `include_properties` is set, a list of properties to skip processing on can b
|
|
606
723
|
|
607
724
|
Any System properties that the JMS provider requires can be set either in a Hash here, or in `jvm.options`
|
608
725
|
|
726
|
+
[id="plugins-{type}s-{plugin}-target"]
|
727
|
+
===== `target`
|
728
|
+
|
729
|
+
* Value type is <<string,string>>
|
730
|
+
* There is no default value for this setting.
|
731
|
+
|
732
|
+
The name of the field to assign MapMessage data into.
|
733
|
+
If not specified data will be stored in the root of the event.
|
734
|
+
|
735
|
+
NOTE: For TextMessage and BytesMessage the `target` has no effect. Use the codec setting instead
|
736
|
+
e.g. `codec => json { target => "[jms]" }`.
|
609
737
|
|
610
738
|
[id="plugins-{type}s-{plugin}-threads"]
|
611
739
|
===== `threads`
|
@@ -613,7 +741,7 @@ Any System properties that the JMS provider requires can be set either in a Hash
|
|
613
741
|
* Value type is <<number,number>>
|
614
742
|
* Default value is `1`
|
615
743
|
|
616
|
-
|
744
|
+
NOTE: If pub_sub is set to `true`, this value *must* be `1`. A configuration error will be thrown otherwise!
|
617
745
|
|
618
746
|
[id="plugins-{type}s-{plugin}-timeout"]
|
619
747
|
===== `timeout`
|
@@ -680,4 +808,4 @@ For some known examples, see https://github.com/reidmorrison/jruby-jms/blob/mast
|
|
680
808
|
[id="plugins-{type}s-{plugin}-common-options"]
|
681
809
|
include::{include_path}/{type}.asciidoc[]
|
682
810
|
|
683
|
-
:default_codec!:
|
811
|
+
:default_codec!:
|