logstash-integration-jdbc 5.0.5 → 5.1.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 39722eec0f50423361c7a332054ffeaab4dde768e0a84d45535d10dc1661fc55
4
- data.tar.gz: cd1f3ed6d68671687d921d4d8e88a8821d9536777f2f5d543954d2848f21340a
3
+ metadata.gz: 34dbd968b687896da6407942201421c91970d025814596b7229609493b8bbcdc
4
+ data.tar.gz: a5155f4d11413ae5440c1c3ec4da5215c3cdd6febd4cb6d6376b2a03bdcbad42
5
5
  SHA512:
6
- metadata.gz: 78b8c924c351b1e80b1456b019c2e7e9fb7305410c748c15f05bc6088f1d9a9d4e0c09cfecaa71804bcad2e33b81a5610ef859023d1bdf26cce3f92db50e6c80
7
- data.tar.gz: 593d2160fc67da503c7f5db0ff42f1c33ed65b9316d398a86c402504c149b266f6e2b1e8fbe2ff965e193341753e23d7d16585597906bb6dadb8b1356e41cb8a
6
+ metadata.gz: dec5e76bf0354e7d46fafcb56725ba581ad31c54d475269faf5562e658f3723426a14011d961d6aedaf6ba921b9a654dbe05b2707bdfad38e6b3ca34f1934b56
7
+ data.tar.gz: 65eb85f0e25e73f2e63de44c3d9e64ad6d834c5af316208acb445efcf1aa8d4443de00f2dcf68e66c90057a46fbddb9c7ff451b80bc46ce03bcf32b43c7a6c4a
data/CHANGELOG.md CHANGED
@@ -1,3 +1,27 @@
1
+ ## 5.1.2
2
+ - Fix `prepared_statement_bind_values` in streaming filter to resolve nested event's fields[#76](https://github.com/logstash-plugins/logstash-integration-jdbc/pull/76)
3
+
4
+ ## 5.1.1
5
+ - [DOC] Changed docs to indicate that logstash-jdbc-static requires local_table [#56](https://github.com/logstash-plugins/logstash-integration-jdbc/pull/56). Fixes [#55](https://github.com/logstash-plugins/logstash-integration-jdbc/issues/55).
6
+
7
+ ## 5.1.0
8
+ - Added `target` option to JDBC input, allowing the row columns to target a specific field instead of being expanded
9
+ at the root of the event. This allows the input to play nicer with the Elastic Common Schema when
10
+ the input does not follow the schema. [#69](https://github.com/logstash-plugins/logstash-integration-jdbc/issues/69)
11
+
12
+ - Added `target` to JDBC filter static `local_lookups` to verify it's properly valued when ECS is enabled.
13
+ [#71](https://github.com/logstash-plugins/logstash-integration-jdbc/issues/71)
14
+
15
+ ## 5.0.7
16
+ - Feat: try hard to log Java cause (chain) [#62](https://github.com/logstash-plugins/logstash-integration-jdbc/pull/62)
17
+
18
+ This allows seeing a full trace from the JDBC driver in case of connection errors.
19
+
20
+ - Refactored Lookup used in jdbc_streaming and jdbc_static to avoid code duplication. [#59](https://github.com/logstash-plugins/logstash-integration-jdbc/pull/59)
21
+
22
+ ## 5.0.6
23
+ - DOC:Replaced plugin_header file with plugin_header-integration file. [#40](https://github.com/logstash-plugins/logstash-integration-jdbc/pull/40)
24
+
1
25
  ## 5.0.5
2
26
  - Fixed user sequel_opts not being passed down properly [#37](https://github.com/logstash-plugins/logstash-integration-jdbc/pull/37)
3
27
  - Refactored jdbc_streaming to share driver loading, so the fixes from the jdbc plugin also effect jdbc_streaming
@@ -22,4 +46,4 @@
22
46
  - [JBDC Input version 4.3.19](https://github.com/logstash-plugins/logstash-input-jdbc/blob/v4.3.19/CHANGELOG.md)
23
47
  - [JDBC Static filter version 1.1.0](https://github.com/logstash-plugins/logstash-filter-jdbc_static/blob/v1.1.0/CHANGELOG.md)
24
48
  - [JDBC Streaming filter version 1.0.10](https://github.com/logstash-plugins/logstash-filter-jdbc_streaming/blob/v1.0.10/CHANGELOG.md)
25
-
49
+
data/README.md CHANGED
@@ -2,7 +2,7 @@
2
2
  Logstash Integration Plugin for JDBC, including Logstash Input and Filter Plugins
3
3
  # Logstash Plugin
4
4
 
5
- [![Travis Build Status](https://travis-ci.org/logstash-plugins/logstash-integration-jdbc.svg)](https://travis-ci.org/logstash-plugins/logstash-integration-jdbc)
5
+ [![Travis Build Status](https://travis-ci.com/logstash-plugins/logstash-integration-jdbc.svg)](https://travis-ci.com/logstash-plugins/logstash-integration-jdbc)
6
6
 
7
7
  This is a plugin for [Logstash](https://github.com/elastic/logstash).
8
8
 
@@ -1,3 +1,4 @@
1
+ :integration: jdbc
1
2
  :plugin: jdbc_static
2
3
  :type: filter
3
4
 
@@ -16,7 +17,7 @@ END - GENERATED VARIABLES, DO NOT EDIT!
16
17
 
17
18
  === Jdbc_static filter plugin
18
19
 
19
- include::{include_path}/plugin_header.asciidoc[]
20
+ include::{include_path}/plugin_header-integration.asciidoc[]
20
21
 
21
22
  ==== Description
22
23
 
@@ -421,7 +422,7 @@ according to the table below.
421
422
  |=======================================================================
422
423
  |Setting |Input type|Required
423
424
  | id|string|No
424
- | table|string|Yes
425
+ | local_table|string|Yes
425
426
  | query|string|Yes
426
427
  | max_rows|number|No
427
428
  | jdbc_connection_string|string|No
@@ -437,7 +438,7 @@ id::
437
438
  An optional identifier. This is used to identify the loader that is
438
439
  generating error messages and log lines.
439
440
 
440
- table::
441
+ local_table::
441
442
  The destination table in the local lookup database that the loader will fill.
442
443
 
443
444
  query::
@@ -1,3 +1,4 @@
1
+ :integration: jdbc
1
2
  :plugin: jdbc_streaming
2
3
  :type: filter
3
4
 
@@ -16,7 +17,7 @@ END - GENERATED VARIABLES, DO NOT EDIT!
16
17
 
17
18
  === Jdbc_streaming filter plugin
18
19
 
19
- include::{include_path}/plugin_header.asciidoc[]
20
+ include::{include_path}/plugin_header-integration.asciidoc[]
20
21
 
21
22
  ==== Description
22
23
 
@@ -1,3 +1,4 @@
1
+ :integration: jdbc
1
2
  :plugin: jdbc
2
3
  :type: input
3
4
  :default_codec: plain
@@ -17,7 +18,7 @@ END - GENERATED VARIABLES, DO NOT EDIT!
17
18
 
18
19
  === Jdbc input plugin
19
20
 
20
- include::{include_path}/plugin_header.asciidoc[]
21
+ include::{include_path}/plugin_header-integration.asciidoc[]
21
22
 
22
23
  ==== Description
23
24
 
@@ -210,6 +211,7 @@ This plugin supports the following configuration options plus the <<plugins-{typ
210
211
  | <<plugins-{type}s-{plugin}-sql_log_level>> |<<string,string>>, one of `["fatal", "error", "warn", "info", "debug"]`|No
211
212
  | <<plugins-{type}s-{plugin}-statement>> |<<string,string>>|No
212
213
  | <<plugins-{type}s-{plugin}-statement_filepath>> |a valid filesystem path|No
214
+ | <<plugins-{type}s-{plugin}-target>> | {logstash-ref}/field-references-deepdive.html[field reference] | No
213
215
  | <<plugins-{type}s-{plugin}-tracking_column>> |<<string,string>>|No
214
216
  | <<plugins-{type}s-{plugin}-tracking_column_type>> |<<string,string>>, one of `["numeric", "timestamp"]`|No
215
217
  | <<plugins-{type}s-{plugin}-use_column_value>> |<<boolean,boolean>>|No
@@ -534,6 +536,17 @@ with the `parameters` setting.
534
536
 
535
537
  Path of file containing statement to execute
536
538
 
539
+ [id="plugins-{type}s-{plugin}-target"]
540
+ ===== `target`
541
+
542
+ * Value type is {logstash-ref}/field-references-deepdive.html[field reference]
543
+ * There is no default value for this setting.
544
+
545
+ Without a `target`, events are created from each row column at the root level.
546
+ When the `target` is set to a field reference, the column of each row is placed in the target field instead.
547
+
548
+ This option can be useful to avoid populating unknown fields when a downstream schema such as ECS is enforced.
549
+
537
550
  [id="plugins-{type}s-{plugin}-tracking_column"]
538
551
  ===== `tracking_column`
539
552
 
@@ -56,6 +56,12 @@ module LogStash module Filters module Jdbc
56
56
  @target = options["target"]
57
57
  @id_used_as_target = @target.nil?
58
58
  if @id_used_as_target
59
+ # target shouldn't be nil if ecs_compatibility is not :disabled
60
+ if globals[:ecs_compatibility] != :disabled
61
+ logger.info('ECS compatibility is enabled but no ``target`` option was specified, it is recommended'\
62
+ ' to set the option to avoid potential schema conflicts (if your data is ECS compliant or'\
63
+ ' non-conflicting feel free to ignore this message)')
64
+ end
59
65
  @target = @id
60
66
  end
61
67
  @options = options
@@ -66,6 +72,7 @@ module LogStash module Filters module Jdbc
66
72
  @prepared_statement = nil
67
73
  @symbol_parameters = nil
68
74
  parse_options
75
+ @load_method_ref = method(:load_data_from_local)
69
76
  end
70
77
 
71
78
  def id_used_as_target?
@@ -81,11 +88,7 @@ module LogStash module Filters module Jdbc
81
88
  end
82
89
 
83
90
  def enhance(local, event)
84
- if @prepared_statement
85
- result = call_prepared(local, event)
86
- else
87
- result = fetch(local, event) # should return a LookupResult
88
- end
91
+ result = retrieve_local_data(local, event, &@load_method_ref) # return a LookupResult
89
92
  if result.failed? || result.parameters_invalid?
90
93
  tag_failure(event)
91
94
  end
@@ -112,6 +115,7 @@ module LogStash module Filters module Jdbc
112
115
  @prepared_parameters.each_with_index { |v, i| hash[:"$p#{i}"] = v }
113
116
  @prepared_param_placeholder_map = hash
114
117
  @prepared_statement = local.prepare(query, hash.keys)
118
+ @load_method_ref = method(:load_data_from_prepared)
115
119
  end
116
120
 
117
121
  private
@@ -128,34 +132,23 @@ module LogStash module Filters module Jdbc
128
132
  end
129
133
  end
130
134
 
131
- def fetch(local, event)
132
- result = LookupResult.new()
133
- if @parameters_specified
134
- params = prepare_parameters_from_event(event, result)
135
- if result.parameters_invalid?
136
- logger.warn? && logger.warn("Parameter field not found in event", :lookup_id => @id, :invalid_parameters => result.invalid_parameters)
137
- return result
138
- end
139
- else
140
- params = {}
135
+ def load_data_from_local(local, query, params, result)
136
+ local.fetch(query, params).each do |row|
137
+ stringified = row.inject({}){|hash,(k,v)| hash[k.to_s] = v; hash} #Stringify row keys
138
+ result.push(stringified)
141
139
  end
142
- begin
143
- logger.debug? && logger.debug("Executing Jdbc query", :lookup_id => @id, :statement => query, :parameters => params)
144
- local.fetch(query, params).each do |row|
145
- stringified = row.inject({}){|hash,(k,v)| hash[k.to_s] = v; hash} #Stringify row keys
146
- result.push(stringified)
147
- end
148
- rescue ::Sequel::Error => e
149
- # all sequel errors are a subclass of this, let all other standard or runtime errors bubble up
150
- result.failed!
151
- logger.warn? && logger.warn("Exception when executing Jdbc query", :lookup_id => @id, :exception => e.message, :backtrace => e.backtrace.take(8))
140
+ end
141
+
142
+ def load_data_from_prepared(_local, _query, params, result)
143
+ @prepared_statement.call(params).each do |row|
144
+ stringified = row.inject({}){|hash,(k,v)| hash[k.to_s] = v; hash} #Stringify row keys
145
+ result.push(stringified)
152
146
  end
153
- # if either of: no records or a Sequel exception occurs the payload is
154
- # empty and the default can be substituted later.
155
- result
156
147
  end
157
148
 
158
- def call_prepared(local, event)
149
+ # the &block is invoked with 4 arguments: local, query[String], params[Hash], result[LookupResult]
150
+ # the result is used as accumulator return variable
151
+ def retrieve_local_data(local, event, &proc)
159
152
  result = LookupResult.new()
160
153
  if @parameters_specified
161
154
  params = prepare_parameters_from_event(event, result)
@@ -168,10 +161,7 @@ module LogStash module Filters module Jdbc
168
161
  end
169
162
  begin
170
163
  logger.debug? && logger.debug("Executing Jdbc query", :lookup_id => @id, :statement => query, :parameters => params)
171
- @prepared_statement.call(params).each do |row|
172
- stringified = row.inject({}){|hash,(k,v)| hash[k.to_s] = v; hash} #Stringify row keys
173
- result.push(stringified)
174
- end
164
+ proc.call(local, query, params, result)
175
165
  rescue ::Sequel::Error => e
176
166
  # all sequel errors are a subclass of this, let all other standard or runtime errors bubble up
177
167
  result.failed!
@@ -2,6 +2,7 @@
2
2
  require "logstash-integration-jdbc_jars"
3
3
  require "logstash/filters/base"
4
4
  require "logstash/namespace"
5
+ require "logstash/plugin_mixins/ecs_compatibility_support"
5
6
  require_relative "jdbc/loader"
6
7
  require_relative "jdbc/loader_schedule"
7
8
  require_relative "jdbc/repeating_load_runner"
@@ -14,6 +15,9 @@ require_relative "jdbc/lookup_processor"
14
15
 
15
16
  #
16
17
  module LogStash module Filters class JdbcStatic < LogStash::Filters::Base
18
+ # adds ecs_compatibility config which could be :disabled or :v1
19
+ include LogStash::PluginMixins::ECSCompatibilitySupport(:disabled, :v1, :v8 => :v1)
20
+
17
21
  config_name "jdbc_static"
18
22
 
19
23
  # Define the loaders, an Array of Hashes, to fetch remote data and create local tables.
@@ -214,6 +218,7 @@ module LogStash module Filters class JdbcStatic < LogStash::Filters::Base
214
218
  options["lookup_jdbc_driver_class"] = @lookup_jdbc_driver_class
215
219
  options["lookup_jdbc_driver_library"] = @lookup_jdbc_driver_library
216
220
  options["lookup_jdbc_connection_string"] = @lookup_jdbc_connection_string
221
+ options["ecs_compatibility"] = ecs_compatibility
217
222
  options
218
223
  end
219
224
 
@@ -3,6 +3,8 @@ require "logstash/inputs/base"
3
3
  require "logstash/namespace"
4
4
  require "logstash/plugin_mixins/jdbc/common"
5
5
  require "logstash/plugin_mixins/jdbc/jdbc"
6
+ require "logstash/plugin_mixins/ecs_compatibility_support"
7
+ require "logstash/plugin_mixins/validator_support/field_reference_validation_adapter"
6
8
 
7
9
  # this require_relative returns early unless the JRuby version is between 9.2.0.0 and 9.2.8.0
8
10
  require_relative "tzinfo_jruby_patch"
@@ -129,6 +131,11 @@ require_relative "tzinfo_jruby_patch"
129
131
  module LogStash module Inputs class Jdbc < LogStash::Inputs::Base
130
132
  include LogStash::PluginMixins::Jdbc::Common
131
133
  include LogStash::PluginMixins::Jdbc::Jdbc
134
+ # adds ecs_compatibility config which could be :disabled or :v1
135
+ include LogStash::PluginMixins::ECSCompatibilitySupport(:disabled,:v1,:v8 => :v1)
136
+ # adds :field_reference validator adapter
137
+ extend LogStash::PluginMixins::ValidatorSupport::FieldReferenceValidationAdapter
138
+
132
139
  config_name "jdbc"
133
140
 
134
141
  # If undefined, Logstash will complain, even if codec is unused.
@@ -209,6 +216,9 @@ module LogStash module Inputs class Jdbc < LogStash::Inputs::Base
209
216
 
210
217
  config :prepared_statement_bind_values, :validate => :array, :default => []
211
218
 
219
+ # Define the target field to store the loaded columns
220
+ config :target, :validate => :field_reference, :required => false
221
+
212
222
  attr_reader :database # for test mocking/stubbing
213
223
 
214
224
  public
@@ -260,6 +270,13 @@ module LogStash module Inputs class Jdbc < LogStash::Inputs::Base
260
270
  converters[encoding] = converter
261
271
  end
262
272
  end
273
+
274
+ # target must be populated if ecs_compatibility is not :disabled
275
+ if @target.nil? && ecs_compatibility != :disabled
276
+ logger.info('ECS compatibility is enabled but no ``target`` option was specified, it is recommended'\
277
+ ' to set the option to avoid potential schema conflicts (if your data is ECS compliant or'\
278
+ ' non-conflicting feel free to ignore this message)')
279
+ end
263
280
  end # def register
264
281
 
265
282
  # test injection points
@@ -318,7 +335,12 @@ module LogStash module Inputs class Jdbc < LogStash::Inputs::Base
318
335
  ## do the necessary conversions to string elements
319
336
  row = Hash[row.map { |k, v| [k.to_s, convert(k, v)] }]
320
337
  end
321
- event = LogStash::Event.new(row)
338
+ if @target
339
+ event = LogStash::Event.new
340
+ event.set(@target, row)
341
+ else
342
+ event = LogStash::Event.new(row)
343
+ end
322
344
  decorate(event)
323
345
  queue << event
324
346
  end
@@ -122,16 +122,32 @@ module LogStash module PluginMixins module Jdbc
122
122
  # rescue Java::JavaSql::SQLException, ::Sequel::Error => e
123
123
  rescue ::Sequel::Error => e
124
124
  if retry_attempts <= 0
125
- @logger.error("Unable to connect to database. Tried #{@connection_retry_attempts} times", :error_message => e.message)
125
+ log_java_exception(e.cause)
126
+ @logger.error("Unable to connect to database. Tried #{@connection_retry_attempts} times", error_details(e, trace: true))
126
127
  raise e
127
128
  else
128
- @logger.error("Unable to connect to database. Trying again", :error_message => e.message)
129
+ @logger.error("Unable to connect to database. Trying again", error_details(e, trace: false))
129
130
  end
130
131
  end
131
132
  sleep(@connection_retry_attempts_wait_time)
132
133
  end
133
134
  end
134
135
 
136
+ def error_details(e, trace: false)
137
+ details = { :message => e.message, :exception => e.class }
138
+ details[:cause] = e.cause if e.cause
139
+ details[:backtrace] = e.backtrace if trace || @logger.debug?
140
+ details
141
+ end
142
+
143
+ def log_java_exception(e)
144
+ return unless e.is_a?(java.lang.Exception)
145
+ # @logger.name using the same convention as LS does
146
+ logger = self.class.name.gsub('::', '.').downcase
147
+ logger = org.apache.logging.log4j.LogManager.getLogger(logger)
148
+ logger.error('', e) # prints nested causes
149
+ end
150
+
135
151
  def open_jdbc_connection
136
152
  # at this point driver is already loaded
137
153
  Sequel.application_timezone = @plugin_timezone.to_sym
@@ -26,7 +26,7 @@ module LogStash module PluginMixins module JdbcStreaming
26
26
  return InterpolatedParameter.new(given_value)
27
27
  end
28
28
 
29
- if given_value =~ /\A\s*\[[^\]]+\]\s*\z/
29
+ if given_value =~ /\A(\s*\[[^\]]+\]\s*)*\z/
30
30
  return FieldParameter.new(given_value)
31
31
  end
32
32
 
@@ -1,6 +1,6 @@
1
1
  Gem::Specification.new do |s|
2
2
  s.name = 'logstash-integration-jdbc'
3
- s.version = '5.0.5'
3
+ s.version = '5.1.2'
4
4
  s.licenses = ['Apache License (2.0)']
5
5
  s.summary = "Integration with JDBC - input and filter plugins"
6
6
  s.description = "This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program"
@@ -36,6 +36,8 @@ Gem::Specification.new do |s|
36
36
  s.add_runtime_dependency 'tzinfo-data'
37
37
  # 3.5 limitation is required for jdbc-static loading schedule
38
38
  s.add_runtime_dependency 'rufus-scheduler', '< 3.5'
39
+ s.add_runtime_dependency 'logstash-mixin-ecs_compatibility_support', '~>1.2'
40
+ s.add_runtime_dependency "logstash-mixin-validator_support", '~> 1.0'
39
41
 
40
42
  s.add_development_dependency "childprocess"
41
43
  s.add_development_dependency 'logstash-devutils'
@@ -59,7 +59,7 @@ module LogStash module Filters
59
59
 
60
60
  let(:plugin) { JdbcStatic.new(settings) }
61
61
 
62
- let(:event) { ::LogStash::Event.new("message" => "some text", "ip" => ipaddr) }
62
+ let(:event) { ::LogStash::Event.new("message" => "some text", "ip" => ipaddr) }
63
63
 
64
64
  let(:ipaddr) { ".3.1.1" }
65
65
 
@@ -248,6 +248,37 @@ module LogStash module Filters module Jdbc
248
248
  expect(subject.valid?).to be_falsey
249
249
  end
250
250
  end
251
+
252
+ describe "validation of target option" do
253
+ let(:lookup_hash) do
254
+ {
255
+ "query" => "select * from servers WHERE ip LIKE ? AND os LIKE ?",
256
+ "prepared_parameters" => ["%%{[ip]}"],
257
+ }
258
+ end
259
+
260
+ it "should log a warn when ECS is enabled and target not defined" do
261
+
262
+ class LoggableLookup < Lookup
263
+
264
+ @@TEST_LOGGER = nil
265
+
266
+ def self.logger=(log)
267
+ @@TEST_LOGGER = log
268
+ end
269
+
270
+ def self.logger
271
+ @@TEST_LOGGER
272
+ end
273
+ end
274
+
275
+ spy_logger = double("logger")
276
+ expect(spy_logger).to receive(:info).once.with(/ECS compatibility is enabled but no .*?target.*? was specified/)
277
+ LoggableLookup.logger = spy_logger
278
+
279
+ LoggableLookup.new(lookup_hash, {:ecs_compatibility => 'v1'}, "lookup-1")
280
+ end
281
+ end
251
282
  end
252
283
  end end end
253
284
 
@@ -45,7 +45,7 @@ describe LogStash::Inputs::Jdbc, :integration => true do
45
45
 
46
46
  context "when supplying a non-existent library" do
47
47
  let(:settings) do
48
- super.merge(
48
+ super().merge(
49
49
  "jdbc_driver_library" => "/no/path/to/postgresql.jar"
50
50
  )
51
51
  end
@@ -61,13 +61,29 @@ describe LogStash::Inputs::Jdbc, :integration => true do
61
61
 
62
62
  context "when connecting to a non-existent server" do
63
63
  let(:settings) do
64
- super.merge(
64
+ super().merge(
65
65
  "jdbc_connection_string" => "jdbc:postgresql://localhost:65000/somedb"
66
66
  )
67
67
  end
68
68
 
69
69
  it "should not register correctly" do
70
70
  plugin.register
71
+ allow( plugin ).to receive(:log_java_exception)
72
+ q = Queue.new
73
+ expect do
74
+ plugin.run(q)
75
+ end.to raise_error(::Sequel::DatabaseConnectionError)
76
+ end
77
+
78
+ it "should log (native) Java driver error" do
79
+ plugin.register
80
+ expect( org.apache.logging.log4j.LogManager ).to receive(:getLogger).and_wrap_original do |m, *args|
81
+ logger = m.call(*args)
82
+ expect( logger ).to receive(:error) do |_, e|
83
+ expect( e ).to be_a org.postgresql.util.PSQLException
84
+ end.and_call_original
85
+ logger
86
+ end
71
87
  q = Queue.new
72
88
  expect do
73
89
  plugin.run(q)
@@ -328,6 +328,48 @@ describe LogStash::Inputs::Jdbc do
328
328
 
329
329
  end
330
330
 
331
+ context "when using target option" do
332
+ let(:settings) do
333
+ {
334
+ "statement" => "SELECT * from test_table FETCH FIRST 1 ROWS ONLY",
335
+ "target" => "sub_field"
336
+ }
337
+ end
338
+
339
+ before do
340
+ plugin.register
341
+ end
342
+
343
+ after do
344
+ plugin.stop
345
+ end
346
+
347
+ it "should put all columns under sub-field" do
348
+ db[:test_table].insert(:num => 1, :custom_time => Time.now.utc, :created_at => Time.now.utc, :string => "Test target option")
349
+
350
+ plugin.run(queue)
351
+
352
+ expect(queue.size).to eq(1)
353
+ event = queue.pop
354
+ expect(event.get("[sub_field][string]")).to eq("Test target option")
355
+ end
356
+ end
357
+
358
+ context "when using target option is not set and ecs_compatibility is enabled" do
359
+ let(:settings) do
360
+ {
361
+ "statement" => "SELECT * from test_table FETCH FIRST 1 ROWS ONLY",
362
+ "ecs_compatibility" => :v1
363
+ }
364
+ end
365
+
366
+ it "should log a warn of missed target usage" do
367
+ expect(plugin.logger).to receive(:info).once.with(/ECS compatibility is enabled but no .*?target.*? was specified/)
368
+
369
+ plugin.register
370
+ end
371
+ end
372
+
331
373
  context "when fetching time data" do
332
374
 
333
375
  let(:settings) do
@@ -0,0 +1,23 @@
1
+ # encoding: utf-8
2
+ require "logstash/devutils/rspec/spec_helper"
3
+ require "logstash/plugin_mixins/jdbc_streaming/parameter_handler"
4
+
5
+
6
+ describe LogStash::PluginMixins::JdbcStreaming::ParameterHandler do
7
+ context "resolve field reference" do
8
+ let(:event) { ::LogStash::Event.new("field" => "field_value") }
9
+
10
+ it "should resolve root field" do
11
+ handler = LogStash::PluginMixins::JdbcStreaming::ParameterHandler.build_bind_value_handler "[field]"
12
+ handler.extract_from(event)
13
+ expect(handler.extract_from(event)).to eq "field_value"
14
+ end
15
+
16
+ it "should resolve nested field" do
17
+ event = ::LogStash::Event.from_json("{\"field\": {\"nested\": \"nested_field\"}}").first
18
+ handler = LogStash::PluginMixins::JdbcStreaming::ParameterHandler.build_bind_value_handler "[field][nested]"
19
+ handler.extract_from(event)
20
+ expect(handler.extract_from(event)).to eq "nested_field"
21
+ end
22
+ end
23
+ end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: logstash-integration-jdbc
3
3
  version: !ruby/object:Gem::Version
4
- version: 5.0.5
4
+ version: 5.1.2
5
5
  platform: ruby
6
6
  authors:
7
7
  - Elastic
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2020-06-16 00:00:00.000000000 Z
11
+ date: 2021-06-28 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  requirement: !ruby/object:Gem::Requirement
@@ -142,6 +142,34 @@ dependencies:
142
142
  - - "<"
143
143
  - !ruby/object:Gem::Version
144
144
  version: '3.5'
145
+ - !ruby/object:Gem::Dependency
146
+ requirement: !ruby/object:Gem::Requirement
147
+ requirements:
148
+ - - "~>"
149
+ - !ruby/object:Gem::Version
150
+ version: '1.2'
151
+ name: logstash-mixin-ecs_compatibility_support
152
+ prerelease: false
153
+ type: :runtime
154
+ version_requirements: !ruby/object:Gem::Requirement
155
+ requirements:
156
+ - - "~>"
157
+ - !ruby/object:Gem::Version
158
+ version: '1.2'
159
+ - !ruby/object:Gem::Dependency
160
+ requirement: !ruby/object:Gem::Requirement
161
+ requirements:
162
+ - - "~>"
163
+ - !ruby/object:Gem::Version
164
+ version: '1.0'
165
+ name: logstash-mixin-validator_support
166
+ prerelease: false
167
+ type: :runtime
168
+ version_requirements: !ruby/object:Gem::Requirement
169
+ requirements:
170
+ - - "~>"
171
+ - !ruby/object:Gem::Version
172
+ version: '1.0'
145
173
  - !ruby/object:Gem::Dependency
146
174
  requirement: !ruby/object:Gem::Requirement
147
175
  requirements:
@@ -264,6 +292,7 @@ files:
264
292
  - spec/helpers/derbyrun.jar
265
293
  - spec/inputs/integration/integ_spec.rb
266
294
  - spec/inputs/jdbc_spec.rb
295
+ - spec/plugin_mixins/jdbc_streaming/parameter_handler_spec.rb
267
296
  - vendor/jar-dependencies/org/apache/derby/derby/10.14.1.0/derby-10.14.1.0.jar
268
297
  - vendor/jar-dependencies/org/apache/derby/derbyclient/10.14.1.0/derbyclient-10.14.1.0.jar
269
298
  homepage: http://www.elastic.co/guide/en/logstash/current/index.html
@@ -316,3 +345,4 @@ test_files:
316
345
  - spec/helpers/derbyrun.jar
317
346
  - spec/inputs/integration/integ_spec.rb
318
347
  - spec/inputs/jdbc_spec.rb
348
+ - spec/plugin_mixins/jdbc_streaming/parameter_handler_spec.rb