logstash-integration-jdbc 5.0.5 → 5.1.2

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 39722eec0f50423361c7a332054ffeaab4dde768e0a84d45535d10dc1661fc55
4
- data.tar.gz: cd1f3ed6d68671687d921d4d8e88a8821d9536777f2f5d543954d2848f21340a
3
+ metadata.gz: 34dbd968b687896da6407942201421c91970d025814596b7229609493b8bbcdc
4
+ data.tar.gz: a5155f4d11413ae5440c1c3ec4da5215c3cdd6febd4cb6d6376b2a03bdcbad42
5
5
  SHA512:
6
- metadata.gz: 78b8c924c351b1e80b1456b019c2e7e9fb7305410c748c15f05bc6088f1d9a9d4e0c09cfecaa71804bcad2e33b81a5610ef859023d1bdf26cce3f92db50e6c80
7
- data.tar.gz: 593d2160fc67da503c7f5db0ff42f1c33ed65b9316d398a86c402504c149b266f6e2b1e8fbe2ff965e193341753e23d7d16585597906bb6dadb8b1356e41cb8a
6
+ metadata.gz: dec5e76bf0354e7d46fafcb56725ba581ad31c54d475269faf5562e658f3723426a14011d961d6aedaf6ba921b9a654dbe05b2707bdfad38e6b3ca34f1934b56
7
+ data.tar.gz: 65eb85f0e25e73f2e63de44c3d9e64ad6d834c5af316208acb445efcf1aa8d4443de00f2dcf68e66c90057a46fbddb9c7ff451b80bc46ce03bcf32b43c7a6c4a
data/CHANGELOG.md CHANGED
@@ -1,3 +1,27 @@
1
+ ## 5.1.2
2
+ - Fix `prepared_statement_bind_values` in streaming filter to resolve nested event's fields[#76](https://github.com/logstash-plugins/logstash-integration-jdbc/pull/76)
3
+
4
+ ## 5.1.1
5
+ - [DOC] Changed docs to indicate that logstash-jdbc-static requires local_table [#56](https://github.com/logstash-plugins/logstash-integration-jdbc/pull/56). Fixes [#55](https://github.com/logstash-plugins/logstash-integration-jdbc/issues/55).
6
+
7
+ ## 5.1.0
8
+ - Added `target` option to JDBC input, allowing the row columns to target a specific field instead of being expanded
9
+ at the root of the event. This allows the input to play nicer with the Elastic Common Schema when
10
+ the input does not follow the schema. [#69](https://github.com/logstash-plugins/logstash-integration-jdbc/issues/69)
11
+
12
+ - Added `target` to JDBC filter static `local_lookups` to verify it's properly valued when ECS is enabled.
13
+ [#71](https://github.com/logstash-plugins/logstash-integration-jdbc/issues/71)
14
+
15
+ ## 5.0.7
16
+ - Feat: try hard to log Java cause (chain) [#62](https://github.com/logstash-plugins/logstash-integration-jdbc/pull/62)
17
+
18
+ This allows seeing a full trace from the JDBC driver in case of connection errors.
19
+
20
+ - Refactored Lookup used in jdbc_streaming and jdbc_static to avoid code duplication. [#59](https://github.com/logstash-plugins/logstash-integration-jdbc/pull/59)
21
+
22
+ ## 5.0.6
23
+ - DOC:Replaced plugin_header file with plugin_header-integration file. [#40](https://github.com/logstash-plugins/logstash-integration-jdbc/pull/40)
24
+
1
25
  ## 5.0.5
2
26
  - Fixed user sequel_opts not being passed down properly [#37](https://github.com/logstash-plugins/logstash-integration-jdbc/pull/37)
3
27
  - Refactored jdbc_streaming to share driver loading, so the fixes from the jdbc plugin also effect jdbc_streaming
@@ -22,4 +46,4 @@
22
46
  - [JBDC Input version 4.3.19](https://github.com/logstash-plugins/logstash-input-jdbc/blob/v4.3.19/CHANGELOG.md)
23
47
  - [JDBC Static filter version 1.1.0](https://github.com/logstash-plugins/logstash-filter-jdbc_static/blob/v1.1.0/CHANGELOG.md)
24
48
  - [JDBC Streaming filter version 1.0.10](https://github.com/logstash-plugins/logstash-filter-jdbc_streaming/blob/v1.0.10/CHANGELOG.md)
25
-
49
+
data/README.md CHANGED
@@ -2,7 +2,7 @@
2
2
  Logstash Integration Plugin for JDBC, including Logstash Input and Filter Plugins
3
3
  # Logstash Plugin
4
4
 
5
- [![Travis Build Status](https://travis-ci.org/logstash-plugins/logstash-integration-jdbc.svg)](https://travis-ci.org/logstash-plugins/logstash-integration-jdbc)
5
+ [![Travis Build Status](https://travis-ci.com/logstash-plugins/logstash-integration-jdbc.svg)](https://travis-ci.com/logstash-plugins/logstash-integration-jdbc)
6
6
 
7
7
  This is a plugin for [Logstash](https://github.com/elastic/logstash).
8
8
 
@@ -1,3 +1,4 @@
1
+ :integration: jdbc
1
2
  :plugin: jdbc_static
2
3
  :type: filter
3
4
 
@@ -16,7 +17,7 @@ END - GENERATED VARIABLES, DO NOT EDIT!
16
17
 
17
18
  === Jdbc_static filter plugin
18
19
 
19
- include::{include_path}/plugin_header.asciidoc[]
20
+ include::{include_path}/plugin_header-integration.asciidoc[]
20
21
 
21
22
  ==== Description
22
23
 
@@ -421,7 +422,7 @@ according to the table below.
421
422
  |=======================================================================
422
423
  |Setting |Input type|Required
423
424
  | id|string|No
424
- | table|string|Yes
425
+ | local_table|string|Yes
425
426
  | query|string|Yes
426
427
  | max_rows|number|No
427
428
  | jdbc_connection_string|string|No
@@ -437,7 +438,7 @@ id::
437
438
  An optional identifier. This is used to identify the loader that is
438
439
  generating error messages and log lines.
439
440
 
440
- table::
441
+ local_table::
441
442
  The destination table in the local lookup database that the loader will fill.
442
443
 
443
444
  query::
@@ -1,3 +1,4 @@
1
+ :integration: jdbc
1
2
  :plugin: jdbc_streaming
2
3
  :type: filter
3
4
 
@@ -16,7 +17,7 @@ END - GENERATED VARIABLES, DO NOT EDIT!
16
17
 
17
18
  === Jdbc_streaming filter plugin
18
19
 
19
- include::{include_path}/plugin_header.asciidoc[]
20
+ include::{include_path}/plugin_header-integration.asciidoc[]
20
21
 
21
22
  ==== Description
22
23
 
@@ -1,3 +1,4 @@
1
+ :integration: jdbc
1
2
  :plugin: jdbc
2
3
  :type: input
3
4
  :default_codec: plain
@@ -17,7 +18,7 @@ END - GENERATED VARIABLES, DO NOT EDIT!
17
18
 
18
19
  === Jdbc input plugin
19
20
 
20
- include::{include_path}/plugin_header.asciidoc[]
21
+ include::{include_path}/plugin_header-integration.asciidoc[]
21
22
 
22
23
  ==== Description
23
24
 
@@ -210,6 +211,7 @@ This plugin supports the following configuration options plus the <<plugins-{typ
210
211
  | <<plugins-{type}s-{plugin}-sql_log_level>> |<<string,string>>, one of `["fatal", "error", "warn", "info", "debug"]`|No
211
212
  | <<plugins-{type}s-{plugin}-statement>> |<<string,string>>|No
212
213
  | <<plugins-{type}s-{plugin}-statement_filepath>> |a valid filesystem path|No
214
+ | <<plugins-{type}s-{plugin}-target>> | {logstash-ref}/field-references-deepdive.html[field reference] | No
213
215
  | <<plugins-{type}s-{plugin}-tracking_column>> |<<string,string>>|No
214
216
  | <<plugins-{type}s-{plugin}-tracking_column_type>> |<<string,string>>, one of `["numeric", "timestamp"]`|No
215
217
  | <<plugins-{type}s-{plugin}-use_column_value>> |<<boolean,boolean>>|No
@@ -534,6 +536,17 @@ with the `parameters` setting.
534
536
 
535
537
  Path of file containing statement to execute
536
538
 
539
+ [id="plugins-{type}s-{plugin}-target"]
540
+ ===== `target`
541
+
542
+ * Value type is {logstash-ref}/field-references-deepdive.html[field reference]
543
+ * There is no default value for this setting.
544
+
545
+ Without a `target`, events are created from each row column at the root level.
546
+ When the `target` is set to a field reference, the column of each row is placed in the target field instead.
547
+
548
+ This option can be useful to avoid populating unknown fields when a downstream schema such as ECS is enforced.
549
+
537
550
  [id="plugins-{type}s-{plugin}-tracking_column"]
538
551
  ===== `tracking_column`
539
552
 
@@ -56,6 +56,12 @@ module LogStash module Filters module Jdbc
56
56
  @target = options["target"]
57
57
  @id_used_as_target = @target.nil?
58
58
  if @id_used_as_target
59
+ # target shouldn't be nil if ecs_compatibility is not :disabled
60
+ if globals[:ecs_compatibility] != :disabled
61
+ logger.info('ECS compatibility is enabled but no ``target`` option was specified, it is recommended'\
62
+ ' to set the option to avoid potential schema conflicts (if your data is ECS compliant or'\
63
+ ' non-conflicting feel free to ignore this message)')
64
+ end
59
65
  @target = @id
60
66
  end
61
67
  @options = options
@@ -66,6 +72,7 @@ module LogStash module Filters module Jdbc
66
72
  @prepared_statement = nil
67
73
  @symbol_parameters = nil
68
74
  parse_options
75
+ @load_method_ref = method(:load_data_from_local)
69
76
  end
70
77
 
71
78
  def id_used_as_target?
@@ -81,11 +88,7 @@ module LogStash module Filters module Jdbc
81
88
  end
82
89
 
83
90
  def enhance(local, event)
84
- if @prepared_statement
85
- result = call_prepared(local, event)
86
- else
87
- result = fetch(local, event) # should return a LookupResult
88
- end
91
+ result = retrieve_local_data(local, event, &@load_method_ref) # return a LookupResult
89
92
  if result.failed? || result.parameters_invalid?
90
93
  tag_failure(event)
91
94
  end
@@ -112,6 +115,7 @@ module LogStash module Filters module Jdbc
112
115
  @prepared_parameters.each_with_index { |v, i| hash[:"$p#{i}"] = v }
113
116
  @prepared_param_placeholder_map = hash
114
117
  @prepared_statement = local.prepare(query, hash.keys)
118
+ @load_method_ref = method(:load_data_from_prepared)
115
119
  end
116
120
 
117
121
  private
@@ -128,34 +132,23 @@ module LogStash module Filters module Jdbc
128
132
  end
129
133
  end
130
134
 
131
- def fetch(local, event)
132
- result = LookupResult.new()
133
- if @parameters_specified
134
- params = prepare_parameters_from_event(event, result)
135
- if result.parameters_invalid?
136
- logger.warn? && logger.warn("Parameter field not found in event", :lookup_id => @id, :invalid_parameters => result.invalid_parameters)
137
- return result
138
- end
139
- else
140
- params = {}
135
+ def load_data_from_local(local, query, params, result)
136
+ local.fetch(query, params).each do |row|
137
+ stringified = row.inject({}){|hash,(k,v)| hash[k.to_s] = v; hash} #Stringify row keys
138
+ result.push(stringified)
141
139
  end
142
- begin
143
- logger.debug? && logger.debug("Executing Jdbc query", :lookup_id => @id, :statement => query, :parameters => params)
144
- local.fetch(query, params).each do |row|
145
- stringified = row.inject({}){|hash,(k,v)| hash[k.to_s] = v; hash} #Stringify row keys
146
- result.push(stringified)
147
- end
148
- rescue ::Sequel::Error => e
149
- # all sequel errors are a subclass of this, let all other standard or runtime errors bubble up
150
- result.failed!
151
- logger.warn? && logger.warn("Exception when executing Jdbc query", :lookup_id => @id, :exception => e.message, :backtrace => e.backtrace.take(8))
140
+ end
141
+
142
+ def load_data_from_prepared(_local, _query, params, result)
143
+ @prepared_statement.call(params).each do |row|
144
+ stringified = row.inject({}){|hash,(k,v)| hash[k.to_s] = v; hash} #Stringify row keys
145
+ result.push(stringified)
152
146
  end
153
- # if either of: no records or a Sequel exception occurs the payload is
154
- # empty and the default can be substituted later.
155
- result
156
147
  end
157
148
 
158
- def call_prepared(local, event)
149
+ # the &block is invoked with 4 arguments: local, query[String], params[Hash], result[LookupResult]
150
+ # the result is used as accumulator return variable
151
+ def retrieve_local_data(local, event, &proc)
159
152
  result = LookupResult.new()
160
153
  if @parameters_specified
161
154
  params = prepare_parameters_from_event(event, result)
@@ -168,10 +161,7 @@ module LogStash module Filters module Jdbc
168
161
  end
169
162
  begin
170
163
  logger.debug? && logger.debug("Executing Jdbc query", :lookup_id => @id, :statement => query, :parameters => params)
171
- @prepared_statement.call(params).each do |row|
172
- stringified = row.inject({}){|hash,(k,v)| hash[k.to_s] = v; hash} #Stringify row keys
173
- result.push(stringified)
174
- end
164
+ proc.call(local, query, params, result)
175
165
  rescue ::Sequel::Error => e
176
166
  # all sequel errors are a subclass of this, let all other standard or runtime errors bubble up
177
167
  result.failed!
@@ -2,6 +2,7 @@
2
2
  require "logstash-integration-jdbc_jars"
3
3
  require "logstash/filters/base"
4
4
  require "logstash/namespace"
5
+ require "logstash/plugin_mixins/ecs_compatibility_support"
5
6
  require_relative "jdbc/loader"
6
7
  require_relative "jdbc/loader_schedule"
7
8
  require_relative "jdbc/repeating_load_runner"
@@ -14,6 +15,9 @@ require_relative "jdbc/lookup_processor"
14
15
 
15
16
  #
16
17
  module LogStash module Filters class JdbcStatic < LogStash::Filters::Base
18
+ # adds ecs_compatibility config which could be :disabled or :v1
19
+ include LogStash::PluginMixins::ECSCompatibilitySupport(:disabled, :v1, :v8 => :v1)
20
+
17
21
  config_name "jdbc_static"
18
22
 
19
23
  # Define the loaders, an Array of Hashes, to fetch remote data and create local tables.
@@ -214,6 +218,7 @@ module LogStash module Filters class JdbcStatic < LogStash::Filters::Base
214
218
  options["lookup_jdbc_driver_class"] = @lookup_jdbc_driver_class
215
219
  options["lookup_jdbc_driver_library"] = @lookup_jdbc_driver_library
216
220
  options["lookup_jdbc_connection_string"] = @lookup_jdbc_connection_string
221
+ options["ecs_compatibility"] = ecs_compatibility
217
222
  options
218
223
  end
219
224
 
@@ -3,6 +3,8 @@ require "logstash/inputs/base"
3
3
  require "logstash/namespace"
4
4
  require "logstash/plugin_mixins/jdbc/common"
5
5
  require "logstash/plugin_mixins/jdbc/jdbc"
6
+ require "logstash/plugin_mixins/ecs_compatibility_support"
7
+ require "logstash/plugin_mixins/validator_support/field_reference_validation_adapter"
6
8
 
7
9
  # this require_relative returns early unless the JRuby version is between 9.2.0.0 and 9.2.8.0
8
10
  require_relative "tzinfo_jruby_patch"
@@ -129,6 +131,11 @@ require_relative "tzinfo_jruby_patch"
129
131
  module LogStash module Inputs class Jdbc < LogStash::Inputs::Base
130
132
  include LogStash::PluginMixins::Jdbc::Common
131
133
  include LogStash::PluginMixins::Jdbc::Jdbc
134
+ # adds ecs_compatibility config which could be :disabled or :v1
135
+ include LogStash::PluginMixins::ECSCompatibilitySupport(:disabled,:v1,:v8 => :v1)
136
+ # adds :field_reference validator adapter
137
+ extend LogStash::PluginMixins::ValidatorSupport::FieldReferenceValidationAdapter
138
+
132
139
  config_name "jdbc"
133
140
 
134
141
  # If undefined, Logstash will complain, even if codec is unused.
@@ -209,6 +216,9 @@ module LogStash module Inputs class Jdbc < LogStash::Inputs::Base
209
216
 
210
217
  config :prepared_statement_bind_values, :validate => :array, :default => []
211
218
 
219
+ # Define the target field to store the loaded columns
220
+ config :target, :validate => :field_reference, :required => false
221
+
212
222
  attr_reader :database # for test mocking/stubbing
213
223
 
214
224
  public
@@ -260,6 +270,13 @@ module LogStash module Inputs class Jdbc < LogStash::Inputs::Base
260
270
  converters[encoding] = converter
261
271
  end
262
272
  end
273
+
274
+ # target must be populated if ecs_compatibility is not :disabled
275
+ if @target.nil? && ecs_compatibility != :disabled
276
+ logger.info('ECS compatibility is enabled but no ``target`` option was specified, it is recommended'\
277
+ ' to set the option to avoid potential schema conflicts (if your data is ECS compliant or'\
278
+ ' non-conflicting feel free to ignore this message)')
279
+ end
263
280
  end # def register
264
281
 
265
282
  # test injection points
@@ -318,7 +335,12 @@ module LogStash module Inputs class Jdbc < LogStash::Inputs::Base
318
335
  ## do the necessary conversions to string elements
319
336
  row = Hash[row.map { |k, v| [k.to_s, convert(k, v)] }]
320
337
  end
321
- event = LogStash::Event.new(row)
338
+ if @target
339
+ event = LogStash::Event.new
340
+ event.set(@target, row)
341
+ else
342
+ event = LogStash::Event.new(row)
343
+ end
322
344
  decorate(event)
323
345
  queue << event
324
346
  end
@@ -122,16 +122,32 @@ module LogStash module PluginMixins module Jdbc
122
122
  # rescue Java::JavaSql::SQLException, ::Sequel::Error => e
123
123
  rescue ::Sequel::Error => e
124
124
  if retry_attempts <= 0
125
- @logger.error("Unable to connect to database. Tried #{@connection_retry_attempts} times", :error_message => e.message)
125
+ log_java_exception(e.cause)
126
+ @logger.error("Unable to connect to database. Tried #{@connection_retry_attempts} times", error_details(e, trace: true))
126
127
  raise e
127
128
  else
128
- @logger.error("Unable to connect to database. Trying again", :error_message => e.message)
129
+ @logger.error("Unable to connect to database. Trying again", error_details(e, trace: false))
129
130
  end
130
131
  end
131
132
  sleep(@connection_retry_attempts_wait_time)
132
133
  end
133
134
  end
134
135
 
136
+ def error_details(e, trace: false)
137
+ details = { :message => e.message, :exception => e.class }
138
+ details[:cause] = e.cause if e.cause
139
+ details[:backtrace] = e.backtrace if trace || @logger.debug?
140
+ details
141
+ end
142
+
143
+ def log_java_exception(e)
144
+ return unless e.is_a?(java.lang.Exception)
145
+ # @logger.name using the same convention as LS does
146
+ logger = self.class.name.gsub('::', '.').downcase
147
+ logger = org.apache.logging.log4j.LogManager.getLogger(logger)
148
+ logger.error('', e) # prints nested causes
149
+ end
150
+
135
151
  def open_jdbc_connection
136
152
  # at this point driver is already loaded
137
153
  Sequel.application_timezone = @plugin_timezone.to_sym
@@ -26,7 +26,7 @@ module LogStash module PluginMixins module JdbcStreaming
26
26
  return InterpolatedParameter.new(given_value)
27
27
  end
28
28
 
29
- if given_value =~ /\A\s*\[[^\]]+\]\s*\z/
29
+ if given_value =~ /\A(\s*\[[^\]]+\]\s*)*\z/
30
30
  return FieldParameter.new(given_value)
31
31
  end
32
32
 
@@ -1,6 +1,6 @@
1
1
  Gem::Specification.new do |s|
2
2
  s.name = 'logstash-integration-jdbc'
3
- s.version = '5.0.5'
3
+ s.version = '5.1.2'
4
4
  s.licenses = ['Apache License (2.0)']
5
5
  s.summary = "Integration with JDBC - input and filter plugins"
6
6
  s.description = "This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program"
@@ -36,6 +36,8 @@ Gem::Specification.new do |s|
36
36
  s.add_runtime_dependency 'tzinfo-data'
37
37
  # 3.5 limitation is required for jdbc-static loading schedule
38
38
  s.add_runtime_dependency 'rufus-scheduler', '< 3.5'
39
+ s.add_runtime_dependency 'logstash-mixin-ecs_compatibility_support', '~>1.2'
40
+ s.add_runtime_dependency "logstash-mixin-validator_support", '~> 1.0'
39
41
 
40
42
  s.add_development_dependency "childprocess"
41
43
  s.add_development_dependency 'logstash-devutils'
@@ -59,7 +59,7 @@ module LogStash module Filters
59
59
 
60
60
  let(:plugin) { JdbcStatic.new(settings) }
61
61
 
62
- let(:event) { ::LogStash::Event.new("message" => "some text", "ip" => ipaddr) }
62
+ let(:event) { ::LogStash::Event.new("message" => "some text", "ip" => ipaddr) }
63
63
 
64
64
  let(:ipaddr) { ".3.1.1" }
65
65
 
@@ -248,6 +248,37 @@ module LogStash module Filters module Jdbc
248
248
  expect(subject.valid?).to be_falsey
249
249
  end
250
250
  end
251
+
252
+ describe "validation of target option" do
253
+ let(:lookup_hash) do
254
+ {
255
+ "query" => "select * from servers WHERE ip LIKE ? AND os LIKE ?",
256
+ "prepared_parameters" => ["%%{[ip]}"],
257
+ }
258
+ end
259
+
260
+ it "should log a warn when ECS is enabled and target not defined" do
261
+
262
+ class LoggableLookup < Lookup
263
+
264
+ @@TEST_LOGGER = nil
265
+
266
+ def self.logger=(log)
267
+ @@TEST_LOGGER = log
268
+ end
269
+
270
+ def self.logger
271
+ @@TEST_LOGGER
272
+ end
273
+ end
274
+
275
+ spy_logger = double("logger")
276
+ expect(spy_logger).to receive(:info).once.with(/ECS compatibility is enabled but no .*?target.*? was specified/)
277
+ LoggableLookup.logger = spy_logger
278
+
279
+ LoggableLookup.new(lookup_hash, {:ecs_compatibility => 'v1'}, "lookup-1")
280
+ end
281
+ end
251
282
  end
252
283
  end end end
253
284
 
@@ -45,7 +45,7 @@ describe LogStash::Inputs::Jdbc, :integration => true do
45
45
 
46
46
  context "when supplying a non-existent library" do
47
47
  let(:settings) do
48
- super.merge(
48
+ super().merge(
49
49
  "jdbc_driver_library" => "/no/path/to/postgresql.jar"
50
50
  )
51
51
  end
@@ -61,13 +61,29 @@ describe LogStash::Inputs::Jdbc, :integration => true do
61
61
 
62
62
  context "when connecting to a non-existent server" do
63
63
  let(:settings) do
64
- super.merge(
64
+ super().merge(
65
65
  "jdbc_connection_string" => "jdbc:postgresql://localhost:65000/somedb"
66
66
  )
67
67
  end
68
68
 
69
69
  it "should not register correctly" do
70
70
  plugin.register
71
+ allow( plugin ).to receive(:log_java_exception)
72
+ q = Queue.new
73
+ expect do
74
+ plugin.run(q)
75
+ end.to raise_error(::Sequel::DatabaseConnectionError)
76
+ end
77
+
78
+ it "should log (native) Java driver error" do
79
+ plugin.register
80
+ expect( org.apache.logging.log4j.LogManager ).to receive(:getLogger).and_wrap_original do |m, *args|
81
+ logger = m.call(*args)
82
+ expect( logger ).to receive(:error) do |_, e|
83
+ expect( e ).to be_a org.postgresql.util.PSQLException
84
+ end.and_call_original
85
+ logger
86
+ end
71
87
  q = Queue.new
72
88
  expect do
73
89
  plugin.run(q)
@@ -328,6 +328,48 @@ describe LogStash::Inputs::Jdbc do
328
328
 
329
329
  end
330
330
 
331
+ context "when using target option" do
332
+ let(:settings) do
333
+ {
334
+ "statement" => "SELECT * from test_table FETCH FIRST 1 ROWS ONLY",
335
+ "target" => "sub_field"
336
+ }
337
+ end
338
+
339
+ before do
340
+ plugin.register
341
+ end
342
+
343
+ after do
344
+ plugin.stop
345
+ end
346
+
347
+ it "should put all columns under sub-field" do
348
+ db[:test_table].insert(:num => 1, :custom_time => Time.now.utc, :created_at => Time.now.utc, :string => "Test target option")
349
+
350
+ plugin.run(queue)
351
+
352
+ expect(queue.size).to eq(1)
353
+ event = queue.pop
354
+ expect(event.get("[sub_field][string]")).to eq("Test target option")
355
+ end
356
+ end
357
+
358
+ context "when using target option is not set and ecs_compatibility is enabled" do
359
+ let(:settings) do
360
+ {
361
+ "statement" => "SELECT * from test_table FETCH FIRST 1 ROWS ONLY",
362
+ "ecs_compatibility" => :v1
363
+ }
364
+ end
365
+
366
+ it "should log a warn of missed target usage" do
367
+ expect(plugin.logger).to receive(:info).once.with(/ECS compatibility is enabled but no .*?target.*? was specified/)
368
+
369
+ plugin.register
370
+ end
371
+ end
372
+
331
373
  context "when fetching time data" do
332
374
 
333
375
  let(:settings) do
@@ -0,0 +1,23 @@
1
+ # encoding: utf-8
2
+ require "logstash/devutils/rspec/spec_helper"
3
+ require "logstash/plugin_mixins/jdbc_streaming/parameter_handler"
4
+
5
+
6
+ describe LogStash::PluginMixins::JdbcStreaming::ParameterHandler do
7
+ context "resolve field reference" do
8
+ let(:event) { ::LogStash::Event.new("field" => "field_value") }
9
+
10
+ it "should resolve root field" do
11
+ handler = LogStash::PluginMixins::JdbcStreaming::ParameterHandler.build_bind_value_handler "[field]"
12
+ handler.extract_from(event)
13
+ expect(handler.extract_from(event)).to eq "field_value"
14
+ end
15
+
16
+ it "should resolve nested field" do
17
+ event = ::LogStash::Event.from_json("{\"field\": {\"nested\": \"nested_field\"}}").first
18
+ handler = LogStash::PluginMixins::JdbcStreaming::ParameterHandler.build_bind_value_handler "[field][nested]"
19
+ handler.extract_from(event)
20
+ expect(handler.extract_from(event)).to eq "nested_field"
21
+ end
22
+ end
23
+ end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: logstash-integration-jdbc
3
3
  version: !ruby/object:Gem::Version
4
- version: 5.0.5
4
+ version: 5.1.2
5
5
  platform: ruby
6
6
  authors:
7
7
  - Elastic
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2020-06-16 00:00:00.000000000 Z
11
+ date: 2021-06-28 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  requirement: !ruby/object:Gem::Requirement
@@ -142,6 +142,34 @@ dependencies:
142
142
  - - "<"
143
143
  - !ruby/object:Gem::Version
144
144
  version: '3.5'
145
+ - !ruby/object:Gem::Dependency
146
+ requirement: !ruby/object:Gem::Requirement
147
+ requirements:
148
+ - - "~>"
149
+ - !ruby/object:Gem::Version
150
+ version: '1.2'
151
+ name: logstash-mixin-ecs_compatibility_support
152
+ prerelease: false
153
+ type: :runtime
154
+ version_requirements: !ruby/object:Gem::Requirement
155
+ requirements:
156
+ - - "~>"
157
+ - !ruby/object:Gem::Version
158
+ version: '1.2'
159
+ - !ruby/object:Gem::Dependency
160
+ requirement: !ruby/object:Gem::Requirement
161
+ requirements:
162
+ - - "~>"
163
+ - !ruby/object:Gem::Version
164
+ version: '1.0'
165
+ name: logstash-mixin-validator_support
166
+ prerelease: false
167
+ type: :runtime
168
+ version_requirements: !ruby/object:Gem::Requirement
169
+ requirements:
170
+ - - "~>"
171
+ - !ruby/object:Gem::Version
172
+ version: '1.0'
145
173
  - !ruby/object:Gem::Dependency
146
174
  requirement: !ruby/object:Gem::Requirement
147
175
  requirements:
@@ -264,6 +292,7 @@ files:
264
292
  - spec/helpers/derbyrun.jar
265
293
  - spec/inputs/integration/integ_spec.rb
266
294
  - spec/inputs/jdbc_spec.rb
295
+ - spec/plugin_mixins/jdbc_streaming/parameter_handler_spec.rb
267
296
  - vendor/jar-dependencies/org/apache/derby/derby/10.14.1.0/derby-10.14.1.0.jar
268
297
  - vendor/jar-dependencies/org/apache/derby/derbyclient/10.14.1.0/derbyclient-10.14.1.0.jar
269
298
  homepage: http://www.elastic.co/guide/en/logstash/current/index.html
@@ -316,3 +345,4 @@ test_files:
316
345
  - spec/helpers/derbyrun.jar
317
346
  - spec/inputs/integration/integ_spec.rb
318
347
  - spec/inputs/jdbc_spec.rb
348
+ - spec/plugin_mixins/jdbc_streaming/parameter_handler_spec.rb