logstash-integration-jdbc 5.0.3 → 5.1.0

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 1c64a823a0973adc161faebb9a8d71fbaa19b8815ebac92dbb5a85dca53a9941
4
- data.tar.gz: effe43eb271c805374fd39084c54d4b522067a1cbaa6bc5f0efe5ad1a9264aa4
3
+ metadata.gz: 59bbd486f1af029b2e7fe07b9356ac1703d864fc9838c3e2275453bf669eae18
4
+ data.tar.gz: 7f2a890f71fe484bd36935491ca5797795b15bad86f9e52317e385c8546316a9
5
5
  SHA512:
6
- metadata.gz: 43b515624423bd156a471b51ed83545b91875cb364277ef79e5685ab0a019744dbe37227ec230a20f11b164df76187e4df3b4f7961aa1672ee8b36d1338e8ec8
7
- data.tar.gz: abf96d5cd6c97f3896e60395f5f390f6e67e308e168bdbc996d095ecb5a7e0eef6e89f224a91ee0f2908210b78e852b85ab3441f660fdda2cb6cd43b3374d74c
6
+ metadata.gz: dfb59eb31fd43a12e8baaa3c861f8558ed6158ed1693b7994f147ee035ee46e09c7f041648376cf11c510f86ef72221229944fdae1db3bdf3ec273318fbf6279
7
+ data.tar.gz: fae0dcec43219c1c98581ee0ff441818c4cafe8cb24d6cae08cc521f74ba4055a21178657fb3e989d9e9f2572c5a750bce975839f62f5309c70cbc096cb03e35
data/CHANGELOG.md CHANGED
@@ -1,3 +1,28 @@
1
+ ## 5.1.0
2
+ - Added `target` option to JDBC input, allowing the row columns to target a specific field instead of being expanded
3
+ at the root of the event. This allows the input to play nicer with the Elastic Common Schema when
4
+ the input does not follow the schema. [#69](https://github.com/logstash-plugins/logstash-integration-jdbc/issues/69)
5
+
6
+ - Added `target` to JDBC filter static `local_lookups` to verify it's properly valued when ECS is enabled.
7
+ [#71](https://github.com/logstash-plugins/logstash-integration-jdbc/issues/71)
8
+
9
+ ## 5.0.7
10
+ - Feat: try hard to log Java cause (chain) [#62](https://github.com/logstash-plugins/logstash-integration-jdbc/pull/62)
11
+
12
+ This allows seeing a full trace from the JDBC driver in case of connection errors.
13
+
14
+ - Refactored Lookup used in jdbc_streaming and jdbc_static to avoid code duplication. [#59](https://github.com/logstash-plugins/logstash-integration-jdbc/pull/59)
15
+
16
+ ## 5.0.6
17
+ - DOC:Replaced plugin_header file with plugin_header-integration file. [#40](https://github.com/logstash-plugins/logstash-integration-jdbc/pull/40)
18
+
19
+ ## 5.0.5
20
+ - Fixed user sequel_opts not being passed down properly [#37](https://github.com/logstash-plugins/logstash-integration-jdbc/pull/37)
21
+ - Refactored jdbc_streaming to share driver loading, so the fixes from the jdbc plugin also effect jdbc_streaming
22
+
23
+ ## 5.0.4
24
+ - Fixed issue where JDBC Drivers that don't correctly register with Java's DriverManager fail to load (such as Sybase) [#34](https://github.com/logstash-plugins/logstash-integration-jdbc/pull/34)
25
+
1
26
  ## 5.0.3
2
27
  - Fixed issue where a lost connection to the database can cause errors when using prepared statements with the scheduler [#25](https://github.com/logstash-plugins/logstash-integration-jdbc/pull/25)
3
28
 
data/README.md CHANGED
@@ -2,7 +2,7 @@
2
2
  Logstash Integration Plugin for JDBC, including Logstash Input and Filter Plugins
3
3
  # Logstash Plugin
4
4
 
5
- [![Travis Build Status](https://travis-ci.org/logstash-plugins/logstash-integration-jdbc.svg)](https://travis-ci.org/logstash-plugins/logstash-integration-jdbc)
5
+ [![Travis Build Status](https://travis-ci.com/logstash-plugins/logstash-integration-jdbc.svg)](https://travis-ci.com/logstash-plugins/logstash-integration-jdbc)
6
6
 
7
7
  This is a plugin for [Logstash](https://github.com/elastic/logstash).
8
8
 
@@ -1,3 +1,4 @@
1
+ :integration: jdbc
1
2
  :plugin: jdbc_static
2
3
  :type: filter
3
4
 
@@ -16,7 +17,7 @@ END - GENERATED VARIABLES, DO NOT EDIT!
16
17
 
17
18
  === Jdbc_static filter plugin
18
19
 
19
- include::{include_path}/plugin_header.asciidoc[]
20
+ include::{include_path}/plugin_header-integration.asciidoc[]
20
21
 
21
22
  ==== Description
22
23
 
@@ -421,7 +422,7 @@ according to the table below.
421
422
  |=======================================================================
422
423
  |Setting |Input type|Required
423
424
  | id|string|No
424
- | table|string|Yes
425
+ | local_table|string|Yes
425
426
  | query|string|Yes
426
427
  | max_rows|number|No
427
428
  | jdbc_connection_string|string|No
@@ -437,7 +438,7 @@ id::
437
438
  An optional identifier. This is used to identify the loader that is
438
439
  generating error messages and log lines.
439
440
 
440
- table::
441
+ local_table::
441
442
  The destination table in the local lookup database that the loader will fill.
442
443
 
443
444
  query::
@@ -1,3 +1,4 @@
1
+ :integration: jdbc
1
2
  :plugin: jdbc_streaming
2
3
  :type: filter
3
4
 
@@ -16,7 +17,7 @@ END - GENERATED VARIABLES, DO NOT EDIT!
16
17
 
17
18
  === Jdbc_streaming filter plugin
18
19
 
19
- include::{include_path}/plugin_header.asciidoc[]
20
+ include::{include_path}/plugin_header-integration.asciidoc[]
20
21
 
21
22
  ==== Description
22
23
 
@@ -1,3 +1,4 @@
1
+ :integration: jdbc
1
2
  :plugin: jdbc
2
3
  :type: input
3
4
  :default_codec: plain
@@ -17,7 +18,7 @@ END - GENERATED VARIABLES, DO NOT EDIT!
17
18
 
18
19
  === Jdbc input plugin
19
20
 
20
- include::{include_path}/plugin_header.asciidoc[]
21
+ include::{include_path}/plugin_header-integration.asciidoc[]
21
22
 
22
23
  ==== Description
23
24
 
@@ -210,6 +211,7 @@ This plugin supports the following configuration options plus the <<plugins-{typ
210
211
  | <<plugins-{type}s-{plugin}-sql_log_level>> |<<string,string>>, one of `["fatal", "error", "warn", "info", "debug"]`|No
211
212
  | <<plugins-{type}s-{plugin}-statement>> |<<string,string>>|No
212
213
  | <<plugins-{type}s-{plugin}-statement_filepath>> |a valid filesystem path|No
214
+ | <<plugins-{type}s-{plugin}-target>> | {logstash-ref}/field-references-deepdive.html[field reference] | No
213
215
  | <<plugins-{type}s-{plugin}-tracking_column>> |<<string,string>>|No
214
216
  | <<plugins-{type}s-{plugin}-tracking_column_type>> |<<string,string>>, one of `["numeric", "timestamp"]`|No
215
217
  | <<plugins-{type}s-{plugin}-use_column_value>> |<<boolean,boolean>>|No
@@ -534,6 +536,17 @@ with the `parameters` setting.
534
536
 
535
537
  Path of file containing statement to execute
536
538
 
539
+ [id="plugins-{type}s-{plugin}-target"]
540
+ ===== `target`
541
+
542
+ * Value type is {logstash-ref}/field-references-deepdive.html[field reference]
543
+ * There is no default value for this setting.
544
+
545
+ Without a `target`, events are created from each row column at the root level.
546
+ When the `target` is set to a field reference, the column of each row is placed in the target field instead.
547
+
548
+ This option can be useful to avoid populating unknown fields when a downstream schema such as ECS is enforced.
549
+
537
550
  [id="plugins-{type}s-{plugin}-tracking_column"]
538
551
  ===== `tracking_column`
539
552
 
@@ -95,7 +95,7 @@ module LogStash module Filters module Jdbc
95
95
  end
96
96
  begin
97
97
  db = nil
98
- ::Sequel::JDBC.load_driver(driver_class)
98
+ @options_hash[:driver] = ::Sequel::JDBC.load_driver(driver_class)
99
99
  @connection_string = connection_string
100
100
  if user
101
101
  @options_hash[:user] = user
@@ -56,6 +56,12 @@ module LogStash module Filters module Jdbc
56
56
  @target = options["target"]
57
57
  @id_used_as_target = @target.nil?
58
58
  if @id_used_as_target
59
+ # target shouldn't be nil if ecs_compatibility is not :disabled
60
+ if globals[:ecs_compatibility] != :disabled
61
+ logger.info('ECS compatibility is enabled but no ``target`` option was specified, it is recommended'\
62
+ ' to set the option to avoid potential schema conflicts (if your data is ECS compliant or'\
63
+ ' non-conflicting feel free to ignore this message)')
64
+ end
59
65
  @target = @id
60
66
  end
61
67
  @options = options
@@ -66,6 +72,7 @@ module LogStash module Filters module Jdbc
66
72
  @prepared_statement = nil
67
73
  @symbol_parameters = nil
68
74
  parse_options
75
+ @load_method_ref = method(:load_data_from_local)
69
76
  end
70
77
 
71
78
  def id_used_as_target?
@@ -81,11 +88,7 @@ module LogStash module Filters module Jdbc
81
88
  end
82
89
 
83
90
  def enhance(local, event)
84
- if @prepared_statement
85
- result = call_prepared(local, event)
86
- else
87
- result = fetch(local, event) # should return a LookupResult
88
- end
91
+ result = retrieve_local_data(local, event, &@load_method_ref) # return a LookupResult
89
92
  if result.failed? || result.parameters_invalid?
90
93
  tag_failure(event)
91
94
  end
@@ -112,6 +115,7 @@ module LogStash module Filters module Jdbc
112
115
  @prepared_parameters.each_with_index { |v, i| hash[:"$p#{i}"] = v }
113
116
  @prepared_param_placeholder_map = hash
114
117
  @prepared_statement = local.prepare(query, hash.keys)
118
+ @load_method_ref = method(:load_data_from_prepared)
115
119
  end
116
120
 
117
121
  private
@@ -128,34 +132,23 @@ module LogStash module Filters module Jdbc
128
132
  end
129
133
  end
130
134
 
131
- def fetch(local, event)
132
- result = LookupResult.new()
133
- if @parameters_specified
134
- params = prepare_parameters_from_event(event, result)
135
- if result.parameters_invalid?
136
- logger.warn? && logger.warn("Parameter field not found in event", :lookup_id => @id, :invalid_parameters => result.invalid_parameters)
137
- return result
138
- end
139
- else
140
- params = {}
135
+ def load_data_from_local(local, query, params, result)
136
+ local.fetch(query, params).each do |row|
137
+ stringified = row.inject({}){|hash,(k,v)| hash[k.to_s] = v; hash} #Stringify row keys
138
+ result.push(stringified)
141
139
  end
142
- begin
143
- logger.debug? && logger.debug("Executing Jdbc query", :lookup_id => @id, :statement => query, :parameters => params)
144
- local.fetch(query, params).each do |row|
145
- stringified = row.inject({}){|hash,(k,v)| hash[k.to_s] = v; hash} #Stringify row keys
146
- result.push(stringified)
147
- end
148
- rescue ::Sequel::Error => e
149
- # all sequel errors are a subclass of this, let all other standard or runtime errors bubble up
150
- result.failed!
151
- logger.warn? && logger.warn("Exception when executing Jdbc query", :lookup_id => @id, :exception => e.message, :backtrace => e.backtrace.take(8))
140
+ end
141
+
142
+ def load_data_from_prepared(_local, _query, params, result)
143
+ @prepared_statement.call(params).each do |row|
144
+ stringified = row.inject({}){|hash,(k,v)| hash[k.to_s] = v; hash} #Stringify row keys
145
+ result.push(stringified)
152
146
  end
153
- # if either of: no records or a Sequel exception occurs the payload is
154
- # empty and the default can be substituted later.
155
- result
156
147
  end
157
148
 
158
- def call_prepared(local, event)
149
+ # the &block is invoked with 4 arguments: local, query[String], params[Hash], result[LookupResult]
150
+ # the result is used as accumulator return variable
151
+ def retrieve_local_data(local, event, &proc)
159
152
  result = LookupResult.new()
160
153
  if @parameters_specified
161
154
  params = prepare_parameters_from_event(event, result)
@@ -168,10 +161,7 @@ module LogStash module Filters module Jdbc
168
161
  end
169
162
  begin
170
163
  logger.debug? && logger.debug("Executing Jdbc query", :lookup_id => @id, :statement => query, :parameters => params)
171
- @prepared_statement.call(params).each do |row|
172
- stringified = row.inject({}){|hash,(k,v)| hash[k.to_s] = v; hash} #Stringify row keys
173
- result.push(stringified)
174
- end
164
+ proc.call(local, query, params, result)
175
165
  rescue ::Sequel::Error => e
176
166
  # all sequel errors are a subclass of this, let all other standard or runtime errors bubble up
177
167
  result.failed!
@@ -2,6 +2,7 @@
2
2
  require "logstash-integration-jdbc_jars"
3
3
  require "logstash/filters/base"
4
4
  require "logstash/namespace"
5
+ require "logstash/plugin_mixins/ecs_compatibility_support"
5
6
  require_relative "jdbc/loader"
6
7
  require_relative "jdbc/loader_schedule"
7
8
  require_relative "jdbc/repeating_load_runner"
@@ -14,6 +15,9 @@ require_relative "jdbc/lookup_processor"
14
15
 
15
16
  #
16
17
  module LogStash module Filters class JdbcStatic < LogStash::Filters::Base
18
+ # adds ecs_compatibility config which could be :disabled or :v1
19
+ include LogStash::PluginMixins::ECSCompatibilitySupport(:disabled, :v1, :v8 => :v1)
20
+
17
21
  config_name "jdbc_static"
18
22
 
19
23
  # Define the loaders, an Array of Hashes, to fetch remote data and create local tables.
@@ -214,6 +218,7 @@ module LogStash module Filters class JdbcStatic < LogStash::Filters::Base
214
218
  options["lookup_jdbc_driver_class"] = @lookup_jdbc_driver_class
215
219
  options["lookup_jdbc_driver_library"] = @lookup_jdbc_driver_library
216
220
  options["lookup_jdbc_connection_string"] = @lookup_jdbc_connection_string
221
+ options["ecs_compatibility"] = ecs_compatibility
217
222
  options
218
223
  end
219
224
 
@@ -1,6 +1,7 @@
1
1
  # encoding: utf-8
2
2
  require "logstash/filters/base"
3
3
  require "logstash/namespace"
4
+ require "logstash/plugin_mixins/jdbc/common"
4
5
  require "logstash/plugin_mixins/jdbc_streaming"
5
6
  require "logstash/plugin_mixins/jdbc_streaming/cache_payload"
6
7
  require "logstash/plugin_mixins/jdbc_streaming/statement_handler"
@@ -46,6 +47,7 @@ require "lru_redux"
46
47
  # }
47
48
  #
48
49
  module LogStash module Filters class JdbcStreaming < LogStash::Filters::Base
50
+ include LogStash::PluginMixins::Jdbc::Common
49
51
  include LogStash::PluginMixins::JdbcStreaming
50
52
 
51
53
  config_name "jdbc_streaming"
@@ -1,7 +1,10 @@
1
1
  # encoding: utf-8
2
2
  require "logstash/inputs/base"
3
3
  require "logstash/namespace"
4
+ require "logstash/plugin_mixins/jdbc/common"
4
5
  require "logstash/plugin_mixins/jdbc/jdbc"
6
+ require "logstash/plugin_mixins/ecs_compatibility_support"
7
+ require "logstash/plugin_mixins/validator_support/field_reference_validation_adapter"
5
8
 
6
9
  # this require_relative returns early unless the JRuby version is between 9.2.0.0 and 9.2.8.0
7
10
  require_relative "tzinfo_jruby_patch"
@@ -126,7 +129,13 @@ require_relative "tzinfo_jruby_patch"
126
129
  # ---------------------------------------------------------------------------------------------------
127
130
  #
128
131
  module LogStash module Inputs class Jdbc < LogStash::Inputs::Base
132
+ include LogStash::PluginMixins::Jdbc::Common
129
133
  include LogStash::PluginMixins::Jdbc::Jdbc
134
+ # adds ecs_compatibility config which could be :disabled or :v1
135
+ include LogStash::PluginMixins::ECSCompatibilitySupport(:disabled,:v1,:v8 => :v1)
136
+ # adds :field_reference validator adapter
137
+ extend LogStash::PluginMixins::ValidatorSupport::FieldReferenceValidationAdapter
138
+
130
139
  config_name "jdbc"
131
140
 
132
141
  # If undefined, Logstash will complain, even if codec is unused.
@@ -207,6 +216,9 @@ module LogStash module Inputs class Jdbc < LogStash::Inputs::Base
207
216
 
208
217
  config :prepared_statement_bind_values, :validate => :array, :default => []
209
218
 
219
+ # Define the target field to store the loaded columns
220
+ config :target, :validate => :field_reference, :required => false
221
+
210
222
  attr_reader :database # for test mocking/stubbing
211
223
 
212
224
  public
@@ -258,6 +270,13 @@ module LogStash module Inputs class Jdbc < LogStash::Inputs::Base
258
270
  converters[encoding] = converter
259
271
  end
260
272
  end
273
+
274
+ # target must be populated if ecs_compatibility is not :disabled
275
+ if @target.nil? && ecs_compatibility != :disabled
276
+ logger.info('ECS compatibility is enabled but no ``target`` option was specified, it is recommended'\
277
+ ' to set the option to avoid potential schema conflicts (if your data is ECS compliant or'\
278
+ ' non-conflicting feel free to ignore this message)')
279
+ end
261
280
  end # def register
262
281
 
263
282
  # test injection points
@@ -316,7 +335,12 @@ module LogStash module Inputs class Jdbc < LogStash::Inputs::Base
316
335
  ## do the necessary conversions to string elements
317
336
  row = Hash[row.map { |k, v| [k.to_s, convert(k, v)] }]
318
337
  end
319
- event = LogStash::Event.new(row)
338
+ if @target
339
+ event = LogStash::Event.new
340
+ event.set(@target, row)
341
+ else
342
+ event = LogStash::Event.new(row)
343
+ end
320
344
  decorate(event)
321
345
  queue << event
322
346
  end
@@ -0,0 +1,66 @@
1
+
2
+ module LogStash module PluginMixins module Jdbc
3
+ module Common
4
+
5
+ private
6
+
7
+ def complete_sequel_opts(defaults = {})
8
+ sequel_opts = @sequel_opts.
9
+ map { |key,val| [key.is_a?(String) ? key.to_sym : key, val] }.
10
+ map { |key,val| [key, val.eql?('true') ? true : (val.eql?('false') ? false : val)] }
11
+ sequel_opts = defaults.merge Hash[sequel_opts]
12
+ sequel_opts[:user] = @jdbc_user unless @jdbc_user.nil? || @jdbc_user.empty?
13
+ sequel_opts[:password] = @jdbc_password.value unless @jdbc_password.nil?
14
+ sequel_opts[:driver] = @driver_impl # Sequel uses this as a fallback, if given URI doesn't auto-load the driver correctly
15
+ sequel_opts
16
+ end
17
+
18
+ def load_driver
19
+ return @driver_impl if @driver_impl ||= nil
20
+
21
+ require "java"
22
+ require "sequel"
23
+ require "sequel/adapters/jdbc"
24
+
25
+ load_driver_jars
26
+ begin
27
+ @driver_impl = Sequel::JDBC.load_driver(@jdbc_driver_class)
28
+ rescue Sequel::AdapterNotFound => e # Sequel::AdapterNotFound, "#{@jdbc_driver_class} not loaded"
29
+ # fix this !!!
30
+ message = if jdbc_driver_library_set?
31
+ "Are you sure you've included the correct jdbc driver in :jdbc_driver_library?"
32
+ else
33
+ ":jdbc_driver_library is not set, are you sure you included " +
34
+ "the proper driver client libraries in your classpath?"
35
+ end
36
+ raise LogStash::PluginLoadingError, "#{e}. #{message}"
37
+ end
38
+ end
39
+
40
+ def load_driver_jars
41
+ if jdbc_driver_library_set?
42
+ @jdbc_driver_library.split(",").each do |driver_jar|
43
+ @logger.debug("loading #{driver_jar}")
44
+ # load 'driver.jar' is different than load 'some.rb' as it only causes the file to be added to
45
+ # JRuby's class-loader lookup (class) path - won't raise a LoadError when file is not readable
46
+ unless FileTest.readable?(driver_jar)
47
+ raise LogStash::PluginLoadingError, "unable to load #{driver_jar} from :jdbc_driver_library, " +
48
+ "file not readable (please check user and group permissions for the path)"
49
+ end
50
+ begin
51
+ require driver_jar
52
+ rescue LoadError => e
53
+ raise LogStash::PluginLoadingError, "unable to load #{driver_jar} from :jdbc_driver_library, #{e.message}"
54
+ rescue StandardError => e
55
+ raise LogStash::PluginLoadingError, "unable to load #{driver_jar} from :jdbc_driver_library, #{e}"
56
+ end
57
+ end
58
+ end
59
+ end
60
+
61
+ def jdbc_driver_library_set?
62
+ !@jdbc_driver_library.nil? && !@jdbc_driver_library.empty?
63
+ end
64
+
65
+ end
66
+ end end end
@@ -106,17 +106,12 @@ module LogStash module PluginMixins module Jdbc
106
106
 
107
107
  private
108
108
  def jdbc_connect
109
- opts = {
110
- :user => @jdbc_user,
111
- :password => @jdbc_password.nil? ? nil : @jdbc_password.value,
112
- :pool_timeout => @jdbc_pool_timeout,
113
- :keep_reference => false
114
- }.merge(@sequel_opts)
109
+ sequel_opts = complete_sequel_opts(:pool_timeout => @jdbc_pool_timeout, :keep_reference => false)
115
110
  retry_attempts = @connection_retry_attempts
116
111
  loop do
117
112
  retry_attempts -= 1
118
113
  begin
119
- return Sequel.connect(@jdbc_connection_string, opts=opts)
114
+ return Sequel.connect(@jdbc_connection_string, sequel_opts)
120
115
  rescue Sequel::PoolTimeout => e
121
116
  if retry_attempts <= 0
122
117
  @logger.error("Failed to connect to database. #{@jdbc_pool_timeout} second timeout exceeded. Tried #{@connection_retry_attempts} times.")
@@ -127,64 +122,30 @@ module LogStash module PluginMixins module Jdbc
127
122
  # rescue Java::JavaSql::SQLException, ::Sequel::Error => e
128
123
  rescue ::Sequel::Error => e
129
124
  if retry_attempts <= 0
130
- @logger.error("Unable to connect to database. Tried #{@connection_retry_attempts} times", :error_message => e.message, )
125
+ log_java_exception(e.cause)
126
+ @logger.error("Unable to connect to database. Tried #{@connection_retry_attempts} times", error_details(e, trace: true))
131
127
  raise e
132
128
  else
133
- @logger.error("Unable to connect to database. Trying again", :error_message => e.message)
129
+ @logger.error("Unable to connect to database. Trying again", error_details(e, trace: false))
134
130
  end
135
131
  end
136
132
  sleep(@connection_retry_attempts_wait_time)
137
133
  end
138
134
  end
139
135
 
140
- private
141
-
142
- def load_driver
143
- if @drivers_loaded.false?
144
- require "java"
145
- require "sequel"
146
- require "sequel/adapters/jdbc"
147
-
148
- load_driver_jars
149
- begin
150
- Sequel::JDBC.load_driver(@jdbc_driver_class)
151
- rescue Sequel::AdapterNotFound => e # Sequel::AdapterNotFound, "#{@jdbc_driver_class} not loaded"
152
- # fix this !!!
153
- message = if jdbc_driver_library_set?
154
- "Are you sure you've included the correct jdbc driver in :jdbc_driver_library?"
155
- else
156
- ":jdbc_driver_library is not set, are you sure you included " +
157
- "the proper driver client libraries in your classpath?"
158
- end
159
- raise LogStash::PluginLoadingError, "#{e}. #{message}"
160
- end
161
- @drivers_loaded.make_true
162
- end
163
- end
164
-
165
- def load_driver_jars
166
- if jdbc_driver_library_set?
167
- @jdbc_driver_library.split(",").each do |driver_jar|
168
- @logger.debug("loading #{driver_jar}")
169
- # load 'driver.jar' is different than load 'some.rb' as it only causes the file to be added to
170
- # JRuby's class-loader lookup (class) path - won't raise a LoadError when file is not readable
171
- unless FileTest.readable?(driver_jar)
172
- raise LogStash::PluginLoadingError, "unable to load #{driver_jar} from :jdbc_driver_library, " +
173
- "file not readable (please check user and group permissions for the path)"
174
- end
175
- begin
176
- require driver_jar
177
- rescue LoadError => e
178
- raise LogStash::PluginLoadingError, "unable to load #{driver_jar} from :jdbc_driver_library, #{e.message}"
179
- rescue StandardError => e
180
- raise LogStash::PluginLoadingError, "unable to load #{driver_jar} from :jdbc_driver_library, #{e}"
181
- end
182
- end
183
- end
136
+ def error_details(e, trace: false)
137
+ details = { :message => e.message, :exception => e.class }
138
+ details[:cause] = e.cause if e.cause
139
+ details[:backtrace] = e.backtrace if trace || @logger.debug?
140
+ details
184
141
  end
185
142
 
186
- def jdbc_driver_library_set?
187
- !@jdbc_driver_library.nil? && !@jdbc_driver_library.empty?
143
+ def log_java_exception(e)
144
+ return unless e.is_a?(java.lang.Exception)
145
+ # @logger.name using the same convention as LS does
146
+ logger = self.class.name.gsub('::', '.').downcase
147
+ logger = org.apache.logging.log4j.LogManager.getLogger(logger)
148
+ logger.error('', e) # prints nested causes
188
149
  end
189
150
 
190
151
  def open_jdbc_connection
@@ -229,7 +190,6 @@ module LogStash module PluginMixins module Jdbc
229
190
  public
230
191
  def prepare_jdbc_connection
231
192
  @connection_lock = ReentrantLock.new
232
- @drivers_loaded = Concurrent::AtomicBoolean.new
233
193
  end
234
194
 
235
195
  public
@@ -55,37 +55,11 @@ module LogStash module PluginMixins module JdbcStreaming
55
55
  config :jdbc_validation_timeout, :validate => :number, :default => 3600
56
56
  end
57
57
 
58
- private
59
-
60
- def load_driver_jars
61
- unless @jdbc_driver_library.nil? || @jdbc_driver_library.empty?
62
- @jdbc_driver_library.split(",").each do |driver_jar|
63
- begin
64
- @logger.debug("loading #{driver_jar}")
65
- # Use https://github.com/jruby/jruby/wiki/CallingJavaFromJRuby#from-jar-files to make classes from jar
66
- # available
67
- require driver_jar
68
- rescue LoadError => e
69
- raise LogStash::PluginLoadingError, "unable to load #{driver_jar} from :jdbc_driver_library, #{e.message}"
70
- end
71
- end
72
- end
73
- end
74
-
75
58
  public
76
59
  def prepare_jdbc_connection
77
- require "sequel"
78
- require "sequel/adapters/jdbc"
79
- require "java"
80
-
81
- load_driver_jars
82
-
83
- @sequel_opts_symbols = @sequel_opts.inject({}) {|hash, (k,v)| hash[k.to_sym] = v; hash}
84
- @sequel_opts_symbols[:user] = @jdbc_user unless @jdbc_user.nil? || @jdbc_user.empty?
85
- @sequel_opts_symbols[:password] = @jdbc_password.value unless @jdbc_password.nil?
60
+ load_driver
86
61
 
87
- Sequel::JDBC.load_driver(@jdbc_driver_class)
88
- @database = Sequel.connect(@jdbc_connection_string, @sequel_opts_symbols)
62
+ @database = Sequel.connect(@jdbc_connection_string, complete_sequel_opts)
89
63
  if @jdbc_validate_connection
90
64
  @database.extension(:connection_validator)
91
65
  @database.pool.connection_validation_timeout = @jdbc_validation_timeout
@@ -1,6 +1,6 @@
1
1
  Gem::Specification.new do |s|
2
2
  s.name = 'logstash-integration-jdbc'
3
- s.version = '5.0.3'
3
+ s.version = '5.1.0'
4
4
  s.licenses = ['Apache License (2.0)']
5
5
  s.summary = "Integration with JDBC - input and filter plugins"
6
6
  s.description = "This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program"
@@ -36,6 +36,8 @@ Gem::Specification.new do |s|
36
36
  s.add_runtime_dependency 'tzinfo-data'
37
37
  # 3.5 limitation is required for jdbc-static loading schedule
38
38
  s.add_runtime_dependency 'rufus-scheduler', '< 3.5'
39
+ s.add_runtime_dependency 'logstash-mixin-ecs_compatibility_support', '~>1.2'
40
+ s.add_runtime_dependency "logstash-mixin-validator_support", '~> 1.0'
39
41
 
40
42
  s.add_development_dependency "childprocess"
41
43
  s.add_development_dependency 'logstash-devutils'
@@ -4,7 +4,6 @@ require "logstash/filters/jdbc_static"
4
4
  require "sequel"
5
5
  require "sequel/adapters/jdbc"
6
6
  require "stud/temporary"
7
- require "timecop"
8
7
 
9
8
  module LogStash module Filters
10
9
  describe JdbcStatic, :integration => true do
@@ -60,7 +59,7 @@ module LogStash module Filters
60
59
 
61
60
  let(:plugin) { JdbcStatic.new(settings) }
62
61
 
63
- let(:event) { ::LogStash::Event.new("message" => "some text", "ip" => ipaddr) }
62
+ let(:event) { ::LogStash::Event.new("message" => "some text", "ip" => ipaddr) }
64
63
 
65
64
  let(:ipaddr) { ".3.1.1" }
66
65
 
@@ -136,18 +135,16 @@ module LogStash module Filters
136
135
  describe "scheduled operation" do
137
136
  context "given a loader_schedule" do
138
137
  it "should properly schedule" do
139
- settings["loader_schedule"] = "*/10 * * * * * UTC"
140
- Timecop.travel(Time.now.utc - 3600)
141
- Timecop.scale(60)
138
+ settings["loader_schedule"] = "*/3 * * * * * UTC"
142
139
  static_filter = JdbcStatic.new(settings)
143
140
  runner = Thread.new(static_filter) do |filter|
144
141
  filter.register
145
142
  end
146
- sleep 3
143
+ runner.join(4)
144
+ sleep 4
147
145
  static_filter.filter(event)
148
146
  expect(static_filter.loader_runner.reload_count).to be > 1
149
147
  static_filter.close
150
- Timecop.return
151
148
  expect(event.get("server")).to eq([{"ip"=>"10.3.1.1", "name"=>"mv-server-1", "location"=>"MV-9-6-4"}])
152
149
  end
153
150
  end
@@ -248,6 +248,37 @@ module LogStash module Filters module Jdbc
248
248
  expect(subject.valid?).to be_falsey
249
249
  end
250
250
  end
251
+
252
+ describe "validation of target option" do
253
+ let(:lookup_hash) do
254
+ {
255
+ "query" => "select * from servers WHERE ip LIKE ? AND os LIKE ?",
256
+ "prepared_parameters" => ["%%{[ip]}"],
257
+ }
258
+ end
259
+
260
+ it "should log a warn when ECS is enabled and target not defined" do
261
+
262
+ class LoggableLookup < Lookup
263
+
264
+ @@TEST_LOGGER = nil
265
+
266
+ def self.logger=(log)
267
+ @@TEST_LOGGER = log
268
+ end
269
+
270
+ def self.logger
271
+ @@TEST_LOGGER
272
+ end
273
+ end
274
+
275
+ spy_logger = double("logger")
276
+ expect(spy_logger).to receive(:info).once.with(/ECS compatibility is enabled but no .*?target.*? was specified/)
277
+ LoggableLookup.logger = spy_logger
278
+
279
+ LoggableLookup.new(lookup_hash, {:ecs_compatibility => 'v1'}, "lookup-1")
280
+ end
281
+ end
251
282
  end
252
283
  end end end
253
284
 
@@ -10,28 +10,37 @@ module LogStash module Filters module Jdbc
10
10
  let(:driver_class) { "org.apache.derby.jdbc.EmbeddedDriver" }
11
11
  let(:driver_library) { nil }
12
12
  subject(:read_only_db) { described_class.create(connection_string, driver_class, driver_library) }
13
+ let(:stub_driver_class) { double(driver_class).as_null_object }
13
14
 
14
15
  describe "basic operations" do
15
16
  describe "initializing" do
17
+ before(:each) do
18
+ allow(Sequel::JDBC).to receive(:load_driver).and_return(stub_driver_class)
19
+ end
20
+
16
21
  it "tests the connection with defaults" do
17
22
  expect(Sequel::JDBC).to receive(:load_driver).once.with(driver_class)
18
- expect(Sequel).to receive(:connect).once.with(connection_string, {:test => true})
23
+ expect(Sequel).to receive(:connect).once.with(connection_string, {:test => true, :driver => stub_driver_class})
19
24
  expect(read_only_db.empty_record_set).to eq([])
20
25
  end
21
26
 
22
- it "tests the connection with fully specified arguments" do
23
- connection_str = "a connection string"
24
- user = "a user"
25
- password = Util::Password.new("secret")
26
- expect(Sequel::JDBC).to receive(:load_driver).once.with("a driver class")
27
- expect(Sequel).to receive(:connect).once.with(connection_str, {:user => user, :password => password.value, :test => true}).and_return(db)
28
- described_class.create(connection_str, "a driver class", nil, user, password)
27
+ context 'with fully-specified arguments' do
28
+ let(:connection_string) { "a connection string" }
29
+ let(:user) { "a user" }
30
+ let(:password) { Util::Password.new("secret") }
31
+ let(:driver_class) { "a driver class" }
32
+
33
+ it "tests the connection" do
34
+ expect(Sequel::JDBC).to receive(:load_driver).once.with(driver_class)
35
+ expect(Sequel).to receive(:connect).once.with(connection_string, {:user => user, :password => password.value, :test => true, :driver => stub_driver_class}).and_return(db)
36
+ described_class.create(connection_string, driver_class, nil, user, password)
37
+ end
29
38
  end
30
39
 
31
40
  it "connects with defaults" do
32
41
  expect(Sequel::JDBC).to receive(:load_driver).once.with(driver_class)
33
- expect(Sequel).to receive(:connect).once.with(connection_string, {:test => true}).and_return(db)
34
- expect(Sequel).to receive(:connect).once.with(connection_string, {}).and_return(db)
42
+ expect(Sequel).to receive(:connect).once.with(connection_string, {:test => true, :driver => stub_driver_class}).and_return(db)
43
+ expect(Sequel).to receive(:connect).once.with(connection_string, {:driver => stub_driver_class}).and_return(db)
35
44
  expect(read_only_db.connected?).to be_falsey
36
45
  read_only_db.connect("a caller specific error message")
37
46
  expect(read_only_db.connected?).to be_truthy
@@ -16,10 +16,11 @@ module LogStash module Filters module Jdbc
16
16
  describe "basic operations" do
17
17
  context "connecting to a db" do
18
18
  it "connects with defaults" do
19
- expect(::Sequel::JDBC).to receive(:load_driver).once.with("org.apache.derby.jdbc.EmbeddedDriver")
19
+ stub_driver_class = double('org.apache.derby.jdbc.EmbeddedDriver').as_null_object
20
+ expect(::Sequel::JDBC).to receive(:load_driver).once.with("org.apache.derby.jdbc.EmbeddedDriver").and_return(stub_driver_class)
20
21
  # two calls to connect because ReadWriteDatabase does verify_connection and connect
21
- expect(::Sequel).to receive(:connect).once.with(connection_string_regex, {:test => true}).and_return(db)
22
- expect(::Sequel).to receive(:connect).once.with(connection_string_regex, {}).and_return(db)
22
+ expect(::Sequel).to receive(:connect).once.with(connection_string_regex, {:driver => stub_driver_class, :test => true}).and_return(db)
23
+ expect(::Sequel).to receive(:connect).once.with(connection_string_regex, {:driver => stub_driver_class}).and_return(db)
23
24
  expect(read_write_db.empty_record_set).to eq([])
24
25
  end
25
26
 
@@ -27,9 +28,10 @@ module LogStash module Filters module Jdbc
27
28
  connection_str = "a connection string"
28
29
  user = "a user"
29
30
  password = Util::Password.new("secret")
30
- expect(::Sequel::JDBC).to receive(:load_driver).once.with("a driver class")
31
- expect(::Sequel).to receive(:connect).once.with(connection_str, {:user => user, :password => password.value, :test => true}).and_return(db)
32
- expect(::Sequel).to receive(:connect).once.with(connection_str, {:user => user, :password => password.value}).and_return(db)
31
+ stub_driver_class = double('com.example.Driver')
32
+ expect(::Sequel::JDBC).to receive(:load_driver).once.with("a driver class").and_return(stub_driver_class)
33
+ expect(::Sequel).to receive(:connect).once.with(connection_str, {:driver => stub_driver_class, :user => user, :password => password.value, :test => true}).and_return(db)
34
+ expect(::Sequel).to receive(:connect).once.with(connection_str, {:driver => stub_driver_class, :user => user, :password => password.value}).and_return(db)
33
35
  described_class.create(connection_str, "a driver class", nil, user, password)
34
36
  end
35
37
  end
@@ -45,7 +45,7 @@ describe LogStash::Inputs::Jdbc, :integration => true do
45
45
 
46
46
  context "when supplying a non-existent library" do
47
47
  let(:settings) do
48
- super.merge(
48
+ super().merge(
49
49
  "jdbc_driver_library" => "/no/path/to/postgresql.jar"
50
50
  )
51
51
  end
@@ -61,13 +61,29 @@ describe LogStash::Inputs::Jdbc, :integration => true do
61
61
 
62
62
  context "when connecting to a non-existent server" do
63
63
  let(:settings) do
64
- super.merge(
64
+ super().merge(
65
65
  "jdbc_connection_string" => "jdbc:postgresql://localhost:65000/somedb"
66
66
  )
67
67
  end
68
68
 
69
69
  it "should not register correctly" do
70
70
  plugin.register
71
+ allow( plugin ).to receive(:log_java_exception)
72
+ q = Queue.new
73
+ expect do
74
+ plugin.run(q)
75
+ end.to raise_error(::Sequel::DatabaseConnectionError)
76
+ end
77
+
78
+ it "should log (native) Java driver error" do
79
+ plugin.register
80
+ expect( org.apache.logging.log4j.LogManager ).to receive(:getLogger).and_wrap_original do |m, *args|
81
+ logger = m.call(*args)
82
+ expect( logger ).to receive(:error) do |_, e|
83
+ expect( e ).to be_a org.postgresql.util.PSQLException
84
+ end.and_call_original
85
+ logger
86
+ end
71
87
  q = Queue.new
72
88
  expect do
73
89
  plugin.run(q)
@@ -13,9 +13,13 @@ require "date"
13
13
  # We do not need to set TZ env var anymore because we can have 'Sequel.application_timezone' set to utc by default now.
14
14
 
15
15
  describe LogStash::Inputs::Jdbc do
16
+ let(:connection_string) { "jdbc:derby:memory:testdb;create=true" }
16
17
  let(:mixin_settings) do
17
- { "jdbc_user" => ENV['USER'], "jdbc_driver_class" => "org.apache.derby.jdbc.EmbeddedDriver",
18
- "jdbc_connection_string" => "jdbc:derby:memory:testdb;create=true"}
18
+ {
19
+ "jdbc_user" => ENV['USER'],
20
+ "jdbc_driver_class" => "org.apache.derby.jdbc.EmbeddedDriver",
21
+ "jdbc_connection_string" => connection_string
22
+ }
19
23
  end
20
24
  let(:settings) { {} }
21
25
  let(:plugin) { LogStash::Inputs::Jdbc.new(mixin_settings.merge(settings)) }
@@ -112,6 +116,57 @@ describe LogStash::Inputs::Jdbc do
112
116
  end
113
117
  end
114
118
 
119
+ context "when sequel opts has user option" do
120
+ let(:settings) do
121
+ {
122
+ "jdbc_user" => 'system',
123
+ "statement" => "SELECT * from test_table",
124
+ "sequel_opts" => { "user" => 'from-opts' }
125
+ }
126
+ end
127
+
128
+ before do
129
+ plugin.register
130
+ end
131
+
132
+ after do
133
+ plugin.stop
134
+ end
135
+
136
+ it "should honor set jdbc-user when connecting" do
137
+ expect( Sequel ).to receive(:connect).with connection_string, hash_including(:user=>"system")
138
+ plugin.send(:jdbc_connect)
139
+ end
140
+ end
141
+
142
+ context "with sequel opts" do
143
+ let(:settings) do
144
+ {
145
+ "jdbc_user" => 'system',
146
+ "statement" => "SELECT * from test_table",
147
+ "sequel_opts" => {
148
+ "truthy" => 'true',
149
+ "falsey" => 'false',
150
+ "foo" => 'bar',
151
+ "jdbc_properties" => { "some" => 'true' }
152
+ }
153
+ }
154
+ end
155
+
156
+ before do
157
+ plugin.register
158
+ end
159
+
160
+ after do
161
+ plugin.stop
162
+ end
163
+
164
+ it "should symbolize keys" do
165
+ expect( Sequel ).to receive(:connect).with connection_string,
166
+ hash_including(:truthy => true, :falsey => false, :foo => 'bar', :jdbc_properties => { 'some' => 'true' })
167
+ plugin.send(:jdbc_connect)
168
+ end
169
+ end
115
170
 
116
171
  context "when neither statement and statement_filepath arguments are passed" do
117
172
  it "should fail to register" do
@@ -273,6 +328,48 @@ describe LogStash::Inputs::Jdbc do
273
328
 
274
329
  end
275
330
 
331
+ context "when using target option" do
332
+ let(:settings) do
333
+ {
334
+ "statement" => "SELECT * from test_table FETCH FIRST 1 ROWS ONLY",
335
+ "target" => "sub_field"
336
+ }
337
+ end
338
+
339
+ before do
340
+ plugin.register
341
+ end
342
+
343
+ after do
344
+ plugin.stop
345
+ end
346
+
347
+ it "should put all columns under sub-field" do
348
+ db[:test_table].insert(:num => 1, :custom_time => Time.now.utc, :created_at => Time.now.utc, :string => "Test target option")
349
+
350
+ plugin.run(queue)
351
+
352
+ expect(queue.size).to eq(1)
353
+ event = queue.pop
354
+ expect(event.get("[sub_field][string]")).to eq("Test target option")
355
+ end
356
+ end
357
+
358
+ context "when using target option is not set and ecs_compatibility is enabled" do
359
+ let(:settings) do
360
+ {
361
+ "statement" => "SELECT * from test_table FETCH FIRST 1 ROWS ONLY",
362
+ "ecs_compatibility" => :v1
363
+ }
364
+ end
365
+
366
+ it "should log a warn of missed target usage" do
367
+ expect(plugin.logger).to receive(:info).once.with(/ECS compatibility is enabled but no .*?target.*? was specified/)
368
+
369
+ plugin.register
370
+ end
371
+ end
372
+
276
373
  context "when fetching time data" do
277
374
 
278
375
  let(:settings) do
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: logstash-integration-jdbc
3
3
  version: !ruby/object:Gem::Version
4
- version: 5.0.3
4
+ version: 5.1.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Elastic
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2020-05-20 00:00:00.000000000 Z
11
+ date: 2021-05-26 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  requirement: !ruby/object:Gem::Requirement
@@ -142,6 +142,34 @@ dependencies:
142
142
  - - "<"
143
143
  - !ruby/object:Gem::Version
144
144
  version: '3.5'
145
+ - !ruby/object:Gem::Dependency
146
+ requirement: !ruby/object:Gem::Requirement
147
+ requirements:
148
+ - - "~>"
149
+ - !ruby/object:Gem::Version
150
+ version: '1.2'
151
+ name: logstash-mixin-ecs_compatibility_support
152
+ prerelease: false
153
+ type: :runtime
154
+ version_requirements: !ruby/object:Gem::Requirement
155
+ requirements:
156
+ - - "~>"
157
+ - !ruby/object:Gem::Version
158
+ version: '1.2'
159
+ - !ruby/object:Gem::Dependency
160
+ requirement: !ruby/object:Gem::Requirement
161
+ requirements:
162
+ - - "~>"
163
+ - !ruby/object:Gem::Version
164
+ version: '1.0'
165
+ name: logstash-mixin-validator_support
166
+ prerelease: false
167
+ type: :runtime
168
+ version_requirements: !ruby/object:Gem::Requirement
169
+ requirements:
170
+ - - "~>"
171
+ - !ruby/object:Gem::Version
172
+ version: '1.0'
145
173
  - !ruby/object:Gem::Dependency
146
174
  requirement: !ruby/object:Gem::Requirement
147
175
  requirements:
@@ -234,6 +262,7 @@ files:
234
262
  - lib/logstash/inputs/jdbc.rb
235
263
  - lib/logstash/inputs/tzinfo_jruby_patch.rb
236
264
  - lib/logstash/plugin_mixins/jdbc/checked_count_logger.rb
265
+ - lib/logstash/plugin_mixins/jdbc/common.rb
237
266
  - lib/logstash/plugin_mixins/jdbc/jdbc.rb
238
267
  - lib/logstash/plugin_mixins/jdbc/statement_handler.rb
239
268
  - lib/logstash/plugin_mixins/jdbc/value_tracking.rb