logstash-integration-jdbc 5.1.10 → 5.2.0
Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: a41fe10947372ac69f9db7a66dd1c285000f27bba0cb77ef7ec62629821fa5b3
|
4
|
+
data.tar.gz: 74b411ebe9f448259da6751def456535bf6819f84c80027a629354140fc5c779
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: '029c86dfb0a4099979c77443c79edc7ebfb1cc3572d54c43c9864642aa018892cede9e749685846ad3451defabb1cea1cc770f3c4062afaa06690a67c0a1d498'
|
7
|
+
data.tar.gz: 3c60fa57ef6c77bee21be8c3eac9b12c5f32af0b9fa7b4874dd8503ce6926b87ff139a2293a2276156df0ae6ec20ce238ed467ee803bc510452ab2d732d090a7
|
data/CHANGELOG.md
CHANGED
@@ -1,7 +1,13 @@
|
|
1
|
+
## 5.2.0
|
2
|
+
- Added `jdbc_paging_mode` option to choose if use `explicit` pagination in statements and avoid the initial count
|
3
|
+
query or use `auto` to delegate to the underlying library [#95](https://github.com/logstash-plugins/logstash-integration-jdbc/pull/95)
|
4
|
+
|
1
5
|
## 5.1.10
|
2
6
|
- Refactor: to explicit Java (driver) class name loading [#96](https://github.com/logstash-plugins/logstash-integration-jdbc/pull/96),
|
3
7
|
the change is expected to provide a more robust fix for the driver loading issue [#83](https://github.com/logstash-plugins/logstash-integration-jdbc/issues/83).
|
4
8
|
|
9
|
+
NOTE: a fatal driver error will no longer keep reloading the pipeline and now leads to a system exit.
|
10
|
+
|
5
11
|
- Fix: regression due returning the Java driver class [#98](https://github.com/logstash-plugins/logstash-integration-jdbc/pull/98)
|
6
12
|
|
7
13
|
## 5.1.9 (yanked)
|
data/docs/input-jdbc.asciidoc
CHANGED
@@ -129,6 +129,9 @@ Here is the list:
|
|
129
129
|
|sql_last_value | The value used to calculate which rows to query. Before any query is run,
|
130
130
|
this is set to Thursday, 1 January 1970, or 0 if `use_column_value` is true and
|
131
131
|
`tracking_column` is set. It is updated accordingly after subsequent queries are run.
|
132
|
+
|offset, size| Values used with manual paging mode to explicitly implement the paging.
|
133
|
+
Supported only if <<plugins-{type}s-{plugin}-jdbc_paging_enabled>> is enabled and
|
134
|
+
<<plugins-{type}s-{plugin}-jdbc_paging_mode>> has the `explicit` value.
|
132
135
|
|==========================================================
|
133
136
|
|
134
137
|
Example:
|
@@ -153,7 +156,7 @@ NOTE: Not all JDBC accessible technologies will support prepared statements.
|
|
153
156
|
With the introduction of Prepared Statement support comes a different code execution path and some new settings. Most of the existing settings are still useful but there are several new settings for Prepared Statements to read up on.
|
154
157
|
Use the boolean setting `use_prepared_statements` to enable this execution mode. Use the `prepared_statement_name` setting to specify a name for the Prepared Statement, this identifies the prepared statement locally and remotely and it should be unique in your config and on the database. Use the `prepared_statement_bind_values` array setting to specify the bind values, use the exact string `:sql_last_value` (multiple times if necessary) for the predefined parameter mentioned before. The `statement` (or `statement_path`) setting still holds the SQL statement but to use bind variables you must use the `?` character as a placeholder in the exact order found in the `prepared_statement_bind_values` array.
|
155
158
|
|
156
|
-
NOTE: Building count queries around a prepared statement is not supported at this time
|
159
|
+
NOTE: Building count queries around a prepared statement is not supported at this time. Because jdbc paging uses count queries when `jdbc_paging_mode` has value `auto`,jdbc paging is not supported with prepared statements at this time either. Therefore, `jdbc_paging_enabled`, `jdbc_page_size` settings are ignored when using prepared statements.
|
157
160
|
|
158
161
|
Example:
|
159
162
|
[source,ruby]
|
@@ -193,6 +196,7 @@ This plugin supports the following configuration options plus the <<plugins-{typ
|
|
193
196
|
| <<plugins-{type}s-{plugin}-jdbc_fetch_size>> |<<number,number>>|No
|
194
197
|
| <<plugins-{type}s-{plugin}-jdbc_page_size>> |<<number,number>>|No
|
195
198
|
| <<plugins-{type}s-{plugin}-jdbc_paging_enabled>> |<<boolean,boolean>>|No
|
199
|
+
| <<plugins-{type}s-{plugin}-jdbc_paging_mode>> |<<string,string>>, one of `["auto", "explicit"]`|No
|
196
200
|
| <<plugins-{type}s-{plugin}-jdbc_password>> |<<password,password>>|No
|
197
201
|
| <<plugins-{type}s-{plugin}-jdbc_password_filepath>> |a valid filesystem path|No
|
198
202
|
| <<plugins-{type}s-{plugin}-jdbc_pool_timeout>> |<<number,number>>|No
|
@@ -373,6 +377,52 @@ result-set. The limit size is set with `jdbc_page_size`.
|
|
373
377
|
|
374
378
|
Be aware that ordering is not guaranteed between queries.
|
375
379
|
|
380
|
+
[id="plugins-{type}s-{plugin}-jdbc_paging_mode"]
|
381
|
+
===== `jdbc_paging_mode`
|
382
|
+
|
383
|
+
* Value can be any of: `auto`, `explicit`
|
384
|
+
* Default value is `"auto"`
|
385
|
+
|
386
|
+
Whether to use `explicit` or `auto` mode during the JDBC paging
|
387
|
+
|
388
|
+
If `auto`, your statement will be automatically surrounded by a count query and subsequent multiple paged queries (with `LIMIT` statement, etc.).
|
389
|
+
|
390
|
+
If `explicit`, multiple queries (without a count query ahead) will be performed with your statement, until no more rows are retrieved.
|
391
|
+
You have to write your own paging conditions in your statement configuration.
|
392
|
+
The `offset` and `size` parameters can be used in your statement (`size` equal to `jdbc_page_size`, and `offset` incremented by `size` for each query).
|
393
|
+
When the number of rows returned by the query is not equal to `size`, SQL paging will be ended.
|
394
|
+
Example:
|
395
|
+
|
396
|
+
[source, ruby]
|
397
|
+
------------------------------------------------------
|
398
|
+
input {
|
399
|
+
jdbc {
|
400
|
+
statement => "SELECT id, mycolumn1, mycolumn2 FROM my_table WHERE id > :sql_last_value LIMIT :size OFFSET :offset",
|
401
|
+
jdbc_paging_enabled => true,
|
402
|
+
jdbc_paging_mode => "explicit",
|
403
|
+
jdbc_page_size => 100000
|
404
|
+
}
|
405
|
+
}
|
406
|
+
------------------------------------------------------
|
407
|
+
|
408
|
+
[source, ruby]
|
409
|
+
------------------------------------------------------
|
410
|
+
input {
|
411
|
+
jdbc {
|
412
|
+
statement => "CALL fetch_my_data(:sql_last_value, :offset, :size)",
|
413
|
+
jdbc_paging_enabled => true,
|
414
|
+
jdbc_paging_mode => "explicit",
|
415
|
+
jdbc_page_size => 100000
|
416
|
+
}
|
417
|
+
}
|
418
|
+
------------------------------------------------------
|
419
|
+
|
420
|
+
This mode can be considered in the following situations:
|
421
|
+
|
422
|
+
. Performance issues encountered in default paging mode.
|
423
|
+
. Your SQL statement is complex, so simply surrounding it with paging statements is not what you want.
|
424
|
+
. Your statement is a stored procedure, and the actual paging statement is inside it.
|
425
|
+
|
376
426
|
[id="plugins-{type}s-{plugin}-jdbc_password"]
|
377
427
|
===== `jdbc_password`
|
378
428
|
|
@@ -55,6 +55,9 @@ module LogStash module PluginMixins module Jdbc
|
|
55
55
|
# Be aware that ordering is not guaranteed between queries.
|
56
56
|
config :jdbc_paging_enabled, :validate => :boolean, :default => false
|
57
57
|
|
58
|
+
# Which pagination mode to use, automatic pagination or explicitly defined in the query.
|
59
|
+
config :jdbc_paging_mode, :validate => [ "auto", "explicit" ], :default => "auto"
|
60
|
+
|
58
61
|
# JDBC page size
|
59
62
|
config :jdbc_page_size, :validate => :number, :default => 100000
|
60
63
|
|
@@ -3,7 +3,15 @@
|
|
3
3
|
module LogStash module PluginMixins module Jdbc
|
4
4
|
class StatementHandler
|
5
5
|
def self.build_statement_handler(plugin, logger)
|
6
|
-
|
6
|
+
if plugin.use_prepared_statements
|
7
|
+
klass = PreparedStatementHandler
|
8
|
+
else
|
9
|
+
if plugin.jdbc_paging_enabled && plugin.jdbc_paging_mode == "explicit"
|
10
|
+
klass = ExplicitPagingModeStatementHandler
|
11
|
+
else
|
12
|
+
klass = NormalStatementHandler
|
13
|
+
end
|
14
|
+
end
|
7
15
|
klass.new(plugin, logger)
|
8
16
|
end
|
9
17
|
|
@@ -27,7 +35,9 @@ module LogStash module PluginMixins module Jdbc
|
|
27
35
|
class NormalStatementHandler < StatementHandler
|
28
36
|
# Performs the query, respecting our pagination settings, yielding once per row of data
|
29
37
|
# @param db [Sequel::Database]
|
30
|
-
# @param sql_last_value [
|
38
|
+
# @param sql_last_value [Integer|DateTime|Time]
|
39
|
+
# @param jdbc_paging_enabled [Boolean]
|
40
|
+
# @param jdbc_page_size [Integer]
|
31
41
|
# @yieldparam row [Hash{Symbol=>Object}]
|
32
42
|
def perform_query(db, sql_last_value, jdbc_paging_enabled, jdbc_page_size)
|
33
43
|
query = build_query(db, sql_last_value)
|
@@ -67,6 +77,28 @@ module LogStash module PluginMixins module Jdbc
|
|
67
77
|
end
|
68
78
|
end
|
69
79
|
|
80
|
+
class ExplicitPagingModeStatementHandler < NormalStatementHandler
|
81
|
+
# Performs the query, respecting our pagination settings, yielding once per row of data
|
82
|
+
# @param db [Sequel::Database]
|
83
|
+
# @param sql_last_value [Integer|DateTime|Time]
|
84
|
+
# @param jdbc_paging_enabled [Boolean]
|
85
|
+
# @param jdbc_page_size [Integer]
|
86
|
+
# @yieldparam row [Hash{Symbol=>Object}]
|
87
|
+
def perform_query(db, sql_last_value, jdbc_paging_enabled, jdbc_page_size)
|
88
|
+
query = build_query(db, sql_last_value)
|
89
|
+
offset = 0
|
90
|
+
loop do
|
91
|
+
rows_in_page = 0
|
92
|
+
query.with_sql(query.sql, offset: offset, size: jdbc_page_size).each do |row|
|
93
|
+
yield row
|
94
|
+
rows_in_page += 1
|
95
|
+
end
|
96
|
+
break unless rows_in_page == jdbc_page_size
|
97
|
+
offset += jdbc_page_size
|
98
|
+
end
|
99
|
+
end
|
100
|
+
end
|
101
|
+
|
70
102
|
class PreparedStatementHandler < StatementHandler
|
71
103
|
attr_reader :name, :bind_values_array, :statement_prepared, :prepared
|
72
104
|
|
@@ -1,6 +1,6 @@
|
|
1
1
|
Gem::Specification.new do |s|
|
2
2
|
s.name = 'logstash-integration-jdbc'
|
3
|
-
s.version = '5.
|
3
|
+
s.version = '5.2.0'
|
4
4
|
s.licenses = ['Apache License (2.0)']
|
5
5
|
s.summary = "Integration with JDBC - input and filter plugins"
|
6
6
|
s.description = "This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program"
|
data/spec/inputs/jdbc_spec.rb
CHANGED
@@ -329,6 +329,39 @@ describe LogStash::Inputs::Jdbc do
|
|
329
329
|
|
330
330
|
end
|
331
331
|
|
332
|
+
context "when iterating result-set via explicit paging mode" do
|
333
|
+
|
334
|
+
let(:settings) do
|
335
|
+
{
|
336
|
+
"statement" => "SELECT * from test_table OFFSET :offset ROWS FETCH NEXT :size ROWS ONLY",
|
337
|
+
"jdbc_paging_enabled" => true,
|
338
|
+
"jdbc_paging_mode" => "explicit",
|
339
|
+
"jdbc_page_size" => 10
|
340
|
+
}
|
341
|
+
end
|
342
|
+
|
343
|
+
let(:num_rows) { 15 }
|
344
|
+
|
345
|
+
before do
|
346
|
+
plugin.register
|
347
|
+
end
|
348
|
+
|
349
|
+
after do
|
350
|
+
plugin.stop
|
351
|
+
end
|
352
|
+
|
353
|
+
it "should fetch all rows" do
|
354
|
+
num_rows.times do
|
355
|
+
db[:test_table].insert(:num => 1, :custom_time => Time.now.utc, :created_at => Time.now.utc)
|
356
|
+
end
|
357
|
+
|
358
|
+
plugin.run(queue)
|
359
|
+
|
360
|
+
expect(queue.size).to eq(num_rows)
|
361
|
+
end
|
362
|
+
|
363
|
+
end
|
364
|
+
|
332
365
|
context "when using target option" do
|
333
366
|
let(:settings) do
|
334
367
|
{
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: logstash-integration-jdbc
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 5.
|
4
|
+
version: 5.2.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Elastic
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2021-12-
|
11
|
+
date: 2021-12-22 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
requirement: !ruby/object:Gem::Requirement
|