logstash-integration-jdbc 5.1.10 → 5.2.0
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: a41fe10947372ac69f9db7a66dd1c285000f27bba0cb77ef7ec62629821fa5b3
|
4
|
+
data.tar.gz: 74b411ebe9f448259da6751def456535bf6819f84c80027a629354140fc5c779
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: '029c86dfb0a4099979c77443c79edc7ebfb1cc3572d54c43c9864642aa018892cede9e749685846ad3451defabb1cea1cc770f3c4062afaa06690a67c0a1d498'
|
7
|
+
data.tar.gz: 3c60fa57ef6c77bee21be8c3eac9b12c5f32af0b9fa7b4874dd8503ce6926b87ff139a2293a2276156df0ae6ec20ce238ed467ee803bc510452ab2d732d090a7
|
data/CHANGELOG.md
CHANGED
@@ -1,7 +1,13 @@
|
|
1
|
+
## 5.2.0
|
2
|
+
- Added `jdbc_paging_mode` option to choose if use `explicit` pagination in statements and avoid the initial count
|
3
|
+
query or use `auto` to delegate to the underlying library [#95](https://github.com/logstash-plugins/logstash-integration-jdbc/pull/95)
|
4
|
+
|
1
5
|
## 5.1.10
|
2
6
|
- Refactor: to explicit Java (driver) class name loading [#96](https://github.com/logstash-plugins/logstash-integration-jdbc/pull/96),
|
3
7
|
the change is expected to provide a more robust fix for the driver loading issue [#83](https://github.com/logstash-plugins/logstash-integration-jdbc/issues/83).
|
4
8
|
|
9
|
+
NOTE: a fatal driver error will no longer keep reloading the pipeline and now leads to a system exit.
|
10
|
+
|
5
11
|
- Fix: regression due returning the Java driver class [#98](https://github.com/logstash-plugins/logstash-integration-jdbc/pull/98)
|
6
12
|
|
7
13
|
## 5.1.9 (yanked)
|
data/docs/input-jdbc.asciidoc
CHANGED
@@ -129,6 +129,9 @@ Here is the list:
|
|
129
129
|
|sql_last_value | The value used to calculate which rows to query. Before any query is run,
|
130
130
|
this is set to Thursday, 1 January 1970, or 0 if `use_column_value` is true and
|
131
131
|
`tracking_column` is set. It is updated accordingly after subsequent queries are run.
|
132
|
+
|offset, size| Values used with manual paging mode to explicitly implement the paging.
|
133
|
+
Supported only if <<plugins-{type}s-{plugin}-jdbc_paging_enabled>> is enabled and
|
134
|
+
<<plugins-{type}s-{plugin}-jdbc_paging_mode>> has the `explicit` value.
|
132
135
|
|==========================================================
|
133
136
|
|
134
137
|
Example:
|
@@ -153,7 +156,7 @@ NOTE: Not all JDBC accessible technologies will support prepared statements.
|
|
153
156
|
With the introduction of Prepared Statement support comes a different code execution path and some new settings. Most of the existing settings are still useful but there are several new settings for Prepared Statements to read up on.
|
154
157
|
Use the boolean setting `use_prepared_statements` to enable this execution mode. Use the `prepared_statement_name` setting to specify a name for the Prepared Statement, this identifies the prepared statement locally and remotely and it should be unique in your config and on the database. Use the `prepared_statement_bind_values` array setting to specify the bind values, use the exact string `:sql_last_value` (multiple times if necessary) for the predefined parameter mentioned before. The `statement` (or `statement_path`) setting still holds the SQL statement but to use bind variables you must use the `?` character as a placeholder in the exact order found in the `prepared_statement_bind_values` array.
|
155
158
|
|
156
|
-
NOTE: Building count queries around a prepared statement is not supported at this time
|
159
|
+
NOTE: Building count queries around a prepared statement is not supported at this time. Because jdbc paging uses count queries when `jdbc_paging_mode` has value `auto`,jdbc paging is not supported with prepared statements at this time either. Therefore, `jdbc_paging_enabled`, `jdbc_page_size` settings are ignored when using prepared statements.
|
157
160
|
|
158
161
|
Example:
|
159
162
|
[source,ruby]
|
@@ -193,6 +196,7 @@ This plugin supports the following configuration options plus the <<plugins-{typ
|
|
193
196
|
| <<plugins-{type}s-{plugin}-jdbc_fetch_size>> |<<number,number>>|No
|
194
197
|
| <<plugins-{type}s-{plugin}-jdbc_page_size>> |<<number,number>>|No
|
195
198
|
| <<plugins-{type}s-{plugin}-jdbc_paging_enabled>> |<<boolean,boolean>>|No
|
199
|
+
| <<plugins-{type}s-{plugin}-jdbc_paging_mode>> |<<string,string>>, one of `["auto", "explicit"]`|No
|
196
200
|
| <<plugins-{type}s-{plugin}-jdbc_password>> |<<password,password>>|No
|
197
201
|
| <<plugins-{type}s-{plugin}-jdbc_password_filepath>> |a valid filesystem path|No
|
198
202
|
| <<plugins-{type}s-{plugin}-jdbc_pool_timeout>> |<<number,number>>|No
|
@@ -373,6 +377,52 @@ result-set. The limit size is set with `jdbc_page_size`.
|
|
373
377
|
|
374
378
|
Be aware that ordering is not guaranteed between queries.
|
375
379
|
|
380
|
+
[id="plugins-{type}s-{plugin}-jdbc_paging_mode"]
|
381
|
+
===== `jdbc_paging_mode`
|
382
|
+
|
383
|
+
* Value can be any of: `auto`, `explicit`
|
384
|
+
* Default value is `"auto"`
|
385
|
+
|
386
|
+
Whether to use `explicit` or `auto` mode during the JDBC paging
|
387
|
+
|
388
|
+
If `auto`, your statement will be automatically surrounded by a count query and subsequent multiple paged queries (with `LIMIT` statement, etc.).
|
389
|
+
|
390
|
+
If `explicit`, multiple queries (without a count query ahead) will be performed with your statement, until no more rows are retrieved.
|
391
|
+
You have to write your own paging conditions in your statement configuration.
|
392
|
+
The `offset` and `size` parameters can be used in your statement (`size` equal to `jdbc_page_size`, and `offset` incremented by `size` for each query).
|
393
|
+
When the number of rows returned by the query is not equal to `size`, SQL paging will be ended.
|
394
|
+
Example:
|
395
|
+
|
396
|
+
[source, ruby]
|
397
|
+
------------------------------------------------------
|
398
|
+
input {
|
399
|
+
jdbc {
|
400
|
+
statement => "SELECT id, mycolumn1, mycolumn2 FROM my_table WHERE id > :sql_last_value LIMIT :size OFFSET :offset",
|
401
|
+
jdbc_paging_enabled => true,
|
402
|
+
jdbc_paging_mode => "explicit",
|
403
|
+
jdbc_page_size => 100000
|
404
|
+
}
|
405
|
+
}
|
406
|
+
------------------------------------------------------
|
407
|
+
|
408
|
+
[source, ruby]
|
409
|
+
------------------------------------------------------
|
410
|
+
input {
|
411
|
+
jdbc {
|
412
|
+
statement => "CALL fetch_my_data(:sql_last_value, :offset, :size)",
|
413
|
+
jdbc_paging_enabled => true,
|
414
|
+
jdbc_paging_mode => "explicit",
|
415
|
+
jdbc_page_size => 100000
|
416
|
+
}
|
417
|
+
}
|
418
|
+
------------------------------------------------------
|
419
|
+
|
420
|
+
This mode can be considered in the following situations:
|
421
|
+
|
422
|
+
. Performance issues encountered in default paging mode.
|
423
|
+
. Your SQL statement is complex, so simply surrounding it with paging statements is not what you want.
|
424
|
+
. Your statement is a stored procedure, and the actual paging statement is inside it.
|
425
|
+
|
376
426
|
[id="plugins-{type}s-{plugin}-jdbc_password"]
|
377
427
|
===== `jdbc_password`
|
378
428
|
|
@@ -55,6 +55,9 @@ module LogStash module PluginMixins module Jdbc
|
|
55
55
|
# Be aware that ordering is not guaranteed between queries.
|
56
56
|
config :jdbc_paging_enabled, :validate => :boolean, :default => false
|
57
57
|
|
58
|
+
# Which pagination mode to use, automatic pagination or explicitly defined in the query.
|
59
|
+
config :jdbc_paging_mode, :validate => [ "auto", "explicit" ], :default => "auto"
|
60
|
+
|
58
61
|
# JDBC page size
|
59
62
|
config :jdbc_page_size, :validate => :number, :default => 100000
|
60
63
|
|
@@ -3,7 +3,15 @@
|
|
3
3
|
module LogStash module PluginMixins module Jdbc
|
4
4
|
class StatementHandler
|
5
5
|
def self.build_statement_handler(plugin, logger)
|
6
|
-
|
6
|
+
if plugin.use_prepared_statements
|
7
|
+
klass = PreparedStatementHandler
|
8
|
+
else
|
9
|
+
if plugin.jdbc_paging_enabled && plugin.jdbc_paging_mode == "explicit"
|
10
|
+
klass = ExplicitPagingModeStatementHandler
|
11
|
+
else
|
12
|
+
klass = NormalStatementHandler
|
13
|
+
end
|
14
|
+
end
|
7
15
|
klass.new(plugin, logger)
|
8
16
|
end
|
9
17
|
|
@@ -27,7 +35,9 @@ module LogStash module PluginMixins module Jdbc
|
|
27
35
|
class NormalStatementHandler < StatementHandler
|
28
36
|
# Performs the query, respecting our pagination settings, yielding once per row of data
|
29
37
|
# @param db [Sequel::Database]
|
30
|
-
# @param sql_last_value [
|
38
|
+
# @param sql_last_value [Integer|DateTime|Time]
|
39
|
+
# @param jdbc_paging_enabled [Boolean]
|
40
|
+
# @param jdbc_page_size [Integer]
|
31
41
|
# @yieldparam row [Hash{Symbol=>Object}]
|
32
42
|
def perform_query(db, sql_last_value, jdbc_paging_enabled, jdbc_page_size)
|
33
43
|
query = build_query(db, sql_last_value)
|
@@ -67,6 +77,28 @@ module LogStash module PluginMixins module Jdbc
|
|
67
77
|
end
|
68
78
|
end
|
69
79
|
|
80
|
+
class ExplicitPagingModeStatementHandler < NormalStatementHandler
|
81
|
+
# Performs the query, respecting our pagination settings, yielding once per row of data
|
82
|
+
# @param db [Sequel::Database]
|
83
|
+
# @param sql_last_value [Integer|DateTime|Time]
|
84
|
+
# @param jdbc_paging_enabled [Boolean]
|
85
|
+
# @param jdbc_page_size [Integer]
|
86
|
+
# @yieldparam row [Hash{Symbol=>Object}]
|
87
|
+
def perform_query(db, sql_last_value, jdbc_paging_enabled, jdbc_page_size)
|
88
|
+
query = build_query(db, sql_last_value)
|
89
|
+
offset = 0
|
90
|
+
loop do
|
91
|
+
rows_in_page = 0
|
92
|
+
query.with_sql(query.sql, offset: offset, size: jdbc_page_size).each do |row|
|
93
|
+
yield row
|
94
|
+
rows_in_page += 1
|
95
|
+
end
|
96
|
+
break unless rows_in_page == jdbc_page_size
|
97
|
+
offset += jdbc_page_size
|
98
|
+
end
|
99
|
+
end
|
100
|
+
end
|
101
|
+
|
70
102
|
class PreparedStatementHandler < StatementHandler
|
71
103
|
attr_reader :name, :bind_values_array, :statement_prepared, :prepared
|
72
104
|
|
@@ -1,6 +1,6 @@
|
|
1
1
|
Gem::Specification.new do |s|
|
2
2
|
s.name = 'logstash-integration-jdbc'
|
3
|
-
s.version = '5.
|
3
|
+
s.version = '5.2.0'
|
4
4
|
s.licenses = ['Apache License (2.0)']
|
5
5
|
s.summary = "Integration with JDBC - input and filter plugins"
|
6
6
|
s.description = "This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program"
|
data/spec/inputs/jdbc_spec.rb
CHANGED
@@ -329,6 +329,39 @@ describe LogStash::Inputs::Jdbc do
|
|
329
329
|
|
330
330
|
end
|
331
331
|
|
332
|
+
context "when iterating result-set via explicit paging mode" do
|
333
|
+
|
334
|
+
let(:settings) do
|
335
|
+
{
|
336
|
+
"statement" => "SELECT * from test_table OFFSET :offset ROWS FETCH NEXT :size ROWS ONLY",
|
337
|
+
"jdbc_paging_enabled" => true,
|
338
|
+
"jdbc_paging_mode" => "explicit",
|
339
|
+
"jdbc_page_size" => 10
|
340
|
+
}
|
341
|
+
end
|
342
|
+
|
343
|
+
let(:num_rows) { 15 }
|
344
|
+
|
345
|
+
before do
|
346
|
+
plugin.register
|
347
|
+
end
|
348
|
+
|
349
|
+
after do
|
350
|
+
plugin.stop
|
351
|
+
end
|
352
|
+
|
353
|
+
it "should fetch all rows" do
|
354
|
+
num_rows.times do
|
355
|
+
db[:test_table].insert(:num => 1, :custom_time => Time.now.utc, :created_at => Time.now.utc)
|
356
|
+
end
|
357
|
+
|
358
|
+
plugin.run(queue)
|
359
|
+
|
360
|
+
expect(queue.size).to eq(num_rows)
|
361
|
+
end
|
362
|
+
|
363
|
+
end
|
364
|
+
|
332
365
|
context "when using target option" do
|
333
366
|
let(:settings) do
|
334
367
|
{
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: logstash-integration-jdbc
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 5.
|
4
|
+
version: 5.2.0
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Elastic
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2021-12-
|
11
|
+
date: 2021-12-22 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
requirement: !ruby/object:Gem::Requirement
|