logstash-output-jdbc 0.2.0.rc3 → 0.2.0.rc4
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/README.md +34 -15
- data/lib/logstash-output-jdbc_ring-buffer.rb +19 -0
- data/lib/logstash/outputs/jdbc.rb +54 -38
- data/spec/outputs/jdbc_spec.rb +13 -0
- metadata +8 -4
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 1dc858c14f1596f4af6dd35a63139642c6a217c4
|
4
|
+
data.tar.gz: 3fc25dcb61a10107aef131240f4394ec4bf17814
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: c2d778ce352a48b759a314b9c138b6e7368091324659227cafe0ae4e2a4e5ee1b0cfe02b7046ff380094bbe9ea7af5d059898d9404417532324896d088e81a3f
|
7
|
+
data.tar.gz: cc1dee650cd5c497bf6fc0c0b01c2444fe1cfab211a89e212be018998f1d5a2f80a59f380d88b0de9252830e81073df51bc73313213d7e2c93c508f0bc935132
|
data/README.md
CHANGED
@@ -6,9 +6,16 @@ See below for tested adapters, and example configurations.
|
|
6
6
|
|
7
7
|
This has not yet been extensively tested with all JDBC drivers and may not yet work for you.
|
8
8
|
|
9
|
+
If you do find this works for a JDBC driver not listed, let me know and provide a small example configuration.
|
10
|
+
|
9
11
|
This plugin does not bundle any JDBC jar files, and does expect them to be in a
|
10
12
|
particular location. Please ensure you read the 4 installation lines below.
|
11
13
|
|
14
|
+
## Headlines
|
15
|
+
- Support for connection pooling added in 0.2.0 [unreleased until #10 is resolved]
|
16
|
+
- Support for unsafe statement handling (allowing dynamic queries) in 0.2.0 [unreleased until #10 is resolved]
|
17
|
+
- Altered exception handling to now count sequential flushes with exceptions thrown in 0.2.0 [untested and unreleased until #10 is resolved]
|
18
|
+
|
12
19
|
## Versions
|
13
20
|
- See master branch for logstash v2+
|
14
21
|
- See v1.5 branch for logstash v1.5
|
@@ -16,20 +23,32 @@ particular location. Please ensure you read the 4 installation lines below.
|
|
16
23
|
|
17
24
|
## Installation
|
18
25
|
- Run `bin/plugin install logstash-output-jdbc` in your logstash installation directory
|
19
|
-
-
|
20
|
-
|
21
|
-
-
|
26
|
+
- Now either:
|
27
|
+
- Use driver_class in your configuraton to specify a path to your jar file
|
28
|
+
- Or:
|
29
|
+
- Create the directory vendor/jar/jdbc in your logstash installation (`mkdir -p vendor/jar/jdbc/`)
|
30
|
+
- Add JDBC jar files to vendor/jar/jdbc in your logstash installation
|
31
|
+
- And then configure (examples below)
|
22
32
|
|
23
33
|
## Configuration options
|
24
|
-
|
25
|
-
|
26
|
-
|
27
|
-
|
28
|
-
|
29
|
-
|
30
|
-
|
34
|
+
|
35
|
+
| Option | Type | Description | Required? | Default |
|
36
|
+
| ------ | ---- | ----------- | --------- | ------- |
|
37
|
+
| driver_path | String | File path to jar file containing your JDBC driver. This is optional, and all JDBC jars may be placed in $LOGSTASH_HOME/vendor/jar/jdbc instead. | No | |
|
38
|
+
| connection_string | String | JDBC connection URL | Yes | |
|
39
|
+
| username | String | JDBC username - this is optional as it may be included in the connection string, for many drivers | No | |
|
40
|
+
| password | String | JDBC password - this is optional as it may be included in the connection string, for many drivers | No | |
|
41
|
+
| statement | Array | An array of strings representing the SQL statement to run. Index 0 is the SQL statement that is prepared, all other array entries are passed in as parameters (in order). A parameter may either be a property of the event (i.e. "@timestamp", or "host") or a formatted string (i.e. "%{host} - %{message}" or "%{message}"). If a key is passed then it will be automatically converted as required for insertion into SQL. If it's a formatted string then it will be passed in verbatim. | Yes | |
|
42
|
+
| unsafe_statement | Boolean | If yes, the statement is evaluated for event fields - this allows you to use dynamic table names, etc. **This is highly dangerous** and you should **not** use this unless you are 100% sure that the field(s) you are passing in are 100% safe. Failure to do so will result in possible SQL injections. Please be aware that there is also a potential performance penalty as each event must be evaluated and inserted into SQL one at a time, where as when this is false multiple events are inserted at once. Example statement: [ "insert into %{table_name_field} (column) values(?)", "fieldname" ] | No | False |
|
43
|
+
| max_pool_size | Number | Maximum number of connections to open to the SQL server at any 1 time | No | 5 |
|
44
|
+
| connection_timeout | Number | Number of seconds before a SQL connection is closed | No | 2800 |
|
45
|
+
| flush_size | Number | Maximum number of entries to buffer before sending to SQL - if this is reached before idle_flush_time | No | 1000 |
|
46
|
+
| idle_flush_time | Number | Number of idle seconds before sending data to SQL - even if the flush_size has not yet been reached | No | 1 |
|
47
|
+
| max_flush_exceptions | Number | Number of sequential flushes which cause an exception, before we stop logstash. Set to a value less than 1 if you never want it to stop. This should be carefully configured with relation to idle_flush_time if your SQL instance is not highly available. | No | 0 |
|
31
48
|
|
32
49
|
## Example configurations
|
50
|
+
If you have a working sample configuration, for a DB thats not listed, pull requests are welcome.
|
51
|
+
|
33
52
|
### SQLite3
|
34
53
|
* Tested using https://bitbucket.org/xerial/sqlite-jdbc
|
35
54
|
* SQLite setup - `echo "CREATE table log (host text, timestamp datetime, message text);" | sqlite3 test.db`
|
@@ -42,7 +61,6 @@ output {
|
|
42
61
|
stdout { }
|
43
62
|
|
44
63
|
jdbc {
|
45
|
-
driver_class => 'org.sqlite.JDBC'
|
46
64
|
connection_string => 'jdbc:sqlite:test.db'
|
47
65
|
statement => [ "INSERT INTO log (host, timestamp, message) VALUES(?, ?, ?)", "host", "@timestamp", "message" ]
|
48
66
|
}
|
@@ -58,7 +76,6 @@ input
|
|
58
76
|
}
|
59
77
|
output {
|
60
78
|
jdbc {
|
61
|
-
driver_class => 'com.microsoft.sqlserver.jdbc.SQLServerDriver'
|
62
79
|
connection_string => "jdbc:sqlserver://server:1433;databaseName=databasename;user=username;password=password;autoReconnect=true;"
|
63
80
|
statement => [ "INSERT INTO log (host, timestamp, message) VALUES(?, ?, ?)", "host", "@timestamp", "message" ]
|
64
81
|
}
|
@@ -74,7 +91,6 @@ input
|
|
74
91
|
}
|
75
92
|
output {
|
76
93
|
jdbc {
|
77
|
-
driver_class => 'org.postgresql.Driver'
|
78
94
|
connection_string => 'jdbc:postgresql://hostname:5432/database?user=username&password=password'
|
79
95
|
statement => [ "INSERT INTO log (host, timestamp, message) VALUES(?, CAST (? AS timestamp), ?)", "host", "@timestamp", "message" ]
|
80
96
|
}
|
@@ -92,7 +108,6 @@ input
|
|
92
108
|
}
|
93
109
|
output {
|
94
110
|
jdbc {
|
95
|
-
driver_class => "oracle.jdbc.driver.OracleDriver"
|
96
111
|
connection_string => "jdbc:oracle:thin:USER/PASS@HOST:PORT:SID"
|
97
112
|
statement => [ "INSERT INTO log (host, timestamp, message) VALUES(?, CAST (? AS timestamp), ?)", "host", "@timestamp", "message" ]
|
98
113
|
}
|
@@ -110,9 +125,13 @@ input
|
|
110
125
|
}
|
111
126
|
output {
|
112
127
|
jdbc {
|
113
|
-
driver_class => "com.mysql.jdbc.Driver"
|
114
128
|
connection_string => "jdbc:mysql://HOSTNAME/DATABASE?user=USER&password=PASSWORD"
|
115
129
|
statement => [ "INSERT INTO log (host, timestamp, message) VALUES(?, CAST (? AS timestamp), ?)", "host", "@timestamp", "message" ]
|
116
130
|
}
|
117
131
|
}
|
118
132
|
```
|
133
|
+
|
134
|
+
### MariaDB
|
135
|
+
This is reportedly working, according to [@db2882](https://github.com/db2882) in issue #20.
|
136
|
+
No example configuration provided.
|
137
|
+
If you have a working sample, pull requests are welcome.
|
@@ -0,0 +1,19 @@
|
|
1
|
+
class RingBuffer < Array
|
2
|
+
attr_reader :max_size
|
3
|
+
|
4
|
+
def initialize(max_size, enum = nil)
|
5
|
+
@max_size = max_size
|
6
|
+
enum.each { |e| self << e } if enum
|
7
|
+
end
|
8
|
+
|
9
|
+
def <<(el)
|
10
|
+
if self.size < @max_size || @max_size.nil?
|
11
|
+
super
|
12
|
+
else
|
13
|
+
self.shift
|
14
|
+
self.push(el)
|
15
|
+
end
|
16
|
+
end
|
17
|
+
|
18
|
+
alias :push :<<
|
19
|
+
end
|
@@ -4,6 +4,7 @@ require "logstash/namespace"
|
|
4
4
|
require "stud/buffer"
|
5
5
|
require "java"
|
6
6
|
require "logstash-output-jdbc_jars"
|
7
|
+
require "logstash-output-jdbc_ring-buffer"
|
7
8
|
|
8
9
|
class LogStash::Outputs::Jdbc < LogStash::Outputs::Base
|
9
10
|
# Adds buffer support
|
@@ -12,7 +13,7 @@ class LogStash::Outputs::Jdbc < LogStash::Outputs::Base
|
|
12
13
|
config_name "jdbc"
|
13
14
|
|
14
15
|
# Driver class - No longer required
|
15
|
-
config :driver_class, :obsolete =>
|
16
|
+
config :driver_class, :obsolete => "driver_class is no longer required and can be removed from your configuration"
|
16
17
|
|
17
18
|
# Where to find the jar
|
18
19
|
# Defaults to not required, and to the original behaviour
|
@@ -58,17 +59,20 @@ class LogStash::Outputs::Jdbc < LogStash::Outputs::Base
|
|
58
59
|
# a timely manner.
|
59
60
|
#
|
60
61
|
# If you change this value please ensure that you change
|
61
|
-
#
|
62
|
+
# max_flush_exceptions accordingly.
|
62
63
|
config :idle_flush_time, :validate => :number, :default => 1
|
63
64
|
|
64
|
-
# Maximum number of
|
65
|
+
# Maximum number of sequential flushes which encounter exceptions, before we stop retrying.
|
65
66
|
# If set to < 1, then it will infinitely retry.
|
66
|
-
|
67
|
+
#
|
68
|
+
# You should carefully tune this in relation to idle_flush_time if your SQL server
|
69
|
+
# is not highly available.
|
70
|
+
# i.e. If your idle_flush_time is 1, and your max_flush_exceptions is 200, and your SQL server takes
|
71
|
+
# longer than 200 seconds to reboot, then logstash will stop.
|
72
|
+
config :max_flush_exceptions, :validate => :number, :default => 0
|
67
73
|
|
68
|
-
|
69
|
-
|
70
|
-
# This value should be carefully considered in respect to idle_flush_time.
|
71
|
-
config :max_repeat_exceptions_time, :validate => :number, :default => 30
|
74
|
+
config :max_repeat_exceptions, :obsolete => "This has been replaced by max_flush_exceptions - which behaves slightly differently. Please check the documentation."
|
75
|
+
config :max_repeat_exceptions_time, :obsolete => "This is no longer required"
|
72
76
|
|
73
77
|
public
|
74
78
|
def register
|
@@ -85,17 +89,12 @@ class LogStash::Outputs::Jdbc < LogStash::Outputs::Base
|
|
85
89
|
@pool.setMaximumPoolSize(@max_pool_size)
|
86
90
|
@pool.setConnectionTimeout(@connection_timeout)
|
87
91
|
|
92
|
+
@exceptions_tracker = RingBuffer.new(@max_flush_exceptions)
|
93
|
+
|
88
94
|
if (@flush_size > 1000)
|
89
95
|
@logger.warn("JDBC - Flush size is set to > 1000")
|
90
96
|
end
|
91
97
|
|
92
|
-
@repeat_exception_count = 0
|
93
|
-
@last_exception_time = Time.now
|
94
|
-
|
95
|
-
if (@max_repeat_exceptions > 0) and ((@idle_flush_time * @max_repeat_exceptions) > @max_repeat_exceptions_time)
|
96
|
-
@logger.warn("JDBC - max_repeat_exceptions_time is set such that it may still permit a looping exception. You probably changed idle_flush_time. Considering increasing max_repeat_exceptions_time.")
|
97
|
-
end
|
98
|
-
|
99
98
|
buffer_initialize(
|
100
99
|
:max_items => @flush_size,
|
101
100
|
:max_interval => @idle_flush_time,
|
@@ -104,7 +103,7 @@ class LogStash::Outputs::Jdbc < LogStash::Outputs::Base
|
|
104
103
|
end
|
105
104
|
|
106
105
|
def receive(event)
|
107
|
-
return unless output?(event)
|
106
|
+
return unless output?(event) or event.cancelled?
|
108
107
|
return unless @statement.length > 0
|
109
108
|
|
110
109
|
buffer_receive(event)
|
@@ -119,21 +118,14 @@ class LogStash::Outputs::Jdbc < LogStash::Outputs::Base
|
|
119
118
|
end
|
120
119
|
|
121
120
|
def on_flush_error(e)
|
122
|
-
return if @
|
121
|
+
return if @max_flush_exceptions < 1
|
123
122
|
|
124
|
-
|
125
|
-
@repeat_exception_count += 1
|
126
|
-
else
|
127
|
-
@repeat_exception_count = 0
|
128
|
-
end
|
123
|
+
@exceptions_tracker << e.class
|
129
124
|
|
130
|
-
if
|
131
|
-
@logger.error("JDBC -
|
125
|
+
if @exceptions_tracker.reject { |i| i.nil? }.count >= @max_flush_exceptions
|
126
|
+
@logger.error("JDBC - max_flush_exceptions has been reached")
|
132
127
|
raise e
|
133
128
|
end
|
134
|
-
|
135
|
-
@last_exception_time = Time.now
|
136
|
-
@last_exception = e.to_s
|
137
129
|
end
|
138
130
|
|
139
131
|
def teardown
|
@@ -173,10 +165,10 @@ class LogStash::Outputs::Jdbc < LogStash::Outputs::Base
|
|
173
165
|
|
174
166
|
def safe_flush(events, teardown=false)
|
175
167
|
connection = @pool.getConnection()
|
176
|
-
|
177
168
|
statement = connection.prepareStatement(@statement[0])
|
178
169
|
|
179
170
|
events.each do |event|
|
171
|
+
next if event.cancelled?
|
180
172
|
next if @statement.length < 2
|
181
173
|
statement = add_statement_event_params(statement, event)
|
182
174
|
|
@@ -184,17 +176,15 @@ class LogStash::Outputs::Jdbc < LogStash::Outputs::Base
|
|
184
176
|
end
|
185
177
|
|
186
178
|
begin
|
187
|
-
@logger.debug("JDBC - Sending SQL", :sql => statement.toString())
|
188
179
|
statement.executeBatch()
|
189
180
|
statement.close()
|
181
|
+
@exceptions_tracker << nil
|
182
|
+
|
190
183
|
rescue => e
|
191
184
|
# Raising an exception will incur a retry from Stud::Buffer.
|
192
185
|
# Since the exceutebatch failed this should mean any events failed to be
|
193
186
|
# inserted will be re-run. We're going to log it for the lols anyway.
|
194
|
-
|
195
|
-
if e.getNextException() != nil
|
196
|
-
@logger.warn("JDBC - Exception. Will automatically retry", :exception => e.getNextException())
|
197
|
-
end
|
187
|
+
log_jdbc_exception(e)
|
198
188
|
ensure
|
199
189
|
connection.close();
|
200
190
|
end
|
@@ -204,13 +194,26 @@ class LogStash::Outputs::Jdbc < LogStash::Outputs::Base
|
|
204
194
|
connection = @pool.getConnection()
|
205
195
|
|
206
196
|
events.each do |event|
|
197
|
+
next if event.cancelled?
|
198
|
+
|
207
199
|
statement = connection.prepareStatement(event.sprintf(@statement[0]))
|
208
|
-
|
209
200
|
statement = add_statement_event_params(statement, event) if @statement.length > 1
|
210
201
|
|
211
|
-
|
212
|
-
|
213
|
-
|
202
|
+
begin
|
203
|
+
statement.execute()
|
204
|
+
|
205
|
+
# cancel the event, since we may end up outputting the same event multiple times
|
206
|
+
# if an exception happens later down the line
|
207
|
+
event.cancel
|
208
|
+
@exceptions_tracker << nil
|
209
|
+
rescue => e
|
210
|
+
# Raising an exception will incur a retry from Stud::Buffer.
|
211
|
+
# We log for the lols.
|
212
|
+
log_jdbc_exception(e)
|
213
|
+
ensure
|
214
|
+
statement.close()
|
215
|
+
connection.close()
|
216
|
+
end
|
214
217
|
end
|
215
218
|
end
|
216
219
|
|
@@ -231,10 +234,23 @@ class LogStash::Outputs::Jdbc < LogStash::Outputs::Base
|
|
231
234
|
when false
|
232
235
|
statement.setBoolean(idx + 1, false)
|
233
236
|
else
|
234
|
-
|
237
|
+
if event[i].nil? and i =~ /%\{/
|
238
|
+
statement.setString(idx + 1, event.sprintf(i))
|
239
|
+
else
|
240
|
+
statement.setString(idx + 1, nil)
|
241
|
+
end
|
235
242
|
end
|
236
243
|
end
|
237
244
|
|
238
245
|
statement
|
239
246
|
end
|
247
|
+
|
248
|
+
def log_jdbc_exception(e)
|
249
|
+
ce = e
|
250
|
+
loop do
|
251
|
+
@logger.error("JDBC Exception encountered: Will automatically retry.", :exception => ce)
|
252
|
+
ce = e.getNextException()
|
253
|
+
break if ce == nil
|
254
|
+
end
|
255
|
+
end
|
240
256
|
end # class LogStash::Outputs::jdbc
|
@@ -0,0 +1,13 @@
|
|
1
|
+
require "logstash/devutils/rspec/spec_helper"
|
2
|
+
require "logstash/outputs/jdbc"
|
3
|
+
require "stud/temporary"
|
4
|
+
|
5
|
+
describe LogStash::Outputs::Jdbc do
|
6
|
+
|
7
|
+
it "should register without errors" do
|
8
|
+
plugin = LogStash::Plugin.lookup("output", "jdbc").new({})
|
9
|
+
expect { plugin.register }.to_not raise_error
|
10
|
+
|
11
|
+
end
|
12
|
+
|
13
|
+
end
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: logstash-output-jdbc
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.2.0.
|
4
|
+
version: 0.2.0.rc4
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- the_angry_angel
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2015-11-
|
11
|
+
date: 2015-11-19 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: logstash-core
|
@@ -82,9 +82,11 @@ extra_rdoc_files: []
|
|
82
82
|
files:
|
83
83
|
- lib/logstash/outputs/jdbc.rb
|
84
84
|
- lib/logstash-output-jdbc_jars.rb
|
85
|
+
- lib/logstash-output-jdbc_ring-buffer.rb
|
85
86
|
- vendor/jar-dependencies/runtime-jars/HikariCP-2.4.2.jar
|
86
87
|
- vendor/jar-dependencies/runtime-jars/slf4j-api-1.7.13.jar
|
87
88
|
- vendor/jar-dependencies/runtime-jars/slf4j-nop-1.7.13.jar
|
89
|
+
- spec/outputs/jdbc_spec.rb
|
88
90
|
- LICENSE.txt
|
89
91
|
- README.md
|
90
92
|
homepage: https://github.com/theangryangel/logstash-output-jdbc
|
@@ -93,7 +95,8 @@ licenses:
|
|
93
95
|
metadata:
|
94
96
|
logstash_plugin: 'true'
|
95
97
|
logstash_group: output
|
96
|
-
post_install_message:
|
98
|
+
post_install_message: logstash-output-jdbc 0.2.0 introduces several new features -
|
99
|
+
please ensure you check the documentation in the README file
|
97
100
|
rdoc_options: []
|
98
101
|
require_paths:
|
99
102
|
- lib
|
@@ -113,4 +116,5 @@ rubygems_version: 2.0.14
|
|
113
116
|
signing_key:
|
114
117
|
specification_version: 4
|
115
118
|
summary: This plugin allows you to output to SQL, via JDBC
|
116
|
-
test_files:
|
119
|
+
test_files:
|
120
|
+
- spec/outputs/jdbc_spec.rb
|