logstash-output-jdbc 0.2.0.rc3 → 0.2.0.rc4

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: a54974a84429e4dd7ae8f0b0ab08a2eca7ea0217
4
- data.tar.gz: fde4c6acdf269342be353facc5caa8402656359c
3
+ metadata.gz: 1dc858c14f1596f4af6dd35a63139642c6a217c4
4
+ data.tar.gz: 3fc25dcb61a10107aef131240f4394ec4bf17814
5
5
  SHA512:
6
- metadata.gz: e1e2792041f4caab27a0774cb8d63f3513bfe5add1debaeb1d08b634c806b21ff1f344d7c4c0905019e97fa2e5ce3ca92a0c0c590a4066bdc2e94aec1bc59a19
7
- data.tar.gz: afc7f0e8a9027e098cd887ae96c4dd0625a8aac95d1ca4ce3bf49ec4aa2d0a16f9cde9feec3b1717ed09e317d0937f4a2ed3c3b6ac58bce6d9bd048af878b130
6
+ metadata.gz: c2d778ce352a48b759a314b9c138b6e7368091324659227cafe0ae4e2a4e5ee1b0cfe02b7046ff380094bbe9ea7af5d059898d9404417532324896d088e81a3f
7
+ data.tar.gz: cc1dee650cd5c497bf6fc0c0b01c2444fe1cfab211a89e212be018998f1d5a2f80a59f380d88b0de9252830e81073df51bc73313213d7e2c93c508f0bc935132
data/README.md CHANGED
@@ -6,9 +6,16 @@ See below for tested adapters, and example configurations.
6
6
 
7
7
  This has not yet been extensively tested with all JDBC drivers and may not yet work for you.
8
8
 
9
+ If you do find this works for a JDBC driver not listed, let me know and provide a small example configuration.
10
+
9
11
  This plugin does not bundle any JDBC jar files, and does expect them to be in a
10
12
  particular location. Please ensure you read the 4 installation lines below.
11
13
 
14
+ ## Headlines
15
+ - Support for connection pooling added in 0.2.0 [unreleased until #10 is resolved]
16
+ - Support for unsafe statement handling (allowing dynamic queries) in 0.2.0 [unreleased until #10 is resolved]
17
+ - Altered exception handling to now count sequential flushes with exceptions thrown in 0.2.0 [untested and unreleased until #10 is resolved]
18
+
12
19
  ## Versions
13
20
  - See master branch for logstash v2+
14
21
  - See v1.5 branch for logstash v1.5
@@ -16,20 +23,32 @@ particular location. Please ensure you read the 4 installation lines below.
16
23
 
17
24
  ## Installation
18
25
  - Run `bin/plugin install logstash-output-jdbc` in your logstash installation directory
19
- - Create the directory vendor/jar/jdbc in your logstash installation (`mkdir -p vendor/jar/jdbc/`)
20
- - Add JDBC jar files to vendor/jar/jdbc in your logstash installation
21
- - Configure
26
+ - Now either:
27
+ - Use driver_class in your configuraton to specify a path to your jar file
28
+ - Or:
29
+ - Create the directory vendor/jar/jdbc in your logstash installation (`mkdir -p vendor/jar/jdbc/`)
30
+ - Add JDBC jar files to vendor/jar/jdbc in your logstash installation
31
+ - And then configure (examples below)
22
32
 
23
33
  ## Configuration options
24
- * driver_class, string, JDBC driver class to load
25
- * connection_string, string, JDBC connection string
26
- * statement, array, an array of strings representing the SQL statement to run. Index 0 is the SQL statement that is prepared, all other array entries are passed in as parameters (in order). A parameter may either be a property of the event (i.e. "@timestamp", or "host") or a formatted string (i.e. "%{host} - %{message}" or "%{message}"). If a key is passed then it will be automatically converted as required for insertion into SQL. If it's a formatted string then it will be passed in verbatim.
27
- * flush_size, number, default = 1000, number of entries to buffer before sending to SQL
28
- * idle_flush_time, number, default = 1, number of idle seconds before sending data to SQL, even if the flush_size has not been reached. If you modify this value you should also consider altering max_repeat_exceptions_time
29
- * max_repeat_exceptions, number, default = 5, number of times the same exception can repeat before we stop logstash. Set to a value less than 1 if you never want it to stop
30
- * max_repeat_exceptions_time, number, default = 30, maxium number of seconds between exceptions before they're considered "different" exceptions. If you modify idle_flush_time you should consider this value
34
+
35
+ | Option | Type | Description | Required? | Default |
36
+ | ------ | ---- | ----------- | --------- | ------- |
37
+ | driver_path | String | File path to jar file containing your JDBC driver. This is optional, and all JDBC jars may be placed in $LOGSTASH_HOME/vendor/jar/jdbc instead. | No | |
38
+ | connection_string | String | JDBC connection URL | Yes | |
39
+ | username | String | JDBC username - this is optional as it may be included in the connection string, for many drivers | No | |
40
+ | password | String | JDBC password - this is optional as it may be included in the connection string, for many drivers | No | |
41
+ | statement | Array | An array of strings representing the SQL statement to run. Index 0 is the SQL statement that is prepared, all other array entries are passed in as parameters (in order). A parameter may either be a property of the event (i.e. "@timestamp", or "host") or a formatted string (i.e. "%{host} - %{message}" or "%{message}"). If a key is passed then it will be automatically converted as required for insertion into SQL. If it's a formatted string then it will be passed in verbatim. | Yes | |
42
+ | unsafe_statement | Boolean | If yes, the statement is evaluated for event fields - this allows you to use dynamic table names, etc. **This is highly dangerous** and you should **not** use this unless you are 100% sure that the field(s) you are passing in are 100% safe. Failure to do so will result in possible SQL injections. Please be aware that there is also a potential performance penalty as each event must be evaluated and inserted into SQL one at a time, where as when this is false multiple events are inserted at once. Example statement: [ "insert into %{table_name_field} (column) values(?)", "fieldname" ] | No | False |
43
+ | max_pool_size | Number | Maximum number of connections to open to the SQL server at any 1 time | No | 5 |
44
+ | connection_timeout | Number | Number of seconds before a SQL connection is closed | No | 2800 |
45
+ | flush_size | Number | Maximum number of entries to buffer before sending to SQL - if this is reached before idle_flush_time | No | 1000 |
46
+ | idle_flush_time | Number | Number of idle seconds before sending data to SQL - even if the flush_size has not yet been reached | No | 1 |
47
+ | max_flush_exceptions | Number | Number of sequential flushes which cause an exception, before we stop logstash. Set to a value less than 1 if you never want it to stop. This should be carefully configured with relation to idle_flush_time if your SQL instance is not highly available. | No | 0 |
31
48
 
32
49
  ## Example configurations
50
+ If you have a working sample configuration, for a DB thats not listed, pull requests are welcome.
51
+
33
52
  ### SQLite3
34
53
  * Tested using https://bitbucket.org/xerial/sqlite-jdbc
35
54
  * SQLite setup - `echo "CREATE table log (host text, timestamp datetime, message text);" | sqlite3 test.db`
@@ -42,7 +61,6 @@ output {
42
61
  stdout { }
43
62
 
44
63
  jdbc {
45
- driver_class => 'org.sqlite.JDBC'
46
64
  connection_string => 'jdbc:sqlite:test.db'
47
65
  statement => [ "INSERT INTO log (host, timestamp, message) VALUES(?, ?, ?)", "host", "@timestamp", "message" ]
48
66
  }
@@ -58,7 +76,6 @@ input
58
76
  }
59
77
  output {
60
78
  jdbc {
61
- driver_class => 'com.microsoft.sqlserver.jdbc.SQLServerDriver'
62
79
  connection_string => "jdbc:sqlserver://server:1433;databaseName=databasename;user=username;password=password;autoReconnect=true;"
63
80
  statement => [ "INSERT INTO log (host, timestamp, message) VALUES(?, ?, ?)", "host", "@timestamp", "message" ]
64
81
  }
@@ -74,7 +91,6 @@ input
74
91
  }
75
92
  output {
76
93
  jdbc {
77
- driver_class => 'org.postgresql.Driver'
78
94
  connection_string => 'jdbc:postgresql://hostname:5432/database?user=username&password=password'
79
95
  statement => [ "INSERT INTO log (host, timestamp, message) VALUES(?, CAST (? AS timestamp), ?)", "host", "@timestamp", "message" ]
80
96
  }
@@ -92,7 +108,6 @@ input
92
108
  }
93
109
  output {
94
110
  jdbc {
95
- driver_class => "oracle.jdbc.driver.OracleDriver"
96
111
  connection_string => "jdbc:oracle:thin:USER/PASS@HOST:PORT:SID"
97
112
  statement => [ "INSERT INTO log (host, timestamp, message) VALUES(?, CAST (? AS timestamp), ?)", "host", "@timestamp", "message" ]
98
113
  }
@@ -110,9 +125,13 @@ input
110
125
  }
111
126
  output {
112
127
  jdbc {
113
- driver_class => "com.mysql.jdbc.Driver"
114
128
  connection_string => "jdbc:mysql://HOSTNAME/DATABASE?user=USER&password=PASSWORD"
115
129
  statement => [ "INSERT INTO log (host, timestamp, message) VALUES(?, CAST (? AS timestamp), ?)", "host", "@timestamp", "message" ]
116
130
  }
117
131
  }
118
132
  ```
133
+
134
+ ### MariaDB
135
+ This is reportedly working, according to [@db2882](https://github.com/db2882) in issue #20.
136
+ No example configuration provided.
137
+ If you have a working sample, pull requests are welcome.
@@ -0,0 +1,19 @@
1
+ class RingBuffer < Array
2
+ attr_reader :max_size
3
+
4
+ def initialize(max_size, enum = nil)
5
+ @max_size = max_size
6
+ enum.each { |e| self << e } if enum
7
+ end
8
+
9
+ def <<(el)
10
+ if self.size < @max_size || @max_size.nil?
11
+ super
12
+ else
13
+ self.shift
14
+ self.push(el)
15
+ end
16
+ end
17
+
18
+ alias :push :<<
19
+ end
@@ -4,6 +4,7 @@ require "logstash/namespace"
4
4
  require "stud/buffer"
5
5
  require "java"
6
6
  require "logstash-output-jdbc_jars"
7
+ require "logstash-output-jdbc_ring-buffer"
7
8
 
8
9
  class LogStash::Outputs::Jdbc < LogStash::Outputs::Base
9
10
  # Adds buffer support
@@ -12,7 +13,7 @@ class LogStash::Outputs::Jdbc < LogStash::Outputs::Base
12
13
  config_name "jdbc"
13
14
 
14
15
  # Driver class - No longer required
15
- config :driver_class, :obsolete => true
16
+ config :driver_class, :obsolete => "driver_class is no longer required and can be removed from your configuration"
16
17
 
17
18
  # Where to find the jar
18
19
  # Defaults to not required, and to the original behaviour
@@ -58,17 +59,20 @@ class LogStash::Outputs::Jdbc < LogStash::Outputs::Base
58
59
  # a timely manner.
59
60
  #
60
61
  # If you change this value please ensure that you change
61
- # max_repeat_exceptions_time accordingly.
62
+ # max_flush_exceptions accordingly.
62
63
  config :idle_flush_time, :validate => :number, :default => 1
63
64
 
64
- # Maximum number of repeating (sequential) exceptions, before we stop retrying
65
+ # Maximum number of sequential flushes which encounter exceptions, before we stop retrying.
65
66
  # If set to < 1, then it will infinitely retry.
66
- config :max_repeat_exceptions, :validate => :number, :default => 4
67
+ #
68
+ # You should carefully tune this in relation to idle_flush_time if your SQL server
69
+ # is not highly available.
70
+ # i.e. If your idle_flush_time is 1, and your max_flush_exceptions is 200, and your SQL server takes
71
+ # longer than 200 seconds to reboot, then logstash will stop.
72
+ config :max_flush_exceptions, :validate => :number, :default => 0
67
73
 
68
- # The max number of seconds since the last exception, before we consider it
69
- # a different cause.
70
- # This value should be carefully considered in respect to idle_flush_time.
71
- config :max_repeat_exceptions_time, :validate => :number, :default => 30
74
+ config :max_repeat_exceptions, :obsolete => "This has been replaced by max_flush_exceptions - which behaves slightly differently. Please check the documentation."
75
+ config :max_repeat_exceptions_time, :obsolete => "This is no longer required"
72
76
 
73
77
  public
74
78
  def register
@@ -85,17 +89,12 @@ class LogStash::Outputs::Jdbc < LogStash::Outputs::Base
85
89
  @pool.setMaximumPoolSize(@max_pool_size)
86
90
  @pool.setConnectionTimeout(@connection_timeout)
87
91
 
92
+ @exceptions_tracker = RingBuffer.new(@max_flush_exceptions)
93
+
88
94
  if (@flush_size > 1000)
89
95
  @logger.warn("JDBC - Flush size is set to > 1000")
90
96
  end
91
97
 
92
- @repeat_exception_count = 0
93
- @last_exception_time = Time.now
94
-
95
- if (@max_repeat_exceptions > 0) and ((@idle_flush_time * @max_repeat_exceptions) > @max_repeat_exceptions_time)
96
- @logger.warn("JDBC - max_repeat_exceptions_time is set such that it may still permit a looping exception. You probably changed idle_flush_time. Considering increasing max_repeat_exceptions_time.")
97
- end
98
-
99
98
  buffer_initialize(
100
99
  :max_items => @flush_size,
101
100
  :max_interval => @idle_flush_time,
@@ -104,7 +103,7 @@ class LogStash::Outputs::Jdbc < LogStash::Outputs::Base
104
103
  end
105
104
 
106
105
  def receive(event)
107
- return unless output?(event)
106
+ return unless output?(event) or event.cancelled?
108
107
  return unless @statement.length > 0
109
108
 
110
109
  buffer_receive(event)
@@ -119,21 +118,14 @@ class LogStash::Outputs::Jdbc < LogStash::Outputs::Base
119
118
  end
120
119
 
121
120
  def on_flush_error(e)
122
- return if @max_repeat_exceptions < 1
121
+ return if @max_flush_exceptions < 1
123
122
 
124
- if @last_exception == e.to_s
125
- @repeat_exception_count += 1
126
- else
127
- @repeat_exception_count = 0
128
- end
123
+ @exceptions_tracker << e.class
129
124
 
130
- if (@repeat_exception_count >= @max_repeat_exceptions) and (Time.now - @last_exception_time) < @max_repeat_exceptions_time
131
- @logger.error("JDBC - Exception repeated more than the maximum configured", :exception => e, :max_repeat_exceptions => @max_repeat_exceptions, :max_repeat_exceptions_time => @max_repeat_exceptions_time)
125
+ if @exceptions_tracker.reject { |i| i.nil? }.count >= @max_flush_exceptions
126
+ @logger.error("JDBC - max_flush_exceptions has been reached")
132
127
  raise e
133
128
  end
134
-
135
- @last_exception_time = Time.now
136
- @last_exception = e.to_s
137
129
  end
138
130
 
139
131
  def teardown
@@ -173,10 +165,10 @@ class LogStash::Outputs::Jdbc < LogStash::Outputs::Base
173
165
 
174
166
  def safe_flush(events, teardown=false)
175
167
  connection = @pool.getConnection()
176
-
177
168
  statement = connection.prepareStatement(@statement[0])
178
169
 
179
170
  events.each do |event|
171
+ next if event.cancelled?
180
172
  next if @statement.length < 2
181
173
  statement = add_statement_event_params(statement, event)
182
174
 
@@ -184,17 +176,15 @@ class LogStash::Outputs::Jdbc < LogStash::Outputs::Base
184
176
  end
185
177
 
186
178
  begin
187
- @logger.debug("JDBC - Sending SQL", :sql => statement.toString())
188
179
  statement.executeBatch()
189
180
  statement.close()
181
+ @exceptions_tracker << nil
182
+
190
183
  rescue => e
191
184
  # Raising an exception will incur a retry from Stud::Buffer.
192
185
  # Since the exceutebatch failed this should mean any events failed to be
193
186
  # inserted will be re-run. We're going to log it for the lols anyway.
194
- @logger.warn("JDBC - Exception. Will automatically retry", :exception => e)
195
- if e.getNextException() != nil
196
- @logger.warn("JDBC - Exception. Will automatically retry", :exception => e.getNextException())
197
- end
187
+ log_jdbc_exception(e)
198
188
  ensure
199
189
  connection.close();
200
190
  end
@@ -204,13 +194,26 @@ class LogStash::Outputs::Jdbc < LogStash::Outputs::Base
204
194
  connection = @pool.getConnection()
205
195
 
206
196
  events.each do |event|
197
+ next if event.cancelled?
198
+
207
199
  statement = connection.prepareStatement(event.sprintf(@statement[0]))
208
-
209
200
  statement = add_statement_event_params(statement, event) if @statement.length > 1
210
201
 
211
- statement.execute()
212
- statement.close()
213
- connection.close()
202
+ begin
203
+ statement.execute()
204
+
205
+ # cancel the event, since we may end up outputting the same event multiple times
206
+ # if an exception happens later down the line
207
+ event.cancel
208
+ @exceptions_tracker << nil
209
+ rescue => e
210
+ # Raising an exception will incur a retry from Stud::Buffer.
211
+ # We log for the lols.
212
+ log_jdbc_exception(e)
213
+ ensure
214
+ statement.close()
215
+ connection.close()
216
+ end
214
217
  end
215
218
  end
216
219
 
@@ -231,10 +234,23 @@ class LogStash::Outputs::Jdbc < LogStash::Outputs::Base
231
234
  when false
232
235
  statement.setBoolean(idx + 1, false)
233
236
  else
234
- statement.setString(idx + 1, event.sprintf(i))
237
+ if event[i].nil? and i =~ /%\{/
238
+ statement.setString(idx + 1, event.sprintf(i))
239
+ else
240
+ statement.setString(idx + 1, nil)
241
+ end
235
242
  end
236
243
  end
237
244
 
238
245
  statement
239
246
  end
247
+
248
+ def log_jdbc_exception(e)
249
+ ce = e
250
+ loop do
251
+ @logger.error("JDBC Exception encountered: Will automatically retry.", :exception => ce)
252
+ ce = e.getNextException()
253
+ break if ce == nil
254
+ end
255
+ end
240
256
  end # class LogStash::Outputs::jdbc
@@ -0,0 +1,13 @@
1
+ require "logstash/devutils/rspec/spec_helper"
2
+ require "logstash/outputs/jdbc"
3
+ require "stud/temporary"
4
+
5
+ describe LogStash::Outputs::Jdbc do
6
+
7
+ it "should register without errors" do
8
+ plugin = LogStash::Plugin.lookup("output", "jdbc").new({})
9
+ expect { plugin.register }.to_not raise_error
10
+
11
+ end
12
+
13
+ end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: logstash-output-jdbc
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.2.0.rc3
4
+ version: 0.2.0.rc4
5
5
  platform: ruby
6
6
  authors:
7
7
  - the_angry_angel
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2015-11-17 00:00:00.000000000 Z
11
+ date: 2015-11-19 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: logstash-core
@@ -82,9 +82,11 @@ extra_rdoc_files: []
82
82
  files:
83
83
  - lib/logstash/outputs/jdbc.rb
84
84
  - lib/logstash-output-jdbc_jars.rb
85
+ - lib/logstash-output-jdbc_ring-buffer.rb
85
86
  - vendor/jar-dependencies/runtime-jars/HikariCP-2.4.2.jar
86
87
  - vendor/jar-dependencies/runtime-jars/slf4j-api-1.7.13.jar
87
88
  - vendor/jar-dependencies/runtime-jars/slf4j-nop-1.7.13.jar
89
+ - spec/outputs/jdbc_spec.rb
88
90
  - LICENSE.txt
89
91
  - README.md
90
92
  homepage: https://github.com/theangryangel/logstash-output-jdbc
@@ -93,7 +95,8 @@ licenses:
93
95
  metadata:
94
96
  logstash_plugin: 'true'
95
97
  logstash_group: output
96
- post_install_message:
98
+ post_install_message: logstash-output-jdbc 0.2.0 introduces several new features -
99
+ please ensure you check the documentation in the README file
97
100
  rdoc_options: []
98
101
  require_paths:
99
102
  - lib
@@ -113,4 +116,5 @@ rubygems_version: 2.0.14
113
116
  signing_key:
114
117
  specification_version: 4
115
118
  summary: This plugin allows you to output to SQL, via JDBC
116
- test_files: []
119
+ test_files:
120
+ - spec/outputs/jdbc_spec.rb