logstash-output-jdbc 0.2.2.rc1 → 0.2.2
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +6 -117
- data/lib/logstash/outputs/jdbc.rb +33 -13
- metadata +4 -4
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA1:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 9d9c04c7056a8af5071a1da2710a14339093abf3
|
4
|
+
data.tar.gz: 4dd0af7eca9417f9c51944b0fb325c42c4cc4ea7
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: abb87d28ba5e8cb5031d5a9a3b23ab44d925608cd8ce81646406366a4f45bac690c356618c74e1b28b57fe63f3c48b6c0cdea78e095882126da36ef73d52634c
|
7
|
+
data.tar.gz: 7ea9fc64a0a90155228dc276af38d394ac135e5cbb91af37ee05cad8b64d8355c339da076eaabdc5e2e3fa404ad86af2fc16d3744ca079a38fb830516bb4384e
|
data/README.md
CHANGED
@@ -17,6 +17,9 @@ particular location. Please ensure you read the 4 installation lines below.
|
|
17
17
|
- Altered exception handling to now count sequential flushes with exceptions thrown in 0.2.0
|
18
18
|
|
19
19
|
## Versions
|
20
|
+
Released versions are are tagged as of v0.2.1, and available via rubygems.
|
21
|
+
|
22
|
+
For development:
|
20
23
|
- See master branch for logstash v2+
|
21
24
|
- See v1.5 branch for logstash v1.5
|
22
25
|
- See v1.4 branch for logstash 1.4
|
@@ -24,7 +27,7 @@ particular location. Please ensure you read the 4 installation lines below.
|
|
24
27
|
## Installation
|
25
28
|
- Run `bin/plugin install logstash-output-jdbc` in your logstash installation directory
|
26
29
|
- Now either:
|
27
|
-
- Use
|
30
|
+
- Use driver_path in your configuraton to specify a path to your jar file
|
28
31
|
- Or:
|
29
32
|
- Create the directory vendor/jar/jdbc in your logstash installation (`mkdir -p vendor/jar/jdbc/`)
|
30
33
|
- Add JDBC jar files to vendor/jar/jdbc in your logstash installation
|
@@ -56,120 +59,6 @@ Tests are not yet 100% complete.
|
|
56
59
|
| max_flush_exceptions | Number | Number of sequential flushes which cause an exception, before we stop logstash. Set to a value less than 1 if you never want it to stop. This should be carefully configured with relation to idle_flush_time if your SQL instance is not highly available. | No | 0 |
|
57
60
|
|
58
61
|
## Example configurations
|
59
|
-
|
60
|
-
|
61
|
-
### SQLite3
|
62
|
-
* Tested using https://bitbucket.org/xerial/sqlite-jdbc
|
63
|
-
* SQLite setup - `echo "CREATE table log (host text, timestamp datetime, message text);" | sqlite3 test.db`
|
64
|
-
```
|
65
|
-
input
|
66
|
-
{
|
67
|
-
stdin { }
|
68
|
-
}
|
69
|
-
output {
|
70
|
-
stdout { }
|
71
|
-
|
72
|
-
jdbc {
|
73
|
-
connection_string => 'jdbc:sqlite:test.db'
|
74
|
-
statement => [ "INSERT INTO log (host, timestamp, message) VALUES(?, ?, ?)", "host", "@timestamp", "message" ]
|
75
|
-
}
|
76
|
-
}
|
77
|
-
```
|
78
|
-
|
79
|
-
### SQL Server
|
80
|
-
* Tested using http://msdn.microsoft.com/en-gb/sqlserver/aa937724.aspx
|
81
|
-
```
|
82
|
-
input
|
83
|
-
{
|
84
|
-
stdin { }
|
85
|
-
}
|
86
|
-
output {
|
87
|
-
jdbc {
|
88
|
-
connection_string => "jdbc:sqlserver://server:1433;databaseName=databasename;user=username;password=password;autoReconnect=true;"
|
89
|
-
statement => [ "INSERT INTO log (host, timestamp, message) VALUES(?, ?, ?)", "host", "@timestamp", "message" ]
|
90
|
-
}
|
91
|
-
}
|
92
|
-
```
|
93
|
-
|
94
|
-
### Postgres
|
95
|
-
With thanks to [@roflmao](https://github.com/roflmao)
|
96
|
-
```
|
97
|
-
input
|
98
|
-
{
|
99
|
-
stdin { }
|
100
|
-
}
|
101
|
-
output {
|
102
|
-
jdbc {
|
103
|
-
connection_string => 'jdbc:postgresql://hostname:5432/database?user=username&password=password'
|
104
|
-
statement => [ "INSERT INTO log (host, timestamp, message) VALUES(?, CAST (? AS timestamp), ?)", "host", "@timestamp", "message" ]
|
105
|
-
}
|
106
|
-
}
|
107
|
-
```
|
62
|
+
Example logstash configurations, can now be found in the examples directory. Where possible we try to link every configuration with a tested jar.
|
108
63
|
|
109
|
-
|
110
|
-
With thanks to [@josemazo](https://github.com/josemazo)
|
111
|
-
* Tested with Express Edition 11g Release 2
|
112
|
-
* Tested using http://www.oracle.com/technetwork/database/enterprise-edition/jdbc-112010-090769.html (ojdbc6.jar)
|
113
|
-
```
|
114
|
-
input
|
115
|
-
{
|
116
|
-
stdin { }
|
117
|
-
}
|
118
|
-
output {
|
119
|
-
jdbc {
|
120
|
-
connection_string => "jdbc:oracle:thin:USER/PASS@HOST:PORT:SID"
|
121
|
-
statement => [ "INSERT INTO log (host, timestamp, message) VALUES(?, CAST (? AS timestamp), ?)", "host", "@timestamp", "message" ]
|
122
|
-
}
|
123
|
-
}
|
124
|
-
```
|
125
|
-
|
126
|
-
### Mysql
|
127
|
-
With thanks to [@jMonsinjon](https://github.com/jMonsinjon)
|
128
|
-
* Tested with Version 14.14 Distrib 5.5.43, for debian-linux-gnu (x86_64)
|
129
|
-
* Tested using http://dev.mysql.com/downloads/file.php?id=457911 (mysql-connector-java-5.1.36-bin.jar)
|
130
|
-
```
|
131
|
-
input
|
132
|
-
{
|
133
|
-
stdin { }
|
134
|
-
}
|
135
|
-
output {
|
136
|
-
jdbc {
|
137
|
-
connection_string => "jdbc:mysql://HOSTNAME/DATABASE?user=USER&password=PASSWORD"
|
138
|
-
statement => [ "INSERT INTO log (host, timestamp, message) VALUES(?, CAST (? AS timestamp), ?)", "host", "@timestamp", "message" ]
|
139
|
-
}
|
140
|
-
}
|
141
|
-
```
|
142
|
-
|
143
|
-
### MariaDB
|
144
|
-
* Tested with Ubuntu 14.04.3 LTS, Server version: 10.1.9-MariaDB-1~trusty-log mariadb.org binary distribution
|
145
|
-
* Tested using https://downloads.mariadb.com/enterprise/tqge-whfa/connectors/java/connector-java-1.3.2/mariadb-java-client-1.3.2.jar (mariadb-java-client-1.3.2.jar)
|
146
|
-
```
|
147
|
-
input
|
148
|
-
{
|
149
|
-
stdin { }
|
150
|
-
}
|
151
|
-
output {
|
152
|
-
jdbc {
|
153
|
-
connection_string => "jdbc:mariadb://HOSTNAME/DATABASE?user=USER&password=PASSWORD"
|
154
|
-
statement => [ "INSERT INTO log (host, timestamp, message) VALUES(?, ?, ?)", "host", "@timestamp", "message" ]
|
155
|
-
}
|
156
|
-
|
157
|
-
}
|
158
|
-
```
|
159
|
-
|
160
|
-
### Apache Phoenix (HBase SQL)
|
161
|
-
* Tested with Ubuntu 14.04.03 / Logstash 2.1 / Apache Phoenix 4.6
|
162
|
-
* <!> HBase and Zookeeper must be both accessible from logstash machine <!>
|
163
|
-
```
|
164
|
-
input
|
165
|
-
{
|
166
|
-
stdin { }
|
167
|
-
}
|
168
|
-
output {
|
169
|
-
jdbc {
|
170
|
-
connection_string => "jdbc:phoenix:ZOOKEEPER_HOSTNAME"
|
171
|
-
statement => [ "UPSERT INTO EVENTS log (host, timestamp, message) VALUES(?, ?, ?)", "host", "@timestamp", "message" ]
|
172
|
-
}
|
173
|
-
|
174
|
-
}
|
175
|
-
```
|
64
|
+
If you have a working sample configuration, for a DB thats not listed, pull requests are welcome.
|
@@ -83,25 +83,22 @@ class LogStash::Outputs::Jdbc < LogStash::Outputs::Base
|
|
83
83
|
|
84
84
|
load_jar_files!
|
85
85
|
|
86
|
-
@pool = Java::ComZaxxerHikari::HikariDataSource.new
|
87
|
-
|
88
|
-
@pool.setAutoCommit(@driver_auto_commit)
|
89
|
-
@pool.setDriverClassName(@driver_class) if @driver_class
|
90
|
-
|
91
|
-
@pool.setJdbcUrl(@connection_string)
|
92
|
-
|
93
|
-
@pool.setUsername(@username) if @username
|
94
|
-
@pool.setPassword(@password) if @password
|
95
|
-
|
96
|
-
@pool.setMaximumPoolSize(@max_pool_size)
|
97
|
-
@pool.setConnectionTimeout(@connection_timeout)
|
98
|
-
|
99
86
|
@exceptions_tracker = RingBuffer.new(@max_flush_exceptions)
|
100
87
|
|
101
88
|
if (@flush_size > 1000)
|
102
89
|
@logger.warn("JDBC - Flush size is set to > 1000")
|
103
90
|
end
|
104
91
|
|
92
|
+
if @statement.length < 1
|
93
|
+
@logger.error("JDBC - No statement provided. Configuration error.")
|
94
|
+
end
|
95
|
+
|
96
|
+
if (!@unsafe_statement and @statement.length < 2)
|
97
|
+
@logger.error("JDBC - Statement has no parameters. No events will be inserted into SQL as you're not passing any event data. Likely configuration error.")
|
98
|
+
end
|
99
|
+
|
100
|
+
setup_and_test_pool!
|
101
|
+
|
105
102
|
buffer_initialize(
|
106
103
|
:max_items => @flush_size,
|
107
104
|
:max_interval => @idle_flush_time,
|
@@ -144,6 +141,29 @@ class LogStash::Outputs::Jdbc < LogStash::Outputs::Base
|
|
144
141
|
|
145
142
|
private
|
146
143
|
|
144
|
+
def setup_and_test_pool!
|
145
|
+
# Setup pool
|
146
|
+
@pool = Java::ComZaxxerHikari::HikariDataSource.new
|
147
|
+
|
148
|
+
@pool.setAutoCommit(@driver_auto_commit)
|
149
|
+
@pool.setDriverClassName(@driver_class) if @driver_class
|
150
|
+
|
151
|
+
@pool.setJdbcUrl(@connection_string)
|
152
|
+
|
153
|
+
@pool.setUsername(@username) if @username
|
154
|
+
@pool.setPassword(@password) if @password
|
155
|
+
|
156
|
+
@pool.setMaximumPoolSize(@max_pool_size)
|
157
|
+
@pool.setConnectionTimeout(@connection_timeout)
|
158
|
+
|
159
|
+
# Test connection
|
160
|
+
test_connection = @pool.getConnection()
|
161
|
+
unless test_connection.isValid(10)
|
162
|
+
@logger.error("JDBC - Connection is not valid. Please check connection string or that your JDBC endpoint is available.")
|
163
|
+
end
|
164
|
+
test_connection.close()
|
165
|
+
end
|
166
|
+
|
147
167
|
def load_jar_files!
|
148
168
|
# Load jar from driver path
|
149
169
|
unless @driver_jar_path.nil?
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: logstash-output-jdbc
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.2.2
|
4
|
+
version: 0.2.2
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- the_angry_angel
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2015-12-
|
11
|
+
date: 2015-12-30 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: logstash-core
|
@@ -107,9 +107,9 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
107
107
|
version: '0'
|
108
108
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
109
109
|
requirements:
|
110
|
-
- - '
|
110
|
+
- - '>='
|
111
111
|
- !ruby/object:Gem::Version
|
112
|
-
version:
|
112
|
+
version: '0'
|
113
113
|
requirements: []
|
114
114
|
rubyforge_project:
|
115
115
|
rubygems_version: 2.0.14
|