logstash-input-tcp 4.2.2-java
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +7 -0
- data/CHANGELOG.md +64 -0
- data/CONTRIBUTORS +25 -0
- data/Gemfile +11 -0
- data/LICENSE +13 -0
- data/NOTICE.TXT +5 -0
- data/README.md +98 -0
- data/docs/index.asciidoc +205 -0
- data/lib/logstash-input-tcp_jars.rb +4 -0
- data/lib/logstash/inputs/tcp.rb +387 -0
- data/lib/logstash/inputs/tcp/decoder_impl.rb +55 -0
- data/logstash-input-tcp.gemspec +36 -0
- data/spec/inputs/tcp_spec.rb +421 -0
- data/spec/spec_helper.rb +91 -0
- data/vendor/jar-dependencies/org/logstash/inputs/logstash-input-tcp/4.2.2/logstash-input-tcp-4.2.2.jar +0 -0
- data/version +1 -0
- metadata +181 -0
checksums.yaml
ADDED
@@ -0,0 +1,7 @@
|
|
1
|
+
---
|
2
|
+
SHA1:
|
3
|
+
metadata.gz: 9c6d50fef2efef38a82073875b776b27440e7af2
|
4
|
+
data.tar.gz: e4770a4a7f75dcf1ea81b15835a3f803755e5426
|
5
|
+
SHA512:
|
6
|
+
metadata.gz: 50874dd44e9a968eb27ca855f7960e77e16d21d137432c262cc2d6c8c0c845aef41366b72ec81cb1d711c2fe8b231f863234945dfdffede13299a42263f93dc2
|
7
|
+
data.tar.gz: 7f257c7bc838ddc5f95e4510005e1797ad5aff903eb354966606cfe150a0fc352fa26b9c9c050c5f60fc8a5079d292ba412fe4ba1da55d09915b189be84832ef
|
data/CHANGELOG.md
ADDED
@@ -0,0 +1,64 @@
|
|
1
|
+
## 4.2.2
|
2
|
+
- Fixed regression causing incoming connection host ips being accidentally resolved to hostnames
|
3
|
+
- Implemented plain socket server in a non-blocking way improving performance and fixing issues for use cases with a large number of concurrent connections
|
4
|
+
|
5
|
+
## 4.2.1
|
6
|
+
- Version yanked from RubyGems for accidental behaviour change causing unwanted reverse lookups on connections
|
7
|
+
|
8
|
+
## 4.2.0
|
9
|
+
- Version yanked from RubyGems for packaging issues
|
10
|
+
|
11
|
+
## 4.1.2
|
12
|
+
- Add documentation for how to use tcp input to accept log4j2 data.
|
13
|
+
|
14
|
+
## 4.1.0
|
15
|
+
- Add support for proxy protocol
|
16
|
+
|
17
|
+
## 4.0.3
|
18
|
+
- Relax constraint on logstash-core-plugin-api to >= 1.60 <= 2.99
|
19
|
+
|
20
|
+
## 4.0.2
|
21
|
+
- Change the log level of the SSLError for the handshake from **error** to **debug** https://github.com/logstash-plugins/logstash-input-tcp/pull/53
|
22
|
+
## 4.0.1
|
23
|
+
- Republish all the gems under jruby.
|
24
|
+
## 4.0.0
|
25
|
+
- Update the plugin to the version 2.0 of the plugin api, this change is required for Logstash 5.0 compatibility. See https://github.com/elastic/logstash/issues/5141
|
26
|
+
# 3.0.5
|
27
|
+
- Fixed a bug where using a certificate with a passphrase wouldn't work.
|
28
|
+
# 3.0.4
|
29
|
+
- Depend on logstash-core-plugin-api instead of logstash-core, removing the need to mass update plugins on major releases of logstash
|
30
|
+
# 3.0.3
|
31
|
+
- New dependency requirements for logstash-core for the 5.0 release
|
32
|
+
## 3.0.2
|
33
|
+
- Fixed a bug where previous connection would accidentally be closed when accepting new socket connection
|
34
|
+
- Fixed an issue with log message which used a closed socket's peer address
|
35
|
+
|
36
|
+
## 3.0.1
|
37
|
+
- properly convert sslsubject to string before assigning to event field, added specs, see https://github.com/logstash-plugins/logstash-input-tcp/pull/38
|
38
|
+
|
39
|
+
## 3.0.0
|
40
|
+
- Deprecate ssl_cacert as it's confusing, does it job but when willing to add a chain of certificated the name and behaviour is a bit confusing.
|
41
|
+
- Add ssl_extra_chain_certs that allows you to specify a list of certificates path that will be added to the CAStore.
|
42
|
+
- Make ssl_verify=true as a default value, if using ssl and performing validation is not reasonable as security might be compromised.
|
43
|
+
- Add tests to verify behaviour under different SSL connection circumstances.
|
44
|
+
- Fixes #3 and #9.
|
45
|
+
|
46
|
+
## 2.1.0
|
47
|
+
- Added the receiving port in the event payload, fixes #4
|
48
|
+
|
49
|
+
## 2.0.5
|
50
|
+
- Fixed malformed SSL crashing Logstash, see https://github.com/logstash-plugins/logstash-input-tcp/pull/25
|
51
|
+
|
52
|
+
## 2.0.4
|
53
|
+
- Dependency on logstash-core update to >= 2.0.0.beta2 < 3.0.0
|
54
|
+
|
55
|
+
## 2.0.3
|
56
|
+
- removed usage of RSpec.configure, see https://github.com/logstash-plugins/logstash-input-tcp/pull/21
|
57
|
+
|
58
|
+
## 2.0.2
|
59
|
+
- refactored & cleaned up plugin structure, see https://github.com/logstash-plugins/logstash-input-tcp/pull/18
|
60
|
+
|
61
|
+
## 2.0.0
|
62
|
+
- Plugins were updated to follow the new shutdown semantic, this mainly allows Logstash to instruct input plugins to terminate gracefully,
|
63
|
+
instead of using Thread.raise on the plugins' threads. Ref: https://github.com/elastic/logstash/pull/3895
|
64
|
+
- Dependency on logstash-core update to 2.0
|
data/CONTRIBUTORS
ADDED
@@ -0,0 +1,25 @@
|
|
1
|
+
The following is a list of people who have contributed ideas, code, bug
|
2
|
+
reports, or in general have helped logstash along its way.
|
3
|
+
|
4
|
+
Contributors:
|
5
|
+
* Avishai Ish-Shalom (avishai-ish-shalom)
|
6
|
+
* Bernd Ahlers (bernd)
|
7
|
+
* Colin Surprenant (colinsurprenant)
|
8
|
+
* Dan Peterson (dpiddy)
|
9
|
+
* John E. Vincent (lusis)
|
10
|
+
* Jordan Sissel (jordansissel)
|
11
|
+
* Kurt Hurtado (kurtado)
|
12
|
+
* Ludovicus Maior (Ludovicus)
|
13
|
+
* Matthew Richardson (mrichar1)
|
14
|
+
* Nick Ethier (nickethier)
|
15
|
+
* Pete Fritchman (fetep)
|
16
|
+
* Philippe Weber (wiibaa)
|
17
|
+
* Pier-Hugues Pellerin (ph)
|
18
|
+
* Richard Pijnenburg (electrical)
|
19
|
+
* Scott Wilkerson (scottwilkerson)
|
20
|
+
* Alex Baker (AlexBaker-)
|
21
|
+
|
22
|
+
Note: If you've sent us patches, bug reports, or otherwise contributed to
|
23
|
+
Logstash, and you aren't on the list above and want to be, please let us know
|
24
|
+
and we'll make sure you're here. Contributions from folks like you are what make
|
25
|
+
open source awesome.
|
data/Gemfile
ADDED
@@ -0,0 +1,11 @@
|
|
1
|
+
source 'https://rubygems.org'
|
2
|
+
|
3
|
+
gemspec
|
4
|
+
|
5
|
+
logstash_path = ENV["LOGSTASH_PATH"] || "../../logstash"
|
6
|
+
use_logstash_source = ENV["LOGSTASH_SOURCE"] && ENV["LOGSTASH_SOURCE"].to_s == "1"
|
7
|
+
|
8
|
+
if Dir.exist?(logstash_path) && use_logstash_source
|
9
|
+
gem 'logstash-core', :path => "#{logstash_path}/logstash-core"
|
10
|
+
gem 'logstash-core-plugin-api', :path => "#{logstash_path}/logstash-core-plugin-api"
|
11
|
+
end
|
data/LICENSE
ADDED
@@ -0,0 +1,13 @@
|
|
1
|
+
Copyright (c) 2012–2016 Elasticsearch <http://www.elastic.co>
|
2
|
+
|
3
|
+
Licensed under the Apache License, Version 2.0 (the "License");
|
4
|
+
you may not use this file except in compliance with the License.
|
5
|
+
You may obtain a copy of the License at
|
6
|
+
|
7
|
+
http://www.apache.org/licenses/LICENSE-2.0
|
8
|
+
|
9
|
+
Unless required by applicable law or agreed to in writing, software
|
10
|
+
distributed under the License is distributed on an "AS IS" BASIS,
|
11
|
+
WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
12
|
+
See the License for the specific language governing permissions and
|
13
|
+
limitations under the License.
|
data/NOTICE.TXT
ADDED
data/README.md
ADDED
@@ -0,0 +1,98 @@
|
|
1
|
+
# Logstash Plugin
|
2
|
+
|
3
|
+
[![Travis Build Status](https://travis-ci.org/logstash-plugins/logstash-input-tcp.svg)](https://travis-ci.org/logstash-plugins/logstash-input-tcp)
|
4
|
+
|
5
|
+
This is a plugin for [Logstash](https://github.com/elastic/logstash).
|
6
|
+
|
7
|
+
It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
|
8
|
+
|
9
|
+
## Documentation
|
10
|
+
|
11
|
+
Logstash provides infrastructure to automatically generate documentation for this plugin. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html. All plugin documentation are placed under one [central location](http://www.elastic.co/guide/en/logstash/current/).
|
12
|
+
|
13
|
+
- For formatting code or config example, you can use the asciidoc `[source,ruby]` directive
|
14
|
+
- For more asciidoc formatting tips, see the excellent reference here https://github.com/elastic/docs#asciidoc-guide
|
15
|
+
|
16
|
+
## Need Help?
|
17
|
+
|
18
|
+
Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/logstash discussion forum.
|
19
|
+
|
20
|
+
## Developing
|
21
|
+
|
22
|
+
### 1. Plugin Developement and Testing
|
23
|
+
|
24
|
+
#### Code
|
25
|
+
- To get started, you'll need JRuby with the Bundler gem installed.
|
26
|
+
|
27
|
+
- Create a new plugin or clone and existing from the GitHub [logstash-plugins](https://github.com/logstash-plugins) organization. We also provide [example plugins](https://github.com/logstash-plugins?query=example).
|
28
|
+
|
29
|
+
- Install dependencies
|
30
|
+
```sh
|
31
|
+
bundle install
|
32
|
+
```
|
33
|
+
|
34
|
+
#### Test
|
35
|
+
|
36
|
+
- Update your dependencies
|
37
|
+
|
38
|
+
```sh
|
39
|
+
bundle install
|
40
|
+
```
|
41
|
+
|
42
|
+
- Run tests
|
43
|
+
|
44
|
+
```sh
|
45
|
+
bundle exec rspec
|
46
|
+
```
|
47
|
+
|
48
|
+
### 2. Running your unpublished Plugin in Logstash
|
49
|
+
|
50
|
+
#### 2.1 Run in a local Logstash clone
|
51
|
+
|
52
|
+
- Edit Logstash `Gemfile` and add the local plugin path, for example:
|
53
|
+
```ruby
|
54
|
+
gem "logstash-filter-awesome", :path => "/your/local/logstash-filter-awesome"
|
55
|
+
```
|
56
|
+
- Install plugin
|
57
|
+
```sh
|
58
|
+
# Logstash 2.3 and higher
|
59
|
+
bin/logstash-plugin install --no-verify
|
60
|
+
|
61
|
+
# Prior to Logstash 2.3
|
62
|
+
bin/plugin install --no-verify
|
63
|
+
|
64
|
+
```
|
65
|
+
- Run Logstash with your plugin
|
66
|
+
```sh
|
67
|
+
bin/logstash -e 'filter {awesome {}}'
|
68
|
+
```
|
69
|
+
At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.
|
70
|
+
|
71
|
+
#### 2.2 Run in an installed Logstash
|
72
|
+
|
73
|
+
You can use the same **2.1** method to run your plugin in an installed Logstash by editing its `Gemfile` and pointing the `:path` to your local plugin development directory or you can build the gem and install it using:
|
74
|
+
|
75
|
+
- Build your plugin gem
|
76
|
+
```sh
|
77
|
+
gem build logstash-filter-awesome.gemspec
|
78
|
+
```
|
79
|
+
- Install the plugin from the Logstash home
|
80
|
+
```sh
|
81
|
+
# Logstash 2.3 and higher
|
82
|
+
bin/logstash-plugin install --no-verify
|
83
|
+
|
84
|
+
# Prior to Logstash 2.3
|
85
|
+
bin/plugin install --no-verify
|
86
|
+
|
87
|
+
```
|
88
|
+
- Start Logstash and proceed to test the plugin
|
89
|
+
|
90
|
+
## Contributing
|
91
|
+
|
92
|
+
All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.
|
93
|
+
|
94
|
+
Programming is not a required skill. Whatever you've seen about open source and maintainers or community members saying "send patches or die" - you will not see that here.
|
95
|
+
|
96
|
+
It is more important to the community that you are able to contribute.
|
97
|
+
|
98
|
+
For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file.
|
data/docs/index.asciidoc
ADDED
@@ -0,0 +1,205 @@
|
|
1
|
+
:plugin: tcp
|
2
|
+
:type: input
|
3
|
+
|
4
|
+
///////////////////////////////////////////
|
5
|
+
START - GENERATED VARIABLES, DO NOT EDIT!
|
6
|
+
///////////////////////////////////////////
|
7
|
+
:version: %VERSION%
|
8
|
+
:release_date: %RELEASE_DATE%
|
9
|
+
:changelog_url: %CHANGELOG_URL%
|
10
|
+
:include_path: ../../../../logstash/docs/include
|
11
|
+
///////////////////////////////////////////
|
12
|
+
END - GENERATED VARIABLES, DO NOT EDIT!
|
13
|
+
///////////////////////////////////////////
|
14
|
+
|
15
|
+
[id="plugins-{type}s-{plugin}"]
|
16
|
+
|
17
|
+
=== Tcp input plugin
|
18
|
+
|
19
|
+
include::{include_path}/plugin_header.asciidoc[]
|
20
|
+
|
21
|
+
==== Description
|
22
|
+
|
23
|
+
Read events over a TCP socket.
|
24
|
+
|
25
|
+
Like stdin and file inputs, each event is assumed to be one line of text.
|
26
|
+
|
27
|
+
Can either accept connections from clients or connect to a server,
|
28
|
+
depending on `mode`.
|
29
|
+
|
30
|
+
#### Accepting log4j2 logs
|
31
|
+
|
32
|
+
Log4j2 can send JSON over a socket, and we can use that combined with our tcp
|
33
|
+
input to accept the logs.
|
34
|
+
|
35
|
+
First, we need to configure your application to send logs in JSON over a
|
36
|
+
socket. The following log4j2.xml accomplishes this task.
|
37
|
+
|
38
|
+
Note, you will want to change the `host` and `port` settings in this
|
39
|
+
configuration to match your needs.
|
40
|
+
|
41
|
+
<Configuration>
|
42
|
+
<Appenders>
|
43
|
+
<Socket name="Socket" host="localhost" port="12345">
|
44
|
+
<JsonLayout compact="true" eventEol="true" />
|
45
|
+
</Socket>
|
46
|
+
</Appenders>
|
47
|
+
<Loggers>
|
48
|
+
<Root level="info">
|
49
|
+
<AppenderRef ref="Socket"/>
|
50
|
+
</Root>
|
51
|
+
</Loggers>
|
52
|
+
</Configuration>
|
53
|
+
|
54
|
+
To accept this in Logstash, you will want tcp input and a date filter:
|
55
|
+
|
56
|
+
input {
|
57
|
+
tcp {
|
58
|
+
port => 12345
|
59
|
+
codec => json
|
60
|
+
}
|
61
|
+
}
|
62
|
+
|
63
|
+
and add a date filter to take log4j2's `timeMillis` field and use it as the
|
64
|
+
event timestamp
|
65
|
+
|
66
|
+
filter {
|
67
|
+
date {
|
68
|
+
match => [ "timeMillis", "UNIX_MS" ]
|
69
|
+
}
|
70
|
+
}
|
71
|
+
|
72
|
+
[id="plugins-{type}s-{plugin}-options"]
|
73
|
+
==== Tcp Input Configuration Options
|
74
|
+
|
75
|
+
This plugin supports the following configuration options plus the <<plugins-{type}s-{plugin}-common-options>> described later.
|
76
|
+
|
77
|
+
[cols="<,<,<",options="header",]
|
78
|
+
|=======================================================================
|
79
|
+
|Setting |Input type|Required
|
80
|
+
| <<plugins-{type}s-{plugin}-host>> |<<string,string>>|No
|
81
|
+
| <<plugins-{type}s-{plugin}-mode>> |<<string,string>>, one of `["server", "client"]`|No
|
82
|
+
| <<plugins-{type}s-{plugin}-port>> |<<number,number>>|Yes
|
83
|
+
| <<plugins-{type}s-{plugin}-proxy_protocol>> |<<boolean,boolean>>|No
|
84
|
+
| <<plugins-{type}s-{plugin}-ssl_cert>> |a valid filesystem path|No
|
85
|
+
| <<plugins-{type}s-{plugin}-ssl_enable>> |<<boolean,boolean>>|No
|
86
|
+
| <<plugins-{type}s-{plugin}-ssl_extra_chain_certs>> |<<array,array>>|No
|
87
|
+
| <<plugins-{type}s-{plugin}-ssl_key>> |a valid filesystem path|No
|
88
|
+
| <<plugins-{type}s-{plugin}-ssl_key_passphrase>> |<<password,password>>|No
|
89
|
+
| <<plugins-{type}s-{plugin}-ssl_verify>> |<<boolean,boolean>>|No
|
90
|
+
|=======================================================================
|
91
|
+
|
92
|
+
Also see <<plugins-{type}s-{plugin}-common-options>> for a list of options supported by all
|
93
|
+
input plugins.
|
94
|
+
|
95
|
+
|
96
|
+
|
97
|
+
[id="plugins-{type}s-{plugin}-data_timeout"]
|
98
|
+
===== `data_timeout` (DEPRECATED)
|
99
|
+
|
100
|
+
* DEPRECATED WARNING: This configuration item is deprecated and may not be available in future versions.
|
101
|
+
* Value type is <<number,number>>
|
102
|
+
* Default value is `-1`
|
103
|
+
|
104
|
+
|
105
|
+
|
106
|
+
[id="plugins-{type}s-{plugin}-host"]
|
107
|
+
===== `host`
|
108
|
+
|
109
|
+
* Value type is <<string,string>>
|
110
|
+
* Default value is `"0.0.0.0"`
|
111
|
+
|
112
|
+
When mode is `server`, the address to listen on.
|
113
|
+
When mode is `client`, the address to connect to.
|
114
|
+
|
115
|
+
[id="plugins-{type}s-{plugin}-mode"]
|
116
|
+
===== `mode`
|
117
|
+
|
118
|
+
* Value can be any of: `server`, `client`
|
119
|
+
* Default value is `"server"`
|
120
|
+
|
121
|
+
Mode to operate in. `server` listens for client connections,
|
122
|
+
`client` connects to a server.
|
123
|
+
|
124
|
+
[id="plugins-{type}s-{plugin}-port"]
|
125
|
+
===== `port`
|
126
|
+
|
127
|
+
* This is a required setting.
|
128
|
+
* Value type is <<number,number>>
|
129
|
+
* There is no default value for this setting.
|
130
|
+
|
131
|
+
When mode is `server`, the port to listen on.
|
132
|
+
When mode is `client`, the port to connect to.
|
133
|
+
|
134
|
+
[id="plugins-{type}s-{plugin}-proxy_protocol"]
|
135
|
+
===== `proxy_protocol`
|
136
|
+
|
137
|
+
* Value type is <<boolean,boolean>>
|
138
|
+
* Default value is `false`
|
139
|
+
|
140
|
+
Proxy protocol support, only v1 is supported at this time
|
141
|
+
http://www.haproxy.org/download/1.5/doc/proxy-protocol.txt
|
142
|
+
|
143
|
+
[id="plugins-{type}s-{plugin}-ssl_cacert"]
|
144
|
+
===== `ssl_cacert` (DEPRECATED)
|
145
|
+
|
146
|
+
* DEPRECATED WARNING: This configuration item is deprecated and may not be available in future versions.
|
147
|
+
* Value type is <<path,path>>
|
148
|
+
* There is no default value for this setting.
|
149
|
+
|
150
|
+
The SSL CA certificate, chainfile or CA path. The system CA path is automatically included.
|
151
|
+
|
152
|
+
[id="plugins-{type}s-{plugin}-ssl_cert"]
|
153
|
+
===== `ssl_cert`
|
154
|
+
|
155
|
+
* Value type is <<path,path>>
|
156
|
+
* There is no default value for this setting.
|
157
|
+
|
158
|
+
SSL certificate path
|
159
|
+
|
160
|
+
[id="plugins-{type}s-{plugin}-ssl_enable"]
|
161
|
+
===== `ssl_enable`
|
162
|
+
|
163
|
+
* Value type is <<boolean,boolean>>
|
164
|
+
* Default value is `false`
|
165
|
+
|
166
|
+
Enable SSL (must be set for other `ssl_` options to take effect).
|
167
|
+
|
168
|
+
[id="plugins-{type}s-{plugin}-ssl_extra_chain_certs"]
|
169
|
+
===== `ssl_extra_chain_certs`
|
170
|
+
|
171
|
+
* Value type is <<array,array>>
|
172
|
+
* Default value is `[]`
|
173
|
+
|
174
|
+
An Array of extra X509 certificates to be added to the certificate chain.
|
175
|
+
Useful when the CA chain is not necessary in the system store.
|
176
|
+
|
177
|
+
[id="plugins-{type}s-{plugin}-ssl_key"]
|
178
|
+
===== `ssl_key`
|
179
|
+
|
180
|
+
* Value type is <<path,path>>
|
181
|
+
* There is no default value for this setting.
|
182
|
+
|
183
|
+
SSL key path
|
184
|
+
|
185
|
+
[id="plugins-{type}s-{plugin}-ssl_key_passphrase"]
|
186
|
+
===== `ssl_key_passphrase`
|
187
|
+
|
188
|
+
* Value type is <<password,password>>
|
189
|
+
* Default value is `nil`
|
190
|
+
|
191
|
+
SSL key passphrase
|
192
|
+
|
193
|
+
[id="plugins-{type}s-{plugin}-ssl_verify"]
|
194
|
+
===== `ssl_verify`
|
195
|
+
|
196
|
+
* Value type is <<boolean,boolean>>
|
197
|
+
* Default value is `true`
|
198
|
+
|
199
|
+
Verify the identity of the other end of the SSL connection against the CA.
|
200
|
+
For input, sets the field `sslsubject` to that of the client certificate.
|
201
|
+
|
202
|
+
|
203
|
+
|
204
|
+
[id="plugins-{type}s-{plugin}-common-options"]
|
205
|
+
include::{include_path}/{type}.asciidoc[]
|
@@ -0,0 +1,387 @@
|
|
1
|
+
# encoding: utf-8
|
2
|
+
|
3
|
+
require "java"
|
4
|
+
|
5
|
+
require "logstash/inputs/base"
|
6
|
+
require "logstash/util/socket_peer"
|
7
|
+
require "logstash-input-tcp_jars"
|
8
|
+
require "logstash/inputs/tcp/decoder_impl"
|
9
|
+
|
10
|
+
require "socket"
|
11
|
+
require "openssl"
|
12
|
+
|
13
|
+
java_import org.logstash.tcp.InputLoop
|
14
|
+
|
15
|
+
# Read events over a TCP socket.
|
16
|
+
#
|
17
|
+
# Like stdin and file inputs, each event is assumed to be one line of text.
|
18
|
+
#
|
19
|
+
# Can either accept connections from clients or connect to a server,
|
20
|
+
# depending on `mode`.
|
21
|
+
#
|
22
|
+
# #### Accepting log4j2 logs
|
23
|
+
#
|
24
|
+
# Log4j2 can send JSON over a socket, and we can use that combined with our tcp
|
25
|
+
# input to accept the logs.
|
26
|
+
#
|
27
|
+
# First, we need to configure your application to send logs in JSON over a
|
28
|
+
# socket. The following log4j2.xml accomplishes this task.
|
29
|
+
#
|
30
|
+
# Note, you will want to change the `host` and `port` settings in this
|
31
|
+
# configuration to match your needs.
|
32
|
+
#
|
33
|
+
# <Configuration>
|
34
|
+
# <Appenders>
|
35
|
+
# <Socket name="Socket" host="localhost" port="12345">
|
36
|
+
# <JsonLayout compact="true" eventEol="true" />
|
37
|
+
# </Socket>
|
38
|
+
# </Appenders>
|
39
|
+
# <Loggers>
|
40
|
+
# <Root level="info">
|
41
|
+
# <AppenderRef ref="Socket"/>
|
42
|
+
# </Root>
|
43
|
+
# </Loggers>
|
44
|
+
# </Configuration>
|
45
|
+
#
|
46
|
+
# To accept this in Logstash, you will want tcp input and a date filter:
|
47
|
+
#
|
48
|
+
# input {
|
49
|
+
# tcp {
|
50
|
+
# port => 12345
|
51
|
+
# codec => json
|
52
|
+
# }
|
53
|
+
# }
|
54
|
+
#
|
55
|
+
# and add a date filter to take log4j2's `timeMillis` field and use it as the
|
56
|
+
# event timestamp
|
57
|
+
#
|
58
|
+
# filter {
|
59
|
+
# date {
|
60
|
+
# match => [ "timeMillis", "UNIX_MS" ]
|
61
|
+
# }
|
62
|
+
# }
|
63
|
+
class LogStash::Inputs::Tcp < LogStash::Inputs::Base
|
64
|
+
config_name "tcp"
|
65
|
+
|
66
|
+
default :codec, "line"
|
67
|
+
|
68
|
+
# When mode is `server`, the address to listen on.
|
69
|
+
# When mode is `client`, the address to connect to.
|
70
|
+
config :host, :validate => :string, :default => "0.0.0.0"
|
71
|
+
|
72
|
+
# When mode is `server`, the port to listen on.
|
73
|
+
# When mode is `client`, the port to connect to.
|
74
|
+
config :port, :validate => :number, :required => true
|
75
|
+
|
76
|
+
config :data_timeout, :validate => :number, :default => -1, :deprecated => "This setting is not used by this plugin. It will be removed soon."
|
77
|
+
|
78
|
+
# Mode to operate in. `server` listens for client connections,
|
79
|
+
# `client` connects to a server.
|
80
|
+
config :mode, :validate => ["server", "client"], :default => "server"
|
81
|
+
|
82
|
+
# Proxy protocol support, only v1 is supported at this time
|
83
|
+
# http://www.haproxy.org/download/1.5/doc/proxy-protocol.txt
|
84
|
+
config :proxy_protocol, :validate => :boolean, :default => false
|
85
|
+
|
86
|
+
# Enable SSL (must be set for other `ssl_` options to take effect).
|
87
|
+
config :ssl_enable, :validate => :boolean, :default => false
|
88
|
+
|
89
|
+
# Verify the identity of the other end of the SSL connection against the CA.
|
90
|
+
# For input, sets the field `sslsubject` to that of the client certificate.
|
91
|
+
config :ssl_verify, :validate => :boolean, :default => true
|
92
|
+
|
93
|
+
# The SSL CA certificate, chainfile or CA path. The system CA path is automatically included.
|
94
|
+
config :ssl_cacert, :validate => :path, :deprecated => "This setting is deprecated in favor of ssl_extra_chain_certs as it sets a more clear expectation to add more X509 certificates to the store"
|
95
|
+
|
96
|
+
# SSL certificate path
|
97
|
+
config :ssl_cert, :validate => :path
|
98
|
+
|
99
|
+
# SSL key path
|
100
|
+
config :ssl_key, :validate => :path
|
101
|
+
|
102
|
+
# SSL key passphrase
|
103
|
+
config :ssl_key_passphrase, :validate => :password, :default => nil
|
104
|
+
|
105
|
+
# An Array of extra X509 certificates to be added to the certificate chain.
|
106
|
+
# Useful when the CA chain is not necessary in the system store.
|
107
|
+
config :ssl_extra_chain_certs, :validate => :array, :default => []
|
108
|
+
|
109
|
+
HOST_FIELD = "host".freeze
|
110
|
+
PORT_FIELD = "port".freeze
|
111
|
+
PROXY_HOST_FIELD = "proxy_host".freeze
|
112
|
+
PROXY_PORT_FIELD = "proxy_port".freeze
|
113
|
+
SSLSUBJECT_FIELD = "sslsubject".freeze
|
114
|
+
|
115
|
+
def initialize(*args)
|
116
|
+
super(*args)
|
117
|
+
|
118
|
+
# monkey patch TCPSocket and SSLSocket to include socket peer
|
119
|
+
TCPSocket.module_eval{include ::LogStash::Util::SocketPeer}
|
120
|
+
OpenSSL::SSL::SSLSocket.module_eval{include ::LogStash::Util::SocketPeer}
|
121
|
+
|
122
|
+
# threadsafe socket bookkeeping
|
123
|
+
@server_socket = nil
|
124
|
+
@client_socket = nil
|
125
|
+
@connection_sockets = {}
|
126
|
+
@socket_mutex = Mutex.new
|
127
|
+
|
128
|
+
@ssl_context = nil
|
129
|
+
end
|
130
|
+
|
131
|
+
def register
|
132
|
+
fix_streaming_codecs
|
133
|
+
|
134
|
+
# note that since we are opening a socket in register, we must also make sure we close it
|
135
|
+
# in the close method even if we also close it in the stop method since we could have
|
136
|
+
# a situation where register is called but not run & stop.
|
137
|
+
|
138
|
+
if server?
|
139
|
+
if @ssl_enable
|
140
|
+
self.server_socket = new_server_socket
|
141
|
+
else
|
142
|
+
@loop = InputLoop.new(@host, @port, DecoderImpl.new(@codec, self))
|
143
|
+
end
|
144
|
+
end
|
145
|
+
end
|
146
|
+
|
147
|
+
def run(output_queue)
|
148
|
+
@output_queue = output_queue
|
149
|
+
if server?
|
150
|
+
if @ssl_enable
|
151
|
+
run_ssl_server
|
152
|
+
else
|
153
|
+
@loop.run
|
154
|
+
end
|
155
|
+
else
|
156
|
+
run_client()
|
157
|
+
end
|
158
|
+
end
|
159
|
+
|
160
|
+
def stop
|
161
|
+
# force close all sockets which will escape any blocking read with a IO exception
|
162
|
+
# and any thread using them will exit.
|
163
|
+
# catch all rescue nil on close to discard any close errors or invalid socket
|
164
|
+
server_socket.close rescue nil
|
165
|
+
@loop.close rescue nil
|
166
|
+
client_socket.close rescue nil
|
167
|
+
connection_sockets.each{|socket| socket.close rescue nil}
|
168
|
+
end
|
169
|
+
|
170
|
+
def close
|
171
|
+
# see related comment in register: we must make sure to close the server socket here
|
172
|
+
# because it is created in the register method and we could be in the context of having
|
173
|
+
# register called but never run & stop, only close.
|
174
|
+
# catch all rescue nil on close to discard any close errors or invalid socket
|
175
|
+
server_socket.close rescue nil
|
176
|
+
@loop.close rescue nil
|
177
|
+
end
|
178
|
+
|
179
|
+
def decode_buffer(client_address, client_port, codec, proxy_address, proxy_port, tbuf)
|
180
|
+
codec.decode(tbuf) do |event|
|
181
|
+
if @proxy_protocol
|
182
|
+
event.set(PROXY_HOST_FIELD, proxy_address) unless event.get(PROXY_HOST_FIELD)
|
183
|
+
event.set(PROXY_PORT_FIELD, proxy_port) unless event.get(PROXY_PORT_FIELD)
|
184
|
+
end
|
185
|
+
enqueue_decorated(event, client_address, client_port)
|
186
|
+
end
|
187
|
+
end
|
188
|
+
|
189
|
+
private
|
190
|
+
|
191
|
+
def run_ssl_server()
|
192
|
+
while !stop?
|
193
|
+
begin
|
194
|
+
socket = add_connection_socket(server_socket.accept)
|
195
|
+
# start a new thread for each connection.
|
196
|
+
server_connection_thread(socket)
|
197
|
+
rescue OpenSSL::SSL::SSLError => e
|
198
|
+
# log error, close socket, accept next connection
|
199
|
+
@logger.debug? && @logger.debug("SSL Error", :exception => e, :backtrace => e.backtrace)
|
200
|
+
rescue => e
|
201
|
+
# if this exception occured while the plugin is stopping
|
202
|
+
# just ignore and exit
|
203
|
+
raise e unless stop?
|
204
|
+
end
|
205
|
+
end
|
206
|
+
ensure
|
207
|
+
# catch all rescue nil on close to discard any close errors or invalid socket
|
208
|
+
server_socket.close rescue nil
|
209
|
+
end
|
210
|
+
|
211
|
+
def run_client()
|
212
|
+
while !stop?
|
213
|
+
self.client_socket = new_client_socket
|
214
|
+
handle_socket(client_socket)
|
215
|
+
end
|
216
|
+
ensure
|
217
|
+
# catch all rescue nil on close to discard any close errors or invalid socket
|
218
|
+
client_socket.close rescue nil
|
219
|
+
end
|
220
|
+
|
221
|
+
def server_connection_thread(socket)
|
222
|
+
Thread.new(socket) do |s|
|
223
|
+
begin
|
224
|
+
@logger.debug? && @logger.debug("Accepted connection", :client => s.peer, :server => "#{@host}:#{@port}")
|
225
|
+
handle_socket(s)
|
226
|
+
ensure
|
227
|
+
delete_connection_socket(s)
|
228
|
+
end
|
229
|
+
end
|
230
|
+
end
|
231
|
+
|
232
|
+
def handle_socket(socket)
|
233
|
+
client_address = socket.peeraddr[3]
|
234
|
+
client_port = socket.peeraddr[1]
|
235
|
+
peer = "#{client_address}:#{client_port}"
|
236
|
+
first_read = true
|
237
|
+
codec = @codec.clone
|
238
|
+
while !stop?
|
239
|
+
tbuf = socket.sysread(16384)
|
240
|
+
if @proxy_protocol && first_read
|
241
|
+
first_read = false
|
242
|
+
pp_hdr, tbuf = tbuf.split("\r\n", 2)
|
243
|
+
|
244
|
+
pp_info = pp_hdr.split(/\s/)
|
245
|
+
# PROXY proto clientip proxyip clientport proxyport
|
246
|
+
if pp_info[0] != "PROXY"
|
247
|
+
@logger.error("invalid proxy protocol header label", :hdr => pp_hdr)
|
248
|
+
raise IOError
|
249
|
+
else
|
250
|
+
proxy_address = pp_info[3]
|
251
|
+
proxy_port = pp_info[5]
|
252
|
+
client_address = pp_info[2]
|
253
|
+
client_port = pp_info[4]
|
254
|
+
end
|
255
|
+
end
|
256
|
+
decode_buffer(client_address, client_port, codec, proxy_address, proxy_port, tbuf)
|
257
|
+
end
|
258
|
+
rescue EOFError
|
259
|
+
@logger.debug? && @logger.debug("Connection closed", :client => peer)
|
260
|
+
rescue Errno::ECONNRESET
|
261
|
+
@logger.debug? && @logger.debug("Connection reset by peer", :client => peer)
|
262
|
+
rescue OpenSSL::SSL::SSLError => e
|
263
|
+
# Fixes issue #23
|
264
|
+
@logger.error("SSL Error", :exception => e, :backtrace => e.backtrace)
|
265
|
+
socket.close rescue nil
|
266
|
+
rescue => e
|
267
|
+
# if plugin is stopping, don't bother logging it as an error
|
268
|
+
!stop? && @logger.error("An error occurred. Closing connection", :client => peer, :exception => e, :backtrace => e.backtrace)
|
269
|
+
ensure
|
270
|
+
# catch all rescue nil on close to discard any close errors or invalid socket
|
271
|
+
socket.close rescue nil
|
272
|
+
|
273
|
+
codec.respond_to?(:flush) && codec.flush do |event|
|
274
|
+
enqueue_decorated(event, client_address, client_port)
|
275
|
+
end
|
276
|
+
end
|
277
|
+
|
278
|
+
def enqueue_decorated(event, client_address, client_port)
|
279
|
+
event.set(HOST_FIELD, client_address) unless event.get(HOST_FIELD)
|
280
|
+
event.set(PORT_FIELD, client_port) unless event.get(PORT_FIELD)
|
281
|
+
decorate(event)
|
282
|
+
@output_queue << event
|
283
|
+
end
|
284
|
+
|
285
|
+
def server?
|
286
|
+
@mode == "server"
|
287
|
+
end
|
288
|
+
|
289
|
+
def ssl_context
|
290
|
+
return @ssl_context if @ssl_context
|
291
|
+
|
292
|
+
begin
|
293
|
+
@ssl_context = OpenSSL::SSL::SSLContext.new
|
294
|
+
@ssl_context.cert = OpenSSL::X509::Certificate.new(File.read(@ssl_cert))
|
295
|
+
@ssl_context.key = OpenSSL::PKey::RSA.new(File.read(@ssl_key),@ssl_key_passphrase.value)
|
296
|
+
if @ssl_verify
|
297
|
+
@ssl_context.cert_store = load_cert_store
|
298
|
+
@ssl_context.verify_mode = OpenSSL::SSL::VERIFY_PEER|OpenSSL::SSL::VERIFY_FAIL_IF_NO_PEER_CERT
|
299
|
+
end
|
300
|
+
rescue => e
|
301
|
+
@logger.error("Could not inititalize SSL context", :exception => e, :backtrace => e.backtrace)
|
302
|
+
raise e
|
303
|
+
end
|
304
|
+
|
305
|
+
@ssl_context
|
306
|
+
end
|
307
|
+
|
308
|
+
def load_cert_store
|
309
|
+
cert_store = OpenSSL::X509::Store.new
|
310
|
+
cert_store.set_default_paths
|
311
|
+
if File.directory?(@ssl_cacert)
|
312
|
+
cert_store.add_path(@ssl_cacert)
|
313
|
+
else
|
314
|
+
cert_store.add_file(@ssl_cacert)
|
315
|
+
end if @ssl_cacert
|
316
|
+
@ssl_extra_chain_certs.each do |cert|
|
317
|
+
cert_store.add_file(cert)
|
318
|
+
end
|
319
|
+
cert_store
|
320
|
+
end
|
321
|
+
|
322
|
+
def new_server_socket
|
323
|
+
@logger.info("Starting tcp input listener", :address => "#{@host}:#{@port}")
|
324
|
+
begin
|
325
|
+
socket = TCPServer.new(@host, @port)
|
326
|
+
rescue Errno::EADDRINUSE
|
327
|
+
@logger.error("Could not start TCP server: Address in use", :host => @host, :port => @port)
|
328
|
+
raise
|
329
|
+
end
|
330
|
+
|
331
|
+
@ssl_enable ? OpenSSL::SSL::SSLServer.new(socket, ssl_context) : socket
|
332
|
+
end
|
333
|
+
|
334
|
+
def new_client_socket
|
335
|
+
socket = TCPSocket.new(@host, @port)
|
336
|
+
|
337
|
+
if @ssl_enable
|
338
|
+
socket = OpenSSL::SSL::SSLSocket.new(socket, ssl_context)
|
339
|
+
socket.connect
|
340
|
+
end
|
341
|
+
|
342
|
+
@logger.debug? && @logger.debug("Opened connection", :client => "#{socket.peer}")
|
343
|
+
|
344
|
+
socket
|
345
|
+
rescue OpenSSL::SSL::SSLError => e
|
346
|
+
@logger.error("SSL Error", :exception => e, :backtrace => e.backtrace)
|
347
|
+
# catch all rescue nil on close to discard any close errors or invalid socket
|
348
|
+
socket.close rescue nil
|
349
|
+
sleep(1) # prevent hammering peer
|
350
|
+
retry
|
351
|
+
rescue
|
352
|
+
# if this exception occured while the plugin is stopping
|
353
|
+
# just ignore and exit
|
354
|
+
raise unless stop?
|
355
|
+
end
|
356
|
+
|
357
|
+
# threadsafe sockets bookkeeping
|
358
|
+
|
359
|
+
def client_socket=(socket)
|
360
|
+
@socket_mutex.synchronize{@client_socket = socket}
|
361
|
+
end
|
362
|
+
|
363
|
+
def client_socket
|
364
|
+
@socket_mutex.synchronize{@client_socket}
|
365
|
+
end
|
366
|
+
|
367
|
+
def server_socket=(socket)
|
368
|
+
@socket_mutex.synchronize{@server_socket = socket}
|
369
|
+
end
|
370
|
+
|
371
|
+
def server_socket
|
372
|
+
@socket_mutex.synchronize{@server_socket}
|
373
|
+
end
|
374
|
+
|
375
|
+
def add_connection_socket(socket)
|
376
|
+
@socket_mutex.synchronize{@connection_sockets[socket] = true}
|
377
|
+
socket
|
378
|
+
end
|
379
|
+
|
380
|
+
def delete_connection_socket(socket)
|
381
|
+
@socket_mutex.synchronize{@connection_sockets.delete(socket)}
|
382
|
+
end
|
383
|
+
|
384
|
+
def connection_sockets
|
385
|
+
@socket_mutex.synchronize{@connection_sockets.keys.dup}
|
386
|
+
end
|
387
|
+
end
|