fluent-plugin-syslog-tls-with-backoff-test 2.1.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
@@ -0,0 +1,139 @@
1
+ # Configuration
2
+
3
+ This plugin allows to configure all syslog message attributes.
4
+
5
+ ## Static configuration
6
+
7
+ Static configuration allow to define connection details, facility and hostname that will
8
+ apply to all messages.
9
+
10
+ ### host
11
+
12
+ Host represents DNS name of endpoint where should be data sent. Example: `syslog.collection.us1.sumologic.com` or `logs1.papertrailapp.com`
13
+
14
+ ### port
15
+
16
+ Example: `6514`
17
+
18
+ ### idle_timeout
19
+
20
+ If a given tag has gone this many seconds between log messages, disconnect and reconnect before sending logs. Useful in low-traffic logging situations with remote hosts that disconnect after a period of time. Disabled by default. Example: `600`
21
+
22
+ ### ca_cert
23
+
24
+ Whether and how to verify the server's TLS certificate signing chain. Examples:
25
+ * ca_cert system - Default; use the system CA certificate store (which must then be configured correctly)
26
+ * ca_cert false - Disable verification; not recommended
27
+ * ca_cert /path/to/file - A path+filename to a single CA file
28
+ * ca_cert /path/to/dir/ - A directory of CA files (in format that OpenSSL can parse); must end with /
29
+
30
+ ### verify_cert_name
31
+
32
+ Whether to verify that the server's cert matches `host`. Enabled by default (except when `ca_cert false`). Recommended; helps prevent MitM attacks. Example: `true`
33
+
34
+ ### token
35
+
36
+ Some services require a token to identify the account. Example: `ABABABABABABA@99999`. Not required for Papertrail.
37
+
38
+ ### client_cert
39
+
40
+ Optionally path to client certificate for TLS connection. Example: `/path/to/crt/file.crt`
41
+
42
+ ### client_key
43
+
44
+ Optionally path to client private key for TLS connection. Example: `/path/to/key/file.key`
45
+
46
+ ### hostname
47
+
48
+ Default hostname that is going to be sent in syslog message. Example: `ip-10-0-0-10`
49
+
50
+ ### facility
51
+
52
+ Default syslog facility for all records. Example: `LOCAL0`
53
+
54
+ ## Per message configuration
55
+
56
+ Its possible to configure the plugin to extract various parts of syslog message header
57
+ from processed record itself. That allows to dynamically set app_name, procid, msgid from
58
+ records if single match is being used to log messages from multiple sources.
59
+
60
+ ### severity_key
61
+
62
+ Optionally record key where to get severity from the record. If not provided default `INFO` will be used.
63
+
64
+ ### facility_key
65
+
66
+ Optionally record key where to get syslog facility from the record. If not provided default `LOCAL0` will be used.
67
+
68
+ ### hostname_key
69
+
70
+ Optionally record key where to get hostname from the record. If not provided hostname is determined by system hostname.
71
+
72
+ ### app_name_key
73
+
74
+ Optionally record key where to get app_name from the record. If not provided nil value will be sent.
75
+
76
+ ### procid_key
77
+
78
+ Optionally record key where to get procid from the record. If not provided nil value will be sent.
79
+
80
+ ### msgid_key
81
+
82
+ Optionally record key where to get msgid from the record. If not provided nil value will be sent.
83
+
84
+ ## Example
85
+
86
+ ```
87
+ <match>
88
+ @type syslog_tls
89
+ host logs1.papertrailapp.com
90
+ port 12345
91
+ idle_timeout 720
92
+
93
+ hostname static-hostname
94
+ facility SYSLOG
95
+
96
+ # You can configure syslog headers to be picked from actual message
97
+ # processed by plugin. If key is not provided '-' value will be sent
98
+ # which is NIL by syslog specification.
99
+ severity_key RECORD_SEVERITY_KEY
100
+ facility_key RECORD_FACILITY_KEY
101
+ hostname_key ...
102
+ app_name_key ...
103
+ procid_key ...
104
+ msgid_key ...
105
+
106
+ # Fluent's standard formatting options are supported. Default is 'json'.
107
+ # Example: For Docker logs sent to Papertrail, send only the log text:
108
+ format single_value
109
+ message_key log
110
+ </match>
111
+ ```
112
+
113
+ ```
114
+ <match>
115
+ @type syslog_tls
116
+ host syslog.collection.us1.sumologic.com
117
+ port 6514
118
+ token [token]@[iana-id]
119
+ client_cert /path/to/cert/file.crt
120
+ client_key /path/to/key/file.key
121
+ verify_cert_name true
122
+
123
+ hostname static-hostname
124
+ facility SYSLOG
125
+
126
+ # You can configure syslog headers to be picked from actual message
127
+ # processed by plugin. If key is not provided '-' value will be sent
128
+ # which is NIL by syslog specification.
129
+ severity_key RECORD_SEVERITY_KEY
130
+ facility_key RECORD_FACILITY_KEY
131
+ hostname_key ...
132
+ app_name_key ...
133
+ procid_key ...
134
+ msgid_key ...
135
+
136
+ # Fluent's standard formatting options are supported. Default is 'json'.
137
+ format json
138
+ </match>
139
+ ```
@@ -0,0 +1,43 @@
1
+ # Copyright 2016 Acquia, Inc.
2
+ # Copyright 2016-2023 t.e.morgan.
3
+ #
4
+ # Licensed under the Apache License, Version 2.0 (the "License");
5
+ # you may not use this file except in compliance with the License.
6
+ # You may obtain a copy of the License at
7
+ #
8
+ # http://www.apache.org/licenses/LICENSE-2.0
9
+ #
10
+ # Unless required by applicable law or agreed to in writing, software
11
+ # distributed under the License is distributed on an "AS IS" BASIS,
12
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13
+ # See the License for the specific language governing permissions and
14
+ # limitations under the License.
15
+
16
+ lib = File.expand_path('../lib', __FILE__)
17
+ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
18
+ require 'syslog_tls/version'
19
+
20
+ Gem::Specification.new do |s|
21
+ s.name = 'fluent-plugin-syslog-tls-with-backoff-test'
22
+ s.version = SyslogTls::VERSION
23
+ s.authors = ['thomas morgan', 'muhammad adil ghaffar']
24
+ s.email = ['tm@iprog.com', 'muhammad.sdil.ghaffar@est.tech']
25
+ s.summary = %q{Fluent Syslog TLS output plugin}
26
+ s.description = %q{Syslog TLS output plugin with formatting support, for Fluentd. This is cutom version backoff.}
27
+ s.homepage = 'https://github.com/Nordix/fluent-plugin-syslog-tls/tree/add-expo-backoff/adil'
28
+ s.license = 'Apache v2'
29
+ s.files = `git ls-files`.split($/)
30
+ s.executables = s.files.grep(%r{^bin/}).map{ |f| File.basename(f) }
31
+ s.test_files = s.files.grep(%r{^(test|spec|features)/})
32
+ s.require_paths = ['lib']
33
+ s.required_ruby_version = '>= 2.5'
34
+
35
+ s.add_runtime_dependency 'fluentd', [">= 0.14.0", "< 2"]
36
+
37
+ s.add_development_dependency 'minitest', '~> 5.8'
38
+ s.add_development_dependency 'minitest-stub_any_instance', '~> 1.0.0'
39
+ s.add_development_dependency 'rake'
40
+ s.add_development_dependency 'test-unit', '~> 3.1'
41
+ s.add_development_dependency 'webmock', '~> 3.0'
42
+ s.add_development_dependency 'simplecov', '~> 0.11'
43
+ end
@@ -0,0 +1,216 @@
1
+ # Copyright 2016 Acquia, Inc.
2
+ # Copyright 2016-2019 t.e.morgan.
3
+ #
4
+ # Licensed under the Apache License, Version 2.0 (the "License");
5
+ # you may not use this file except in compliance with the License.
6
+ # You may obtain a copy of the License at
7
+ #
8
+ # http://www.apache.org/licenses/LICENSE-2.0
9
+ #
10
+ # Unless required by applicable law or agreed to in writing, software
11
+ # distributed under the License is distributed on an "AS IS" BASIS,
12
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13
+ # See the License for the specific language governing permissions and
14
+ # limitations under the License.
15
+
16
+ require 'socket'
17
+ require 'syslog_tls/logger'
18
+ require 'fluent/plugin/output'
19
+ require 'syslog_tls/host_backoff_specs'
20
+ require 'pp'
21
+
22
+ $host_backoff_specs_list = []
23
+
24
+ def add_host_backoff_spec(retries_to_do, host_ip_port)
25
+ # Check if an element with the same hostIPport already exists
26
+ return if $host_backoff_specs_list.any? { |spec| spec.hostIPport == host_ip_port }
27
+
28
+ # If not, add a new HostBackoffSpecs instance to the global array
29
+ $host_backoff_specs_list << ::SyslogTls::HostBackoffSpecs.new(retries_to_do, host_ip_port)
30
+ end
31
+
32
+ def can_write(host_ip_port)
33
+ if !conatins_host(host_ip_port)
34
+ add_host_backoff_spec(0, host_ip_port)
35
+ end
36
+ begin
37
+ $host_backoff_specs_list.each do |backoff_specs|
38
+ if backoff_specs.hostIPport == host_ip_port
39
+ return backoff_specs.canwrite
40
+ end
41
+ end
42
+ rescue => e
43
+ pp "Error in can_write: #{e.message}"
44
+ return 0
45
+ end
46
+ return 1
47
+ end
48
+
49
+ def increase_retry(host_ip_port)
50
+ if !conatins_host(host_ip_port)
51
+ add_host_backoff_spec(0, host_ip_port)
52
+ end
53
+ begin
54
+ $host_backoff_specs_list.each do |backoff_specs|
55
+ if backoff_specs.hostIPport == host_ip_port
56
+ backoff_specs.failtowrite
57
+ end
58
+ end
59
+ rescue => e
60
+ pp "Error in increase_retry: #{e.message}"
61
+ end
62
+ end
63
+
64
+ def reset_tries(host_ip_port)
65
+ if !conatins_host(host_ip_port)
66
+ add_host_backoff_spec(0, host_ip_port)
67
+ end
68
+ $host_backoff_specs_list.each do |backoff_specs|
69
+ if backoff_specs.hostIPport == host_ip_port
70
+ backoff_specs.resetRetries
71
+ end
72
+ end
73
+ end
74
+
75
+ def conatins_host(host_ip_port)
76
+ $host_backoff_specs_list.each do |backoff_specs|
77
+ if backoff_specs.hostIPport == host_ip_port
78
+ return true
79
+ end
80
+ end
81
+ return false
82
+ end
83
+
84
+ module Fluent::Plugin
85
+ class SyslogTlsOutput < Output
86
+ Fluent::Plugin.register_output('syslog_tls', self)
87
+
88
+ helpers :inject, :formatter, :compat_parameters
89
+
90
+ DEFAULT_FORMAT_TYPE = 'json'
91
+
92
+ config_param :host, :string
93
+ config_param :port, :integer
94
+ config_param :idle_timeout, :integer, default: nil
95
+ config_param :ca_cert, :string, default: 'system'
96
+ config_param :verify_cert_name, :bool, default: true
97
+ config_param :token, :string, default: nil
98
+ config_param :client_cert, :string, default: nil
99
+ config_param :client_key, :string, default: nil
100
+ config_param :hostname, :string, default: nil
101
+ config_param :facility, :string, default: 'LOCAL0'
102
+
103
+ # Allow to map keys from record to syslog message headers
104
+ SYSLOG_HEADERS = [
105
+ :severity, :facility, :hostname, :app_name, :procid, :msgid
106
+ ]
107
+
108
+ SYSLOG_HEADERS.each do |key_name|
109
+ config_param "#{key_name}_key".to_sym, :string, default: nil
110
+ end
111
+
112
+ config_section :format do
113
+ config_set_default :@type, DEFAULT_FORMAT_TYPE
114
+ end
115
+
116
+ attr_accessor :formatter
117
+
118
+
119
+ def initialize
120
+ super
121
+ @loggers = {}
122
+ end
123
+
124
+ def shutdown
125
+ @loggers.values.each(&:close)
126
+ super
127
+ end
128
+
129
+ # This method is called before starting.
130
+ def configure(conf)
131
+ if conf['output_type'] && !conf['format']
132
+ conf['format'] = conf['output_type']
133
+ end
134
+ compat_parameters_convert(conf, :inject, :formatter)
135
+
136
+ super
137
+ @host = conf['host']
138
+ @port = conf['port']
139
+ @token = conf['token']
140
+ @hostname = conf['hostname'] || Socket.gethostname.split('.').first
141
+
142
+ # Determine mapping of record keys to syslog keys
143
+ @mappings = {}
144
+ SYSLOG_HEADERS.each do |key_name|
145
+ conf_key = "#{key_name}_key"
146
+ @mappings[key_name] = conf[conf_key] if conf.key?(conf_key)
147
+ end
148
+
149
+ add_host_backoff_spec(0, conf['host']+":"+conf['port'])
150
+
151
+ @formatter = formatter_create(conf: conf.elements('format').first, default_type: DEFAULT_FORMAT_TYPE)
152
+ end
153
+
154
+ # Get logger for given tag
155
+ def logger(tag)
156
+ # Try to reuse existing logger
157
+ @loggers[tag] ||= new_logger(tag)
158
+
159
+ # Create new logger if old one is closed
160
+ if @loggers[tag].closed?
161
+ @loggers[tag] = new_logger(tag)
162
+ end
163
+
164
+ @loggers[tag]
165
+ end
166
+
167
+ def new_logger(tag)
168
+ transport = ::SyslogTls::SSLTransport.new(host, port,
169
+ idle_timeout: idle_timeout,
170
+ ca_cert: ca_cert,
171
+ client_cert: client_cert,
172
+ client_key: client_key,
173
+ verify_cert_name: verify_cert_name,
174
+ max_retries: 3,
175
+ )
176
+ logger = ::SyslogTls::Logger.new(transport, token)
177
+ logger.facility(facility)
178
+ logger.hostname(hostname)
179
+ logger.app_name(tag)
180
+ logger
181
+ end
182
+
183
+ def format(tag, time, record)
184
+ record = inject_values_to_record(tag, time, record)
185
+ @formatter.format(tag, time, record)
186
+ end
187
+
188
+ def process(tag, es)
189
+ es.each do |time, record|
190
+ record.each_pair do |_, v|
191
+ v.force_encoding('utf-8') if v.is_a?(String)
192
+ end
193
+
194
+ # Check if severity has been provided in record otherwise use INFO
195
+ # by default.
196
+ severity = if @mappings.key?(:severity)
197
+ record[@mappings[:severity]] || 'INFO'
198
+ else
199
+ 'INFO'
200
+ end
201
+
202
+ # Send message to Syslog
203
+ begin
204
+ logger(tag).log(severity, format(tag, time, record), time: Time.at(time)) do |header|
205
+ # Map syslog headers from record
206
+ @mappings.each do |name, record_key|
207
+ header.send("#{name}=", record[record_key]) unless record[record_key].nil?
208
+ end
209
+ end
210
+ rescue => e
211
+ log.error e.to_s
212
+ end
213
+ end
214
+ end
215
+ end
216
+ end
@@ -0,0 +1,46 @@
1
+ # Copyright 2016 Acquia, Inc.
2
+ #
3
+ # Licensed under the Apache License, Version 2.0 (the "License");
4
+ # you may not use this file except in compliance with the License.
5
+ # You may obtain a copy of the License at
6
+ #
7
+ # http://www.apache.org/licenses/LICENSE-2.0
8
+ #
9
+ # Unless required by applicable law or agreed to in writing, software
10
+ # distributed under the License is distributed on an "AS IS" BASIS,
11
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12
+ # See the License for the specific language governing permissions and
13
+ # limitations under the License.
14
+
15
+ require_relative 'lookup_from_const'
16
+
17
+ module SyslogTls
18
+ module Facility
19
+ extend LookupFromConst
20
+ KERN = 0
21
+ USER = 1
22
+ MAIL = 2
23
+ DAEMON = 3
24
+ AUTH = 4
25
+ SYSLOG = 5
26
+ LPR = 6
27
+ NEWS = 7
28
+ UUCP = 8
29
+ CRON = 9
30
+ AUTHPRIV = 10
31
+ FTP = 11
32
+ NTP = 12
33
+ SECURITY = 13
34
+ CONSOLE = 14
35
+ RAS = 15
36
+ LOCAL0 = 16
37
+ LOCAL1 = 17
38
+ LOCAL2 = 18
39
+ LOCAL3 = 19
40
+ LOCAL4 = 20
41
+ LOCAL5 = 21
42
+ LOCAL6 = 22
43
+ LOCAL7 = 23
44
+ NONE = SYSLOG
45
+ end
46
+ end
@@ -0,0 +1,45 @@
1
+ require 'pp'
2
+
3
+ module SyslogTls
4
+ class HostBackoffSpecs
5
+ attr_accessor :retriesToDo, :hostIPport
6
+
7
+ def initialize(retries_to_do, host_ip_port)
8
+ @retriesToDo = retries_to_do
9
+ @hostIPport = host_ip_port
10
+ @failTime = nil
11
+ @baseThreshold = 2
12
+ end
13
+
14
+ def canwrite
15
+ time_passed_since_failure = -1
16
+ if @failTime != nil
17
+ time_passed_since_failure = Time.now - @failTime
18
+ time_passed_since_failure = time_passed_since_failure.round(2)
19
+ end
20
+ if time_passed_since_failure == -1
21
+ return 1
22
+ end
23
+ backoffTime = @baseThreshold ** @retriesToDo
24
+ if backoffTime > 1800
25
+ backoffTime = 1800
26
+ end
27
+ if time_passed_since_failure > backoffTime
28
+ pp "canwrite writting after backoff :: time_passed_since_failure" + time_passed_since_failure.to_s + " time :: " + Time.now.to_s
29
+ return 1
30
+ else
31
+ return 0
32
+ end
33
+ end
34
+
35
+ def failtowrite
36
+ @retriesToDo += 1
37
+ @failTime = Time.now
38
+ end
39
+
40
+ def resetRetries
41
+ @retriesToDo = 0
42
+ @failTime = nil
43
+ end
44
+ end
45
+ end
@@ -0,0 +1,82 @@
1
+ # Copyright 2016 Acquia, Inc.
2
+ # Copyright 2016 t.e.morgan.
3
+ #
4
+ # Licensed under the Apache License, Version 2.0 (the "License");
5
+ # you may not use this file except in compliance with the License.
6
+ # You may obtain a copy of the License at
7
+ #
8
+ # http://www.apache.org/licenses/LICENSE-2.0
9
+ #
10
+ # Unless required by applicable law or agreed to in writing, software
11
+ # distributed under the License is distributed on an "AS IS" BASIS,
12
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
13
+ # See the License for the specific language governing permissions and
14
+ # limitations under the License.
15
+
16
+ require_relative 'protocol'
17
+ require_relative 'ssl_transport'
18
+
19
+ module SyslogTls
20
+ class Logger
21
+ attr_reader :token
22
+ attr_accessor :transport
23
+
24
+ # Logger accepts transport which should implement IO methods
25
+ # close, closed? and write
26
+ def initialize(transport, token=nil)
27
+ @transport = transport
28
+ @default_header = SyslogTls::Header.new
29
+ @default_structured_data = SyslogTls::StructuredData.new(token)
30
+ end
31
+
32
+ # Sets default facility for each message
33
+ def facility(val)
34
+ @default_header.facility = val
35
+ end
36
+
37
+ # Sets default hostname for each message
38
+ def hostname(val)
39
+ @default_header.hostname = val
40
+ end
41
+
42
+ # Sets default app_name for each message
43
+ def app_name(val)
44
+ @default_header.app_name = val
45
+ end
46
+
47
+ # Sets default procid for message
48
+ def procid(val)
49
+ @default_header.procid = val
50
+ end
51
+
52
+ # Check if IO is closed
53
+ def closed?
54
+ transport && transport.closed?
55
+ end
56
+
57
+ def close
58
+ transport.close
59
+ end
60
+
61
+ # Send log message with severity to syslog
62
+ def log(severity, message, time: nil)
63
+ time ||= Time.now
64
+
65
+ m = SyslogTls::Message.new
66
+
67
+ # Include authentication header
68
+ m.structured_data << @default_structured_data
69
+
70
+ # Adjust header with current timestamp and severity
71
+ m.header = @default_header.dup
72
+ m.header.severity = severity
73
+ m.header.timestamp = time
74
+
75
+ yield m.header if block_given?
76
+
77
+ m.msg = message
78
+
79
+ transport.write(m.to_s)
80
+ end
81
+ end
82
+ end
@@ -0,0 +1,36 @@
1
+ # Copyright 2016 Acquia, Inc.
2
+ #
3
+ # Licensed under the Apache License, Version 2.0 (the "License");
4
+ # you may not use this file except in compliance with the License.
5
+ # You may obtain a copy of the License at
6
+ #
7
+ # http://www.apache.org/licenses/LICENSE-2.0
8
+ #
9
+ # Unless required by applicable law or agreed to in writing, software
10
+ # distributed under the License is distributed on an "AS IS" BASIS,
11
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12
+ # See the License for the specific language governing permissions and
13
+ # limitations under the License.
14
+
15
+ module SyslogTls
16
+ module LookupFromConst
17
+ def setup_constants(dst)
18
+ constants.each do |pri|
19
+ cval = const_get pri
20
+
21
+ dst[pri] = cval
22
+ dst[pri.downcase] = cval
23
+
24
+ dst[:"LOG_#{pri.to_s}"] = cval
25
+ dst[:"LOG_#{pri.downcase.to_s}"] = cval
26
+ const_set :"LOG_#{pri.to_s}", cval
27
+
28
+ dst[pri.to_s] = cval
29
+ dst[pri.downcase.to_s] = cval
30
+
31
+ dst[cval] = cval
32
+ dst[cval.to_s] = cval
33
+ end
34
+ end
35
+ end
36
+ end