fluent-plugin-scalyr 0.8.6 → 0.8.11
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/Gemfile +2 -0
- data/README.md +38 -11
- data/Rakefile +9 -6
- data/VERSION +1 -1
- data/fluent-plugin-scalyr.gemspec +11 -8
- data/fluent.conf.sample +1 -1
- data/lib/fluent/plugin/out_scalyr.rb +200 -228
- data/lib/fluent/plugin/{scalyr-exceptions.rb → scalyr_exceptions.rb} +2 -2
- data/test/helper.rb +12 -6
- data/test/test_config.rb +24 -21
- data/test/test_events.rb +156 -154
- data/test/test_handle_response.rb +34 -35
- data/test/test_ssl_verify.rb +101 -10
- metadata +46 -33
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: 2d9fa05bade6ec51a39896e90bce7ddfdf233c8888d5ca2ac41393966c8a924b
|
4
|
+
data.tar.gz: b6406c122b1180faa48294b47ed2ebb1b61aad08c64c0310385a00c0ae8a044a
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 3b083816314fdf992eaca0cb0216795e2c7a6c1cc7e2f32669452201c45b29d6254dcbc2f69a665d6a0852242423bd87cdf62a88feda5f463e6f774b142d0520
|
7
|
+
data.tar.gz: 8a94b4b649cfd45fa1143e6866d119811236771980f508d0643b0b51b1d6bbdb2e241233a469086a0b5a3e22529c08d1c23b7825966fe78e367defd4672d20c2
|
data/Gemfile
CHANGED
data/README.md
CHANGED
@@ -1,5 +1,5 @@
|
|
1
1
|
Scalyr output plugin for Fluentd
|
2
|
-
|
2
|
+
================================
|
3
3
|
|
4
4
|
**Note:** Fluentd introduced breaking changes to their plugin API between
|
5
5
|
version 0.12 and 0.14.
|
@@ -24,7 +24,7 @@ Fluentd may format log messages into json or some other format. If you want to
|
|
24
24
|
format none
|
25
25
|
```
|
26
26
|
|
27
|
-
The Scalyr output plugin assigns a unique Scalyr session id for each Fluentd <match> block. It is recommended that a single machine doesn't create too many simultaneous Scalyr sessions, so if possible you should try to have a single match for all logs you wish to send to Scalyr.
|
27
|
+
The Scalyr output plugin assigns a unique Scalyr session id for each Fluentd <match> block, or for each worker. It is recommended that a single machine doesn't create too many simultaneous Scalyr sessions, so if possible you should try to have a single match for all logs you wish to send to Scalyr.
|
28
28
|
|
29
29
|
This can be done by specifying tags such as scalyr.apache, scalyr.maillog etc and matching on scalyr.\*
|
30
30
|
|
@@ -33,7 +33,7 @@ Fluentd tag names will be used for the logfile name in Scalyr.
|
|
33
33
|
Scalyr Parsers and Custom Fields
|
34
34
|
--------------------------------
|
35
35
|
|
36
|
-
You may also need to specify a Scalyr parser for your log message or add custom fields to each log event. This can be done using Fluentd's filter mechanism, in particular the [record_transformer filter](
|
36
|
+
You may also need to specify a Scalyr parser for your log message or add custom fields to each log event. This can be done using Fluentd's filter mechanism, in particular the [record_transformer filter](https://docs.fluentd.org/filter/record_transformer).
|
37
37
|
|
38
38
|
For example, if you want to use Scalyr's ```accessLog``` parser for all events with the ```scalyr.access``` tag you would add the following to your fluent.conf file:
|
39
39
|
|
@@ -66,7 +66,9 @@ The following configuration options are also supported:
|
|
66
66
|
|
67
67
|
#scalyr specific options
|
68
68
|
api_write_token YOUR_SCALYR_WRITE_TOKEN
|
69
|
-
compression_type
|
69
|
+
compression_type deflate
|
70
|
+
compression_level 6
|
71
|
+
use_hostname_for_serverhost true
|
70
72
|
server_attributes {
|
71
73
|
"serverHost": "front-1",
|
72
74
|
"serverType": "frontend",
|
@@ -79,7 +81,7 @@ The following configuration options are also supported:
|
|
79
81
|
ssl_verify_depth 5
|
80
82
|
message_field message
|
81
83
|
|
82
|
-
max_request_buffer
|
84
|
+
max_request_buffer 5500000
|
83
85
|
|
84
86
|
force_message_encoding nil
|
85
87
|
replace_invalid_utf8 false
|
@@ -91,14 +93,18 @@ The following configuration options are also supported:
|
|
91
93
|
retry_max_interval 30s
|
92
94
|
flush_interval 5s
|
93
95
|
flush_thread_count 1
|
94
|
-
chunk_limit_size
|
96
|
+
chunk_limit_size 2.5m
|
95
97
|
queue_limit_length 1024
|
96
98
|
</buffer>
|
97
99
|
|
98
100
|
</match>
|
99
101
|
```
|
100
102
|
|
101
|
-
|
103
|
+
For some additional examples of configuration for different setups, please refer to the
|
104
|
+
[examples/configs/](https://github.com/scalyr/scalyr-fluentd/tree/master/examples/configs/)
|
105
|
+
directory.
|
106
|
+
|
107
|
+
### Scalyr specific options
|
102
108
|
|
103
109
|
***compression_type*** - compress Scalyr traffic to reduce network traffic. Options are `bz2` and `deflate`. See [here](https://www.scalyr.com/help/scalyr-agent#compressing) for more details. This feature is optional.
|
104
110
|
|
@@ -106,9 +112,11 @@ The following configuration options are also supported:
|
|
106
112
|
|
107
113
|
***server_attributes*** - a JSON hash containing custom server attributes you want to include with each log request. This value is optional and defaults to *nil*.
|
108
114
|
|
115
|
+
***use_hostname_for_serverhost*** - if `true` then if `server_attributes` is nil or it does *not* include a field called `serverHost` then the plugin will add the `serverHost` field with the value set to the hostname that fluentd is running on. Defaults to `true`.
|
116
|
+
|
109
117
|
***scalyr_server*** - the Scalyr server to send API requests to. This value is optional and defaults to https://agent.scalyr.com/
|
110
118
|
|
111
|
-
***ssl_ca_bundle_path*** - a path on your server pointing to a valid certificate bundle. This value is optional and defaults to
|
119
|
+
***ssl_ca_bundle_path*** - a path on your server pointing to a valid certificate bundle. This value is optional and defaults to *nil*, which means it will look for a valid certificate bundle on its own.
|
112
120
|
|
113
121
|
**Note:** if the certificate bundle does not contain a certificate chain that verifies the Scalyr SSL certificate then all requests to Scalyr will fail unless ***ssl_verify_peer*** is set to false. If you suspect logging to Scalyr is failing due to an invalid certificate chain, you can grep through the Fluentd output for warnings that contain the message 'certificate verification failed'. The full text of such warnings will look something like this:
|
114
122
|
|
@@ -126,13 +134,13 @@ The cURL project maintains CA certificate bundles automatically converted from m
|
|
126
134
|
|
127
135
|
***message_field*** - Scalyr expects all log events to have a 'message' field containing the contents of a log message. If your event has the log message stored in another field, you can specify the field name here, and the plugin will rename that field to 'message' before sending the data to Scalyr. **Note:** this will override any existing 'message' field if the log record contains both a 'message' field and the field specified by this config option.
|
128
136
|
|
129
|
-
***max_request_buffer*** - The maximum size in bytes of each request to send to Scalyr. Defaults to
|
137
|
+
***max_request_buffer*** - The maximum size in bytes of each request to send to Scalyr. Defaults to 5,500,000 (5.5MB). Fluentd chunks that generate JSON requests larger than the max_request_buffer will be split in to multiple separate requests. **Note:** The maximum size the Scalyr servers accept for this value is 6MB and requests containing data larger than this will be rejected.
|
130
138
|
|
131
139
|
***force_message_encoding*** - Set a specific encoding for all your log messages (defaults to nil). If your log messages are not in UTF-8, this can cause problems when converting the message to JSON in order to send to the Scalyr server. You can avoid these problems by setting an encoding for your log messages so they can be correctly converted.
|
132
140
|
|
133
141
|
***replace_invalid_utf8*** - If this value is true and ***force_message_encoding*** is set to 'UTF-8' then all invalid UTF-8 sequences in log messages will be replaced with <?>. Defaults to false. This flag has no effect if ***force_message_encoding*** is not set to 'UTF-8'.
|
134
142
|
|
135
|
-
|
143
|
+
### Buffer options
|
136
144
|
|
137
145
|
***retry_max_times*** - the maximum number of times to retry a failed post request before giving up. Defaults to *40*.
|
138
146
|
|
@@ -144,7 +152,7 @@ The cURL project maintains CA certificate bundles automatically converted from m
|
|
144
152
|
|
145
153
|
***flush_thread_count*** - the number of threads to use to upload logs. This is currently fixed to 1 will cause fluentd to fail with a ConfigError if set to anything greater.
|
146
154
|
|
147
|
-
***chunk_limit_size*** - the maximum amount of log data to send to Scalyr in a single request. Defaults to *
|
155
|
+
***chunk_limit_size*** - the maximum amount of log data to send to Scalyr in a single request. Defaults to *2.5MB*. **Note:** if you set this value too large, then Scalyr may reject your requests. Requests smaller than 6 MB will typically be accepted by Scalyr, but note that the 6 MB limit also includes the entire request body and all associated JSON keys and punctuation, which may be considerably larger than the raw log data. This value should be set lower than the `max_request_buffer` option.
|
148
156
|
|
149
157
|
***queue_limit_length*** - the maximum number of chunks to buffer before dropping new log requests. Defaults to *1024*. Combines with ***chunk_limit_size*** to give you the total amount of buffer to use in the event of request failures before dropping requests.
|
150
158
|
|
@@ -172,3 +180,22 @@ Which builds the gem and puts it in the pkg directory, then install the Gem usin
|
|
172
180
|
```
|
173
181
|
fluent-gem install pkg/fluent-plugin-scalyr-<VERSION>.gem
|
174
182
|
```
|
183
|
+
|
184
|
+
Publishing a new release to RubyGems
|
185
|
+
------------------------------------
|
186
|
+
|
187
|
+
(for project maintainers)
|
188
|
+
|
189
|
+
To publish a new version to RubyGems, simply make your changes, make sure all the lint checks and
|
190
|
+
tests pass and merge your changes into master.
|
191
|
+
|
192
|
+
After that's done, bump a version in ``VERSION`` file, update ``CHANGELOG.md`` file, add a tag
|
193
|
+
which matches a version in VERSION file (e.g. ``v0.8.10``) and push that tag to the remote:
|
194
|
+
|
195
|
+
```bash
|
196
|
+
git tag v0.8.10
|
197
|
+
git push origin v0.8.10
|
198
|
+
```
|
199
|
+
|
200
|
+
Push of this tag will trigger a Circle CI job which will build the latest version of the gem and
|
201
|
+
publish it to RubyGems.
|
data/Rakefile
CHANGED
@@ -1,11 +1,14 @@
|
|
1
|
-
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
require "bundler"
|
2
4
|
Bundler::GemHelper.install_tasks
|
3
5
|
|
4
|
-
require
|
6
|
+
require "rake/testtask"
|
5
7
|
|
6
|
-
Rake::TestTask.new do |
|
7
|
-
|
8
|
-
|
8
|
+
Rake::TestTask.new(:test) do |test|
|
9
|
+
test.libs << "lib" << "test"
|
10
|
+
test.test_files = FileList["test/test_*.rb"]
|
11
|
+
test.verbose = true
|
9
12
|
end
|
10
13
|
|
11
|
-
task :
|
14
|
+
task default: [:build]
|
data/VERSION
CHANGED
@@ -1 +1 @@
|
|
1
|
-
0.8.
|
1
|
+
0.8.11
|
@@ -1,4 +1,6 @@
|
|
1
|
-
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
$LOAD_PATH.push File.expand_path("lib", __dir__)
|
2
4
|
|
3
5
|
Gem::Specification.new do |gem|
|
4
6
|
gem.name = "fluent-plugin-scalyr"
|
@@ -9,18 +11,19 @@ Gem::Specification.new do |gem|
|
|
9
11
|
gem.authors = ["Imron Alston"]
|
10
12
|
gem.licenses = ["Apache-2.0"]
|
11
13
|
gem.email = "imron@scalyr.com"
|
12
|
-
gem.has_rdoc = false
|
13
14
|
gem.platform = Gem::Platform::RUBY
|
14
|
-
gem.files = Dir[
|
15
|
+
gem.files = Dir["AUTHORS", "Gemfile", "LICENSE", "README.md", "Rakefile", "VERSION",
|
16
|
+
"fluent-plugin-scalyr.gemspec", "fluent.conf.sample", "lib/**/*", "test/**/*"]
|
15
17
|
gem.test_files = Dir.glob("{test,spec,features}/**/*")
|
16
|
-
gem.executables = Dir.glob("bin/*").map{
|
17
|
-
gem.require_paths = [
|
18
|
-
gem.add_dependency "fluentd", [">= 0.14.0", "< 2"]
|
18
|
+
gem.executables = Dir.glob("bin/*").map {|f| File.basename(f) }
|
19
|
+
gem.require_paths = ["lib"]
|
19
20
|
gem.add_dependency "ffi", "1.9.25"
|
21
|
+
gem.add_dependency "fluentd", [">= 0.14.0", "< 2"]
|
20
22
|
gem.add_dependency "rbzip2", "0.3.0"
|
21
23
|
gem.add_dependency "zlib"
|
24
|
+
gem.add_development_dependency "bundler", "~> 1.9"
|
25
|
+
gem.add_development_dependency "flexmock", "~> 1.2"
|
22
26
|
gem.add_development_dependency "rake", "~> 0.9"
|
27
|
+
gem.add_development_dependency "rubocop", "~> 0.4"
|
23
28
|
gem.add_development_dependency "test-unit", "~> 3.0"
|
24
|
-
gem.add_development_dependency "flexmock", "~> 1.2"
|
25
|
-
gem.add_development_dependency "bundler", "~> 1.9"
|
26
29
|
end
|
data/fluent.conf.sample
CHANGED
@@ -1,3 +1,5 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
1
3
|
#
|
2
4
|
# Scalyr Output Plugin for Fluentd
|
3
5
|
#
|
@@ -15,45 +17,44 @@
|
|
15
17
|
# See the License for the specific language governing permissions and
|
16
18
|
# limitations under the License.
|
17
19
|
|
18
|
-
|
19
|
-
require
|
20
|
-
require
|
21
|
-
require
|
22
|
-
require
|
23
|
-
require
|
24
|
-
require
|
25
|
-
require
|
26
|
-
require
|
27
|
-
require
|
28
|
-
require
|
29
|
-
require 'thread'
|
30
|
-
|
20
|
+
require "fluent/plugin/output"
|
21
|
+
require "fluent/plugin/scalyr_exceptions"
|
22
|
+
require "fluent/plugin_helper/compat_parameters"
|
23
|
+
require "json"
|
24
|
+
require "net/http"
|
25
|
+
require "net/https"
|
26
|
+
require "rbzip2"
|
27
|
+
require "stringio"
|
28
|
+
require "zlib"
|
29
|
+
require "securerandom"
|
30
|
+
require "socket"
|
31
31
|
module Scalyr
|
32
32
|
class ScalyrOut < Fluent::Plugin::Output
|
33
|
-
Fluent::Plugin.register_output(
|
33
|
+
Fluent::Plugin.register_output("scalyr", self)
|
34
34
|
helpers :compat_parameters
|
35
35
|
helpers :event_emitter
|
36
36
|
|
37
37
|
config_param :api_write_token, :string
|
38
|
-
config_param :server_attributes, :hash, :
|
39
|
-
config_param :
|
40
|
-
config_param :
|
41
|
-
config_param :
|
42
|
-
config_param :
|
43
|
-
config_param :
|
44
|
-
config_param :
|
45
|
-
config_param :
|
46
|
-
config_param :
|
47
|
-
config_param :
|
48
|
-
config_param :
|
38
|
+
config_param :server_attributes, :hash, default: nil
|
39
|
+
config_param :use_hostname_for_serverhost, :bool, default: true
|
40
|
+
config_param :scalyr_server, :string, default: "https://agent.scalyr.com/"
|
41
|
+
config_param :ssl_ca_bundle_path, :string, default: nil
|
42
|
+
config_param :ssl_verify_peer, :bool, default: true
|
43
|
+
config_param :ssl_verify_depth, :integer, default: 5
|
44
|
+
config_param :message_field, :string, default: "message"
|
45
|
+
config_param :max_request_buffer, :integer, default: 5_500_000
|
46
|
+
config_param :force_message_encoding, :string, default: nil
|
47
|
+
config_param :replace_invalid_utf8, :bool, default: false
|
48
|
+
config_param :compression_type, :string, default: nil # Valid options are bz2, deflate or None. Defaults to None.
|
49
|
+
config_param :compression_level, :integer, default: 6 # An int containing the compression level of compression to use, from 1-9. Defaults to 6
|
49
50
|
|
50
51
|
config_section :buffer do
|
51
|
-
config_set_default :retry_max_times, 40 #try a maximum of 40 times before discarding
|
52
|
-
config_set_default :retry_max_interval,
|
53
|
-
config_set_default :retry_wait, 5 #wait a minimum of 5 seconds per retry
|
54
|
-
config_set_default :flush_interval, 5 #default flush interval of 5 seconds
|
55
|
-
config_set_default :chunk_limit_size,
|
56
|
-
config_set_default :queue_limit_length, 1024 #default queue size of 1024
|
52
|
+
config_set_default :retry_max_times, 40 # try a maximum of 40 times before discarding
|
53
|
+
config_set_default :retry_max_interval, 30 # wait a maximum of 30 seconds per retry
|
54
|
+
config_set_default :retry_wait, 5 # wait a minimum of 5 seconds per retry
|
55
|
+
config_set_default :flush_interval, 5 # default flush interval of 5 seconds
|
56
|
+
config_set_default :chunk_limit_size, 2_500_000 # default chunk size of 2.5mb
|
57
|
+
config_set_default :queue_limit_length, 1024 # default queue size of 1024
|
57
58
|
end
|
58
59
|
|
59
60
|
# support for version 0.14.0:
|
@@ -65,180 +66,182 @@ module Scalyr
|
|
65
66
|
true
|
66
67
|
end
|
67
68
|
|
68
|
-
def
|
69
|
+
def multi_workers_ready?
|
70
|
+
true
|
71
|
+
end
|
69
72
|
|
70
|
-
|
71
|
-
|
73
|
+
def configure(conf)
|
74
|
+
if conf.elements("buffer").empty?
|
75
|
+
$log.warn "Pre 0.14.0 configuration file detected. Please consider updating your configuration file" # rubocop:disable Layout/LineLength, Lint/RedundantCopDisableDirective
|
72
76
|
end
|
73
77
|
|
74
|
-
compat_parameters_buffer(
|
78
|
+
compat_parameters_buffer(conf, default_chunk_key: "")
|
75
79
|
|
76
80
|
super
|
77
81
|
|
78
|
-
if @buffer.chunk_limit_size >
|
79
|
-
$log.warn "Buffer chunk size is greater than
|
82
|
+
if @buffer.chunk_limit_size > 6_000_000
|
83
|
+
$log.warn "Buffer chunk size is greater than 6Mb. This may result in requests being rejected by Scalyr" # rubocop:disable Layout/LineLength, Lint/RedundantCopDisableDirective
|
80
84
|
end
|
81
85
|
|
82
|
-
if @max_request_buffer >
|
83
|
-
$log.warn "Maximum request buffer >
|
86
|
+
if @max_request_buffer > 6_000_000
|
87
|
+
$log.warn "Maximum request buffer > 6Mb. This may result in requests being rejected by Scalyr" # rubocop:disable Layout/LineLength, Lint/RedundantCopDisableDirective
|
84
88
|
end
|
85
89
|
|
86
90
|
@message_encoding = nil
|
87
|
-
if @force_message_encoding.to_s !=
|
91
|
+
if @force_message_encoding.to_s != ""
|
88
92
|
begin
|
89
|
-
@message_encoding = Encoding.find(
|
93
|
+
@message_encoding = Encoding.find(@force_message_encoding)
|
90
94
|
$log.debug "Forcing message encoding to '#{@force_message_encoding}'"
|
91
95
|
rescue ArgumentError
|
92
96
|
$log.warn "No encoding '#{@force_message_encoding}' found. Ignoring"
|
93
97
|
end
|
94
98
|
end
|
95
99
|
|
96
|
-
#evaluate any statements in string value of the server_attributes object
|
100
|
+
# evaluate any statements in string value of the server_attributes object
|
97
101
|
if @server_attributes
|
98
102
|
new_attributes = {}
|
99
103
|
@server_attributes.each do |key, value|
|
100
|
-
|
101
|
-
|
102
|
-
|
103
|
-
|
104
|
-
|
105
|
-
|
106
|
-
|
107
|
-
|
104
|
+
next unless value.is_a?(String)
|
105
|
+
|
106
|
+
m = /^\#{(.*)}$/.match(value)
|
107
|
+
new_attributes[key] = if m
|
108
|
+
eval(m[1]) # rubocop:disable Security/Eval
|
109
|
+
else
|
110
|
+
value
|
111
|
+
end
|
108
112
|
end
|
109
113
|
@server_attributes = new_attributes
|
110
114
|
end
|
111
115
|
|
112
|
-
|
116
|
+
# See if we should use the hostname as the server_attributes.serverHost
|
117
|
+
if @use_hostname_for_serverhost
|
118
|
+
|
119
|
+
# ensure server_attributes is not nil
|
120
|
+
@server_attributes = {} if @server_attributes.nil?
|
121
|
+
|
122
|
+
# only set serverHost if it doesn't currently exist in server_attributes
|
123
|
+
# Note: Use strings rather than symbols for the key, because keys coming
|
124
|
+
# from the config file will be strings
|
125
|
+
unless @server_attributes.key? "serverHost"
|
126
|
+
@server_attributes["serverHost"] = Socket.gethostname
|
127
|
+
end
|
128
|
+
end
|
129
|
+
|
130
|
+
@scalyr_server << "/" unless @scalyr_server.end_with?("/")
|
113
131
|
|
114
132
|
@add_events_uri = URI @scalyr_server + "addEvents"
|
115
133
|
|
116
134
|
num_threads = @buffer_config.flush_thread_count
|
117
135
|
|
118
|
-
#forcibly limit the number of threads to 1 for now, to ensure requests always have incrementing timestamps
|
119
|
-
|
136
|
+
# forcibly limit the number of threads to 1 for now, to ensure requests always have incrementing timestamps
|
137
|
+
if num_threads > 1
|
138
|
+
raise Fluent::ConfigError, "num_threads is currently limited to 1. You specified #{num_threads}."
|
139
|
+
end
|
120
140
|
end
|
121
141
|
|
122
142
|
def start
|
123
143
|
super
|
124
|
-
|
125
|
-
#Generate a session id. This will be called once for each <match> in fluent.conf that uses scalyr
|
144
|
+
# Generate a session id. This will be called once for each <match> in fluent.conf that uses scalyr
|
126
145
|
@session = SecureRandom.uuid
|
127
146
|
|
128
|
-
@
|
129
|
-
#the following variables are all under the control of the above mutex
|
130
|
-
@thread_ids = Hash.new #hash of tags -> id
|
131
|
-
@next_id = 1 #incrementing thread id for the session
|
132
|
-
@last_timestamp = 0 #timestamp of most recent event in nanoseconds since epoch
|
133
|
-
|
147
|
+
$log.info "Scalyr Fluentd Plugin ID id=#{plugin_id} worker=#{fluentd_worker_id} session=#{@session}" # rubocop:disable Layout/LineLength, Lint/RedundantCopDisableDirective
|
134
148
|
end
|
135
149
|
|
136
|
-
def format(
|
137
|
-
|
138
|
-
|
139
|
-
|
140
|
-
|
141
|
-
|
142
|
-
|
143
|
-
|
144
|
-
|
145
|
-
|
146
|
-
|
147
|
-
|
148
|
-
sec = components[0].to_i
|
149
|
-
nsec = (components[1] * 10**9).to_i
|
150
|
-
time = Fluent::EventTime.new( sec, nsec )
|
151
|
-
end
|
150
|
+
def format(tag, time, record)
|
151
|
+
time = Fluent::Engine.now if time.nil?
|
152
|
+
|
153
|
+
# handle timestamps that are not EventTime types
|
154
|
+
if time.is_a?(Integer)
|
155
|
+
time = Fluent::EventTime.new(time)
|
156
|
+
elsif time.is_a?(Float)
|
157
|
+
components = time.divmod 1 # get integer and decimal components
|
158
|
+
sec = components[0].to_i
|
159
|
+
nsec = (components[1] * 10**9).to_i
|
160
|
+
time = Fluent::EventTime.new(sec, nsec)
|
161
|
+
end
|
152
162
|
|
153
|
-
|
154
|
-
|
155
|
-
|
156
|
-
|
157
|
-
end
|
158
|
-
record["message"] = record[@message_field]
|
159
|
-
record.delete( @message_field )
|
163
|
+
if @message_field != "message"
|
164
|
+
if record.key? @message_field
|
165
|
+
if record.key? "message"
|
166
|
+
$log.warn "Overwriting log record field 'message'. You are seeing this warning because in your fluentd config file you have configured the '#{@message_field}' field to be converted to the 'message' field, but the log record already contains a field called 'message' and this is now being overwritten." # rubocop:disable Layout/LineLength, Lint/RedundantCopDisableDirective
|
160
167
|
end
|
168
|
+
record["message"] = record[@message_field]
|
169
|
+
record.delete(@message_field)
|
161
170
|
end
|
171
|
+
end
|
162
172
|
|
163
|
-
|
164
|
-
|
165
|
-
|
166
|
-
|
167
|
-
|
168
|
-
end
|
173
|
+
if @message_encoding && record.key?("message") && record["message"]
|
174
|
+
if @replace_invalid_utf8 && (@message_encoding == Encoding::UTF_8)
|
175
|
+
record["message"] = record["message"].encode("UTF-8", invalid: :replace, undef: :replace, replace: "<?>").force_encoding("UTF-8") # rubocop:disable Layout/LineLength, Lint/RedundantCopDisableDirective
|
176
|
+
else
|
177
|
+
record["message"].force_encoding(@message_encoding)
|
169
178
|
end
|
170
|
-
[tag, time.sec, time.nsec, record].to_msgpack
|
171
|
-
|
172
|
-
rescue JSON::GeneratorError
|
173
|
-
$log.warn "Unable to format message due to JSON::GeneratorError. Record is:\n\t#{record.to_s}"
|
174
|
-
raise
|
175
179
|
end
|
180
|
+
[tag, time.sec, time.nsec, record].to_msgpack
|
181
|
+
rescue JSON::GeneratorError
|
182
|
+
$log.warn "Unable to format message due to JSON::GeneratorError. Record is:\n\t#{record}"
|
183
|
+
raise
|
176
184
|
end
|
177
185
|
|
178
|
-
#called by fluentd when a chunk of log messages is ready
|
179
|
-
def write(
|
180
|
-
|
181
|
-
|
182
|
-
|
183
|
-
$log.debug "Chunk split into #{requests.size} request(s)."
|
184
|
-
|
185
|
-
requests.each_with_index { |request, index|
|
186
|
-
$log.debug "Request #{index + 1}/#{requests.size}: #{request[:body].bytesize} bytes"
|
187
|
-
begin
|
188
|
-
response = self.post_request( @add_events_uri, request[:body] )
|
189
|
-
self.handle_response( response )
|
190
|
-
rescue OpenSSL::SSL::SSLError => e
|
191
|
-
if e.message.include? "certificate verify failed"
|
192
|
-
$log.warn "SSL certificate verification failed. Please make sure your certificate bundle is configured correctly and points to a valid file. You can configure this with the ssl_ca_bundle_path configuration option. The current value of ssl_ca_bundle_path is '#{@ssl_ca_bundle_path}'"
|
193
|
-
end
|
194
|
-
$log.warn e.message
|
195
|
-
$log.warn "Discarding buffer chunk without retrying or logging to <secondary>"
|
196
|
-
rescue Scalyr::Client4xxError => e
|
197
|
-
$log.warn "4XX status code received for request #{index + 1}/#{requests.size}. Discarding buffer without retrying or logging.\n\t#{response.code} - #{e.message}\n\tChunk Size: #{chunk.size}\n\tLog messages this request: #{request[:record_count]}\n\tJSON payload size: #{request[:body].bytesize}\n\tSample: #{request[:body][0,1024]}..."
|
186
|
+
# called by fluentd when a chunk of log messages is ready
|
187
|
+
def write(chunk)
|
188
|
+
$log.debug "Size of chunk is: #{chunk.size}"
|
189
|
+
requests = build_add_events_body(chunk)
|
190
|
+
$log.debug "Chunk split into #{requests.size} request(s)."
|
198
191
|
|
192
|
+
requests.each_with_index {|request, index|
|
193
|
+
$log.debug "Request #{index + 1}/#{requests.size}: #{request[:body].bytesize} bytes"
|
194
|
+
begin
|
195
|
+
response = post_request(@add_events_uri, request[:body])
|
196
|
+
handle_response(response)
|
197
|
+
rescue OpenSSL::SSL::SSLError => e
|
198
|
+
if e.message.include? "certificate verify failed"
|
199
|
+
$log.warn "SSL certificate verification failed. Please make sure your certificate bundle is configured correctly and points to a valid file. You can configure this with the ssl_ca_bundle_path configuration option. The current value of ssl_ca_bundle_path is '#{@ssl_ca_bundle_path}'" # rubocop:disable Layout/LineLength, Lint/RedundantCopDisableDirective
|
199
200
|
end
|
200
|
-
|
201
|
-
|
202
|
-
|
203
|
-
|
204
|
-
|
205
|
-
|
201
|
+
$log.warn e.message
|
202
|
+
$log.warn "Discarding buffer chunk without retrying or logging to <secondary>"
|
203
|
+
rescue Scalyr::Client4xxError => e
|
204
|
+
$log.warn "4XX status code received for request #{index + 1}/#{requests.size}. Discarding buffer without retrying or logging.\n\t#{response.code} - #{e.message}\n\tChunk Size: #{chunk.size}\n\tLog messages this request: #{request[:record_count]}\n\tJSON payload size: #{request[:body].bytesize}\n\tSample: #{request[:body][0, 1024]}..."
|
205
|
+
end
|
206
|
+
}
|
207
|
+
rescue JSON::GeneratorError
|
208
|
+
$log.warn "Unable to format message due to JSON::GeneratorError."
|
209
|
+
raise
|
206
210
|
end
|
207
211
|
|
208
|
-
|
209
|
-
#
|
210
|
-
|
211
|
-
def to_nanos( seconds, nsec )
|
212
|
+
# explicit function to convert to nanoseconds
|
213
|
+
# will make things easier to maintain if/when fluentd supports higher than second resolutions
|
214
|
+
def to_nanos(seconds, nsec)
|
212
215
|
(seconds * 10**9) + nsec
|
213
216
|
end
|
214
217
|
|
215
|
-
#explicit function to convert to milliseconds
|
216
|
-
#will make things easier to maintain if/when fluentd supports higher than second resolutions
|
217
|
-
def to_millis(
|
218
|
+
# explicit function to convert to milliseconds
|
219
|
+
# will make things easier to maintain if/when fluentd supports higher than second resolutions
|
220
|
+
def to_millis(timestamp)
|
218
221
|
(timestamp.sec * 10**3) + (timestamp.nsec / 10**6)
|
219
222
|
end
|
220
223
|
|
221
|
-
def post_request(
|
222
|
-
|
223
|
-
https = Net::HTTP.new( uri.host, uri.port )
|
224
|
+
def post_request(uri, body)
|
225
|
+
https = Net::HTTP.new(uri.host, uri.port)
|
224
226
|
https.use_ssl = true
|
225
227
|
|
226
|
-
#verify peers to prevent potential MITM attacks
|
228
|
+
# verify peers to prevent potential MITM attacks
|
227
229
|
if @ssl_verify_peer
|
228
|
-
https.ca_file = @ssl_ca_bundle_path
|
230
|
+
https.ca_file = @ssl_ca_bundle_path unless @ssl_ca_bundle_path.nil?
|
231
|
+
https.ssl_version = :TLSv1_2
|
229
232
|
https.verify_mode = OpenSSL::SSL::VERIFY_PEER
|
230
233
|
https.verify_depth = @ssl_verify_depth
|
231
234
|
end
|
232
235
|
|
233
|
-
#use compression if enabled
|
236
|
+
# use compression if enabled
|
234
237
|
encoding = nil
|
235
238
|
|
236
239
|
if @compression_type
|
237
|
-
if @compression_type ==
|
238
|
-
encoding =
|
240
|
+
if @compression_type == "deflate"
|
241
|
+
encoding = "deflate"
|
239
242
|
body = Zlib::Deflate.deflate(body, @compression_level)
|
240
|
-
elsif @compression_type ==
|
241
|
-
encoding =
|
243
|
+
elsif @compression_type == "bz2"
|
244
|
+
encoding = "bz2"
|
242
245
|
io = StringIO.new
|
243
246
|
bz2 = RBzip2.default_adapter::Compressor.new io
|
244
247
|
bz2.write body
|
@@ -248,106 +251,82 @@ module Scalyr
|
|
248
251
|
end
|
249
252
|
|
250
253
|
post = Net::HTTP::Post.new uri.path
|
251
|
-
post.add_field(
|
254
|
+
post.add_field("Content-Type", "application/json")
|
252
255
|
|
253
|
-
if @compression_type
|
254
|
-
post.add_field( 'Content-Encoding', encoding )
|
255
|
-
end
|
256
|
+
post.add_field("Content-Encoding", encoding) if @compression_type
|
256
257
|
|
257
258
|
post.body = body
|
258
259
|
|
259
|
-
https.request(
|
260
|
-
|
260
|
+
https.request(post)
|
261
261
|
end
|
262
262
|
|
263
|
-
def handle_response(
|
263
|
+
def handle_response(response)
|
264
264
|
$log.debug "Response Code: #{response.code}"
|
265
265
|
$log.debug "Response Body: #{response.body}"
|
266
266
|
|
267
|
-
response_hash =
|
267
|
+
response_hash = {}
|
268
268
|
|
269
269
|
begin
|
270
|
-
response_hash = JSON.parse(
|
271
|
-
rescue
|
270
|
+
response_hash = JSON.parse(response.body)
|
271
|
+
rescue StandardError
|
272
272
|
response_hash["status"] = "Invalid JSON response from server"
|
273
273
|
end
|
274
274
|
|
275
|
-
#make sure the JSON reponse has a "status" field
|
276
|
-
|
275
|
+
# make sure the JSON reponse has a "status" field
|
276
|
+
unless response_hash.key? "status"
|
277
277
|
$log.debug "JSON response does not contain status message"
|
278
278
|
raise Scalyr::ServerError.new "JSON response does not contain status message"
|
279
279
|
end
|
280
280
|
|
281
281
|
status = response_hash["status"]
|
282
282
|
|
283
|
-
#4xx codes are handled separately
|
283
|
+
# 4xx codes are handled separately
|
284
284
|
if response.code =~ /^4\d\d/
|
285
285
|
raise Scalyr::Client4xxError.new status
|
286
286
|
else
|
287
|
-
if status != "success"
|
287
|
+
if status != "success" # rubocop:disable Style/IfInsideElse
|
288
288
|
if status =~ /discardBuffer/
|
289
289
|
$log.warn "Received 'discardBuffer' message from server. Buffer dropped."
|
290
|
-
elsif status =~ %r
|
290
|
+
elsif status =~ %r{/client/}i
|
291
291
|
raise Scalyr::ClientError.new status
|
292
|
-
else #don't check specifically for server, we assume all non-client errors are server errors
|
292
|
+
else # don't check specifically for server, we assume all non-client errors are server errors
|
293
293
|
raise Scalyr::ServerError.new status
|
294
294
|
end
|
295
|
-
elsif !response.code.include? "200" #response code is a string not an int
|
295
|
+
elsif !response.code.include? "200" # response code is a string not an int
|
296
296
|
raise Scalyr::ServerError
|
297
297
|
end
|
298
298
|
end
|
299
|
-
|
300
299
|
end
|
301
300
|
|
302
|
-
def build_add_events_body(
|
301
|
+
def build_add_events_body(chunk)
|
302
|
+
# requests
|
303
|
+
requests = []
|
303
304
|
|
304
|
-
#
|
305
|
-
|
305
|
+
# set of unique scalyr threads for this chunk
|
306
|
+
current_threads = {}
|
306
307
|
|
307
|
-
#
|
308
|
-
current_threads = Hash.new
|
309
|
-
|
310
|
-
#byte count
|
308
|
+
# byte count
|
311
309
|
total_bytes = 0
|
312
310
|
|
313
|
-
#create a Scalyr event object for each record in the chunk
|
314
|
-
events =
|
315
|
-
chunk.msgpack_each {|(tag, sec, nsec, record)|
|
316
|
-
|
317
|
-
timestamp = self.to_nanos( sec, nsec )
|
318
|
-
|
319
|
-
thread_id = 0
|
311
|
+
# create a Scalyr event object for each record in the chunk
|
312
|
+
events = []
|
313
|
+
chunk.msgpack_each {|(tag, sec, nsec, record)| # rubocop:disable Metrics/BlockLength
|
314
|
+
timestamp = to_nanos(sec, nsec)
|
320
315
|
|
321
|
-
|
322
|
-
#ensure timestamp is at least 1 nanosecond greater than the last one
|
323
|
-
timestamp = [timestamp, @last_timestamp + 1].max
|
324
|
-
@last_timestamp = timestamp
|
316
|
+
thread_id = tag
|
325
317
|
|
326
|
-
|
327
|
-
if @thread_ids.key? tag
|
328
|
-
thread_id = @thread_ids[tag]
|
329
|
-
else
|
330
|
-
thread_id = @next_id
|
331
|
-
@thread_ids[tag] = thread_id
|
332
|
-
@next_id += 1
|
333
|
-
end
|
334
|
-
}
|
335
|
-
|
336
|
-
#then update the map of threads for this chunk
|
318
|
+
# then update the map of threads for this chunk
|
337
319
|
current_threads[tag] = thread_id
|
338
320
|
|
339
|
-
#add a logfile field if one doesn't exist
|
340
|
-
|
341
|
-
record["logfile"] = "/fluentd/#{tag}"
|
342
|
-
end
|
321
|
+
# add a logfile field if one doesn't exist
|
322
|
+
record["logfile"] = "/fluentd/#{tag}" unless record.key? "logfile"
|
343
323
|
|
344
|
-
#append to list of events
|
345
|
-
event = {
|
346
|
-
|
347
|
-
|
348
|
-
}
|
324
|
+
# append to list of events
|
325
|
+
event = {thread: thread_id.to_s,
|
326
|
+
ts: timestamp,
|
327
|
+
attrs: record}
|
349
328
|
|
350
|
-
#get json string of event to keep track of how many bytes we are sending
|
329
|
+
# get json string of event to keep track of how many bytes we are sending
|
351
330
|
|
352
331
|
begin
|
353
332
|
event_json = event.to_json
|
@@ -355,72 +334,65 @@ module Scalyr
|
|
355
334
|
$log.warn "#{e.class}: #{e.message}"
|
356
335
|
|
357
336
|
# Send the faulty event to a label @ERROR block and allow to handle it there (output to exceptions file for ex)
|
358
|
-
time = Fluent::EventTime.new(
|
337
|
+
time = Fluent::EventTime.new(sec, nsec)
|
359
338
|
router.emit_error_event(tag, time, record, e)
|
360
339
|
|
361
340
|
event[:attrs].each do |key, value|
|
362
341
|
$log.debug "\t#{key} (#{value.encoding.name}): '#{value}'"
|
363
|
-
event[:attrs][key] = value.encode("UTF-8", :
|
342
|
+
event[:attrs][key] = value.encode("UTF-8", invalid: :replace, undef: :replace, replace: "<?>").force_encoding("UTF-8") # rubocop:disable Layout/LineLength, Lint/RedundantCopDisableDirective
|
364
343
|
end
|
365
344
|
event_json = event.to_json
|
366
345
|
end
|
367
346
|
|
368
|
-
#generate new request if json size of events in the array exceed maximum request buffer size
|
347
|
+
# generate new request if json size of events in the array exceed maximum request buffer size
|
369
348
|
append_event = true
|
370
349
|
if total_bytes + event_json.bytesize > @max_request_buffer
|
371
|
-
#make sure we always have at least one event
|
372
|
-
if events.
|
350
|
+
# make sure we always have at least one event
|
351
|
+
if events.empty?
|
373
352
|
events << event
|
374
353
|
append_event = false
|
375
354
|
end
|
376
|
-
request =
|
355
|
+
request = create_request(events, current_threads)
|
377
356
|
requests << request
|
378
357
|
|
379
358
|
total_bytes = 0
|
380
|
-
current_threads =
|
381
|
-
events =
|
359
|
+
current_threads = {}
|
360
|
+
events = []
|
382
361
|
end
|
383
362
|
|
384
|
-
#if we haven't consumed the current event already
|
385
|
-
#add it to the end of our array and keep track of the json bytesize
|
363
|
+
# if we haven't consumed the current event already
|
364
|
+
# add it to the end of our array and keep track of the json bytesize
|
386
365
|
if append_event
|
387
366
|
events << event
|
388
367
|
total_bytes += event_json.bytesize
|
389
368
|
end
|
390
|
-
|
391
369
|
}
|
392
370
|
|
393
|
-
#create a final request with any left over events
|
394
|
-
request =
|
371
|
+
# create a final request with any left over events
|
372
|
+
request = create_request(events, current_threads)
|
395
373
|
requests << request
|
396
|
-
|
397
374
|
end
|
398
375
|
|
399
|
-
def create_request(
|
400
|
-
#build the scalyr thread objects
|
401
|
-
threads =
|
376
|
+
def create_request(events, current_threads)
|
377
|
+
# build the scalyr thread objects
|
378
|
+
threads = []
|
402
379
|
current_threads.each do |tag, id|
|
403
|
-
threads << {
|
404
|
-
|
405
|
-
}
|
380
|
+
threads << {id: id.to_s,
|
381
|
+
name: "Fluentd: #{tag}"}
|
406
382
|
end
|
407
383
|
|
408
|
-
current_time =
|
384
|
+
current_time = to_millis(Fluent::Engine.now)
|
409
385
|
|
410
|
-
body = {
|
411
|
-
|
412
|
-
|
413
|
-
|
414
|
-
|
415
|
-
}
|
386
|
+
body = {token: @api_write_token,
|
387
|
+
client_timestamp: current_time.to_s,
|
388
|
+
session: @session,
|
389
|
+
events: events,
|
390
|
+
threads: threads}
|
416
391
|
|
417
|
-
#add server_attributes hash if it exists
|
418
|
-
if @server_attributes
|
419
|
-
body[:sessionInfo] = @server_attributes
|
420
|
-
end
|
392
|
+
# add server_attributes hash if it exists
|
393
|
+
body[:sessionInfo] = @server_attributes if @server_attributes
|
421
394
|
|
422
|
-
{
|
395
|
+
{body: body.to_json, record_count: events.size}
|
423
396
|
end
|
424
|
-
|
425
397
|
end
|
426
398
|
end
|