fluent-plugin-scalyr 0.8.11 → 0.8.16
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +29 -16
- data/Rakefile +1 -0
- data/VERSION +1 -1
- data/lib/fluent/plugin/out_scalyr.rb +56 -11
- data/lib/fluent/plugin/scalyr_utils.rb +65 -0
- data/test/test_config.rb +5 -0
- data/test/test_events.rb +154 -0
- data/test/test_utils.rb +100 -0
- metadata +6 -3
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: cf7d29d7bd880724b9541793c3b1ee368d47172b88fa919c8ef10c4d8ec9028c
|
4
|
+
data.tar.gz: ad42c0dcb664d9d4c9c2417557733dce293b5af51a52978b5a2ab05604515887
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: de05664b8aa2a23c158815c73159fe3466aef2f17697093558407b022becf095b737e05bd23e20bfb3ed1923b41405297b9658f91a6ecb1ae1f943e5022bcebd
|
7
|
+
data.tar.gz: 2ec30ad8a9e1a3ebc7ea16080729684117d38033109b7e829a959e11802b909fd14202fd85d248f38f0770b083e5b8b95e25f9a83d8659487d80598ed5c5795b
|
data/README.md
CHANGED
@@ -30,22 +30,6 @@ This can be done by specifying tags such as scalyr.apache, scalyr.maillog etc an
|
|
30
30
|
|
31
31
|
Fluentd tag names will be used for the logfile name in Scalyr.
|
32
32
|
|
33
|
-
Scalyr Parsers and Custom Fields
|
34
|
-
--------------------------------
|
35
|
-
|
36
|
-
You may also need to specify a Scalyr parser for your log message or add custom fields to each log event. This can be done using Fluentd's filter mechanism, in particular the [record_transformer filter](https://docs.fluentd.org/filter/record_transformer).
|
37
|
-
|
38
|
-
For example, if you want to use Scalyr's ```accessLog``` parser for all events with the ```scalyr.access``` tag you would add the following to your fluent.conf file:
|
39
|
-
|
40
|
-
```
|
41
|
-
<filter scalyr.access>
|
42
|
-
@type record_transformer
|
43
|
-
<record>
|
44
|
-
parser accessLog
|
45
|
-
</record>
|
46
|
-
</filter>
|
47
|
-
```
|
48
|
-
|
49
33
|
Plugin Configuration
|
50
34
|
-------------
|
51
35
|
|
@@ -80,6 +64,7 @@ The following configuration options are also supported:
|
|
80
64
|
ssl_verify_peer true
|
81
65
|
ssl_verify_depth 5
|
82
66
|
message_field message
|
67
|
+
parser nil
|
83
68
|
|
84
69
|
max_request_buffer 5500000
|
85
70
|
|
@@ -104,6 +89,32 @@ For some additional examples of configuration for different setups, please refer
|
|
104
89
|
[examples/configs/](https://github.com/scalyr/scalyr-fluentd/tree/master/examples/configs/)
|
105
90
|
directory.
|
106
91
|
|
92
|
+
Scalyr Parsers and Custom Fields
|
93
|
+
--------------------------------
|
94
|
+
|
95
|
+
You may also need to specify a Scalyr parser for your log message or add custom fields to each log event. This can be done using Fluentd's filter mechanism, in particular the [record_transformer filter](https://docs.fluentd.org/filter/record_transformer).
|
96
|
+
|
97
|
+
For example, if you want to use Scalyr's ```accessLog``` parser for all events with the ```scalyr.access``` tag you would add the following to your fluent.conf file:
|
98
|
+
|
99
|
+
```
|
100
|
+
<filter scalyr.access>
|
101
|
+
@type record_transformer
|
102
|
+
<record>
|
103
|
+
parser accessLog
|
104
|
+
</record>
|
105
|
+
</filter>
|
106
|
+
```
|
107
|
+
|
108
|
+
Alternatively, if the same parser can be used for all events sent by the Fluentd output plugin, the parser can be set using the ```parser``` plugin configuration. For example, to use Scalyr's ```accessLog``` parser for all events, use the output plugin configuration:
|
109
|
+
|
110
|
+
```
|
111
|
+
<match scalyr.*>
|
112
|
+
@type scalyr
|
113
|
+
api_write_token YOUR_SCALYR_WRITE_LOGS_TOKEN
|
114
|
+
parser accessLog
|
115
|
+
</match>
|
116
|
+
```
|
117
|
+
|
107
118
|
### Scalyr specific options
|
108
119
|
|
109
120
|
***compression_type*** - compress Scalyr traffic to reduce network traffic. Options are `bz2` and `deflate`. See [here](https://www.scalyr.com/help/scalyr-agent#compressing) for more details. This feature is optional.
|
@@ -134,6 +145,8 @@ The cURL project maintains CA certificate bundles automatically converted from m
|
|
134
145
|
|
135
146
|
***message_field*** - Scalyr expects all log events to have a 'message' field containing the contents of a log message. If your event has the log message stored in another field, you can specify the field name here, and the plugin will rename that field to 'message' before sending the data to Scalyr. **Note:** this will override any existing 'message' field if the log record contains both a 'message' field and the field specified by this config option.
|
136
147
|
|
148
|
+
***parser*** - The Scalyr parser to use to parse the 'message' field for each event. This value is optional and defaults to *nil*.
|
149
|
+
|
137
150
|
***max_request_buffer*** - The maximum size in bytes of each request to send to Scalyr. Defaults to 5,500,000 (5.5MB). Fluentd chunks that generate JSON requests larger than the max_request_buffer will be split in to multiple separate requests. **Note:** The maximum size the Scalyr servers accept for this value is 6MB and requests containing data larger than this will be rejected.
|
138
151
|
|
139
152
|
***force_message_encoding*** - Set a specific encoding for all your log messages (defaults to nil). If your log messages are not in UTF-8, this can cause problems when converting the message to JSON in order to send to the Scalyr server. You can avoid these problems by setting an encoding for your log messages so they can be correctly converted.
|
data/Rakefile
CHANGED
data/VERSION
CHANGED
@@ -1 +1 @@
|
|
1
|
-
0.8.
|
1
|
+
0.8.16
|
@@ -19,6 +19,7 @@
|
|
19
19
|
|
20
20
|
require "fluent/plugin/output"
|
21
21
|
require "fluent/plugin/scalyr_exceptions"
|
22
|
+
require "fluent/plugin/scalyr_utils"
|
22
23
|
require "fluent/plugin_helper/compat_parameters"
|
23
24
|
require "json"
|
24
25
|
require "net/http"
|
@@ -36,6 +37,7 @@ module Scalyr
|
|
36
37
|
|
37
38
|
config_param :api_write_token, :string
|
38
39
|
config_param :server_attributes, :hash, default: nil
|
40
|
+
config_param :parser, :string, default: nil # Set the "parser" field to this, per event.
|
39
41
|
config_param :use_hostname_for_serverhost, :bool, default: true
|
40
42
|
config_param :scalyr_server, :string, default: "https://agent.scalyr.com/"
|
41
43
|
config_param :ssl_ca_bundle_path, :string, default: nil
|
@@ -71,6 +73,12 @@ module Scalyr
|
|
71
73
|
end
|
72
74
|
|
73
75
|
def configure(conf)
|
76
|
+
@version = if Gem.loaded_specs.key?("fluent-plugin-scalyr")
|
77
|
+
Gem.loaded_specs["fluent-plugin-scalyr"].version
|
78
|
+
else
|
79
|
+
"unknown"
|
80
|
+
end
|
81
|
+
|
74
82
|
if conf.elements("buffer").empty?
|
75
83
|
$log.warn "Pre 0.14.0 configuration file detected. Please consider updating your configuration file" # rubocop:disable Layout/LineLength, Lint/RedundantCopDisableDirective
|
76
84
|
end
|
@@ -144,7 +152,7 @@ module Scalyr
|
|
144
152
|
# Generate a session id. This will be called once for each <match> in fluent.conf that uses scalyr
|
145
153
|
@session = SecureRandom.uuid
|
146
154
|
|
147
|
-
$log.info "Scalyr Fluentd Plugin ID id=#{plugin_id} worker=#{fluentd_worker_id} session=#{@session}" # rubocop:disable Layout/LineLength, Lint/RedundantCopDisableDirective
|
155
|
+
$log.info "Scalyr Fluentd Plugin ID id=#{plugin_id} worker=#{fluentd_worker_id} session=#{@session} version=#{@version}" # rubocop:disable Layout/LineLength, Lint/RedundantCopDisableDirective
|
148
156
|
end
|
149
157
|
|
150
158
|
def format(tag, time, record)
|
@@ -252,8 +260,8 @@ module Scalyr
|
|
252
260
|
|
253
261
|
post = Net::HTTP::Post.new uri.path
|
254
262
|
post.add_field("Content-Type", "application/json")
|
255
|
-
|
256
263
|
post.add_field("Content-Encoding", encoding) if @compression_type
|
264
|
+
post.add_field("User-Agent", "fluent-plugin-scalyr;#{@version}")
|
257
265
|
|
258
266
|
post.body = body
|
259
267
|
|
@@ -321,6 +329,9 @@ module Scalyr
|
|
321
329
|
# add a logfile field if one doesn't exist
|
322
330
|
record["logfile"] = "/fluentd/#{tag}" unless record.key? "logfile"
|
323
331
|
|
332
|
+
# set per-event parser if it is configured
|
333
|
+
record["parser"] = @parser unless @parser.nil?
|
334
|
+
|
324
335
|
# append to list of events
|
325
336
|
event = {thread: thread_id.to_s,
|
326
337
|
ts: timestamp,
|
@@ -331,29 +342,59 @@ module Scalyr
|
|
331
342
|
begin
|
332
343
|
event_json = event.to_json
|
333
344
|
rescue JSON::GeneratorError, Encoding::UndefinedConversionError => e
|
334
|
-
$log.warn "#{e.class}: #{e.message}"
|
345
|
+
$log.warn "JSON serialization of the event failed: #{e.class}: #{e.message}"
|
335
346
|
|
336
347
|
# Send the faulty event to a label @ERROR block and allow to handle it there (output to exceptions file for ex)
|
337
348
|
time = Fluent::EventTime.new(sec, nsec)
|
338
349
|
router.emit_error_event(tag, time, record, e)
|
339
350
|
|
351
|
+
# Print attribute values for debugging / troubleshooting purposes
|
352
|
+
$log.debug "Event attributes:"
|
353
|
+
|
340
354
|
event[:attrs].each do |key, value|
|
341
|
-
|
342
|
-
|
355
|
+
# NOTE: value doesn't always value.encoding attribute so we use .class which is always available
|
356
|
+
$log.debug "\t#{key} (#{value.class}): '#{value}'"
|
343
357
|
end
|
358
|
+
|
359
|
+
# Recursively re-encode and sanitize potentially bad string values
|
360
|
+
event[:attrs] = sanitize_and_reencode_value(event[:attrs])
|
344
361
|
event_json = event.to_json
|
345
362
|
end
|
346
363
|
|
347
364
|
# generate new request if json size of events in the array exceed maximum request buffer size
|
348
365
|
append_event = true
|
349
366
|
if total_bytes + event_json.bytesize > @max_request_buffer
|
350
|
-
#
|
367
|
+
# the case where a single event causes us to exceed the @max_request_buffer
|
351
368
|
if events.empty?
|
352
|
-
|
369
|
+
# if we are able to truncate the content (and append an ellipsis)
|
370
|
+
# inside the @message_field we do so here
|
371
|
+
if record.key?(@message_field) &&
|
372
|
+
record[@message_field].is_a?(String) &&
|
373
|
+
record[@message_field].bytesize > event_json.bytesize - @max_request_buffer &&
|
374
|
+
record[@message_field].bytesize >= 3
|
375
|
+
|
376
|
+
@log.warn "Received a record that cannot fit within max_request_buffer "\
|
377
|
+
"(#{@max_request_buffer}) from #{record['logfile']}, serialized event size "\
|
378
|
+
"is #{event_json.bytesize}. The #{@message_field} field will be truncated to fit."
|
379
|
+
max_msg_size = @max_request_buffer - event_json.bytesize - 3
|
380
|
+
truncated_msg = event[:attrs][@message_field][0...max_msg_size] + "..."
|
381
|
+
event[:attrs][@message_field] = truncated_msg
|
382
|
+
events << event
|
383
|
+
|
384
|
+
# otherwise we drop the event and save ourselves hitting a 4XX response from the server
|
385
|
+
else
|
386
|
+
@log.warn "Received a record that cannot fit within max_request_buffer "\
|
387
|
+
"(#{@max_request_buffer}) from #{record['logfile']}, serialized event size "\
|
388
|
+
"is #{event_json.bytesize}. The #{@message_field} field too short to truncate, "\
|
389
|
+
"dropping event."
|
390
|
+
end
|
353
391
|
append_event = false
|
354
392
|
end
|
355
|
-
|
356
|
-
|
393
|
+
|
394
|
+
unless events.empty?
|
395
|
+
request = create_request(events, current_threads)
|
396
|
+
requests << request
|
397
|
+
end
|
357
398
|
|
358
399
|
total_bytes = 0
|
359
400
|
current_threads = {}
|
@@ -369,8 +410,12 @@ module Scalyr
|
|
369
410
|
}
|
370
411
|
|
371
412
|
# create a final request with any left over events
|
372
|
-
|
373
|
-
|
413
|
+
unless events.empty?
|
414
|
+
request = create_request(events, current_threads)
|
415
|
+
requests << request
|
416
|
+
end
|
417
|
+
|
418
|
+
requests
|
374
419
|
end
|
375
420
|
|
376
421
|
def create_request(events, current_threads)
|
@@ -0,0 +1,65 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
#
|
4
|
+
# Scalyr Output Plugin for Fluentd
|
5
|
+
#
|
6
|
+
# Copyright (C) 2020 Scalyr, Inc.
|
7
|
+
#
|
8
|
+
# Licensed under the Apache License, Version 2.0 (the "License");
|
9
|
+
# you may not use this file except in compliance with the License.
|
10
|
+
# You may obtain a copy of the License at
|
11
|
+
#
|
12
|
+
# http://www.apache.org/licenses/LICENSE-2.0
|
13
|
+
#
|
14
|
+
# Unless required by applicable law or agreed to in writing, software
|
15
|
+
# distributed under the License is distributed on an "AS IS" BASIS,
|
16
|
+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
17
|
+
# See the License for the specific language governing permissions and
|
18
|
+
# limitations under the License.
|
19
|
+
|
20
|
+
module Scalyr
|
21
|
+
def sanitize_and_reencode_value(value)
|
22
|
+
# Method which recursively sanitizes the provided value and tries to re-encode all the strings as
|
23
|
+
# UTF-8 ignoring any bad unicode sequences
|
24
|
+
case value
|
25
|
+
when Hash
|
26
|
+
return sanitize_and_reencode_hash(value)
|
27
|
+
when Array
|
28
|
+
return sanitize_and_reencode_array(value)
|
29
|
+
when String
|
30
|
+
value = sanitize_and_reencode_string(value)
|
31
|
+
return value
|
32
|
+
end
|
33
|
+
|
34
|
+
# We only need to re-encode strings, for other value types (ints, nils,
|
35
|
+
# etc. no reencoding is needed)
|
36
|
+
value
|
37
|
+
end
|
38
|
+
|
39
|
+
def sanitize_and_reencode_array(array)
|
40
|
+
array.each_with_index do |value, index|
|
41
|
+
value = sanitize_and_reencode_value(value)
|
42
|
+
array[index] = value
|
43
|
+
end
|
44
|
+
|
45
|
+
array
|
46
|
+
end
|
47
|
+
|
48
|
+
def sanitize_and_reencode_hash(hash)
|
49
|
+
hash.each do |key, value|
|
50
|
+
hash[key] = sanitize_and_reencode_value(value)
|
51
|
+
end
|
52
|
+
|
53
|
+
hash
|
54
|
+
end
|
55
|
+
|
56
|
+
def sanitize_and_reencode_string(value)
|
57
|
+
# Function which sanitized the provided string value and tries to re-encode it as UTF-8
|
58
|
+
# ignoring any encoding error which could arise due to bad or partial unicode sequence
|
59
|
+
begin # rubocop:disable Style/RedundantBegin
|
60
|
+
value.encode("UTF-8", invalid: :replace, undef: :replace, replace: "<?>").force_encoding("UTF-8") # rubocop:disable Layout/LineLength, Lint/RedundantCopDisableDirective
|
61
|
+
rescue # rubocop:disable Style/RescueStandardError
|
62
|
+
"failed-to-reencode-as-utf8"
|
63
|
+
end
|
64
|
+
end
|
65
|
+
end
|
data/test/test_config.rb
CHANGED
@@ -57,5 +57,10 @@ class ConfigTest < Scalyr::ScalyrOutTest
|
|
57
57
|
d = create_driver CONFIG + 'server_attributes { "test":"value" }'
|
58
58
|
assert_equal("value", d.instance.server_attributes["test"], "Config failed to set server_attributes")
|
59
59
|
end
|
60
|
+
|
61
|
+
def test_configure_parser
|
62
|
+
d = create_driver CONFIG + "parser access_log"
|
63
|
+
assert_equal("access_log", d.instance.parser, "Config failed to set parser")
|
64
|
+
end
|
60
65
|
end
|
61
66
|
# rubocop:enable Layout/LineLength
|
data/test/test_events.rb
CHANGED
@@ -136,6 +136,34 @@ class EventsTest < Scalyr::ScalyrOutTest
|
|
136
136
|
assert_equal(mock_called, true, "mock method was never called!")
|
137
137
|
end
|
138
138
|
|
139
|
+
def test_build_add_events_body_with_parser
|
140
|
+
d = create_driver CONFIG + "parser test_parser"
|
141
|
+
|
142
|
+
time = event_time("2015-04-01 10:00:00 UTC")
|
143
|
+
attrs = {"a" => 1}
|
144
|
+
|
145
|
+
response = flexmock(Net::HTTPResponse, code: "200", body: '{ "status":"success" }')
|
146
|
+
mock = flexmock(d.instance)
|
147
|
+
|
148
|
+
mock_called = false
|
149
|
+
|
150
|
+
mock.should_receive(:post_request).with(
|
151
|
+
URI,
|
152
|
+
on {|request_body|
|
153
|
+
body = JSON.parse(request_body)
|
154
|
+
assert_equal("test_parser", body["events"][0]["attrs"]["parser"])
|
155
|
+
mock_called = true
|
156
|
+
true
|
157
|
+
}
|
158
|
+
).once.and_return(response)
|
159
|
+
|
160
|
+
d.run(default_tag: "test") do
|
161
|
+
d.feed(time, attrs)
|
162
|
+
end
|
163
|
+
|
164
|
+
assert_equal(mock_called, true, "mock method was never called!")
|
165
|
+
end
|
166
|
+
|
139
167
|
def test_build_add_events_body_incrementing_timestamps
|
140
168
|
d = create_driver
|
141
169
|
|
@@ -170,6 +198,88 @@ class EventsTest < Scalyr::ScalyrOutTest
|
|
170
198
|
assert_equal(mock_called, true, "mock method was never called!")
|
171
199
|
end
|
172
200
|
|
201
|
+
def test_build_add_events_body_non_json_serializable_value
|
202
|
+
d = create_driver
|
203
|
+
|
204
|
+
time = event_time("2015-04-01 10:00:00 UTC")
|
205
|
+
attrs = {"a" => 1}
|
206
|
+
attrs["int1"] = 1_601_923_119
|
207
|
+
attrs["int2"] = Integer(1_601_923_119)
|
208
|
+
attrs["int3"] = Integer(9_223_372_036_854_775_807)
|
209
|
+
attrs["int4"] = Integer(-1)
|
210
|
+
attrs["nil"] = nil
|
211
|
+
attrs["array"] = [1, 2, "a", "b", nil]
|
212
|
+
attrs["hash"] = {
|
213
|
+
"a" => "1",
|
214
|
+
"b" => "c"
|
215
|
+
}
|
216
|
+
attrs["logfile"] = "/some/log/file"
|
217
|
+
|
218
|
+
# This partial unicode sequence will fail encoding so we make sure it doesn't break the plugin
|
219
|
+
# and we correctly cast it to a value which we can send to the API
|
220
|
+
attrs["partial_unicode_sequence"] = "\xC2"
|
221
|
+
attrs["array_with_partial_unicode_sequence"] = [1, 2, "a", "b", nil, "7", "\xC2"]
|
222
|
+
attrs["nested_array_with_partial_unicode_sequence"] = [1, 2, "a", "b", nil, "7",
|
223
|
+
[8, 9, [10, "\xC2"]],
|
224
|
+
{"a" => 1, "b" => "\xC2"}]
|
225
|
+
attrs["hash_with_partial_unicode_sequence"] = {
|
226
|
+
"a" => "1",
|
227
|
+
"b" => "\xC2",
|
228
|
+
"c" => nil
|
229
|
+
}
|
230
|
+
attrs["nested_hash_with_partial_unicode_sequence"] = {
|
231
|
+
"a" => "1",
|
232
|
+
"b" => {
|
233
|
+
"c" => "\xC2",
|
234
|
+
"d" => "e",
|
235
|
+
"f" => nil,
|
236
|
+
"g" => {
|
237
|
+
"h" => "\xC2",
|
238
|
+
"b" => 3
|
239
|
+
}
|
240
|
+
}
|
241
|
+
}
|
242
|
+
|
243
|
+
response = flexmock(Net::HTTPResponse, code: "200", body: '{ "status":"success" }')
|
244
|
+
mock = flexmock(d.instance)
|
245
|
+
|
246
|
+
mock_called = false
|
247
|
+
|
248
|
+
# NOTE: We need to perform a deep clone / copy
|
249
|
+
expected_attrs = Marshal.load(Marshal.dump(attrs))
|
250
|
+
|
251
|
+
expected_attrs["partial_unicode_sequence"] = "<?>"
|
252
|
+
expected_attrs["array_with_partial_unicode_sequence"][-1] = "<?>"
|
253
|
+
expected_attrs["nested_array_with_partial_unicode_sequence"][-2][-1][-1] = "<?>"
|
254
|
+
expected_attrs["nested_array_with_partial_unicode_sequence"][-1]["b"] = "<?>"
|
255
|
+
expected_attrs["hash_with_partial_unicode_sequence"]["b"] = "<?>"
|
256
|
+
expected_attrs["nested_hash_with_partial_unicode_sequence"]["b"]["c"] = "<?>"
|
257
|
+
expected_attrs["nested_hash_with_partial_unicode_sequence"]["b"]["g"]["h"] = "<?>"
|
258
|
+
|
259
|
+
# Verify that clone / copy was correct and the original object wasn't modified
|
260
|
+
assert_not_equal(expected_attrs, attrs, "Objects are the same but should be different")
|
261
|
+
assert_not_equal(Marshal.load(Marshal.dump(attrs)), Marshal.load(Marshal.dump(expected_attrs)))
|
262
|
+
assert_equal(attrs["partial_unicode_sequence"], "\xC2")
|
263
|
+
assert_equal(attrs["array_with_partial_unicode_sequence"][-1], "\xC2")
|
264
|
+
assert_equal(attrs["nested_hash_with_partial_unicode_sequence"]["b"]["g"]["h"], "\xC2")
|
265
|
+
|
266
|
+
mock.should_receive(:post_request).with(
|
267
|
+
URI,
|
268
|
+
on {|request_body|
|
269
|
+
body = JSON.parse(request_body)
|
270
|
+
assert_equal(expected_attrs, body["events"][0]["attrs"], "Value of attrs differs from log")
|
271
|
+
mock_called = true
|
272
|
+
true
|
273
|
+
}
|
274
|
+
).once.and_return(response)
|
275
|
+
|
276
|
+
d.run(default_tag: "test") do
|
277
|
+
d.feed(time, attrs)
|
278
|
+
end
|
279
|
+
|
280
|
+
assert_equal(mock_called, true, "mock method was never called!")
|
281
|
+
end
|
282
|
+
|
173
283
|
def test_default_message_field
|
174
284
|
d = create_driver CONFIG
|
175
285
|
|
@@ -247,4 +357,48 @@ class EventsTest < Scalyr::ScalyrOutTest
|
|
247
357
|
d.feed(time, attrs)
|
248
358
|
end
|
249
359
|
end
|
360
|
+
|
361
|
+
def test_truncated_large_event
|
362
|
+
d = create_driver CONFIG + "max_request_buffer 4000"
|
363
|
+
|
364
|
+
time = event_time("2015-04-01 10:00:00 UTC")
|
365
|
+
attrs = {"log" => "this is a test", "message" => "0123456789" * 500}
|
366
|
+
|
367
|
+
response = flexmock(Net::HTTPResponse, code: "200", body: '{ "status":"success" }')
|
368
|
+
mock = flexmock(d.instance)
|
369
|
+
|
370
|
+
mock.should_receive(:post_request).with(
|
371
|
+
URI,
|
372
|
+
on {|request_body|
|
373
|
+
body = JSON.parse(request_body)
|
374
|
+
events = body["events"]
|
375
|
+
assert(events[0]["attrs"].key?("message"), "'message' field not found in event")
|
376
|
+
assert_equal(
|
377
|
+
"0123456789" * 388 + "012...",
|
378
|
+
events[0]["attrs"]["message"],
|
379
|
+
"'message' field incorrect"
|
380
|
+
)
|
381
|
+
true
|
382
|
+
}
|
383
|
+
).once.and_return(response)
|
384
|
+
|
385
|
+
d.run(default_tag: "test") do
|
386
|
+
d.feed(time, attrs)
|
387
|
+
end
|
388
|
+
end
|
389
|
+
|
390
|
+
def test_dropped_large_event
|
391
|
+
d = create_driver CONFIG + "max_request_buffer 4000"
|
392
|
+
|
393
|
+
time = event_time("2015-04-01 10:00:00 UTC")
|
394
|
+
attrs = {"message" => "this is a test", "not_message" => "0123456789" * 500}
|
395
|
+
|
396
|
+
mock = flexmock(d.instance)
|
397
|
+
|
398
|
+
mock.should_receive(:post_request).never
|
399
|
+
|
400
|
+
d.run(default_tag: "test") do
|
401
|
+
d.feed(time, attrs)
|
402
|
+
end
|
403
|
+
end
|
250
404
|
end
|
data/test/test_utils.rb
ADDED
@@ -0,0 +1,100 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
#
|
4
|
+
# Scalyr Output Plugin for Fluentd
|
5
|
+
#
|
6
|
+
# Copyright (C) 2015 Scalyr, Inc.
|
7
|
+
#
|
8
|
+
# Licensed under the Apache License, Version 2.0 (the "License");
|
9
|
+
# you may not use this file except in compliance with the License.
|
10
|
+
# You may obtain a copy of the License at
|
11
|
+
#
|
12
|
+
# http://www.apache.org/licenses/LICENSE-2.0
|
13
|
+
#
|
14
|
+
# Unless required by applicable law or agreed to in writing, software
|
15
|
+
# distributed under the License is distributed on an "AS IS" BASIS,
|
16
|
+
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
|
17
|
+
# See the License for the specific language governing permissions and
|
18
|
+
# limitations under the License.
|
19
|
+
|
20
|
+
require "helper"
|
21
|
+
require "flexmock/test_unit"
|
22
|
+
|
23
|
+
require "fluent/plugin/scalyr_utils"
|
24
|
+
include Scalyr # rubocop:disable Style/MixinUsage
|
25
|
+
|
26
|
+
module Scalyr
|
27
|
+
class UtilsTest < Test::Unit::TestCase
|
28
|
+
def test_sanitize_and_reencode_value_simple_types
|
29
|
+
# Simple value - string
|
30
|
+
result = sanitize_and_reencode_value("test foo")
|
31
|
+
assert_equal("test foo", result)
|
32
|
+
|
33
|
+
# Simple value - string with bad unicode sequences
|
34
|
+
result = sanitize_and_reencode_value("test \xC2 foo \xC2 bar")
|
35
|
+
assert_equal("test <?> foo <?> bar", result)
|
36
|
+
|
37
|
+
# Simple value - int
|
38
|
+
result = sanitize_and_reencode_value(100)
|
39
|
+
assert_equal(100, result)
|
40
|
+
|
41
|
+
# Simple value - nill
|
42
|
+
result = sanitize_and_reencode_value(nil)
|
43
|
+
assert_equal(nil, result)
|
44
|
+
|
45
|
+
# Simple value - bool
|
46
|
+
result = sanitize_and_reencode_value(true)
|
47
|
+
assert_equal(true, result)
|
48
|
+
|
49
|
+
result = sanitize_and_reencode_value(false)
|
50
|
+
assert_equal(false, result)
|
51
|
+
end
|
52
|
+
|
53
|
+
def test_sanitize_and_reencode_value_complex_nested_types
|
54
|
+
actual = [1, 2, "a", "b", nil, "7", "\xC2"]
|
55
|
+
expected = [1, 2, "a", "b", nil, "7", "<?>"]
|
56
|
+
|
57
|
+
result = sanitize_and_reencode_value(actual)
|
58
|
+
assert_equal(expected, result)
|
59
|
+
|
60
|
+
actual = [1, 2, "a", "b", nil, "7",
|
61
|
+
[8, 9, [10, "\xC2"]],
|
62
|
+
{"a" => 1, "b" => "\xC2"}]
|
63
|
+
expected = [1, 2, "a", "b", nil, "7",
|
64
|
+
[8, 9, [10, "<?>"]],
|
65
|
+
{"a" => 1, "b" => "<?>"}]
|
66
|
+
|
67
|
+
result = sanitize_and_reencode_value(actual)
|
68
|
+
assert_equal(expected, result)
|
69
|
+
|
70
|
+
actual = {
|
71
|
+
"a" => "1",
|
72
|
+
"b" => {
|
73
|
+
"c" => "\xC2",
|
74
|
+
"d" => "e",
|
75
|
+
"f" => nil,
|
76
|
+
"g" => {
|
77
|
+
"h" => "bar \xC2",
|
78
|
+
"b" => 3,
|
79
|
+
"l" => [1, 2, "foo\xC2", 3, 4, 5]
|
80
|
+
}
|
81
|
+
}
|
82
|
+
}
|
83
|
+
expected = {
|
84
|
+
"a" => "1",
|
85
|
+
"b" => {
|
86
|
+
"c" => "<?>",
|
87
|
+
"d" => "e",
|
88
|
+
"f" => nil,
|
89
|
+
"g" => {
|
90
|
+
"h" => "bar <?>",
|
91
|
+
"b" => 3,
|
92
|
+
"l" => [1, 2, "foo<?>", 3, 4, 5]
|
93
|
+
}
|
94
|
+
}
|
95
|
+
}
|
96
|
+
result = sanitize_and_reencode_value(actual)
|
97
|
+
assert_equal(expected, result)
|
98
|
+
end
|
99
|
+
end
|
100
|
+
end
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: fluent-plugin-scalyr
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.8.
|
4
|
+
version: 0.8.16
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Imron Alston
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date:
|
11
|
+
date: 2022-02-14 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: ffi
|
@@ -158,11 +158,13 @@ files:
|
|
158
158
|
- fluent.conf.sample
|
159
159
|
- lib/fluent/plugin/out_scalyr.rb
|
160
160
|
- lib/fluent/plugin/scalyr_exceptions.rb
|
161
|
+
- lib/fluent/plugin/scalyr_utils.rb
|
161
162
|
- test/helper.rb
|
162
163
|
- test/test_config.rb
|
163
164
|
- test/test_events.rb
|
164
165
|
- test/test_handle_response.rb
|
165
166
|
- test/test_ssl_verify.rb
|
167
|
+
- test/test_utils.rb
|
166
168
|
homepage: https://github.com/scalyr/scalyr-fluentd
|
167
169
|
licenses:
|
168
170
|
- Apache-2.0
|
@@ -188,7 +190,8 @@ specification_version: 4
|
|
188
190
|
summary: Scalyr plugin for fluentd
|
189
191
|
test_files:
|
190
192
|
- test/helper.rb
|
193
|
+
- test/test_config.rb
|
191
194
|
- test/test_events.rb
|
192
195
|
- test/test_handle_response.rb
|
193
|
-
- test/test_config.rb
|
194
196
|
- test/test_ssl_verify.rb
|
197
|
+
- test/test_utils.rb
|