fluent-plugin-scalyr 0.8.11 → 0.8.16

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 2d9fa05bade6ec51a39896e90bce7ddfdf233c8888d5ca2ac41393966c8a924b
4
- data.tar.gz: b6406c122b1180faa48294b47ed2ebb1b61aad08c64c0310385a00c0ae8a044a
3
+ metadata.gz: cf7d29d7bd880724b9541793c3b1ee368d47172b88fa919c8ef10c4d8ec9028c
4
+ data.tar.gz: ad42c0dcb664d9d4c9c2417557733dce293b5af51a52978b5a2ab05604515887
5
5
  SHA512:
6
- metadata.gz: 3b083816314fdf992eaca0cb0216795e2c7a6c1cc7e2f32669452201c45b29d6254dcbc2f69a665d6a0852242423bd87cdf62a88feda5f463e6f774b142d0520
7
- data.tar.gz: 8a94b4b649cfd45fa1143e6866d119811236771980f508d0643b0b51b1d6bbdb2e241233a469086a0b5a3e22529c08d1c23b7825966fe78e367defd4672d20c2
6
+ metadata.gz: de05664b8aa2a23c158815c73159fe3466aef2f17697093558407b022becf095b737e05bd23e20bfb3ed1923b41405297b9658f91a6ecb1ae1f943e5022bcebd
7
+ data.tar.gz: 2ec30ad8a9e1a3ebc7ea16080729684117d38033109b7e829a959e11802b909fd14202fd85d248f38f0770b083e5b8b95e25f9a83d8659487d80598ed5c5795b
data/README.md CHANGED
@@ -30,22 +30,6 @@ This can be done by specifying tags such as scalyr.apache, scalyr.maillog etc an
30
30
 
31
31
  Fluentd tag names will be used for the logfile name in Scalyr.
32
32
 
33
- Scalyr Parsers and Custom Fields
34
- --------------------------------
35
-
36
- You may also need to specify a Scalyr parser for your log message or add custom fields to each log event. This can be done using Fluentd's filter mechanism, in particular the [record_transformer filter](https://docs.fluentd.org/filter/record_transformer).
37
-
38
- For example, if you want to use Scalyr's ```accessLog``` parser for all events with the ```scalyr.access``` tag you would add the following to your fluent.conf file:
39
-
40
- ```
41
- <filter scalyr.access>
42
- @type record_transformer
43
- <record>
44
- parser accessLog
45
- </record>
46
- </filter>
47
- ```
48
-
49
33
  Plugin Configuration
50
34
  -------------
51
35
 
@@ -80,6 +64,7 @@ The following configuration options are also supported:
80
64
  ssl_verify_peer true
81
65
  ssl_verify_depth 5
82
66
  message_field message
67
+ parser nil
83
68
 
84
69
  max_request_buffer 5500000
85
70
 
@@ -104,6 +89,32 @@ For some additional examples of configuration for different setups, please refer
104
89
  [examples/configs/](https://github.com/scalyr/scalyr-fluentd/tree/master/examples/configs/)
105
90
  directory.
106
91
 
92
+ Scalyr Parsers and Custom Fields
93
+ --------------------------------
94
+
95
+ You may also need to specify a Scalyr parser for your log message or add custom fields to each log event. This can be done using Fluentd's filter mechanism, in particular the [record_transformer filter](https://docs.fluentd.org/filter/record_transformer).
96
+
97
+ For example, if you want to use Scalyr's ```accessLog``` parser for all events with the ```scalyr.access``` tag you would add the following to your fluent.conf file:
98
+
99
+ ```
100
+ <filter scalyr.access>
101
+ @type record_transformer
102
+ <record>
103
+ parser accessLog
104
+ </record>
105
+ </filter>
106
+ ```
107
+
108
+ Alternatively, if the same parser can be used for all events sent by the Fluentd output plugin, the parser can be set using the ```parser``` plugin configuration. For example, to use Scalyr's ```accessLog``` parser for all events, use the output plugin configuration:
109
+
110
+ ```
111
+ <match scalyr.*>
112
+ @type scalyr
113
+ api_write_token YOUR_SCALYR_WRITE_LOGS_TOKEN
114
+ parser accessLog
115
+ </match>
116
+ ```
117
+
107
118
  ### Scalyr specific options
108
119
 
109
120
  ***compression_type*** - compress Scalyr traffic to reduce network traffic. Options are `bz2` and `deflate`. See [here](https://www.scalyr.com/help/scalyr-agent#compressing) for more details. This feature is optional.
@@ -134,6 +145,8 @@ The cURL project maintains CA certificate bundles automatically converted from m
134
145
 
135
146
  ***message_field*** - Scalyr expects all log events to have a 'message' field containing the contents of a log message. If your event has the log message stored in another field, you can specify the field name here, and the plugin will rename that field to 'message' before sending the data to Scalyr. **Note:** this will override any existing 'message' field if the log record contains both a 'message' field and the field specified by this config option.
136
147
 
148
+ ***parser*** - The Scalyr parser to use to parse the 'message' field for each event. This value is optional and defaults to *nil*.
149
+
137
150
  ***max_request_buffer*** - The maximum size in bytes of each request to send to Scalyr. Defaults to 5,500,000 (5.5MB). Fluentd chunks that generate JSON requests larger than the max_request_buffer will be split in to multiple separate requests. **Note:** The maximum size the Scalyr servers accept for this value is 6MB and requests containing data larger than this will be rejected.
138
151
 
139
152
  ***force_message_encoding*** - Set a specific encoding for all your log messages (defaults to nil). If your log messages are not in UTF-8, this can cause problems when converting the message to JSON in order to send to the Scalyr server. You can avoid these problems by setting an encoding for your log messages so they can be correctly converted.
data/Rakefile CHANGED
@@ -9,6 +9,7 @@ Rake::TestTask.new(:test) do |test|
9
9
  test.libs << "lib" << "test"
10
10
  test.test_files = FileList["test/test_*.rb"]
11
11
  test.verbose = true
12
+ test.options = "--verbose=verbose"
12
13
  end
13
14
 
14
15
  task default: [:build]
data/VERSION CHANGED
@@ -1 +1 @@
1
- 0.8.11
1
+ 0.8.16
@@ -19,6 +19,7 @@
19
19
 
20
20
  require "fluent/plugin/output"
21
21
  require "fluent/plugin/scalyr_exceptions"
22
+ require "fluent/plugin/scalyr_utils"
22
23
  require "fluent/plugin_helper/compat_parameters"
23
24
  require "json"
24
25
  require "net/http"
@@ -36,6 +37,7 @@ module Scalyr
36
37
 
37
38
  config_param :api_write_token, :string
38
39
  config_param :server_attributes, :hash, default: nil
40
+ config_param :parser, :string, default: nil # Set the "parser" field to this, per event.
39
41
  config_param :use_hostname_for_serverhost, :bool, default: true
40
42
  config_param :scalyr_server, :string, default: "https://agent.scalyr.com/"
41
43
  config_param :ssl_ca_bundle_path, :string, default: nil
@@ -71,6 +73,12 @@ module Scalyr
71
73
  end
72
74
 
73
75
  def configure(conf)
76
+ @version = if Gem.loaded_specs.key?("fluent-plugin-scalyr")
77
+ Gem.loaded_specs["fluent-plugin-scalyr"].version
78
+ else
79
+ "unknown"
80
+ end
81
+
74
82
  if conf.elements("buffer").empty?
75
83
  $log.warn "Pre 0.14.0 configuration file detected. Please consider updating your configuration file" # rubocop:disable Layout/LineLength, Lint/RedundantCopDisableDirective
76
84
  end
@@ -144,7 +152,7 @@ module Scalyr
144
152
  # Generate a session id. This will be called once for each <match> in fluent.conf that uses scalyr
145
153
  @session = SecureRandom.uuid
146
154
 
147
- $log.info "Scalyr Fluentd Plugin ID id=#{plugin_id} worker=#{fluentd_worker_id} session=#{@session}" # rubocop:disable Layout/LineLength, Lint/RedundantCopDisableDirective
155
+ $log.info "Scalyr Fluentd Plugin ID id=#{plugin_id} worker=#{fluentd_worker_id} session=#{@session} version=#{@version}" # rubocop:disable Layout/LineLength, Lint/RedundantCopDisableDirective
148
156
  end
149
157
 
150
158
  def format(tag, time, record)
@@ -252,8 +260,8 @@ module Scalyr
252
260
 
253
261
  post = Net::HTTP::Post.new uri.path
254
262
  post.add_field("Content-Type", "application/json")
255
-
256
263
  post.add_field("Content-Encoding", encoding) if @compression_type
264
+ post.add_field("User-Agent", "fluent-plugin-scalyr;#{@version}")
257
265
 
258
266
  post.body = body
259
267
 
@@ -321,6 +329,9 @@ module Scalyr
321
329
  # add a logfile field if one doesn't exist
322
330
  record["logfile"] = "/fluentd/#{tag}" unless record.key? "logfile"
323
331
 
332
+ # set per-event parser if it is configured
333
+ record["parser"] = @parser unless @parser.nil?
334
+
324
335
  # append to list of events
325
336
  event = {thread: thread_id.to_s,
326
337
  ts: timestamp,
@@ -331,29 +342,59 @@ module Scalyr
331
342
  begin
332
343
  event_json = event.to_json
333
344
  rescue JSON::GeneratorError, Encoding::UndefinedConversionError => e
334
- $log.warn "#{e.class}: #{e.message}"
345
+ $log.warn "JSON serialization of the event failed: #{e.class}: #{e.message}"
335
346
 
336
347
  # Send the faulty event to a label @ERROR block and allow to handle it there (output to exceptions file for ex)
337
348
  time = Fluent::EventTime.new(sec, nsec)
338
349
  router.emit_error_event(tag, time, record, e)
339
350
 
351
+ # Print attribute values for debugging / troubleshooting purposes
352
+ $log.debug "Event attributes:"
353
+
340
354
  event[:attrs].each do |key, value|
341
- $log.debug "\t#{key} (#{value.encoding.name}): '#{value}'"
342
- event[:attrs][key] = value.encode("UTF-8", invalid: :replace, undef: :replace, replace: "<?>").force_encoding("UTF-8") # rubocop:disable Layout/LineLength, Lint/RedundantCopDisableDirective
355
+ # NOTE: value doesn't always value.encoding attribute so we use .class which is always available
356
+ $log.debug "\t#{key} (#{value.class}): '#{value}'"
343
357
  end
358
+
359
+ # Recursively re-encode and sanitize potentially bad string values
360
+ event[:attrs] = sanitize_and_reencode_value(event[:attrs])
344
361
  event_json = event.to_json
345
362
  end
346
363
 
347
364
  # generate new request if json size of events in the array exceed maximum request buffer size
348
365
  append_event = true
349
366
  if total_bytes + event_json.bytesize > @max_request_buffer
350
- # make sure we always have at least one event
367
+ # the case where a single event causes us to exceed the @max_request_buffer
351
368
  if events.empty?
352
- events << event
369
+ # if we are able to truncate the content (and append an ellipsis)
370
+ # inside the @message_field we do so here
371
+ if record.key?(@message_field) &&
372
+ record[@message_field].is_a?(String) &&
373
+ record[@message_field].bytesize > event_json.bytesize - @max_request_buffer &&
374
+ record[@message_field].bytesize >= 3
375
+
376
+ @log.warn "Received a record that cannot fit within max_request_buffer "\
377
+ "(#{@max_request_buffer}) from #{record['logfile']}, serialized event size "\
378
+ "is #{event_json.bytesize}. The #{@message_field} field will be truncated to fit."
379
+ max_msg_size = @max_request_buffer - event_json.bytesize - 3
380
+ truncated_msg = event[:attrs][@message_field][0...max_msg_size] + "..."
381
+ event[:attrs][@message_field] = truncated_msg
382
+ events << event
383
+
384
+ # otherwise we drop the event and save ourselves hitting a 4XX response from the server
385
+ else
386
+ @log.warn "Received a record that cannot fit within max_request_buffer "\
387
+ "(#{@max_request_buffer}) from #{record['logfile']}, serialized event size "\
388
+ "is #{event_json.bytesize}. The #{@message_field} field too short to truncate, "\
389
+ "dropping event."
390
+ end
353
391
  append_event = false
354
392
  end
355
- request = create_request(events, current_threads)
356
- requests << request
393
+
394
+ unless events.empty?
395
+ request = create_request(events, current_threads)
396
+ requests << request
397
+ end
357
398
 
358
399
  total_bytes = 0
359
400
  current_threads = {}
@@ -369,8 +410,12 @@ module Scalyr
369
410
  }
370
411
 
371
412
  # create a final request with any left over events
372
- request = create_request(events, current_threads)
373
- requests << request
413
+ unless events.empty?
414
+ request = create_request(events, current_threads)
415
+ requests << request
416
+ end
417
+
418
+ requests
374
419
  end
375
420
 
376
421
  def create_request(events, current_threads)
@@ -0,0 +1,65 @@
1
+ # frozen_string_literal: true
2
+
3
+ #
4
+ # Scalyr Output Plugin for Fluentd
5
+ #
6
+ # Copyright (C) 2020 Scalyr, Inc.
7
+ #
8
+ # Licensed under the Apache License, Version 2.0 (the "License");
9
+ # you may not use this file except in compliance with the License.
10
+ # You may obtain a copy of the License at
11
+ #
12
+ # http://www.apache.org/licenses/LICENSE-2.0
13
+ #
14
+ # Unless required by applicable law or agreed to in writing, software
15
+ # distributed under the License is distributed on an "AS IS" BASIS,
16
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
17
+ # See the License for the specific language governing permissions and
18
+ # limitations under the License.
19
+
20
+ module Scalyr
21
+ def sanitize_and_reencode_value(value)
22
+ # Method which recursively sanitizes the provided value and tries to re-encode all the strings as
23
+ # UTF-8 ignoring any bad unicode sequences
24
+ case value
25
+ when Hash
26
+ return sanitize_and_reencode_hash(value)
27
+ when Array
28
+ return sanitize_and_reencode_array(value)
29
+ when String
30
+ value = sanitize_and_reencode_string(value)
31
+ return value
32
+ end
33
+
34
+ # We only need to re-encode strings, for other value types (ints, nils,
35
+ # etc. no reencoding is needed)
36
+ value
37
+ end
38
+
39
+ def sanitize_and_reencode_array(array)
40
+ array.each_with_index do |value, index|
41
+ value = sanitize_and_reencode_value(value)
42
+ array[index] = value
43
+ end
44
+
45
+ array
46
+ end
47
+
48
+ def sanitize_and_reencode_hash(hash)
49
+ hash.each do |key, value|
50
+ hash[key] = sanitize_and_reencode_value(value)
51
+ end
52
+
53
+ hash
54
+ end
55
+
56
+ def sanitize_and_reencode_string(value)
57
+ # Function which sanitized the provided string value and tries to re-encode it as UTF-8
58
+ # ignoring any encoding error which could arise due to bad or partial unicode sequence
59
+ begin # rubocop:disable Style/RedundantBegin
60
+ value.encode("UTF-8", invalid: :replace, undef: :replace, replace: "<?>").force_encoding("UTF-8") # rubocop:disable Layout/LineLength, Lint/RedundantCopDisableDirective
61
+ rescue # rubocop:disable Style/RescueStandardError
62
+ "failed-to-reencode-as-utf8"
63
+ end
64
+ end
65
+ end
data/test/test_config.rb CHANGED
@@ -57,5 +57,10 @@ class ConfigTest < Scalyr::ScalyrOutTest
57
57
  d = create_driver CONFIG + 'server_attributes { "test":"value" }'
58
58
  assert_equal("value", d.instance.server_attributes["test"], "Config failed to set server_attributes")
59
59
  end
60
+
61
+ def test_configure_parser
62
+ d = create_driver CONFIG + "parser access_log"
63
+ assert_equal("access_log", d.instance.parser, "Config failed to set parser")
64
+ end
60
65
  end
61
66
  # rubocop:enable Layout/LineLength
data/test/test_events.rb CHANGED
@@ -136,6 +136,34 @@ class EventsTest < Scalyr::ScalyrOutTest
136
136
  assert_equal(mock_called, true, "mock method was never called!")
137
137
  end
138
138
 
139
+ def test_build_add_events_body_with_parser
140
+ d = create_driver CONFIG + "parser test_parser"
141
+
142
+ time = event_time("2015-04-01 10:00:00 UTC")
143
+ attrs = {"a" => 1}
144
+
145
+ response = flexmock(Net::HTTPResponse, code: "200", body: '{ "status":"success" }')
146
+ mock = flexmock(d.instance)
147
+
148
+ mock_called = false
149
+
150
+ mock.should_receive(:post_request).with(
151
+ URI,
152
+ on {|request_body|
153
+ body = JSON.parse(request_body)
154
+ assert_equal("test_parser", body["events"][0]["attrs"]["parser"])
155
+ mock_called = true
156
+ true
157
+ }
158
+ ).once.and_return(response)
159
+
160
+ d.run(default_tag: "test") do
161
+ d.feed(time, attrs)
162
+ end
163
+
164
+ assert_equal(mock_called, true, "mock method was never called!")
165
+ end
166
+
139
167
  def test_build_add_events_body_incrementing_timestamps
140
168
  d = create_driver
141
169
 
@@ -170,6 +198,88 @@ class EventsTest < Scalyr::ScalyrOutTest
170
198
  assert_equal(mock_called, true, "mock method was never called!")
171
199
  end
172
200
 
201
+ def test_build_add_events_body_non_json_serializable_value
202
+ d = create_driver
203
+
204
+ time = event_time("2015-04-01 10:00:00 UTC")
205
+ attrs = {"a" => 1}
206
+ attrs["int1"] = 1_601_923_119
207
+ attrs["int2"] = Integer(1_601_923_119)
208
+ attrs["int3"] = Integer(9_223_372_036_854_775_807)
209
+ attrs["int4"] = Integer(-1)
210
+ attrs["nil"] = nil
211
+ attrs["array"] = [1, 2, "a", "b", nil]
212
+ attrs["hash"] = {
213
+ "a" => "1",
214
+ "b" => "c"
215
+ }
216
+ attrs["logfile"] = "/some/log/file"
217
+
218
+ # This partial unicode sequence will fail encoding so we make sure it doesn't break the plugin
219
+ # and we correctly cast it to a value which we can send to the API
220
+ attrs["partial_unicode_sequence"] = "\xC2"
221
+ attrs["array_with_partial_unicode_sequence"] = [1, 2, "a", "b", nil, "7", "\xC2"]
222
+ attrs["nested_array_with_partial_unicode_sequence"] = [1, 2, "a", "b", nil, "7",
223
+ [8, 9, [10, "\xC2"]],
224
+ {"a" => 1, "b" => "\xC2"}]
225
+ attrs["hash_with_partial_unicode_sequence"] = {
226
+ "a" => "1",
227
+ "b" => "\xC2",
228
+ "c" => nil
229
+ }
230
+ attrs["nested_hash_with_partial_unicode_sequence"] = {
231
+ "a" => "1",
232
+ "b" => {
233
+ "c" => "\xC2",
234
+ "d" => "e",
235
+ "f" => nil,
236
+ "g" => {
237
+ "h" => "\xC2",
238
+ "b" => 3
239
+ }
240
+ }
241
+ }
242
+
243
+ response = flexmock(Net::HTTPResponse, code: "200", body: '{ "status":"success" }')
244
+ mock = flexmock(d.instance)
245
+
246
+ mock_called = false
247
+
248
+ # NOTE: We need to perform a deep clone / copy
249
+ expected_attrs = Marshal.load(Marshal.dump(attrs))
250
+
251
+ expected_attrs["partial_unicode_sequence"] = "<?>"
252
+ expected_attrs["array_with_partial_unicode_sequence"][-1] = "<?>"
253
+ expected_attrs["nested_array_with_partial_unicode_sequence"][-2][-1][-1] = "<?>"
254
+ expected_attrs["nested_array_with_partial_unicode_sequence"][-1]["b"] = "<?>"
255
+ expected_attrs["hash_with_partial_unicode_sequence"]["b"] = "<?>"
256
+ expected_attrs["nested_hash_with_partial_unicode_sequence"]["b"]["c"] = "<?>"
257
+ expected_attrs["nested_hash_with_partial_unicode_sequence"]["b"]["g"]["h"] = "<?>"
258
+
259
+ # Verify that clone / copy was correct and the original object wasn't modified
260
+ assert_not_equal(expected_attrs, attrs, "Objects are the same but should be different")
261
+ assert_not_equal(Marshal.load(Marshal.dump(attrs)), Marshal.load(Marshal.dump(expected_attrs)))
262
+ assert_equal(attrs["partial_unicode_sequence"], "\xC2")
263
+ assert_equal(attrs["array_with_partial_unicode_sequence"][-1], "\xC2")
264
+ assert_equal(attrs["nested_hash_with_partial_unicode_sequence"]["b"]["g"]["h"], "\xC2")
265
+
266
+ mock.should_receive(:post_request).with(
267
+ URI,
268
+ on {|request_body|
269
+ body = JSON.parse(request_body)
270
+ assert_equal(expected_attrs, body["events"][0]["attrs"], "Value of attrs differs from log")
271
+ mock_called = true
272
+ true
273
+ }
274
+ ).once.and_return(response)
275
+
276
+ d.run(default_tag: "test") do
277
+ d.feed(time, attrs)
278
+ end
279
+
280
+ assert_equal(mock_called, true, "mock method was never called!")
281
+ end
282
+
173
283
  def test_default_message_field
174
284
  d = create_driver CONFIG
175
285
 
@@ -247,4 +357,48 @@ class EventsTest < Scalyr::ScalyrOutTest
247
357
  d.feed(time, attrs)
248
358
  end
249
359
  end
360
+
361
+ def test_truncated_large_event
362
+ d = create_driver CONFIG + "max_request_buffer 4000"
363
+
364
+ time = event_time("2015-04-01 10:00:00 UTC")
365
+ attrs = {"log" => "this is a test", "message" => "0123456789" * 500}
366
+
367
+ response = flexmock(Net::HTTPResponse, code: "200", body: '{ "status":"success" }')
368
+ mock = flexmock(d.instance)
369
+
370
+ mock.should_receive(:post_request).with(
371
+ URI,
372
+ on {|request_body|
373
+ body = JSON.parse(request_body)
374
+ events = body["events"]
375
+ assert(events[0]["attrs"].key?("message"), "'message' field not found in event")
376
+ assert_equal(
377
+ "0123456789" * 388 + "012...",
378
+ events[0]["attrs"]["message"],
379
+ "'message' field incorrect"
380
+ )
381
+ true
382
+ }
383
+ ).once.and_return(response)
384
+
385
+ d.run(default_tag: "test") do
386
+ d.feed(time, attrs)
387
+ end
388
+ end
389
+
390
+ def test_dropped_large_event
391
+ d = create_driver CONFIG + "max_request_buffer 4000"
392
+
393
+ time = event_time("2015-04-01 10:00:00 UTC")
394
+ attrs = {"message" => "this is a test", "not_message" => "0123456789" * 500}
395
+
396
+ mock = flexmock(d.instance)
397
+
398
+ mock.should_receive(:post_request).never
399
+
400
+ d.run(default_tag: "test") do
401
+ d.feed(time, attrs)
402
+ end
403
+ end
250
404
  end
@@ -0,0 +1,100 @@
1
+ # frozen_string_literal: true
2
+
3
+ #
4
+ # Scalyr Output Plugin for Fluentd
5
+ #
6
+ # Copyright (C) 2015 Scalyr, Inc.
7
+ #
8
+ # Licensed under the Apache License, Version 2.0 (the "License");
9
+ # you may not use this file except in compliance with the License.
10
+ # You may obtain a copy of the License at
11
+ #
12
+ # http://www.apache.org/licenses/LICENSE-2.0
13
+ #
14
+ # Unless required by applicable law or agreed to in writing, software
15
+ # distributed under the License is distributed on an "AS IS" BASIS,
16
+ # WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
17
+ # See the License for the specific language governing permissions and
18
+ # limitations under the License.
19
+
20
+ require "helper"
21
+ require "flexmock/test_unit"
22
+
23
+ require "fluent/plugin/scalyr_utils"
24
+ include Scalyr # rubocop:disable Style/MixinUsage
25
+
26
+ module Scalyr
27
+ class UtilsTest < Test::Unit::TestCase
28
+ def test_sanitize_and_reencode_value_simple_types
29
+ # Simple value - string
30
+ result = sanitize_and_reencode_value("test foo")
31
+ assert_equal("test foo", result)
32
+
33
+ # Simple value - string with bad unicode sequences
34
+ result = sanitize_and_reencode_value("test \xC2 foo \xC2 bar")
35
+ assert_equal("test <?> foo <?> bar", result)
36
+
37
+ # Simple value - int
38
+ result = sanitize_and_reencode_value(100)
39
+ assert_equal(100, result)
40
+
41
+ # Simple value - nill
42
+ result = sanitize_and_reencode_value(nil)
43
+ assert_equal(nil, result)
44
+
45
+ # Simple value - bool
46
+ result = sanitize_and_reencode_value(true)
47
+ assert_equal(true, result)
48
+
49
+ result = sanitize_and_reencode_value(false)
50
+ assert_equal(false, result)
51
+ end
52
+
53
+ def test_sanitize_and_reencode_value_complex_nested_types
54
+ actual = [1, 2, "a", "b", nil, "7", "\xC2"]
55
+ expected = [1, 2, "a", "b", nil, "7", "<?>"]
56
+
57
+ result = sanitize_and_reencode_value(actual)
58
+ assert_equal(expected, result)
59
+
60
+ actual = [1, 2, "a", "b", nil, "7",
61
+ [8, 9, [10, "\xC2"]],
62
+ {"a" => 1, "b" => "\xC2"}]
63
+ expected = [1, 2, "a", "b", nil, "7",
64
+ [8, 9, [10, "<?>"]],
65
+ {"a" => 1, "b" => "<?>"}]
66
+
67
+ result = sanitize_and_reencode_value(actual)
68
+ assert_equal(expected, result)
69
+
70
+ actual = {
71
+ "a" => "1",
72
+ "b" => {
73
+ "c" => "\xC2",
74
+ "d" => "e",
75
+ "f" => nil,
76
+ "g" => {
77
+ "h" => "bar \xC2",
78
+ "b" => 3,
79
+ "l" => [1, 2, "foo\xC2", 3, 4, 5]
80
+ }
81
+ }
82
+ }
83
+ expected = {
84
+ "a" => "1",
85
+ "b" => {
86
+ "c" => "<?>",
87
+ "d" => "e",
88
+ "f" => nil,
89
+ "g" => {
90
+ "h" => "bar <?>",
91
+ "b" => 3,
92
+ "l" => [1, 2, "foo<?>", 3, 4, 5]
93
+ }
94
+ }
95
+ }
96
+ result = sanitize_and_reencode_value(actual)
97
+ assert_equal(expected, result)
98
+ end
99
+ end
100
+ end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: fluent-plugin-scalyr
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.8.11
4
+ version: 0.8.16
5
5
  platform: ruby
6
6
  authors:
7
7
  - Imron Alston
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2020-08-19 00:00:00.000000000 Z
11
+ date: 2022-02-14 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: ffi
@@ -158,11 +158,13 @@ files:
158
158
  - fluent.conf.sample
159
159
  - lib/fluent/plugin/out_scalyr.rb
160
160
  - lib/fluent/plugin/scalyr_exceptions.rb
161
+ - lib/fluent/plugin/scalyr_utils.rb
161
162
  - test/helper.rb
162
163
  - test/test_config.rb
163
164
  - test/test_events.rb
164
165
  - test/test_handle_response.rb
165
166
  - test/test_ssl_verify.rb
167
+ - test/test_utils.rb
166
168
  homepage: https://github.com/scalyr/scalyr-fluentd
167
169
  licenses:
168
170
  - Apache-2.0
@@ -188,7 +190,8 @@ specification_version: 4
188
190
  summary: Scalyr plugin for fluentd
189
191
  test_files:
190
192
  - test/helper.rb
193
+ - test/test_config.rb
191
194
  - test/test_events.rb
192
195
  - test/test_handle_response.rb
193
- - test/test_config.rb
194
196
  - test/test_ssl_verify.rb
197
+ - test/test_utils.rb