fluent-plugin-scalyr 0.8.12 → 0.8.14
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/README.md +29 -16
- data/VERSION +1 -1
- data/lib/fluent/plugin/out_scalyr.rb +4 -0
- data/test/test_config.rb +5 -0
- data/test/test_events.rb +28 -0
- metadata +4 -4
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: f9f9a9d2d7c0086b3885e3c3962fcd7271b84b79cbd1dd937cfb27f1d24e16d8
|
4
|
+
data.tar.gz: d99dd0b487a8f4920a2abc53b00fb2f75499647ed6cb75bcd0ffa9a0a0dd0871
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: b6374eb7cb7e4eb1fa3ba3c7342999f176743b07b3293ff3a34f87e127004db50502fb3989117873b68702600fdb6072f4fe39f1f78fef2d280d37a86fa04dc8
|
7
|
+
data.tar.gz: 8964f56c050fa025825ef3bb4c74d7ef593d32eee86930f9bb9847bedd5048a1851767cc1d3d9d67f064dc1b08622b479add48cab6f395519f08e692c2f57154
|
data/README.md
CHANGED
@@ -30,22 +30,6 @@ This can be done by specifying tags such as scalyr.apache, scalyr.maillog etc an
|
|
30
30
|
|
31
31
|
Fluentd tag names will be used for the logfile name in Scalyr.
|
32
32
|
|
33
|
-
Scalyr Parsers and Custom Fields
|
34
|
-
--------------------------------
|
35
|
-
|
36
|
-
You may also need to specify a Scalyr parser for your log message or add custom fields to each log event. This can be done using Fluentd's filter mechanism, in particular the [record_transformer filter](https://docs.fluentd.org/filter/record_transformer).
|
37
|
-
|
38
|
-
For example, if you want to use Scalyr's ```accessLog``` parser for all events with the ```scalyr.access``` tag you would add the following to your fluent.conf file:
|
39
|
-
|
40
|
-
```
|
41
|
-
<filter scalyr.access>
|
42
|
-
@type record_transformer
|
43
|
-
<record>
|
44
|
-
parser accessLog
|
45
|
-
</record>
|
46
|
-
</filter>
|
47
|
-
```
|
48
|
-
|
49
33
|
Plugin Configuration
|
50
34
|
-------------
|
51
35
|
|
@@ -80,6 +64,7 @@ The following configuration options are also supported:
|
|
80
64
|
ssl_verify_peer true
|
81
65
|
ssl_verify_depth 5
|
82
66
|
message_field message
|
67
|
+
parser nil
|
83
68
|
|
84
69
|
max_request_buffer 5500000
|
85
70
|
|
@@ -104,6 +89,32 @@ For some additional examples of configuration for different setups, please refer
|
|
104
89
|
[examples/configs/](https://github.com/scalyr/scalyr-fluentd/tree/master/examples/configs/)
|
105
90
|
directory.
|
106
91
|
|
92
|
+
Scalyr Parsers and Custom Fields
|
93
|
+
--------------------------------
|
94
|
+
|
95
|
+
You may also need to specify a Scalyr parser for your log message or add custom fields to each log event. This can be done using Fluentd's filter mechanism, in particular the [record_transformer filter](https://docs.fluentd.org/filter/record_transformer).
|
96
|
+
|
97
|
+
For example, if you want to use Scalyr's ```accessLog``` parser for all events with the ```scalyr.access``` tag you would add the following to your fluent.conf file:
|
98
|
+
|
99
|
+
```
|
100
|
+
<filter scalyr.access>
|
101
|
+
@type record_transformer
|
102
|
+
<record>
|
103
|
+
parser accessLog
|
104
|
+
</record>
|
105
|
+
</filter>
|
106
|
+
```
|
107
|
+
|
108
|
+
Alternatively, if the same parser can be used for all events sent by the Fluentd output plugin, the parser can be set using the ```parser``` plugin configuration. For example, to use Scalyr's ```accessLog``` parser for all events, use the output plugin configuration:
|
109
|
+
|
110
|
+
```
|
111
|
+
<match scalyr.*>
|
112
|
+
@type scalyr
|
113
|
+
api_write_token YOUR_SCALYR_WRITE_LOGS_TOKEN
|
114
|
+
parser accessLog
|
115
|
+
</match>
|
116
|
+
```
|
117
|
+
|
107
118
|
### Scalyr specific options
|
108
119
|
|
109
120
|
***compression_type*** - compress Scalyr traffic to reduce network traffic. Options are `bz2` and `deflate`. See [here](https://www.scalyr.com/help/scalyr-agent#compressing) for more details. This feature is optional.
|
@@ -134,6 +145,8 @@ The cURL project maintains CA certificate bundles automatically converted from m
|
|
134
145
|
|
135
146
|
***message_field*** - Scalyr expects all log events to have a 'message' field containing the contents of a log message. If your event has the log message stored in another field, you can specify the field name here, and the plugin will rename that field to 'message' before sending the data to Scalyr. **Note:** this will override any existing 'message' field if the log record contains both a 'message' field and the field specified by this config option.
|
136
147
|
|
148
|
+
***parser*** - The Scalyr parser to use to parse the 'message' field for each event. This value is optional and defaults to *nil*.
|
149
|
+
|
137
150
|
***max_request_buffer*** - The maximum size in bytes of each request to send to Scalyr. Defaults to 5,500,000 (5.5MB). Fluentd chunks that generate JSON requests larger than the max_request_buffer will be split in to multiple separate requests. **Note:** The maximum size the Scalyr servers accept for this value is 6MB and requests containing data larger than this will be rejected.
|
138
151
|
|
139
152
|
***force_message_encoding*** - Set a specific encoding for all your log messages (defaults to nil). If your log messages are not in UTF-8, this can cause problems when converting the message to JSON in order to send to the Scalyr server. You can avoid these problems by setting an encoding for your log messages so they can be correctly converted.
|
data/VERSION
CHANGED
@@ -1 +1 @@
|
|
1
|
-
0.8.
|
1
|
+
0.8.14
|
@@ -37,6 +37,7 @@ module Scalyr
|
|
37
37
|
|
38
38
|
config_param :api_write_token, :string
|
39
39
|
config_param :server_attributes, :hash, default: nil
|
40
|
+
config_param :parser, :string, default: nil # Set the "parser" field to this, per event.
|
40
41
|
config_param :use_hostname_for_serverhost, :bool, default: true
|
41
42
|
config_param :scalyr_server, :string, default: "https://agent.scalyr.com/"
|
42
43
|
config_param :ssl_ca_bundle_path, :string, default: nil
|
@@ -322,6 +323,9 @@ module Scalyr
|
|
322
323
|
# add a logfile field if one doesn't exist
|
323
324
|
record["logfile"] = "/fluentd/#{tag}" unless record.key? "logfile"
|
324
325
|
|
326
|
+
# set per-event parser if it is configured
|
327
|
+
record["parser"] = @parser unless @parser.nil?
|
328
|
+
|
325
329
|
# append to list of events
|
326
330
|
event = {thread: thread_id.to_s,
|
327
331
|
ts: timestamp,
|
data/test/test_config.rb
CHANGED
@@ -57,5 +57,10 @@ class ConfigTest < Scalyr::ScalyrOutTest
|
|
57
57
|
d = create_driver CONFIG + 'server_attributes { "test":"value" }'
|
58
58
|
assert_equal("value", d.instance.server_attributes["test"], "Config failed to set server_attributes")
|
59
59
|
end
|
60
|
+
|
61
|
+
def test_configure_parser
|
62
|
+
d = create_driver CONFIG + "parser access_log"
|
63
|
+
assert_equal("access_log", d.instance.parser, "Config failed to set parser")
|
64
|
+
end
|
60
65
|
end
|
61
66
|
# rubocop:enable Layout/LineLength
|
data/test/test_events.rb
CHANGED
@@ -136,6 +136,34 @@ class EventsTest < Scalyr::ScalyrOutTest
|
|
136
136
|
assert_equal(mock_called, true, "mock method was never called!")
|
137
137
|
end
|
138
138
|
|
139
|
+
def test_build_add_events_body_with_parser
|
140
|
+
d = create_driver CONFIG + "parser test_parser"
|
141
|
+
|
142
|
+
time = event_time("2015-04-01 10:00:00 UTC")
|
143
|
+
attrs = {"a" => 1}
|
144
|
+
|
145
|
+
response = flexmock(Net::HTTPResponse, code: "200", body: '{ "status":"success" }')
|
146
|
+
mock = flexmock(d.instance)
|
147
|
+
|
148
|
+
mock_called = false
|
149
|
+
|
150
|
+
mock.should_receive(:post_request).with(
|
151
|
+
URI,
|
152
|
+
on {|request_body|
|
153
|
+
body = JSON.parse(request_body)
|
154
|
+
assert_equal("test_parser", body["events"][0]["attrs"]["parser"])
|
155
|
+
mock_called = true
|
156
|
+
true
|
157
|
+
}
|
158
|
+
).once.and_return(response)
|
159
|
+
|
160
|
+
d.run(default_tag: "test") do
|
161
|
+
d.feed(time, attrs)
|
162
|
+
end
|
163
|
+
|
164
|
+
assert_equal(mock_called, true, "mock method was never called!")
|
165
|
+
end
|
166
|
+
|
139
167
|
def test_build_add_events_body_incrementing_timestamps
|
140
168
|
d = create_driver
|
141
169
|
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: fluent-plugin-scalyr
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.8.
|
4
|
+
version: 0.8.14
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Imron Alston
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date:
|
11
|
+
date: 2021-10-28 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: ffi
|
@@ -190,8 +190,8 @@ specification_version: 4
|
|
190
190
|
summary: Scalyr plugin for fluentd
|
191
191
|
test_files:
|
192
192
|
- test/helper.rb
|
193
|
+
- test/test_config.rb
|
193
194
|
- test/test_events.rb
|
194
195
|
- test/test_handle_response.rb
|
195
|
-
- test/test_config.rb
|
196
|
-
- test/test_utils.rb
|
197
196
|
- test/test_ssl_verify.rb
|
197
|
+
- test/test_utils.rb
|