fluent-plugin-scalyr 0.8.9 → 0.8.14
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +5 -5
- data/Gemfile +2 -0
- data/README.md +59 -22
- data/Rakefile +10 -6
- data/VERSION +1 -1
- data/fluent-plugin-scalyr.gemspec +11 -8
- data/fluent.conf.sample +1 -1
- data/lib/fluent/plugin/out_scalyr.rb +202 -237
- data/lib/fluent/plugin/{scalyr-exceptions.rb → scalyr_exceptions.rb} +2 -2
- data/lib/fluent/plugin/scalyr_utils.rb +65 -0
- data/test/helper.rb +12 -6
- data/test/test_config.rb +24 -30
- data/test/test_events.rb +254 -142
- data/test/test_handle_response.rb +34 -35
- data/test/test_ssl_verify.rb +101 -10
- data/test/test_utils.rb +100 -0
- metadata +50 -34
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
|
-
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
2
|
+
SHA256:
|
3
|
+
metadata.gz: f9f9a9d2d7c0086b3885e3c3962fcd7271b84b79cbd1dd937cfb27f1d24e16d8
|
4
|
+
data.tar.gz: d99dd0b487a8f4920a2abc53b00fb2f75499647ed6cb75bcd0ffa9a0a0dd0871
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: b6374eb7cb7e4eb1fa3ba3c7342999f176743b07b3293ff3a34f87e127004db50502fb3989117873b68702600fdb6072f4fe39f1f78fef2d280d37a86fa04dc8
|
7
|
+
data.tar.gz: 8964f56c050fa025825ef3bb4c74d7ef593d32eee86930f9bb9847bedd5048a1851767cc1d3d9d67f064dc1b08622b479add48cab6f395519f08e692c2f57154
|
data/Gemfile
CHANGED
data/README.md
CHANGED
@@ -1,5 +1,5 @@
|
|
1
1
|
Scalyr output plugin for Fluentd
|
2
|
-
|
2
|
+
================================
|
3
3
|
|
4
4
|
**Note:** Fluentd introduced breaking changes to their plugin API between
|
5
5
|
version 0.12 and 0.14.
|
@@ -24,28 +24,12 @@ Fluentd may format log messages into json or some other format. If you want to
|
|
24
24
|
format none
|
25
25
|
```
|
26
26
|
|
27
|
-
The Scalyr output plugin assigns a unique Scalyr session id for each Fluentd <match> block. It is recommended that a single machine doesn't create too many simultaneous Scalyr sessions, so if possible you should try to have a single match for all logs you wish to send to Scalyr.
|
27
|
+
The Scalyr output plugin assigns a unique Scalyr session id for each Fluentd <match> block, or for each worker. It is recommended that a single machine doesn't create too many simultaneous Scalyr sessions, so if possible you should try to have a single match for all logs you wish to send to Scalyr.
|
28
28
|
|
29
29
|
This can be done by specifying tags such as scalyr.apache, scalyr.maillog etc and matching on scalyr.\*
|
30
30
|
|
31
31
|
Fluentd tag names will be used for the logfile name in Scalyr.
|
32
32
|
|
33
|
-
Scalyr Parsers and Custom Fields
|
34
|
-
--------------------------------
|
35
|
-
|
36
|
-
You may also need to specify a Scalyr parser for your log message or add custom fields to each log event. This can be done using Fluentd's filter mechanism, in particular the [record_transformer filter](https://docs.fluentd.org/filter/record_transformer).
|
37
|
-
|
38
|
-
For example, if you want to use Scalyr's ```accessLog``` parser for all events with the ```scalyr.access``` tag you would add the following to your fluent.conf file:
|
39
|
-
|
40
|
-
```
|
41
|
-
<filter scalyr.access>
|
42
|
-
@type record_transformer
|
43
|
-
<record>
|
44
|
-
parser accessLog
|
45
|
-
</record>
|
46
|
-
</filter>
|
47
|
-
```
|
48
|
-
|
49
33
|
Plugin Configuration
|
50
34
|
-------------
|
51
35
|
|
@@ -66,7 +50,8 @@ The following configuration options are also supported:
|
|
66
50
|
|
67
51
|
#scalyr specific options
|
68
52
|
api_write_token YOUR_SCALYR_WRITE_TOKEN
|
69
|
-
compression_type
|
53
|
+
compression_type deflate
|
54
|
+
compression_level 6
|
70
55
|
use_hostname_for_serverhost true
|
71
56
|
server_attributes {
|
72
57
|
"serverHost": "front-1",
|
@@ -79,8 +64,9 @@ The following configuration options are also supported:
|
|
79
64
|
ssl_verify_peer true
|
80
65
|
ssl_verify_depth 5
|
81
66
|
message_field message
|
67
|
+
parser nil
|
82
68
|
|
83
|
-
max_request_buffer
|
69
|
+
max_request_buffer 5500000
|
84
70
|
|
85
71
|
force_message_encoding nil
|
86
72
|
replace_invalid_utf8 false
|
@@ -99,6 +85,36 @@ The following configuration options are also supported:
|
|
99
85
|
</match>
|
100
86
|
```
|
101
87
|
|
88
|
+
For some additional examples of configuration for different setups, please refer to the
|
89
|
+
[examples/configs/](https://github.com/scalyr/scalyr-fluentd/tree/master/examples/configs/)
|
90
|
+
directory.
|
91
|
+
|
92
|
+
Scalyr Parsers and Custom Fields
|
93
|
+
--------------------------------
|
94
|
+
|
95
|
+
You may also need to specify a Scalyr parser for your log message or add custom fields to each log event. This can be done using Fluentd's filter mechanism, in particular the [record_transformer filter](https://docs.fluentd.org/filter/record_transformer).
|
96
|
+
|
97
|
+
For example, if you want to use Scalyr's ```accessLog``` parser for all events with the ```scalyr.access``` tag you would add the following to your fluent.conf file:
|
98
|
+
|
99
|
+
```
|
100
|
+
<filter scalyr.access>
|
101
|
+
@type record_transformer
|
102
|
+
<record>
|
103
|
+
parser accessLog
|
104
|
+
</record>
|
105
|
+
</filter>
|
106
|
+
```
|
107
|
+
|
108
|
+
Alternatively, if the same parser can be used for all events sent by the Fluentd output plugin, the parser can be set using the ```parser``` plugin configuration. For example, to use Scalyr's ```accessLog``` parser for all events, use the output plugin configuration:
|
109
|
+
|
110
|
+
```
|
111
|
+
<match scalyr.*>
|
112
|
+
@type scalyr
|
113
|
+
api_write_token YOUR_SCALYR_WRITE_LOGS_TOKEN
|
114
|
+
parser accessLog
|
115
|
+
</match>
|
116
|
+
```
|
117
|
+
|
102
118
|
### Scalyr specific options
|
103
119
|
|
104
120
|
***compression_type*** - compress Scalyr traffic to reduce network traffic. Options are `bz2` and `deflate`. See [here](https://www.scalyr.com/help/scalyr-agent#compressing) for more details. This feature is optional.
|
@@ -111,7 +127,7 @@ The following configuration options are also supported:
|
|
111
127
|
|
112
128
|
***scalyr_server*** - the Scalyr server to send API requests to. This value is optional and defaults to https://agent.scalyr.com/
|
113
129
|
|
114
|
-
***ssl_ca_bundle_path*** - a path on your server pointing to a valid certificate bundle. This value is optional and defaults to
|
130
|
+
***ssl_ca_bundle_path*** - a path on your server pointing to a valid certificate bundle. This value is optional and defaults to *nil*, which means it will look for a valid certificate bundle on its own.
|
115
131
|
|
116
132
|
**Note:** if the certificate bundle does not contain a certificate chain that verifies the Scalyr SSL certificate then all requests to Scalyr will fail unless ***ssl_verify_peer*** is set to false. If you suspect logging to Scalyr is failing due to an invalid certificate chain, you can grep through the Fluentd output for warnings that contain the message 'certificate verification failed'. The full text of such warnings will look something like this:
|
117
133
|
|
@@ -129,7 +145,9 @@ The cURL project maintains CA certificate bundles automatically converted from m
|
|
129
145
|
|
130
146
|
***message_field*** - Scalyr expects all log events to have a 'message' field containing the contents of a log message. If your event has the log message stored in another field, you can specify the field name here, and the plugin will rename that field to 'message' before sending the data to Scalyr. **Note:** this will override any existing 'message' field if the log record contains both a 'message' field and the field specified by this config option.
|
131
147
|
|
132
|
-
***
|
148
|
+
***parser*** - The Scalyr parser to use to parse the 'message' field for each event. This value is optional and defaults to *nil*.
|
149
|
+
|
150
|
+
***max_request_buffer*** - The maximum size in bytes of each request to send to Scalyr. Defaults to 5,500,000 (5.5MB). Fluentd chunks that generate JSON requests larger than the max_request_buffer will be split in to multiple separate requests. **Note:** The maximum size the Scalyr servers accept for this value is 6MB and requests containing data larger than this will be rejected.
|
133
151
|
|
134
152
|
***force_message_encoding*** - Set a specific encoding for all your log messages (defaults to nil). If your log messages are not in UTF-8, this can cause problems when converting the message to JSON in order to send to the Scalyr server. You can avoid these problems by setting an encoding for your log messages so they can be correctly converted.
|
135
153
|
|
@@ -175,3 +193,22 @@ Which builds the gem and puts it in the pkg directory, then install the Gem usin
|
|
175
193
|
```
|
176
194
|
fluent-gem install pkg/fluent-plugin-scalyr-<VERSION>.gem
|
177
195
|
```
|
196
|
+
|
197
|
+
Publishing a new release to RubyGems
|
198
|
+
------------------------------------
|
199
|
+
|
200
|
+
(for project maintainers)
|
201
|
+
|
202
|
+
To publish a new version to RubyGems, simply make your changes, make sure all the lint checks and
|
203
|
+
tests pass and merge your changes into master.
|
204
|
+
|
205
|
+
After that's done, bump a version in ``VERSION`` file, update ``CHANGELOG.md`` file, add a tag
|
206
|
+
which matches a version in VERSION file (e.g. ``v0.8.10``) and push that tag to the remote:
|
207
|
+
|
208
|
+
```bash
|
209
|
+
git tag v0.8.10
|
210
|
+
git push origin v0.8.10
|
211
|
+
```
|
212
|
+
|
213
|
+
Push of this tag will trigger a Circle CI job which will build the latest version of the gem and
|
214
|
+
publish it to RubyGems.
|
data/Rakefile
CHANGED
@@ -1,11 +1,15 @@
|
|
1
|
-
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
require "bundler"
|
2
4
|
Bundler::GemHelper.install_tasks
|
3
5
|
|
4
|
-
require
|
6
|
+
require "rake/testtask"
|
5
7
|
|
6
|
-
Rake::TestTask.new do |
|
7
|
-
|
8
|
-
|
8
|
+
Rake::TestTask.new(:test) do |test|
|
9
|
+
test.libs << "lib" << "test"
|
10
|
+
test.test_files = FileList["test/test_*.rb"]
|
11
|
+
test.verbose = true
|
12
|
+
test.options = "--verbose=verbose"
|
9
13
|
end
|
10
14
|
|
11
|
-
task :
|
15
|
+
task default: [:build]
|
data/VERSION
CHANGED
@@ -1 +1 @@
|
|
1
|
-
0.8.
|
1
|
+
0.8.14
|
@@ -1,4 +1,6 @@
|
|
1
|
-
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
$LOAD_PATH.push File.expand_path("lib", __dir__)
|
2
4
|
|
3
5
|
Gem::Specification.new do |gem|
|
4
6
|
gem.name = "fluent-plugin-scalyr"
|
@@ -9,18 +11,19 @@ Gem::Specification.new do |gem|
|
|
9
11
|
gem.authors = ["Imron Alston"]
|
10
12
|
gem.licenses = ["Apache-2.0"]
|
11
13
|
gem.email = "imron@scalyr.com"
|
12
|
-
gem.has_rdoc = false
|
13
14
|
gem.platform = Gem::Platform::RUBY
|
14
|
-
gem.files = Dir[
|
15
|
+
gem.files = Dir["AUTHORS", "Gemfile", "LICENSE", "README.md", "Rakefile", "VERSION",
|
16
|
+
"fluent-plugin-scalyr.gemspec", "fluent.conf.sample", "lib/**/*", "test/**/*"]
|
15
17
|
gem.test_files = Dir.glob("{test,spec,features}/**/*")
|
16
|
-
gem.executables = Dir.glob("bin/*").map{
|
17
|
-
gem.require_paths = [
|
18
|
-
gem.add_dependency "fluentd", [">= 0.14.0", "< 2"]
|
18
|
+
gem.executables = Dir.glob("bin/*").map {|f| File.basename(f) }
|
19
|
+
gem.require_paths = ["lib"]
|
19
20
|
gem.add_dependency "ffi", "1.9.25"
|
21
|
+
gem.add_dependency "fluentd", [">= 0.14.0", "< 2"]
|
20
22
|
gem.add_dependency "rbzip2", "0.3.0"
|
21
23
|
gem.add_dependency "zlib"
|
24
|
+
gem.add_development_dependency "bundler", "~> 1.9"
|
25
|
+
gem.add_development_dependency "flexmock", "~> 1.2"
|
22
26
|
gem.add_development_dependency "rake", "~> 0.9"
|
27
|
+
gem.add_development_dependency "rubocop", "~> 0.4"
|
23
28
|
gem.add_development_dependency "test-unit", "~> 3.0"
|
24
|
-
gem.add_development_dependency "flexmock", "~> 1.2"
|
25
|
-
gem.add_development_dependency "bundler", "~> 1.9"
|
26
29
|
end
|