fluent-plugin-datadog 0.10.5 → 0.13.0

Sign up to get free protection for your applications and to get access to all the features.
Files changed (4) hide show
  1. checksums.yaml +5 -5
  2. data/README.md +66 -9
  3. data/fluent-plugin-datadog.gemspec +9 -4
  4. metadata +72 -11
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
- SHA1:
3
- metadata.gz: 64fbcf3f2a58f4ae91945601d802bfcbbe27a411
4
- data.tar.gz: a80b933ca941bdcaaf15f64f0ff2d4900e7cf967
2
+ SHA256:
3
+ metadata.gz: c796cd3bf58664302c3522ede3ed1518c268a1a557cd00244ceef41dcb97b8a7
4
+ data.tar.gz: 00c4cd64362d94ba967c9239f15cfa830f7e8533345f379433709630754ae2cf
5
5
  SHA512:
6
- metadata.gz: 97682ce0db7b87f629be382e95d9489aec3df163a014f7b74f87642d6741a01cbc5f6ee44d321ea8917023b2499bd0cd958e1dee31a2aa0f0d3502b72d023bae
7
- data.tar.gz: 7ede13e8ae2b91ae16cef2ba7a0a2323acb432ecc69dab99d2d08e0a1884fe68c7fbce426fa3209deff25aaa895a21354912eef3d4141feaf86eb7efd394763a
6
+ metadata.gz: 58ba2c958571d7756c8cbc18c610a324195a2c0f96e1cc4c62c79dee8438e8f6f16df2f77c50169f84d541e6db39530e47ccd67d8c43b9f2e8c590b5d6c0c360
7
+ data.tar.gz: a92d2e8972aa49c15609cefcc5b70c75cf8f6a674734935c4795b7d393d3e5e63c669ae3d2503714ad94860e15dd81851009d250e16ed56d9d3c901b44d8372b
data/README.md CHANGED
@@ -1,11 +1,15 @@
1
1
  # Fluentd output plugin for Datadog
2
2
 
3
- It mainly contains a proper JSON formatter and a socket handler that
4
- streams logs directly to Datadog - so no need to use a log shipper
3
+ This output plugin allows sending logs directly from Fluentd to Datadog - so you don't have to use a separate log shipper
5
4
  if you don't wan't to.
6
5
 
7
6
  ## Pre-requirements
8
7
 
8
+ | fluent-plugin-datadog | Fluentd | Ruby |
9
+ |:--------------------------|:-----------|:-------|
10
+ | \>= 0.12.0 | \>= v1 | \>= 2.4 |
11
+ | < 0.12.0 | \>= v0.12.0 | \>= 2.1 |
12
+
9
13
  To add the plugin to your fluentd agent, use the following command:
10
14
 
11
15
  gem install fluent-plugin-datadog
@@ -19,7 +23,7 @@ If you installed the td-agent instead
19
23
 
20
24
  To match events and send them to Datadog, simply add the following code to your configuration file.
21
25
 
22
- TCP example:
26
+ HTTP example:
23
27
 
24
28
  ```xml
25
29
  # Match events tagged with "datadog.**" and
@@ -35,10 +39,21 @@ TCP example:
35
39
  tag_key 'tag'
36
40
 
37
41
  # Optional parameters
38
- dd_source '<INTEGRATION_NAME>'
39
- dd_tags '<KEY1:VALU1>,<KEY2:VALUE2>'
42
+ dd_source '<INTEGRATION_NAME>'
43
+ dd_tags '<KEY1:VALUE1>,<KEY2:VALUE2>'
40
44
  dd_sourcecategory '<MY_SOURCE_CATEGORY>'
41
45
 
46
+ # Optional http proxy
47
+ http_proxy 'http://my-proxy.example'
48
+
49
+ <buffer>
50
+ @type memory
51
+ flush_thread_count 4
52
+ flush_interval 3s
53
+ chunk_limit_size 5m
54
+ chunk_limit_records 500
55
+ </buffer>
56
+
42
57
  </match>
43
58
  ```
44
59
 
@@ -68,14 +83,25 @@ As fluent-plugin-datadog is an output_buffer, you can set all output_buffer prop
68
83
  |-------------|--------------------------------------------------------------------------|----------------|
69
84
  | **api_key** | This parameter is required in order to authenticate your fluent agent. | nil |
70
85
  | **use_json** | Event format, if true, the event is sent in json format. Othwerwise, in plain text. | true |
71
- | **include_tag_key** | Automatically include tags in the record. | false |
72
- | **tag_key** | Name of the tag attribute, if they are included. | "tag" |
86
+ | **include_tag_key** | Automatically include the Fluentd tag in the record. | false |
87
+ | **tag_key** | Where to store the Fluentd tag. | "tag" |
88
+ | **timestamp_key** | Name of the attribute which will contain timestamp of the log event. If nil, timestamp attribute is not added. | "@timestamp" |
73
89
  | **use_ssl** | If true, the agent initializes a secure connection to Datadog. In clear TCP otherwise. | true |
90
+ | **no_ssl_validation** | Disable SSL validation (useful for proxy forwarding) | false |
91
+ | **ssl_port** | Port used to send logs over a SSL encrypted connection to Datadog. If use_http is disabled, use 10516 for the US region and 443 for the EU region. | 443 |
74
92
  | **max_retries** | The number of retries before the output plugin stops. Set to -1 for unlimited retries | -1 |
75
- | **max_retries** | The number of retries before the output plugin stops. Set to -1 for unlimited retries | -1 |
93
+ | **max_backoff** | The maximum time waited between each retry in seconds | 30 |
94
+ | **use_http** | Enable HTTP forwarding. If you disable it, make sure to change the port to 10514 or ssl_port to 10516 | true |
95
+ | **use_compression** | Enable log compression for HTTP | true |
96
+ | **compression_level** | Set the log compression level for HTTP (1 to 9, 9 being the best ratio) | 6 |
76
97
  | **dd_source** | This tells Datadog what integration it is | nil |
77
- | **dd_sourcecategory** | Multiple value attribute. Can be used to refine the source attribtue | nil |
98
+ | **dd_sourcecategory** | Multiple value attribute. Can be used to refine the source attribute | nil |
78
99
  | **dd_tags** | Custom tags with the following format "key1:value1, key2:value2" | nil |
100
+ | **dd_hostname** | Used by Datadog to identify the host submitting the logs. | `hostname -f` |
101
+ | **service** | Used by Datadog to correlate between logs, traces and metrics. | nil |
102
+ | **port** | Proxy port when logs are not directly forwarded to Datadog and ssl is not used | 80 |
103
+ | **host** | Proxy endpoint when logs are not directly forwarded to Datadog | http-intake.logs.datadoghq.com |
104
+ | **http_proxy** | HTTP proxy, only takes effect if HTTP forwarding is enabled (`use_http`). Defaults to `HTTP_PROXY`/`http_proxy` env vars. | nil |
79
105
 
80
106
  ### Docker and Kubernetes tags
81
107
 
@@ -99,3 +125,34 @@ Configuration example:
99
125
  type kubernetes_metadata
100
126
  </filter>
101
127
  ```
128
+
129
+ ### Encoding
130
+
131
+ Datadog's API expects log messages to be encoded in UTF-8.
132
+ If some of your logs are encoded with a different encoding, we recommend using the [`record_modifier` filter plugin](https://github.com/repeatedly/fluent-plugin-record-modifier#char_encoding)
133
+ to encode these logs to UTF-8.
134
+
135
+ Configuration example:
136
+
137
+ ```
138
+ # Change encoding of logs tagged with "datadog.**"
139
+ <filter datadog.**>
140
+ @type record_modifier
141
+
142
+ # change the encoding from the '<SOURCE_ENCODING>' of your logs to 'utf-8'
143
+ char_encoding <SOURCE_ENCODING>:utf-8
144
+ </filter>
145
+ ```
146
+
147
+ ## Build
148
+
149
+ To build a new version of this plugin and push it to RubyGems:
150
+
151
+ - Update the version in the .gemspec file accordingly
152
+ - `rake build` to build the gem file
153
+ - `rake release` to push the new gem to RubyGems
154
+
155
+ **Note**: The latest command will fail without appropriate credentials configured. You can set those credentials by running the following command:
156
+
157
+ `curl -u <USERNAME> https://rubygems.org/api/v1/api_key.yaml > ~/.gem/credentials`, it will ask for your password.
158
+
data/fluent-plugin-datadog.gemspec CHANGED
@@ -9,19 +9,24 @@ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
9
9
 
10
10
  Gem::Specification.new do |spec|
11
11
  spec.name = "fluent-plugin-datadog"
12
- spec.version = "0.10.5"
12
+ spec.version = "0.13.0"
13
13
  spec.authors = ["Datadog Solutions Team"]
14
14
  spec.email = ["support@datadoghq.com"]
15
15
  spec.summary = "Datadog output plugin for Fluent event collector"
16
16
  spec.homepage = "http://datadoghq.com"
17
- spec.license = "Apache License 2.0"
17
+ spec.license = "Apache-2.0"
18
18
 
19
19
  spec.files = [".gitignore", "Gemfile", "LICENSE", "README.md", "Rakefile", "fluent-plugin-datadog.gemspec", "lib/fluent/plugin/out_datadog.rb"]
20
20
  spec.executables = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
21
21
  spec.test_files = spec.files.grep(%r{^(test|spec|features)/})
22
22
  spec.require_paths = ["lib"]
23
23
 
24
- spec.add_development_dependency "bundler", "~> 1.5"
25
- spec.add_development_dependency "rake"
24
+ spec.add_runtime_dependency "fluentd", [">= 1", "< 2"]
25
+ spec.add_runtime_dependency "net-http-persistent", '~> 3.1'
26
+
27
+ spec.add_development_dependency "bundler", "~> 2.1"
28
+ spec.add_development_dependency "test-unit", '~> 3.1'
29
+ spec.add_development_dependency "rake", "~> 12.0"
26
30
  spec.add_development_dependency "yajl-ruby", "~> 1.2"
31
+ spec.add_development_dependency 'webmock', "~> 3.5.0"
27
32
  end
metadata CHANGED
@@ -1,43 +1,91 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: fluent-plugin-datadog
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.10.5
4
+ version: 0.13.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Datadog Solutions Team
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2018-11-26 00:00:00.000000000 Z
11
+ date: 2021-03-22 00:00:00.000000000 Z
12
12
  dependencies:
13
+ - !ruby/object:Gem::Dependency
14
+ name: fluentd
15
+ requirement: !ruby/object:Gem::Requirement
16
+ requirements:
17
+ - - ">="
18
+ - !ruby/object:Gem::Version
19
+ version: '1'
20
+ - - "<"
21
+ - !ruby/object:Gem::Version
22
+ version: '2'
23
+ type: :runtime
24
+ prerelease: false
25
+ version_requirements: !ruby/object:Gem::Requirement
26
+ requirements:
27
+ - - ">="
28
+ - !ruby/object:Gem::Version
29
+ version: '1'
30
+ - - "<"
31
+ - !ruby/object:Gem::Version
32
+ version: '2'
33
+ - !ruby/object:Gem::Dependency
34
+ name: net-http-persistent
35
+ requirement: !ruby/object:Gem::Requirement
36
+ requirements:
37
+ - - "~>"
38
+ - !ruby/object:Gem::Version
39
+ version: '3.1'
40
+ type: :runtime
41
+ prerelease: false
42
+ version_requirements: !ruby/object:Gem::Requirement
43
+ requirements:
44
+ - - "~>"
45
+ - !ruby/object:Gem::Version
46
+ version: '3.1'
13
47
  - !ruby/object:Gem::Dependency
14
48
  name: bundler
15
49
  requirement: !ruby/object:Gem::Requirement
16
50
  requirements:
17
51
  - - "~>"
18
52
  - !ruby/object:Gem::Version
19
- version: '1.5'
53
+ version: '2.1'
54
+ type: :development
55
+ prerelease: false
56
+ version_requirements: !ruby/object:Gem::Requirement
57
+ requirements:
58
+ - - "~>"
59
+ - !ruby/object:Gem::Version
60
+ version: '2.1'
61
+ - !ruby/object:Gem::Dependency
62
+ name: test-unit
63
+ requirement: !ruby/object:Gem::Requirement
64
+ requirements:
65
+ - - "~>"
66
+ - !ruby/object:Gem::Version
67
+ version: '3.1'
20
68
  type: :development
21
69
  prerelease: false
22
70
  version_requirements: !ruby/object:Gem::Requirement
23
71
  requirements:
24
72
  - - "~>"
25
73
  - !ruby/object:Gem::Version
26
- version: '1.5'
74
+ version: '3.1'
27
75
  - !ruby/object:Gem::Dependency
28
76
  name: rake
29
77
  requirement: !ruby/object:Gem::Requirement
30
78
  requirements:
31
- - - ">="
79
+ - - "~>"
32
80
  - !ruby/object:Gem::Version
33
- version: '0'
81
+ version: '12.0'
34
82
  type: :development
35
83
  prerelease: false
36
84
  version_requirements: !ruby/object:Gem::Requirement
37
85
  requirements:
38
- - - ">="
86
+ - - "~>"
39
87
  - !ruby/object:Gem::Version
40
- version: '0'
88
+ version: '12.0'
41
89
  - !ruby/object:Gem::Dependency
42
90
  name: yajl-ruby
43
91
  requirement: !ruby/object:Gem::Requirement
@@ -52,6 +100,20 @@ dependencies:
52
100
  - - "~>"
53
101
  - !ruby/object:Gem::Version
54
102
  version: '1.2'
103
+ - !ruby/object:Gem::Dependency
104
+ name: webmock
105
+ requirement: !ruby/object:Gem::Requirement
106
+ requirements:
107
+ - - "~>"
108
+ - !ruby/object:Gem::Version
109
+ version: 3.5.0
110
+ type: :development
111
+ prerelease: false
112
+ version_requirements: !ruby/object:Gem::Requirement
113
+ requirements:
114
+ - - "~>"
115
+ - !ruby/object:Gem::Version
116
+ version: 3.5.0
55
117
  description:
56
118
  email:
57
119
  - support@datadoghq.com
@@ -68,7 +130,7 @@ files:
68
130
  - lib/fluent/plugin/out_datadog.rb
69
131
  homepage: http://datadoghq.com
70
132
  licenses:
71
- - Apache License 2.0
133
+ - Apache-2.0
72
134
  metadata: {}
73
135
  post_install_message:
74
136
  rdoc_options: []
@@ -85,8 +147,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
85
147
  - !ruby/object:Gem::Version
86
148
  version: '0'
87
149
  requirements: []
88
- rubyforge_project:
89
- rubygems_version: 2.5.2.3
150
+ rubygems_version: 3.0.3
90
151
  signing_key:
91
152
  specification_version: 4
92
153
  summary: Datadog output plugin for Fluent event collector