fluent-plugin-newrelic 1.1.5 → 1.1.8
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +4 -4
- data/DEVELOPER.md +46 -0
- data/README.md +9 -0
- data/examples/Dockerfile +5 -0
- data/examples/readme.md +71 -0
- data/examples/syslog/fluentd.config +47 -0
- data/lib/fluent/plugin/out_newrelic.rb +3 -0
- data/lib/newrelic-fluentd-output/version.rb +1 -1
- metadata +7 -3
checksums.yaml
CHANGED
@@ -1,7 +1,7 @@
|
|
1
1
|
---
|
2
2
|
SHA256:
|
3
|
-
metadata.gz:
|
4
|
-
data.tar.gz:
|
3
|
+
metadata.gz: caf9616fa22e2338c4df33d70571d6a18850a17dafe98bb78c1e916aa7339ab5
|
4
|
+
data.tar.gz: a23147d65d03ac2de83765856f3ee84cc2afe26fbe24bc8c7e3c8f028316fb37
|
5
5
|
SHA512:
|
6
|
-
metadata.gz:
|
7
|
-
data.tar.gz:
|
6
|
+
metadata.gz: 991cab2d7e3060134f2f31784077d6e409c505a878a2d44a9eb1bf1d6ec8bc79752e3840881b0e7e0cddbb8547d9067bb153f76c650cded1d0c330eb9fc184bf
|
7
|
+
data.tar.gz: 716cb1b8150874ecb42c09bf678baa339d0f7bc65e4ebeb30badd9c31a9258d6a60d31d6376649bbbc64329bdcfc95e3d45fbb9b1bc022ea657940c3c4be9106
|
data/DEVELOPER.md
ADDED
@@ -0,0 +1,46 @@
|
|
1
|
+
# Developing the plugin
|
2
|
+
|
3
|
+
## Developing
|
4
|
+
|
5
|
+
* Install Bundler gem: `gem install bundler`
|
6
|
+
* Install dependencies: `bundle install`
|
7
|
+
* Write tests and production code!
|
8
|
+
* Bump version: edit version file `version.rb`
|
9
|
+
* Run tests: `bundle exec rspec`
|
10
|
+
* Build the gem: `gem build newrelic-fluentd-output.gemspec`
|
11
|
+
|
12
|
+
**NOTE**: Be mindful that if you are using 'match **', that using `log.info` in the plugin can cause an unintended
|
13
|
+
Sorcerer's Apprentice Syndrome issue where exponentially larger copies of log messages are sent until the
|
14
|
+
td-agent is unloaded. To prevent this, use match tags specific to your source (so use `<match tag_from_your_source>`
|
15
|
+
instead of `<match **>`), so that your output plugin does not also pick up things that Fluentd logs.
|
16
|
+
|
17
|
+
## Testing on MacOS
|
18
|
+
|
19
|
+
### Install Fluentd
|
20
|
+
* `brew cask install td-agent`
|
21
|
+
|
22
|
+
### Configure Fluentd
|
23
|
+
* `sudo vi /etc/td-agent/td-agent.conf`
|
24
|
+
* Add the following:
|
25
|
+
```
|
26
|
+
<source>
|
27
|
+
@type tail
|
28
|
+
format none
|
29
|
+
path /usr/local/var/log/test.log
|
30
|
+
tag test
|
31
|
+
</source>
|
32
|
+
|
33
|
+
<match test>
|
34
|
+
@type newrelic
|
35
|
+
api_key (your-api-key)
|
36
|
+
</match>
|
37
|
+
```
|
38
|
+
|
39
|
+
### Testing plugin
|
40
|
+
* Stop Fluentd: `sudo launchctl unload /Library/LaunchDaemons/td-agent.plist`
|
41
|
+
* Remove previous version: `sudo /opt/td-agent/usr/sbin/td-agent-gem uninstall fluent-plugin-newrelic`
|
42
|
+
* Add new version: `sudo /opt/td-agent/usr/sbin/td-agent-gem install fluent-plugin-newrelic-<version>.gem`
|
43
|
+
* Start Fluentd: `sudo launchctl load /Library/LaunchDaemons/td-agent.plist`
|
44
|
+
* Make sure things start up OK: `tail -f /var/log/td-agent/td-agent.log`
|
45
|
+
* Cause a change that you've configured Fluentd to pick up: (`echo "FluentdTest" >> /usr/local/var/log/test.log`
|
46
|
+
* Look in `https://one.newrelic.com/launcher/logger.log-launcher` for your log message ("FluentdTest")
|
data/README.md
CHANGED
@@ -4,11 +4,17 @@ A [Fluentd](https://fluentd.org/) output plugin that sends logs to New Relic
|
|
4
4
|
|
5
5
|
This project is provided AS-IS WITHOUT WARRANTY OR SUPPORT, although you can report issues and contribute to the project here on GitHub.
|
6
6
|
|
7
|
+
## Examples
|
8
|
+
|
9
|
+
Please see the [examples](examples/) directory for ways to build a Docker image with the New Relic output plugin and other configuration types
|
10
|
+
that could be useful in your environment.
|
11
|
+
|
7
12
|
## Prerequisites
|
8
13
|
|
9
14
|
Fluentd >= v1.0
|
10
15
|
|
11
16
|
## Installation
|
17
|
+
|
12
18
|
Add the plugin to your fluentd agent:
|
13
19
|
|
14
20
|
`fluent-gem install fluent-plugin-newrelic`
|
@@ -53,6 +59,7 @@ Add one of the following blocks to your Fluentd config file (with your specific
|
|
53
59
|
#### Using Insights Inserts Key
|
54
60
|
|
55
61
|
Example using Insights Insert key:
|
62
|
+
|
56
63
|
```rb
|
57
64
|
<match **>
|
58
65
|
@type newrelic
|
@@ -64,7 +71,9 @@ Getting your New Relic Insights Insert key:
|
|
64
71
|
`https://insights.newrelic.com/accounts/<ACCOUNT_ID>/manage/api_keys`
|
65
72
|
|
66
73
|
#### Using License Key
|
74
|
+
|
67
75
|
Example using License key:
|
76
|
+
|
68
77
|
```rb
|
69
78
|
<match **>
|
70
79
|
@type newrelic
|
data/examples/Dockerfile
ADDED
data/examples/readme.md
ADDED
@@ -0,0 +1,71 @@
|
|
1
|
+
# Fluentd -> New Relic Examples
|
2
|
+
|
3
|
+
## Creating a Docker Image
|
4
|
+
|
5
|
+
Using [Fluentd](https://www.fluentd.org/) as an image makes it simple to deploy a quick input logging solution for functions such as [Syslogs](https://docs.fluentd.org/input/syslog), [HTTP](https://docs.fluentd.org/input/http), custom [UDP](https://docs.fluentd.org/input/udp) and [TCP](https://docs.fluentd.org/input/tcp) use cases, [SNMP](https://github.com/iij/fluent-plugin-snmp), along with many other functions. The [Fluentd](https://www.fluentd.org/) team has put together a great [set of documents](https://docs.fluentd.org/container-deployment) to help you get their basic configuration setup. After that, you will want to get your logs flowing into [New Relic Logs](https://docs.newrelic.com/docs/logs/new-relic-logs/get-started/introduction-new-relic-logs) to create alerts and monitor your systems.
|
6
|
+
|
7
|
+
If you are able to use the Fluentd image directly, it is really simple build on that image and add the New Relic Fluentd Output Plugin. The below set of steps assumes you have some basic understanding of building a Docker image.
|
8
|
+
|
9
|
+
### Docker Image: Steps
|
10
|
+
|
11
|
+
#### 1. Create a `Dockerfile`
|
12
|
+
|
13
|
+
It doesn't take much to get the New Relic Output Plugin into a docker image. Here is a good example from Docker on how best to create an image, [LINK](https://docs.docker.com/develop/develop-images/dockerfile_best-practices/).
|
14
|
+
|
15
|
+
We have an [example Docker file](Dockerfile) in this folder which can be used to get moving quickly.
|
16
|
+
|
17
|
+
#### 2. Build the Image
|
18
|
+
|
19
|
+
The build process is simple and will register a newly created image in your local Docker repository. If you want to use this image for multiple machines, you will need to publish the image to a location you can access.
|
20
|
+
|
21
|
+
```bash
|
22
|
+
# Run this command in the same directory as the Docker file or point to its location
|
23
|
+
docker build --tag nr-fluent:latest nri-fluentd .
|
24
|
+
|
25
|
+
# Run this command to verify the image was created
|
26
|
+
docker image ls
|
27
|
+
|
28
|
+
REPOSITORY TAG IMAGE ID CREATED SIZE
|
29
|
+
nr-fluent latest 70c388b63afc 1 minute ago 44.9MB
|
30
|
+
```
|
31
|
+
|
32
|
+
#### 3. Run the Docker Image
|
33
|
+
|
34
|
+
The next steps assume that you have already created a `fluentd.conf` file with is ready to being monitoring. If you haven't, you must do so before you continue.
|
35
|
+
|
36
|
+
In the following example can be used if you are going to run a syslog server on the image.
|
37
|
+
|
38
|
+
```bash
|
39
|
+
# Notice that the syntax for exposing the UDP port is a bit different
|
40
|
+
# In testing, it appeared that trying to map the UDP port to from a different one configured in the Fluentd config file didn't work as expected
|
41
|
+
docker run -d --name "syslog" -p 0.0.0.0:5140:5140/udp -p 0.0.0.0:5142:5142/udp -v /etc/fluentd:/fluentd/etc -e FLUENTD_CONF=fluentd.conf nr-fluent:latest
|
42
|
+
```
|
43
|
+
|
44
|
+
## Configuring a Syslog Server with Fluentd and New Relic Logs
|
45
|
+
|
46
|
+
In the below example, we are going to create a `fluentd.conf` which will enable a syslog server. It is a good idea to determine if your syslog server is going
|
47
|
+
to connect to this service using UDP or TCP. By default, most syslog servers will likely use UDP. In the below example, I am setting up two syslog listeners for
|
48
|
+
Ubiquiti [Edgemax router](https://help.ubnt.com/hc/en-us/articles/204975904-EdgeRouter-Remote-Syslog-Server-for-System-Logs) and the [Unifi Security Gateway and Access points](https://community.ui.com/questions/syslog-server-and-unifi-logs/bbde4318-e73f-4efe-b1b9-ae11319cc1d9).
|
49
|
+
|
50
|
+
### Fluentd.Conf: Steps
|
51
|
+
|
52
|
+
#### 1. Create a `fluentd.conf` file in a known directory on the host machine
|
53
|
+
|
54
|
+
Below, I have chosen `/etc/fluentd/` as my directory.
|
55
|
+
|
56
|
+
```bash
|
57
|
+
# Make the directory
|
58
|
+
sudo mkdir /etc/fluentd
|
59
|
+
|
60
|
+
# Edit the file
|
61
|
+
sudo nano /etc/fluentd/fluentd.conf
|
62
|
+
```
|
63
|
+
|
64
|
+
#### 2. Add the contents from the [`syslog\fluentd.conf`](syslog\fluentd.config)
|
65
|
+
|
66
|
+
You can find the contents of the [`syslog\fluentd.conf`](syslog\fluentd.config) in the sub folder `syslog`. These contents should provide a quick start to getting started. In the provided
|
67
|
+
example, the syslog details are coming from the above mentioned devices. You may need to tweak the configuration according to the server sending the syslog traffic.
|
68
|
+
|
69
|
+
#### 3. Check New Relic for New Logs
|
70
|
+
|
71
|
+
That is all it should take to get a new stream of logs coming from those devices.
|
@@ -0,0 +1,47 @@
|
|
1
|
+
# UNIFI
|
2
|
+
<source>
|
3
|
+
@type syslog
|
4
|
+
port 5140
|
5
|
+
bind 0.0.0.0
|
6
|
+
tag unifi
|
7
|
+
</source>
|
8
|
+
|
9
|
+
<filter unifi.**>
|
10
|
+
@type record_transformer
|
11
|
+
renew_record true
|
12
|
+
enable_ruby true
|
13
|
+
<record>
|
14
|
+
timestamp ${time.to_f}
|
15
|
+
hostname ${record["host"][/(^.*?),/, 1]}
|
16
|
+
service ${tag_prefix[1]}
|
17
|
+
log_level ${tag_suffix[2]}
|
18
|
+
message ${record.to_json}
|
19
|
+
</record>
|
20
|
+
</filter>
|
21
|
+
|
22
|
+
# EDGE MAX
|
23
|
+
<source>
|
24
|
+
@type syslog
|
25
|
+
port 5142
|
26
|
+
bind 0.0.0.0
|
27
|
+
tag edge-max
|
28
|
+
</source>
|
29
|
+
|
30
|
+
<filter edge-max.**>
|
31
|
+
@type record_transformer
|
32
|
+
renew_record true
|
33
|
+
enable_ruby true
|
34
|
+
<record>
|
35
|
+
timestamp ${time.to_f}
|
36
|
+
hostname ${record["host"]}
|
37
|
+
service ${tag_prefix[1]}
|
38
|
+
log_level ${tag_suffix[2]}
|
39
|
+
message ${record.to_json}
|
40
|
+
</record>
|
41
|
+
</filter>
|
42
|
+
|
43
|
+
# Send data
|
44
|
+
<match **>
|
45
|
+
@type newrelic
|
46
|
+
api_key <CHANGE TO YOUR LICENSE KEY HERE>
|
47
|
+
</match>
|
@@ -67,6 +67,9 @@ module Fluent
|
|
67
67
|
end
|
68
68
|
|
69
69
|
def package_record(record, timestamp)
|
70
|
+
if defined? timestamp.nsec
|
71
|
+
timestamp = timestamp * 1000 + timestamp.nsec / 1_000_000
|
72
|
+
end
|
70
73
|
packaged = {
|
71
74
|
'timestamp' => timestamp,
|
72
75
|
# non-intrinsic attributes get put into 'attributes'
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: fluent-plugin-newrelic
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 1.1.
|
4
|
+
version: 1.1.8
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- New Relic Logging Team
|
8
8
|
autorequire:
|
9
9
|
bindir: exe
|
10
10
|
cert_chain: []
|
11
|
-
date:
|
11
|
+
date: 2020-03-09 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: fluentd
|
@@ -103,11 +103,15 @@ extra_rdoc_files: []
|
|
103
103
|
files:
|
104
104
|
- ".dockerignore"
|
105
105
|
- ".gitignore"
|
106
|
+
- DEVELOPER.md
|
106
107
|
- Dockerfile
|
107
108
|
- Gemfile
|
108
109
|
- LICENSE
|
109
110
|
- README.md
|
110
111
|
- Rakefile
|
112
|
+
- examples/Dockerfile
|
113
|
+
- examples/readme.md
|
114
|
+
- examples/syslog/fluentd.config
|
111
115
|
- lib/fluent/plugin/out_newrelic.rb
|
112
116
|
- lib/newrelic-fluentd-output/version.rb
|
113
117
|
- merge-to-master-pipeline.yml
|
@@ -132,7 +136,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
|
|
132
136
|
- !ruby/object:Gem::Version
|
133
137
|
version: '0'
|
134
138
|
requirements: []
|
135
|
-
rubygems_version: 3.
|
139
|
+
rubygems_version: 3.1.2
|
136
140
|
signing_key:
|
137
141
|
specification_version: 4
|
138
142
|
summary: Sends FluentD events to New Relic
|