fluent-plugin-mesosphere-filter 0.1.0 → 0.1.1
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +8 -8
- data/.gitignore +1 -0
- data/README.md +107 -1
- data/fluent-plugin-mesosphere-filter.gemspec +1 -1
- data/lib/fluent/plugin/mesosphere.rb +102 -13
- data/test/containers/chronos_bad.json +137 -0
- data/test/containers/marathon.json +2 -2
- data/test/containers/marathon2.json +187 -0
- data/test/plugin/test_mesosphere.rb +49 -8
- metadata +6 -2
checksums.yaml
CHANGED
@@ -1,15 +1,15 @@
|
|
1
1
|
---
|
2
2
|
!binary "U0hBMQ==":
|
3
3
|
metadata.gz: !binary |-
|
4
|
-
|
4
|
+
NTcyNGFiYTRlM2JmMTE4NWU3MTRlNjVhNzUzM2Y0YTYxMjUwODQ4Mg==
|
5
5
|
data.tar.gz: !binary |-
|
6
|
-
|
6
|
+
NmRlODgyMDBhMWZlOTNhNTYyZmRjODA1OGI3YmJhMzUyYTkzMmExOQ==
|
7
7
|
SHA512:
|
8
8
|
metadata.gz: !binary |-
|
9
|
-
|
10
|
-
|
11
|
-
|
9
|
+
MTcyNjE1NGQxMzBkMzFlOGVkMWQ3ZDI2MjI4Yjk3MDU2MWY5ZGYxNzdjMDlh
|
10
|
+
YTU0NzM4ZmVjMWJkNGVmMjNkODk2ZjJiMTIxYTFjNTA0MzBkNzk5ZmY4YWU3
|
11
|
+
ZjA5ZTM2N2YzNTZkMDcyZDc0YTMyNzEwYWZlZTY3ZmZlOTk3OGM=
|
12
12
|
data.tar.gz: !binary |-
|
13
|
-
|
14
|
-
|
15
|
-
|
13
|
+
ZmY0ZDMxYjkxOTFlNGJhNzRlYTY5ODYxYmY4MjNhMTdhM2JiZjM4NGUzNWNl
|
14
|
+
MTBhMDUyMWRmOGY1YmFjZTcyYzQ1MzA3MDI5M2FmNzZhNjRkNjlmZWY4ZGNj
|
15
|
+
ZDc3YTA5NGU0ODRmNjdmNzhlMjZiZjU4NjIxOTM3YmFlYmY1NmY=
|
data/.gitignore
CHANGED
data/README.md
CHANGED
@@ -1,3 +1,109 @@
|
|
1
|
-
#
|
1
|
+
#Mesosphere Fluentd Filter
|
2
2
|
[](https://codeclimate.com/github/joshughes/fluent-plugin-mesosphere-filter)
|
3
3
|
[](https://codeclimate.com/github/joshughes/fluent-plugin-mesosphere-filter/coverage)
|
4
|
+
[](https://badge.fury.io/rb/fluent-plugin-mesosphere-filter)
|
5
|
+
[](http://inch-ci.org/github/joshughes/fluent-plugin-mesosphere-filter)
|
6
|
+
[](https://www.versioneye.com/user/projects/5658d10aaef3b5003e000000)
|
7
|
+
|
8
|
+
Marathon, Chronos and Mesos combined can allow for teams to build a great solution for deploying and scaling containers. The issue is when you have a system like Kibana it can be hard to identify what container or task a log is coming from.
|
9
|
+
|
10
|
+
This filter aims to solve that issue by inspecting containers to inject Mesosphere metatdata into the Fluentd event stream.
|
11
|
+
|
12
|
+
##What gets injected?
|
13
|
+
###Marathon
|
14
|
+
|
15
|
+
|`key`| Description |
|
16
|
+
|---|---|
|
17
|
+
|`app`| The application name in marathon |
|
18
|
+
| `mesos_framework` | marathon |
|
19
|
+
| `mesos_task_id` | The unique Mesos task id running the docker container |
|
20
|
+
|
21
|
+
###Chronos
|
22
|
+
Chronos does not have the idea of an 'application'. Just jobs. We run a lot of jobs that relate to our applications so we use a naming scheme that allows us to extract the marathon application that is running the Chronos task.
|
23
|
+
|
24
|
+
An example of a Chronos Job name is the following.
|
25
|
+
|
26
|
+
`some-task-app2-11182015-1718-deployTasks-1-144786721`
|
27
|
+
|
28
|
+
With the default regex in the plugin we extract the following data and inject it into the event stream.
|
29
|
+
|
30
|
+
|`key`| Description | Value |
|
31
|
+
|---|---|---|
|
32
|
+
|`app`| The application name running the task | some-task-app2 |
|
33
|
+
| `mesos_framework` | chronos | chronos |
|
34
|
+
| `mesos_task_id` | The unique Mesos task id running the docker container | |
|
35
|
+
| `chronos_task_type` | The task type our application is running. We run deployment and scheduled tasks. | deployTasks |
|
36
|
+
|
37
|
+
##Configuration
|
38
|
+
|
39
|
+
If your using the docker fluentd logging plugin your configuration should look something like this.
|
40
|
+
|
41
|
+
```
|
42
|
+
<source>
|
43
|
+
type forward
|
44
|
+
port 24224
|
45
|
+
bind 0.0.0.0
|
46
|
+
</source>
|
47
|
+
|
48
|
+
<filter docker.*>
|
49
|
+
type mesosphere-filter
|
50
|
+
cache_size 1000
|
51
|
+
cache_ttl 3600
|
52
|
+
merge_json_log true
|
53
|
+
cronos_task_regex (?<app>[a-z0-9]([-a-z0-9]*[a-z0-9]))-(?<date>[^-]+)-(?<time>[^-]+)-(?<task_type>[^-]+)-(?<run>[^-]+)-(?<epoc>[^-]+)
|
54
|
+
</filter>
|
55
|
+
|
56
|
+
<match docker.*>
|
57
|
+
type stdout
|
58
|
+
</match>
|
59
|
+
```
|
60
|
+
|
61
|
+
|`key`| Description | Default |
|
62
|
+
|---|---|---|
|
63
|
+
|`cache_size`| This plugin will cache information from the docker daemon. This configuration determine how large that cache is. | 1000 |
|
64
|
+
| `cache_ttl ` | How long to keep items in the cache. | 3600 |
|
65
|
+
| `merge_json_log ` | If your application logs in a valid json format, this will merge that into the event stream. | true |
|
66
|
+
| `cronos_task_regex ` | If you don't provide a valid regex here then you will only get the mesos_task id from until you create a standard chronos job name that is parseable by a ruby regex. | **See example above** |
|
67
|
+
|
68
|
+
|
69
|
+
##Example
|
70
|
+
We have an application called `hello-world`. That application is deployed via marathon and has the following environment variables.
|
71
|
+
|
72
|
+
|`key`| value |
|
73
|
+
|---|---|
|
74
|
+
| `MESOS_TASK_ID` | `hello-world.14b0596d-93ea-11e5-a134-124eefe69197`|
|
75
|
+
| `MARATHON_APP_ID` | `/hello-world`|
|
76
|
+
|
77
|
+
Like all great hello world applications. This docker container just execute:
|
78
|
+
|
79
|
+
```bash
|
80
|
+
echo '{"say":"Hello World"}'
|
81
|
+
```
|
82
|
+
|
83
|
+
Without this filter fluentd would process the docker log and output the following.
|
84
|
+
|
85
|
+
```json
|
86
|
+
{
|
87
|
+
"container_id": "ce327cb0de115f7dbfcd2c6055ba945436dada26035a62587b951332a028a530",
|
88
|
+
"container_name": "/some_random_meaningless_name",
|
89
|
+
"source": "stdout",
|
90
|
+
"log": "{\"say\":\"Hello World\"}\r"
|
91
|
+
}
|
92
|
+
```
|
93
|
+
|
94
|
+
With the filter in place that log will become the following.
|
95
|
+
|
96
|
+
```json
|
97
|
+
{
|
98
|
+
"container_id": "ce327cb0de115f7dbfcd2c6055ba945436dada26035a62587b951332a028a530",
|
99
|
+
"container_name": "/some_random_meaningless_name",
|
100
|
+
"say": "Hello World",
|
101
|
+
"mesos_framework": "marathon",
|
102
|
+
"app": "hello-world",
|
103
|
+
"mesos_task_id": "unquie_task_id",
|
104
|
+
"source": "stdout",
|
105
|
+
"log": "{\"say\":\"Hello World\"}\r"
|
106
|
+
}
|
107
|
+
```
|
108
|
+
|
109
|
+
So now in Kibana you can filter on many more fields and more easily track down issues when `hello-world` may be running in multiple containers that are associated with different Mesos tasks.
|
@@ -4,7 +4,7 @@ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
|
|
4
4
|
|
5
5
|
Gem::Specification.new do |gem|
|
6
6
|
gem.name = 'fluent-plugin-mesosphere-filter'
|
7
|
-
gem.version = '0.1.
|
7
|
+
gem.version = '0.1.1'
|
8
8
|
gem.authors = ['Joseph Hughes']
|
9
9
|
gem.email = ['jjhughes57@gmail.com']
|
10
10
|
gem.description = 'Filter plugin to add Mesosphere metadata'
|
@@ -16,21 +16,26 @@
|
|
16
16
|
# limitations under the License.
|
17
17
|
#
|
18
18
|
module Fluent
|
19
|
+
# Parses Marathon and Chronos data from docker to make fluentd logs more
|
20
|
+
# useful.
|
19
21
|
class MesosphereFilter < Fluent::Filter
|
20
22
|
Fluent::Plugin.register_filter('mesosphere-filter', self)
|
21
23
|
|
22
24
|
config_param :cache_size, :integer, default: 1000
|
23
25
|
config_param :cache_ttl, :integer, default: 60 * 60
|
26
|
+
config_param :get_container_id_tag, :bool, default: true
|
27
|
+
config_param :container_id_attr, :string, default: 'container_id'
|
24
28
|
|
25
29
|
config_param :merge_json_log, :bool, default: true
|
26
30
|
config_param :cronos_task_regex,
|
27
31
|
:string,
|
28
|
-
default: '(?<app>[a-z0-9]([-a-z0-
|
32
|
+
default: '(?<app>[a-z0-9]([-a-z0-9_]*[a-z0-9_]))-(?<date>[^-]+)-(?<time>[^-]+)-(?<task_type>[^-]+)-(?<run>[^-]+)-(?<epoc>[^-]+)'
|
29
33
|
|
30
34
|
def initialize
|
31
35
|
super
|
32
36
|
end
|
33
37
|
|
38
|
+
# Get the configuration for the plugin
|
34
39
|
def configure(conf)
|
35
40
|
super
|
36
41
|
|
@@ -43,53 +48,137 @@ module Fluent
|
|
43
48
|
@cache = LruRedux::TTL::ThreadSafeCache.new(@cache_size, @cache_ttl)
|
44
49
|
|
45
50
|
@chronos_task_regex_compiled = Regexp.compile(@cronos_task_regex)
|
51
|
+
|
52
|
+
marathon_regex = '\/(?<app>[a-z0-9]([-a-z0-9_]*[a-z0-9_]))'
|
53
|
+
@marathon_app_regex_compiled = Regexp.compile(marathon_regex)
|
46
54
|
end
|
47
55
|
|
56
|
+
# Gets the log event stream and moifies it. This is where the plugin hooks
|
57
|
+
# into the fluentd envent stream.
|
48
58
|
def filter_stream(tag, es)
|
49
59
|
new_es = MultiEventStream.new
|
60
|
+
container_id = ''
|
50
61
|
|
51
|
-
container_id = tag
|
52
|
-
mesos_data = @cache.getset(container_id) do
|
53
|
-
get_container_metadata(container_id)
|
54
|
-
end
|
62
|
+
container_id = get_container_id_from_tag(tag) if get_container_id_tag
|
55
63
|
|
56
64
|
es.each do |time, record|
|
57
|
-
|
58
|
-
|
59
|
-
|
65
|
+
container_id =
|
66
|
+
get_container_id_from_record(record) if container_id.empty?
|
67
|
+
next unless container_id
|
68
|
+
new_es.add(time, modify_record(record, get_mesos_data(container_id)))
|
60
69
|
end
|
61
|
-
|
62
70
|
new_es
|
63
71
|
end
|
64
72
|
|
73
|
+
# Injects the meso framework data into the record and also merges
|
74
|
+
# the json log if that configuration is enabled.
|
75
|
+
#
|
76
|
+
# ==== Attributes:
|
77
|
+
# * +record+ - The log record being processed
|
78
|
+
# * +mesos_data+ - The mesos data retrived from the docker container
|
79
|
+
#
|
80
|
+
# ==== Returns:
|
81
|
+
# * A record hash that has mesos data and optinally log data added
|
82
|
+
def modify_record(record, mesos_data)
|
83
|
+
modified_record = record.merge(mesos_data)
|
84
|
+
modified_record = merge_json_log(modified_record) if @merge_json_log
|
85
|
+
modified_record
|
86
|
+
end
|
87
|
+
|
88
|
+
# Gets the mesos data about a container from the cache or calls the Docker
|
89
|
+
# api to retrieve the data about the container and store it in the cache.
|
90
|
+
#
|
91
|
+
# ==== Attributes:
|
92
|
+
# * +container_id+ - The container_id where the log record originated from.
|
93
|
+
# ==== Returns:
|
94
|
+
# * A hash of data that describes a mesos task
|
95
|
+
def get_mesos_data(container_id)
|
96
|
+
@cache.getset(container_id) do
|
97
|
+
get_container_metadata(container_id)
|
98
|
+
end
|
99
|
+
end
|
100
|
+
|
101
|
+
# Goes out to docker to get environment variables for a container.
|
102
|
+
# Then we parse the environment varibles looking for known Marathon
|
103
|
+
# and Chronos environment variables
|
104
|
+
#
|
105
|
+
# ==== Attributes:
|
106
|
+
# * +id+ - The id of the container to look at for mesosphere metadata.
|
107
|
+
# ==== Returns:
|
108
|
+
# * A hash that describes a mesos task gathered from the Docker API
|
65
109
|
def get_container_metadata(id)
|
66
110
|
task_data = {}
|
67
111
|
container = Docker::Container.get(id)
|
68
112
|
if container
|
69
113
|
environment = container.json['Config']['Env']
|
70
114
|
environment.each do |env|
|
115
|
+
# Chronos puts task_id in lowercase, and Marathon does it with
|
116
|
+
# uppercase
|
71
117
|
if env =~ /MESOS_TASK_ID/i
|
72
118
|
task_data['mesos_task_id'] = parse_env(env)
|
73
119
|
elsif env.include? 'MARATHON_APP_ID'
|
120
|
+
match_data = parse_env(env).match(@marathon_app_regex_compiled)
|
74
121
|
task_data['mesos_framework'] = 'marathon'
|
75
|
-
task_data['app'] =
|
122
|
+
task_data['app'] = match_data['app'] if match_data
|
76
123
|
elsif env.include? 'CHRONOS_JOB_NAME'
|
77
124
|
match_data = parse_env(env).match(@chronos_task_regex_compiled)
|
78
125
|
task_data['mesos_framework'] = 'chronos'
|
79
|
-
task_data['app'] = match_data['app']
|
80
|
-
task_data['chronos_task_type'] = match_data['task_type']
|
126
|
+
task_data['app'] = match_data['app'] if match_data
|
127
|
+
task_data['chronos_task_type'] = match_data['task_type'] if match_data
|
81
128
|
end
|
82
129
|
end
|
83
130
|
end
|
84
131
|
task_data
|
85
132
|
end
|
86
133
|
|
134
|
+
# Gets the container id from the last element in the tag. If the user has
|
135
|
+
# configured container_id_attr the container id can be gathered from the
|
136
|
+
# record if it has been inserted there.
|
137
|
+
#
|
138
|
+
# ==== Attributes:
|
139
|
+
# * +tag+ - The tag of the log being processed
|
140
|
+
# ==== Returns:
|
141
|
+
# * A docker container id
|
142
|
+
def get_container_id_from_tag(tag)
|
143
|
+
tag.split('.').last
|
144
|
+
end
|
145
|
+
|
146
|
+
# If the user has configured container_id_attr the container id can be
|
147
|
+
# gathered from the record if it has been inserted there. If no container_id
|
148
|
+
# can be found, the record is not processed.
|
149
|
+
#
|
150
|
+
# ==== Attributes::
|
151
|
+
# * +record+ - The record that is being transformed by the filter
|
152
|
+
# ==== Returns:
|
153
|
+
# * A docker container id
|
154
|
+
def get_container_id_from_record(record)
|
155
|
+
record[@container_id_attr]
|
156
|
+
end
|
157
|
+
|
158
|
+
# Split the env var on = and return the value
|
159
|
+
# ==== Attributes:
|
160
|
+
# * +env+ - The docker environment variable to parse to get the value.
|
161
|
+
# ==== Examples
|
162
|
+
# # For the env value MARATHON_APP_ID the actual string value given to us
|
163
|
+
# # by docker is 'MARATHON_APP_ID=some-app'. We want to return 'some-app'.
|
164
|
+
# ==== Returns:
|
165
|
+
# * The value of an environment varaible
|
87
166
|
def parse_env(env)
|
88
167
|
env.split('=').last
|
89
168
|
end
|
90
169
|
|
170
|
+
# Look at the log value and if it is valid json then we will parse the json
|
171
|
+
# and merge it into the log record.
|
172
|
+
# ==== Attributes:
|
173
|
+
# * +record+ - The record we are transforming in the fluentd event stream.
|
174
|
+
# ==== Examples
|
175
|
+
# # Docker captures stdout and passes it in the 'log' record attribute.
|
176
|
+
# # We try to discover is the value of 'log' is json, if it is then we
|
177
|
+
# # will parse the json and add the keys and values to the record.
|
178
|
+
# ==== Returns:
|
179
|
+
# * A record hash that has json log data merged into the record
|
91
180
|
def merge_json_log(record)
|
92
|
-
if record.
|
181
|
+
if record.key?('log')
|
93
182
|
log = record['log'].strip
|
94
183
|
if log[0].eql?('{') && log[-1].eql?('}')
|
95
184
|
begin
|
@@ -0,0 +1,137 @@
|
|
1
|
+
{
|
2
|
+
"AppArmorProfile": "",
|
3
|
+
"Args": [
|
4
|
+
" task",
|
5
|
+
"task"
|
6
|
+
],
|
7
|
+
"Config": {
|
8
|
+
"AttachStderr": false,
|
9
|
+
"AttachStdin": false,
|
10
|
+
"AttachStdout": false,
|
11
|
+
"Cmd": [
|
12
|
+
" task",
|
13
|
+
"task"
|
14
|
+
],
|
15
|
+
"CpuShares": 153,
|
16
|
+
"Cpuset": "",
|
17
|
+
"Domainname": "",
|
18
|
+
"Entrypoint": [
|
19
|
+
"/minimal"
|
20
|
+
],
|
21
|
+
"Env": [
|
22
|
+
"mesos_task_id=ct:1448508194000:0:recurring-transaction3:task",
|
23
|
+
"CHRONOS_JOB_OWNER=chartreuse-team@itriagehealth.com",
|
24
|
+
"CHRONOS_JOB_NAME=foobarzasdf",
|
25
|
+
"HOST=ip-10-10-30-60.ec2.internal",
|
26
|
+
"CHRONOS_RESOURCE_MEM=1024.0",
|
27
|
+
"CHRONOS_RESOURCE_CPU=0.05",
|
28
|
+
"CHRONOS_RESOURCE_DISK=256.0",
|
29
|
+
"YEAR=2016",
|
30
|
+
"VAULT_HOSTS=https://dev-use1a-pr-01-vault-vault-0001.dev-utopia.com:8200,https://dev-use1b-pr-01-vault-vault-0001.dev-utopia.com:8200,https://dev-use1e-pr-01-vault-vault-0001.dev-utopia.com:8200",
|
31
|
+
"MESOS_SANDBOX=/mnt/mesos/sandbox",
|
32
|
+
"PORT=80"
|
33
|
+
],
|
34
|
+
"ExposedPorts": {
|
35
|
+
"80/tcp": {}
|
36
|
+
},
|
37
|
+
"Hostname": "ip-10-10-30-60",
|
38
|
+
"Image": "docker-dev.itriagehealth.com/utopia/recurring-transaction2-task:564a03fd52f1800005000003",
|
39
|
+
"Labels": {},
|
40
|
+
"MacAddress": "",
|
41
|
+
"Memory": 1107296256,
|
42
|
+
"MemorySwap": -1,
|
43
|
+
"NetworkDisabled": false,
|
44
|
+
"OnBuild": null,
|
45
|
+
"OpenStdin": false,
|
46
|
+
"PortSpecs": null,
|
47
|
+
"StdinOnce": false,
|
48
|
+
"Tty": false,
|
49
|
+
"User": "",
|
50
|
+
"Volumes": null,
|
51
|
+
"WorkingDir": "/"
|
52
|
+
},
|
53
|
+
"Created": "2015-11-26T03:23:15.958740506Z",
|
54
|
+
"Driver": "overlay",
|
55
|
+
"ExecDriver": "native-0.2",
|
56
|
+
"ExecIDs": null,
|
57
|
+
"HostConfig": {
|
58
|
+
"Binds": [
|
59
|
+
"/data/mesos-slave/slaves/20151115-193554-4078832138-5050-1872-S2/docker/links/2e3e0f5c-ac02-40a0-804b-b4c7736935b5:/mnt/mesos/sandbox"
|
60
|
+
],
|
61
|
+
"CapAdd": null,
|
62
|
+
"CapDrop": null,
|
63
|
+
"CgroupParent": "",
|
64
|
+
"ContainerIDFile": "",
|
65
|
+
"CpuShares": 153,
|
66
|
+
"CpusetCpus": "",
|
67
|
+
"Devices": [],
|
68
|
+
"Dns": null,
|
69
|
+
"DnsSearch": null,
|
70
|
+
"ExtraHosts": null,
|
71
|
+
"IpcMode": "",
|
72
|
+
"Links": null,
|
73
|
+
"LogConfig": {
|
74
|
+
"Config": null,
|
75
|
+
"Type": "json-file"
|
76
|
+
},
|
77
|
+
"LxcConf": [],
|
78
|
+
"Memory": 1107296256,
|
79
|
+
"MemorySwap": -1,
|
80
|
+
"NetworkMode": "host",
|
81
|
+
"PidMode": "",
|
82
|
+
"PortBindings": {},
|
83
|
+
"Privileged": false,
|
84
|
+
"PublishAllPorts": false,
|
85
|
+
"ReadonlyRootfs": false,
|
86
|
+
"RestartPolicy": {
|
87
|
+
"MaximumRetryCount": 0,
|
88
|
+
"Name": "no"
|
89
|
+
},
|
90
|
+
"SecurityOpt": null,
|
91
|
+
"Ulimits": null,
|
92
|
+
"VolumesFrom": null
|
93
|
+
},
|
94
|
+
"HostnamePath": "/data/docker/containers/8dac46bdb4c099f8a12c12e06b104e9ca86b839e241b386bf3ddb537c9923642/hostname",
|
95
|
+
"HostsPath": "/data/docker/containers/8dac46bdb4c099f8a12c12e06b104e9ca86b839e241b386bf3ddb537c9923642/hosts",
|
96
|
+
"Id": "foobar124",
|
97
|
+
"Image": "dc41b56ccebd1433c6be3a92b838cc64c8c5f7088237a3da08a380cf43d6d6ac",
|
98
|
+
"LogPath": "/data/docker/containers/8dac46bdb4c099f8a12c12e06b104e9ca86b839e241b386bf3ddb537c9923642/8dac46bdb4c099f8a12c12e06b104e9ca86b839e241b386bf3ddb537c9923642-json.log",
|
99
|
+
"MountLabel": "",
|
100
|
+
"Name": "/mesos-2e3e0f5c-ac02-40a0-804b-b4c7736935b5",
|
101
|
+
"NetworkSettings": {
|
102
|
+
"Bridge": "",
|
103
|
+
"Gateway": "",
|
104
|
+
"GlobalIPv6Address": "",
|
105
|
+
"GlobalIPv6PrefixLen": 0,
|
106
|
+
"IPAddress": "",
|
107
|
+
"IPPrefixLen": 0,
|
108
|
+
"IPv6Gateway": "",
|
109
|
+
"LinkLocalIPv6Address": "",
|
110
|
+
"LinkLocalIPv6PrefixLen": 0,
|
111
|
+
"MacAddress": "",
|
112
|
+
"PortMapping": null,
|
113
|
+
"Ports": null
|
114
|
+
},
|
115
|
+
"Path": "/minimal",
|
116
|
+
"ProcessLabel": "",
|
117
|
+
"ResolvConfPath": "/data/docker/containers/8dac46bdb4c099f8a12c12e06b104e9ca86b839e241b386bf3ddb537c9923642/resolv.conf",
|
118
|
+
"RestartCount": 0,
|
119
|
+
"State": {
|
120
|
+
"Dead": false,
|
121
|
+
"Error": "",
|
122
|
+
"ExitCode": 0,
|
123
|
+
"FinishedAt": "2015-11-26T03:23:16.090077843Z",
|
124
|
+
"OOMKilled": false,
|
125
|
+
"Paused": false,
|
126
|
+
"Pid": 0,
|
127
|
+
"Restarting": false,
|
128
|
+
"Running": false,
|
129
|
+
"StartedAt": "2015-11-26T03:23:16.01885868Z"
|
130
|
+
},
|
131
|
+
"Volumes": {
|
132
|
+
"/mnt/mesos/sandbox": "/data/mesos-slave/slaves/20151115-193554-4078832138-5050-1872-S2/frameworks/20150821-174742-1460275722-5050-14777-0001/executors/ct:1448508194000:0:recurring-transaction2:task/runs/2e3e0f5c-ac02-40a0-804b-b4c7736935b5"
|
133
|
+
},
|
134
|
+
"VolumesRW": {
|
135
|
+
"/mnt/mesos/sandbox": true
|
136
|
+
}
|
137
|
+
}
|
@@ -118,8 +118,8 @@
|
|
118
118
|
"StdinOnce": false,
|
119
119
|
"Env": [
|
120
120
|
"FLUENTD_OPT=-vv",
|
121
|
-
"MESOS_TASK_ID=
|
122
|
-
"MARATHON_APP_ID=/
|
121
|
+
"MESOS_TASK_ID=hello-world.14b0596d-93ea-11e5-a134-124eefe69197",
|
122
|
+
"MARATHON_APP_ID=/hello-world",
|
123
123
|
"FLUENTD_CONF=test.conf",
|
124
124
|
"PATH=/home/ubuntu/ruby/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
|
125
125
|
"FLUENTD_OPT="
|
@@ -0,0 +1,187 @@
|
|
1
|
+
{
|
2
|
+
"Id": "somecontainer123",
|
3
|
+
"Created": "2015-11-26T06:57:56.560006039Z",
|
4
|
+
"Path": "/bin/sh",
|
5
|
+
"Args": [
|
6
|
+
"-c",
|
7
|
+
"exec fluentd -c /fluentd/etc/$FLUENTD_CONF -p /fluentd/plugins $FLUENTD_OPT"
|
8
|
+
],
|
9
|
+
"State": {
|
10
|
+
"Status": "running",
|
11
|
+
"Running": true,
|
12
|
+
"Paused": false,
|
13
|
+
"Restarting": false,
|
14
|
+
"OOMKilled": false,
|
15
|
+
"Dead": false,
|
16
|
+
"Pid": 6563,
|
17
|
+
"ExitCode": 0,
|
18
|
+
"Error": "",
|
19
|
+
"StartedAt": "2015-11-26T07:00:45.707356204Z",
|
20
|
+
"FinishedAt": "2015-11-26T07:00:43.278299772Z"
|
21
|
+
},
|
22
|
+
"Image": "7cd5efd43c5097c8e6a9b6bdc050f887e0e1dee49a79b7eb920e2d741c36b3b5",
|
23
|
+
"ResolvConfPath": "/mnt/sda1/var/lib/docker/containers/110e3bd749e33e3b55bb55bec44a192457d722607afb1e6423e6a45a1c1e8b45/resolv.conf",
|
24
|
+
"HostnamePath": "/mnt/sda1/var/lib/docker/containers/110e3bd749e33e3b55bb55bec44a192457d722607afb1e6423e6a45a1c1e8b45/hostname",
|
25
|
+
"HostsPath": "/mnt/sda1/var/lib/docker/containers/110e3bd749e33e3b55bb55bec44a192457d722607afb1e6423e6a45a1c1e8b45/hosts",
|
26
|
+
"LogPath": "/mnt/sda1/var/lib/docker/containers/110e3bd749e33e3b55bb55bec44a192457d722607afb1e6423e6a45a1c1e8b45/110e3bd749e33e3b55bb55bec44a192457d722607afb1e6423e6a45a1c1e8b45-json.log",
|
27
|
+
"Name": "/fluentddockertest_fluentd_server_1",
|
28
|
+
"RestartCount": 0,
|
29
|
+
"Driver": "aufs",
|
30
|
+
"ExecDriver": "native-0.2",
|
31
|
+
"MountLabel": "",
|
32
|
+
"ProcessLabel": "",
|
33
|
+
"AppArmorProfile": "",
|
34
|
+
"ExecIDs": null,
|
35
|
+
"HostConfig": {
|
36
|
+
"Binds": [
|
37
|
+
"/Users/A658307/fluentd_docker_test/fluentd/etc:/fluentd/etc:rw",
|
38
|
+
"/Users/A658307/fluentd_docker_test/fluentd/logs:/fluentd/log:rw",
|
39
|
+
"/Users/A658307/fluentd_docker_test/fluentd/plugins:/fluentd/plugins:rw"
|
40
|
+
],
|
41
|
+
"ContainerIDFile": "",
|
42
|
+
"LxcConf": null,
|
43
|
+
"Memory": 0,
|
44
|
+
"MemoryReservation": 0,
|
45
|
+
"MemorySwap": 0,
|
46
|
+
"KernelMemory": 0,
|
47
|
+
"CpuShares": 0,
|
48
|
+
"CpuPeriod": 0,
|
49
|
+
"CpusetCpus": "",
|
50
|
+
"CpusetMems": "",
|
51
|
+
"CpuQuota": 0,
|
52
|
+
"BlkioWeight": 0,
|
53
|
+
"OomKillDisable": false,
|
54
|
+
"MemorySwappiness": null,
|
55
|
+
"Privileged": false,
|
56
|
+
"PortBindings": {
|
57
|
+
"24224/tcp": [{
|
58
|
+
"HostIp": "",
|
59
|
+
"HostPort": "24224"
|
60
|
+
}]
|
61
|
+
},
|
62
|
+
"Links": null,
|
63
|
+
"PublishAllPorts": false,
|
64
|
+
"Dns": null,
|
65
|
+
"DnsOptions": null,
|
66
|
+
"DnsSearch": null,
|
67
|
+
"ExtraHosts": [],
|
68
|
+
"VolumesFrom": [],
|
69
|
+
"Devices": null,
|
70
|
+
"NetworkMode": "default",
|
71
|
+
"IpcMode": "",
|
72
|
+
"PidMode": "",
|
73
|
+
"UTSMode": "",
|
74
|
+
"CapAdd": null,
|
75
|
+
"CapDrop": null,
|
76
|
+
"GroupAdd": null,
|
77
|
+
"RestartPolicy": {
|
78
|
+
"Name": "",
|
79
|
+
"MaximumRetryCount": 0
|
80
|
+
},
|
81
|
+
"SecurityOpt": null,
|
82
|
+
"ReadonlyRootfs": false,
|
83
|
+
"Ulimits": null,
|
84
|
+
"LogConfig": {
|
85
|
+
"Type": "json-file",
|
86
|
+
"Config": {}
|
87
|
+
},
|
88
|
+
"CgroupParent": "",
|
89
|
+
"ConsoleSize": [
|
90
|
+
0,
|
91
|
+
0
|
92
|
+
],
|
93
|
+
"VolumeDriver": ""
|
94
|
+
},
|
95
|
+
"GraphDriver": {
|
96
|
+
"Name": "aufs",
|
97
|
+
"Data": null
|
98
|
+
},
|
99
|
+
"Volumes": {
|
100
|
+
"/fluentd/etc": "/Users/A658307/fluentd_docker_test/fluentd/etc",
|
101
|
+
"/fluentd/log": "/Users/A658307/fluentd_docker_test/fluentd/logs",
|
102
|
+
"/fluentd/plugins": "/Users/A658307/fluentd_docker_test/fluentd/plugins"
|
103
|
+
},
|
104
|
+
"VolumesRW": {
|
105
|
+
"/fluentd/etc": true,
|
106
|
+
"/fluentd/log": true,
|
107
|
+
"/fluentd/plugins": true
|
108
|
+
},
|
109
|
+
"Config": {
|
110
|
+
"Hostname": "110e3bd749e3",
|
111
|
+
"Domainname": "",
|
112
|
+
"User": "ubuntu",
|
113
|
+
"AttachStdin": false,
|
114
|
+
"AttachStdout": false,
|
115
|
+
"AttachStderr": false,
|
116
|
+
"Tty": false,
|
117
|
+
"OpenStdin": false,
|
118
|
+
"StdinOnce": false,
|
119
|
+
"Env": [
|
120
|
+
"FLUENTD_OPT=-vv",
|
121
|
+
"MESOS_TASK_ID=hello-world.14b0596d-93ea-11e5-a134-124eefe69197",
|
122
|
+
"MARATHON_APP_ID=/hello-world",
|
123
|
+
"FLUENTD_CONF=test.conf",
|
124
|
+
"PATH=/home/ubuntu/ruby/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
|
125
|
+
"FLUENTD_OPT="
|
126
|
+
],
|
127
|
+
"Cmd": [
|
128
|
+
"/bin/sh",
|
129
|
+
"-c",
|
130
|
+
"exec fluentd -c /fluentd/etc/$FLUENTD_CONF -p /fluentd/plugins $FLUENTD_OPT"
|
131
|
+
],
|
132
|
+
"Image": "fluent/fluentd:latest",
|
133
|
+
"Volumes": {
|
134
|
+
"/fluentd/etc": {},
|
135
|
+
"/fluentd/log": {},
|
136
|
+
"/fluentd/plugins": {}
|
137
|
+
},
|
138
|
+
"WorkingDir": "/home/ubuntu",
|
139
|
+
"Entrypoint": null,
|
140
|
+
"OnBuild": null,
|
141
|
+
"Labels": {
|
142
|
+
"Description": "Fluentd docker image",
|
143
|
+
"Vendor": "Fluent Organization",
|
144
|
+
"Version": "1.0",
|
145
|
+
"com.docker.compose.config-hash": "7a03ce1e88388255c166fa62dcad89a43d43034ce605265094cf9c9951dcc9f0",
|
146
|
+
"com.docker.compose.container-number": "1",
|
147
|
+
"com.docker.compose.oneoff": "False",
|
148
|
+
"com.docker.compose.project": "fluentddockertest",
|
149
|
+
"com.docker.compose.service": "fluentd_server",
|
150
|
+
"com.docker.compose.version": "1.5.1"
|
151
|
+
},
|
152
|
+
"MacAddress": "",
|
153
|
+
"NetworkDisabled": false,
|
154
|
+
"ExposedPorts": {
|
155
|
+
"24224/tcp": {}
|
156
|
+
},
|
157
|
+
"VolumeDriver": "",
|
158
|
+
"Memory": 0,
|
159
|
+
"MemorySwap": 0,
|
160
|
+
"CpuShares": 0,
|
161
|
+
"Cpuset": ""
|
162
|
+
},
|
163
|
+
"NetworkSettings": {
|
164
|
+
"Bridge": "",
|
165
|
+
"SandboxID": "f304aa3b0e20fe1e0edb5984aedd03b7d5df999ff5ff9bd3ef8f831923e5f96a",
|
166
|
+
"HairpinMode": false,
|
167
|
+
"LinkLocalIPv6Address": "",
|
168
|
+
"LinkLocalIPv6PrefixLen": 0,
|
169
|
+
"Ports": {
|
170
|
+
"24224/tcp": [{
|
171
|
+
"HostIp": "0.0.0.0",
|
172
|
+
"HostPort": "24224"
|
173
|
+
}]
|
174
|
+
},
|
175
|
+
"SandboxKey": "/var/run/docker/netns/f304aa3b0e20",
|
176
|
+
"SecondaryIPAddresses": null,
|
177
|
+
"SecondaryIPv6Addresses": null,
|
178
|
+
"EndpointID": "7c6889977686db7c2b2042798b021a03df61c4273dcfd84fb2a9e5830fcb71a9",
|
179
|
+
"Gateway": "172.17.0.1",
|
180
|
+
"GlobalIPv6Address": "",
|
181
|
+
"GlobalIPv6PrefixLen": 0,
|
182
|
+
"IPAddress": "172.17.0.2",
|
183
|
+
"IPPrefixLen": 16,
|
184
|
+
"IPv6Gateway": "",
|
185
|
+
"MacAddress": "02:42:ac:11:00:02"
|
186
|
+
}
|
187
|
+
}
|
@@ -21,6 +21,10 @@ class AmplifierFilterTest < Test::Unit::TestCase
|
|
21
21
|
cache_size 2000
|
22
22
|
cache_ttl 300
|
23
23
|
]
|
24
|
+
CONFIG3 = %[
|
25
|
+
get_container_id_tag false
|
26
|
+
container_id_attr container_id
|
27
|
+
]
|
24
28
|
|
25
29
|
def create_driver(conf = CONFIG, tag = 'test')
|
26
30
|
Fluent::Test::FilterTestDriver.new(Fluent::MesosphereFilter, tag).configure(conf)
|
@@ -31,9 +35,9 @@ class AmplifierFilterTest < Test::Unit::TestCase
|
|
31
35
|
.to_return(status: 200, body: file, headers: {})
|
32
36
|
end
|
33
37
|
|
34
|
-
def setup_marathon_container
|
35
|
-
docker_api_url =
|
36
|
-
file = File.open(
|
38
|
+
def setup_marathon_container(container_id, file_name)
|
39
|
+
docker_api_url = "http://tcp//example.com:5422/v1.16/containers/#{container_id}/json"
|
40
|
+
file = File.open("test/containers/#{file_name}.json", 'rb')
|
37
41
|
setup_docker_stub(file, docker_api_url)
|
38
42
|
end
|
39
43
|
|
@@ -44,7 +48,7 @@ class AmplifierFilterTest < Test::Unit::TestCase
|
|
44
48
|
end
|
45
49
|
|
46
50
|
def test_marathon_filter
|
47
|
-
setup_marathon_container
|
51
|
+
setup_marathon_container('foobar123', 'marathon')
|
48
52
|
|
49
53
|
d1 = create_driver(CONFIG, 'docker.foobar123')
|
50
54
|
d1.run do
|
@@ -52,17 +56,17 @@ class AmplifierFilterTest < Test::Unit::TestCase
|
|
52
56
|
end
|
53
57
|
filtered = d1.filtered_as_array
|
54
58
|
|
55
|
-
task_id = '
|
59
|
+
task_id = 'hello-world.14b0596d-93ea-11e5-a134-124eefe69197'
|
56
60
|
|
57
61
|
log_entry = filtered[0][2]
|
58
62
|
|
59
63
|
assert_equal 'marathon', log_entry['mesos_framework']
|
60
|
-
assert_equal '
|
64
|
+
assert_equal 'hello-world', log_entry['app']
|
61
65
|
assert_equal task_id, log_entry['mesos_task_id']
|
62
66
|
end
|
63
67
|
|
64
68
|
def test_container_cache
|
65
|
-
setup_marathon_container
|
69
|
+
setup_marathon_container('foobar123', 'marathon')
|
66
70
|
|
67
71
|
d1 = create_driver(CONFIG, 'docker.foobar123')
|
68
72
|
d1.run do
|
@@ -77,7 +81,7 @@ class AmplifierFilterTest < Test::Unit::TestCase
|
|
77
81
|
end
|
78
82
|
|
79
83
|
def test_container_cache_expiration
|
80
|
-
setup_marathon_container
|
84
|
+
setup_marathon_container('foobar123', 'marathon')
|
81
85
|
|
82
86
|
d1 = create_driver(CONFIG2, 'docker.foobar123')
|
83
87
|
d1.run do
|
@@ -115,6 +119,26 @@ class AmplifierFilterTest < Test::Unit::TestCase
|
|
115
119
|
assert_equal 'deployTasks', log_entry['chronos_task_type']
|
116
120
|
end
|
117
121
|
|
122
|
+
def test_chronos_bad_match
|
123
|
+
docker_api_url = 'http://tcp//example.com:5422/v1.16/containers/foobar124/json'
|
124
|
+
file = File.open('test/containers/chronos_bad.json', 'rb')
|
125
|
+
setup_docker_stub(file, docker_api_url)
|
126
|
+
|
127
|
+
d1 = create_driver(CONFIG, 'docker.foobar124')
|
128
|
+
d1.run do
|
129
|
+
d1.filter('log' => 'Hello World 1')
|
130
|
+
end
|
131
|
+
filtered = d1.filtered_as_array
|
132
|
+
|
133
|
+
task_id = 'ct:1448508194000:0:recurring-transaction3:task'
|
134
|
+
log_entry = filtered[0][2]
|
135
|
+
|
136
|
+
assert_equal 'chronos', log_entry['mesos_framework']
|
137
|
+
assert_equal task_id, log_entry['mesos_task_id']
|
138
|
+
refute log_entry['app']
|
139
|
+
refute log_entry['chronos_task_type']
|
140
|
+
end
|
141
|
+
|
118
142
|
def test_merge_json
|
119
143
|
setup_chronos_container
|
120
144
|
|
@@ -143,4 +167,21 @@ class AmplifierFilterTest < Test::Unit::TestCase
|
|
143
167
|
assert_equal bad_json1, filtered[0][2]['log']
|
144
168
|
assert_equal bad_json2, filtered[1][2]['log']
|
145
169
|
end
|
170
|
+
|
171
|
+
def test_container_id_from_record
|
172
|
+
setup_marathon_container('somecontainer123', 'marathon2')
|
173
|
+
|
174
|
+
d1 = create_driver(CONFIG3, 'docker')
|
175
|
+
d1.run do
|
176
|
+
d1.filter('log' => 'hello_world', 'container_id' => 'somecontainer123')
|
177
|
+
end
|
178
|
+
filtered = d1.filtered_as_array
|
179
|
+
|
180
|
+
task_id = 'hello-world.14b0596d-93ea-11e5-a134-124eefe69197'
|
181
|
+
log_entry = filtered[0][2]
|
182
|
+
|
183
|
+
assert_equal 'marathon', log_entry['mesos_framework']
|
184
|
+
assert_equal 'hello-world', log_entry['app']
|
185
|
+
assert_equal task_id, log_entry['mesos_task_id']
|
186
|
+
end
|
146
187
|
end
|
metadata
CHANGED
@@ -1,14 +1,14 @@
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
2
2
|
name: fluent-plugin-mesosphere-filter
|
3
3
|
version: !ruby/object:Gem::Version
|
4
|
-
version: 0.1.
|
4
|
+
version: 0.1.1
|
5
5
|
platform: ruby
|
6
6
|
authors:
|
7
7
|
- Joseph Hughes
|
8
8
|
autorequire:
|
9
9
|
bindir: bin
|
10
10
|
cert_chain: []
|
11
|
-
date: 2015-11-
|
11
|
+
date: 2015-11-28 00:00:00.000000000 Z
|
12
12
|
dependencies:
|
13
13
|
- !ruby/object:Gem::Dependency
|
14
14
|
name: fluentd
|
@@ -195,7 +195,9 @@ files:
|
|
195
195
|
- fluent-plugin-mesosphere-filter.gemspec
|
196
196
|
- lib/fluent/plugin/mesosphere.rb
|
197
197
|
- test/containers/chronos.json
|
198
|
+
- test/containers/chronos_bad.json
|
198
199
|
- test/containers/marathon.json
|
200
|
+
- test/containers/marathon2.json
|
199
201
|
- test/helper.rb
|
200
202
|
- test/plugin/test_mesosphere.rb
|
201
203
|
homepage: https://github.com/joshughes/fluent-plugin-mesosphere-filter
|
@@ -224,6 +226,8 @@ specification_version: 4
|
|
224
226
|
summary: Filter plugin to add Mesosphere metadata to fluentd from Chronos and Marathon
|
225
227
|
test_files:
|
226
228
|
- test/containers/chronos.json
|
229
|
+
- test/containers/chronos_bad.json
|
227
230
|
- test/containers/marathon.json
|
231
|
+
- test/containers/marathon2.json
|
228
232
|
- test/helper.rb
|
229
233
|
- test/plugin/test_mesosphere.rb
|