fluent-plugin-mesosphere-filter 0.1.0 → 0.1.1

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,15 +1,15 @@
1
1
  ---
2
2
  !binary "U0hBMQ==":
3
3
  metadata.gz: !binary |-
4
- ODM4MTU1YmFjNGUyYWZjZTIzYTI1ZDUxZTE3MjZiZDY3ZWNhMWViYQ==
4
+ NTcyNGFiYTRlM2JmMTE4NWU3MTRlNjVhNzUzM2Y0YTYxMjUwODQ4Mg==
5
5
  data.tar.gz: !binary |-
6
- YTZhNTVhZWRjZTE3ODMzNGYwMjM4YzU5ZGMyNDNjM2JmY2Q5ZWM3Mg==
6
+ NmRlODgyMDBhMWZlOTNhNTYyZmRjODA1OGI3YmJhMzUyYTkzMmExOQ==
7
7
  SHA512:
8
8
  metadata.gz: !binary |-
9
- MTdkNTFiMjVkZTBkMDdhZDczZjc3ZGZhNzkzZGEyMjFlNTExYTU4ODliNjhk
10
- ZGQ3NGU2NzdiYTZlZmY1NjYwM2MyOWY2ZDMwMTQ2ZDBjYzQyZGRmYzBkMjY2
11
- MmQxZTM0YzdhMWNjNTY2Y2VkYzkwZTBhYzNjNzEzYzcyMjZjOGU=
9
+ MTcyNjE1NGQxMzBkMzFlOGVkMWQ3ZDI2MjI4Yjk3MDU2MWY5ZGYxNzdjMDlh
10
+ YTU0NzM4ZmVjMWJkNGVmMjNkODk2ZjJiMTIxYTFjNTA0MzBkNzk5ZmY4YWU3
11
+ ZjA5ZTM2N2YzNTZkMDcyZDc0YTMyNzEwYWZlZTY3ZmZlOTk3OGM=
12
12
  data.tar.gz: !binary |-
13
- NTNkNTA4OTYwY2JkYzgwYjQyZWJkNmI1YmNmOTAwYTE2MDJjOTk3MTEyZmU3
14
- ZmJjOTA1ZDgyZTdjMWY5MDI5YTExNDhmM2FiNTY4ZmMxODJhMGNmMmNlMDRi
15
- MDgxNDY2MmE3MzQzNDA2ZTQ3NWIyYTBkZjA5NmMwYjU5YzM5MjY=
13
+ ZmY0ZDMxYjkxOTFlNGJhNzRlYTY5ODYxYmY4MjNhMTdhM2JiZjM4NGUzNWNl
14
+ MTBhMDUyMWRmOGY1YmFjZTcyYzQ1MzA3MDI5M2FmNzZhNjRkNjlmZWY4ZGNj
15
+ ZDc3YTA5NGU0ODRmNjdmNzhlMjZiZjU4NjIxOTM3YmFlYmY1NmY=
data/.gitignore CHANGED
@@ -17,3 +17,4 @@ test/version_tmp
17
17
  tmp
18
18
  .idea/
19
19
  *.iml
20
+ .DS_Store
data/README.md CHANGED
@@ -1,3 +1,109 @@
1
- #MEsos Stuff
1
+ #Mesosphere Fluentd Filter
2
2
  [![Code Climate](https://codeclimate.com/github/joshughes/fluent-plugin-mesosphere-filter/badges/gpa.svg)](https://codeclimate.com/github/joshughes/fluent-plugin-mesosphere-filter)
3
3
  [![Test Coverage](https://codeclimate.com/github/joshughes/fluent-plugin-mesosphere-filter/badges/coverage.svg)](https://codeclimate.com/github/joshughes/fluent-plugin-mesosphere-filter/coverage)
4
+ [![Gem Version](https://badge.fury.io/rb/fluent-plugin-mesosphere-filter.svg)](https://badge.fury.io/rb/fluent-plugin-mesosphere-filter)
5
+ [![Inline docs](http://inch-ci.org/github/joshughes/fluent-plugin-mesosphere-filter.svg?branch=master)](http://inch-ci.org/github/joshughes/fluent-plugin-mesosphere-filter)
6
+ [![Dependency Status](https://www.versioneye.com/user/projects/5658d10aaef3b5003e000000/badge.svg?style=flat)](https://www.versioneye.com/user/projects/5658d10aaef3b5003e000000)
7
+
8
+ Marathon, Chronos and Mesos combined can allow for teams to build a great solution for deploying and scaling containers. The issue is when you have a system like Kibana it can be hard to identify what container or task a log is coming from.
9
+
10
+ This filter aims to solve that issue by inspecting containers to inject Mesosphere metatdata into the Fluentd event stream.
11
+
12
+ ##What gets injected?
13
+ ###Marathon
14
+
15
+ |`key`| Description |
16
+ |---|---|
17
+ |`app`| The application name in marathon |
18
+ | `mesos_framework` | marathon |
19
+ | `mesos_task_id` | The unique Mesos task id running the docker container |
20
+
21
+ ###Chronos
22
+ Chronos does not have the idea of an 'application'. Just jobs. We run a lot of jobs that relate to our applications so we use a naming scheme that allows us to extract the marathon application that is running the Chronos task.
23
+
24
+ An example of a Chronos Job name is the following.
25
+
26
+ `some-task-app2-11182015-1718-deployTasks-1-144786721`
27
+
28
+ With the default regex in the plugin we extract the following data and inject it into the event stream.
29
+
30
+ |`key`| Description | Value |
31
+ |---|---|---|
32
+ |`app`| The application name running the task | some-task-app2 |
33
+ | `mesos_framework` | chronos | chronos |
34
+ | `mesos_task_id` | The unique Mesos task id running the docker container | |
35
+ | `chronos_task_type` | The task type our application is running. We run deployment and scheduled tasks. | deployTasks |
36
+
37
+ ##Configuration
38
+
39
+ If your using the docker fluentd logging plugin your configuration should look something like this.
40
+
41
+ ```
42
+ <source>
43
+ type forward
44
+ port 24224
45
+ bind 0.0.0.0
46
+ </source>
47
+
48
+ <filter docker.*>
49
+ type mesosphere-filter
50
+ cache_size 1000
51
+ cache_ttl 3600
52
+ merge_json_log true
53
+ cronos_task_regex (?<app>[a-z0-9]([-a-z0-9]*[a-z0-9]))-(?<date>[^-]+)-(?<time>[^-]+)-(?<task_type>[^-]+)-(?<run>[^-]+)-(?<epoc>[^-]+)
54
+ </filter>
55
+
56
+ <match docker.*>
57
+ type stdout
58
+ </match>
59
+ ```
60
+
61
+ |`key`| Description | Default |
62
+ |---|---|---|
63
+ |`cache_size`| This plugin will cache information from the docker daemon. This configuration determine how large that cache is. | 1000 |
64
+ | `cache_ttl ` | How long to keep items in the cache. | 3600 |
65
+ | `merge_json_log ` | If your application logs in a valid json format, this will merge that into the event stream. | true |
66
+ | `cronos_task_regex ` | If you don't provide a valid regex here then you will only get the mesos_task id from until you create a standard chronos job name that is parseable by a ruby regex. | **See example above** |
67
+
68
+
69
+ ##Example
70
+ We have an application called `hello-world`. That application is deployed via marathon and has the following environment variables.
71
+
72
+ |`key`| value |
73
+ |---|---|
74
+ | `MESOS_TASK_ID` | `hello-world.14b0596d-93ea-11e5-a134-124eefe69197`|
75
+ | `MARATHON_APP_ID` | `/hello-world`|
76
+
77
+ Like all great hello world applications. This docker container just execute:
78
+
79
+ ```bash
80
+ echo '{"say":"Hello World"}'
81
+ ```
82
+
83
+ Without this filter fluentd would process the docker log and output the following.
84
+
85
+ ```json
86
+ {
87
+ "container_id": "ce327cb0de115f7dbfcd2c6055ba945436dada26035a62587b951332a028a530",
88
+ "container_name": "/some_random_meaningless_name",
89
+ "source": "stdout",
90
+ "log": "{\"say\":\"Hello World\"}\r"
91
+ }
92
+ ```
93
+
94
+ With the filter in place that log will become the following.
95
+
96
+ ```json
97
+ {
98
+ "container_id": "ce327cb0de115f7dbfcd2c6055ba945436dada26035a62587b951332a028a530",
99
+ "container_name": "/some_random_meaningless_name",
100
+ "say": "Hello World",
101
+ "mesos_framework": "marathon",
102
+ "app": "hello-world",
103
+ "mesos_task_id": "unquie_task_id",
104
+ "source": "stdout",
105
+ "log": "{\"say\":\"Hello World\"}\r"
106
+ }
107
+ ```
108
+
109
+ So now in Kibana you can filter on many more fields and more easily track down issues when `hello-world` may be running in multiple containers that are associated with different Mesos tasks.
@@ -4,7 +4,7 @@ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
4
4
 
5
5
  Gem::Specification.new do |gem|
6
6
  gem.name = 'fluent-plugin-mesosphere-filter'
7
- gem.version = '0.1.0'
7
+ gem.version = '0.1.1'
8
8
  gem.authors = ['Joseph Hughes']
9
9
  gem.email = ['jjhughes57@gmail.com']
10
10
  gem.description = 'Filter plugin to add Mesosphere metadata'
@@ -16,21 +16,26 @@
16
16
  # limitations under the License.
17
17
  #
18
18
  module Fluent
19
+ # Parses Marathon and Chronos data from docker to make fluentd logs more
20
+ # useful.
19
21
  class MesosphereFilter < Fluent::Filter
20
22
  Fluent::Plugin.register_filter('mesosphere-filter', self)
21
23
 
22
24
  config_param :cache_size, :integer, default: 1000
23
25
  config_param :cache_ttl, :integer, default: 60 * 60
26
+ config_param :get_container_id_tag, :bool, default: true
27
+ config_param :container_id_attr, :string, default: 'container_id'
24
28
 
25
29
  config_param :merge_json_log, :bool, default: true
26
30
  config_param :cronos_task_regex,
27
31
  :string,
28
- default: '(?<app>[a-z0-9]([-a-z0-9]*[a-z0-9]))-(?<date>[^-]+)-(?<time>[^-]+)-(?<task_type>[^-]+)-(?<run>[^-]+)-(?<epoc>[^-]+)'
32
+ default: '(?<app>[a-z0-9]([-a-z0-9_]*[a-z0-9_]))-(?<date>[^-]+)-(?<time>[^-]+)-(?<task_type>[^-]+)-(?<run>[^-]+)-(?<epoc>[^-]+)'
29
33
 
30
34
  def initialize
31
35
  super
32
36
  end
33
37
 
38
+ # Get the configuration for the plugin
34
39
  def configure(conf)
35
40
  super
36
41
 
@@ -43,53 +48,137 @@ module Fluent
43
48
  @cache = LruRedux::TTL::ThreadSafeCache.new(@cache_size, @cache_ttl)
44
49
 
45
50
  @chronos_task_regex_compiled = Regexp.compile(@cronos_task_regex)
51
+
52
+ marathon_regex = '\/(?<app>[a-z0-9]([-a-z0-9_]*[a-z0-9_]))'
53
+ @marathon_app_regex_compiled = Regexp.compile(marathon_regex)
46
54
  end
47
55
 
56
+ # Gets the log event stream and moifies it. This is where the plugin hooks
57
+ # into the fluentd envent stream.
48
58
  def filter_stream(tag, es)
49
59
  new_es = MultiEventStream.new
60
+ container_id = ''
50
61
 
51
- container_id = tag.split('.').last
52
- mesos_data = @cache.getset(container_id) do
53
- get_container_metadata(container_id)
54
- end
62
+ container_id = get_container_id_from_tag(tag) if get_container_id_tag
55
63
 
56
64
  es.each do |time, record|
57
- record = record.merge(mesos_data)
58
- record = merge_json_log(record) if @merge_json_log
59
- new_es.add(time, record)
65
+ container_id =
66
+ get_container_id_from_record(record) if container_id.empty?
67
+ next unless container_id
68
+ new_es.add(time, modify_record(record, get_mesos_data(container_id)))
60
69
  end
61
-
62
70
  new_es
63
71
  end
64
72
 
73
+ # Injects the meso framework data into the record and also merges
74
+ # the json log if that configuration is enabled.
75
+ #
76
+ # ==== Attributes:
77
+ # * +record+ - The log record being processed
78
+ # * +mesos_data+ - The mesos data retrived from the docker container
79
+ #
80
+ # ==== Returns:
81
+ # * A record hash that has mesos data and optinally log data added
82
+ def modify_record(record, mesos_data)
83
+ modified_record = record.merge(mesos_data)
84
+ modified_record = merge_json_log(modified_record) if @merge_json_log
85
+ modified_record
86
+ end
87
+
88
+ # Gets the mesos data about a container from the cache or calls the Docker
89
+ # api to retrieve the data about the container and store it in the cache.
90
+ #
91
+ # ==== Attributes:
92
+ # * +container_id+ - The container_id where the log record originated from.
93
+ # ==== Returns:
94
+ # * A hash of data that describes a mesos task
95
+ def get_mesos_data(container_id)
96
+ @cache.getset(container_id) do
97
+ get_container_metadata(container_id)
98
+ end
99
+ end
100
+
101
+ # Goes out to docker to get environment variables for a container.
102
+ # Then we parse the environment varibles looking for known Marathon
103
+ # and Chronos environment variables
104
+ #
105
+ # ==== Attributes:
106
+ # * +id+ - The id of the container to look at for mesosphere metadata.
107
+ # ==== Returns:
108
+ # * A hash that describes a mesos task gathered from the Docker API
65
109
  def get_container_metadata(id)
66
110
  task_data = {}
67
111
  container = Docker::Container.get(id)
68
112
  if container
69
113
  environment = container.json['Config']['Env']
70
114
  environment.each do |env|
115
+ # Chronos puts task_id in lowercase, and Marathon does it with
116
+ # uppercase
71
117
  if env =~ /MESOS_TASK_ID/i
72
118
  task_data['mesos_task_id'] = parse_env(env)
73
119
  elsif env.include? 'MARATHON_APP_ID'
120
+ match_data = parse_env(env).match(@marathon_app_regex_compiled)
74
121
  task_data['mesos_framework'] = 'marathon'
75
- task_data['app'] = parse_env(env)
122
+ task_data['app'] = match_data['app'] if match_data
76
123
  elsif env.include? 'CHRONOS_JOB_NAME'
77
124
  match_data = parse_env(env).match(@chronos_task_regex_compiled)
78
125
  task_data['mesos_framework'] = 'chronos'
79
- task_data['app'] = match_data['app']
80
- task_data['chronos_task_type'] = match_data['task_type']
126
+ task_data['app'] = match_data['app'] if match_data
127
+ task_data['chronos_task_type'] = match_data['task_type'] if match_data
81
128
  end
82
129
  end
83
130
  end
84
131
  task_data
85
132
  end
86
133
 
134
+ # Gets the container id from the last element in the tag. If the user has
135
+ # configured container_id_attr the container id can be gathered from the
136
+ # record if it has been inserted there.
137
+ #
138
+ # ==== Attributes:
139
+ # * +tag+ - The tag of the log being processed
140
+ # ==== Returns:
141
+ # * A docker container id
142
+ def get_container_id_from_tag(tag)
143
+ tag.split('.').last
144
+ end
145
+
146
+ # If the user has configured container_id_attr the container id can be
147
+ # gathered from the record if it has been inserted there. If no container_id
148
+ # can be found, the record is not processed.
149
+ #
150
+ # ==== Attributes::
151
+ # * +record+ - The record that is being transformed by the filter
152
+ # ==== Returns:
153
+ # * A docker container id
154
+ def get_container_id_from_record(record)
155
+ record[@container_id_attr]
156
+ end
157
+
158
+ # Split the env var on = and return the value
159
+ # ==== Attributes:
160
+ # * +env+ - The docker environment variable to parse to get the value.
161
+ # ==== Examples
162
+ # # For the env value MARATHON_APP_ID the actual string value given to us
163
+ # # by docker is 'MARATHON_APP_ID=some-app'. We want to return 'some-app'.
164
+ # ==== Returns:
165
+ # * The value of an environment varaible
87
166
  def parse_env(env)
88
167
  env.split('=').last
89
168
  end
90
169
 
170
+ # Look at the log value and if it is valid json then we will parse the json
171
+ # and merge it into the log record.
172
+ # ==== Attributes:
173
+ # * +record+ - The record we are transforming in the fluentd event stream.
174
+ # ==== Examples
175
+ # # Docker captures stdout and passes it in the 'log' record attribute.
176
+ # # We try to discover is the value of 'log' is json, if it is then we
177
+ # # will parse the json and add the keys and values to the record.
178
+ # ==== Returns:
179
+ # * A record hash that has json log data merged into the record
91
180
  def merge_json_log(record)
92
- if record.has_key?('log')
181
+ if record.key?('log')
93
182
  log = record['log'].strip
94
183
  if log[0].eql?('{') && log[-1].eql?('}')
95
184
  begin
@@ -0,0 +1,137 @@
1
+ {
2
+ "AppArmorProfile": "",
3
+ "Args": [
4
+ " task",
5
+ "task"
6
+ ],
7
+ "Config": {
8
+ "AttachStderr": false,
9
+ "AttachStdin": false,
10
+ "AttachStdout": false,
11
+ "Cmd": [
12
+ " task",
13
+ "task"
14
+ ],
15
+ "CpuShares": 153,
16
+ "Cpuset": "",
17
+ "Domainname": "",
18
+ "Entrypoint": [
19
+ "/minimal"
20
+ ],
21
+ "Env": [
22
+ "mesos_task_id=ct:1448508194000:0:recurring-transaction3:task",
23
+ "CHRONOS_JOB_OWNER=chartreuse-team@itriagehealth.com",
24
+ "CHRONOS_JOB_NAME=foobarzasdf",
25
+ "HOST=ip-10-10-30-60.ec2.internal",
26
+ "CHRONOS_RESOURCE_MEM=1024.0",
27
+ "CHRONOS_RESOURCE_CPU=0.05",
28
+ "CHRONOS_RESOURCE_DISK=256.0",
29
+ "YEAR=2016",
30
+ "VAULT_HOSTS=https://dev-use1a-pr-01-vault-vault-0001.dev-utopia.com:8200,https://dev-use1b-pr-01-vault-vault-0001.dev-utopia.com:8200,https://dev-use1e-pr-01-vault-vault-0001.dev-utopia.com:8200",
31
+ "MESOS_SANDBOX=/mnt/mesos/sandbox",
32
+ "PORT=80"
33
+ ],
34
+ "ExposedPorts": {
35
+ "80/tcp": {}
36
+ },
37
+ "Hostname": "ip-10-10-30-60",
38
+ "Image": "docker-dev.itriagehealth.com/utopia/recurring-transaction2-task:564a03fd52f1800005000003",
39
+ "Labels": {},
40
+ "MacAddress": "",
41
+ "Memory": 1107296256,
42
+ "MemorySwap": -1,
43
+ "NetworkDisabled": false,
44
+ "OnBuild": null,
45
+ "OpenStdin": false,
46
+ "PortSpecs": null,
47
+ "StdinOnce": false,
48
+ "Tty": false,
49
+ "User": "",
50
+ "Volumes": null,
51
+ "WorkingDir": "/"
52
+ },
53
+ "Created": "2015-11-26T03:23:15.958740506Z",
54
+ "Driver": "overlay",
55
+ "ExecDriver": "native-0.2",
56
+ "ExecIDs": null,
57
+ "HostConfig": {
58
+ "Binds": [
59
+ "/data/mesos-slave/slaves/20151115-193554-4078832138-5050-1872-S2/docker/links/2e3e0f5c-ac02-40a0-804b-b4c7736935b5:/mnt/mesos/sandbox"
60
+ ],
61
+ "CapAdd": null,
62
+ "CapDrop": null,
63
+ "CgroupParent": "",
64
+ "ContainerIDFile": "",
65
+ "CpuShares": 153,
66
+ "CpusetCpus": "",
67
+ "Devices": [],
68
+ "Dns": null,
69
+ "DnsSearch": null,
70
+ "ExtraHosts": null,
71
+ "IpcMode": "",
72
+ "Links": null,
73
+ "LogConfig": {
74
+ "Config": null,
75
+ "Type": "json-file"
76
+ },
77
+ "LxcConf": [],
78
+ "Memory": 1107296256,
79
+ "MemorySwap": -1,
80
+ "NetworkMode": "host",
81
+ "PidMode": "",
82
+ "PortBindings": {},
83
+ "Privileged": false,
84
+ "PublishAllPorts": false,
85
+ "ReadonlyRootfs": false,
86
+ "RestartPolicy": {
87
+ "MaximumRetryCount": 0,
88
+ "Name": "no"
89
+ },
90
+ "SecurityOpt": null,
91
+ "Ulimits": null,
92
+ "VolumesFrom": null
93
+ },
94
+ "HostnamePath": "/data/docker/containers/8dac46bdb4c099f8a12c12e06b104e9ca86b839e241b386bf3ddb537c9923642/hostname",
95
+ "HostsPath": "/data/docker/containers/8dac46bdb4c099f8a12c12e06b104e9ca86b839e241b386bf3ddb537c9923642/hosts",
96
+ "Id": "foobar124",
97
+ "Image": "dc41b56ccebd1433c6be3a92b838cc64c8c5f7088237a3da08a380cf43d6d6ac",
98
+ "LogPath": "/data/docker/containers/8dac46bdb4c099f8a12c12e06b104e9ca86b839e241b386bf3ddb537c9923642/8dac46bdb4c099f8a12c12e06b104e9ca86b839e241b386bf3ddb537c9923642-json.log",
99
+ "MountLabel": "",
100
+ "Name": "/mesos-2e3e0f5c-ac02-40a0-804b-b4c7736935b5",
101
+ "NetworkSettings": {
102
+ "Bridge": "",
103
+ "Gateway": "",
104
+ "GlobalIPv6Address": "",
105
+ "GlobalIPv6PrefixLen": 0,
106
+ "IPAddress": "",
107
+ "IPPrefixLen": 0,
108
+ "IPv6Gateway": "",
109
+ "LinkLocalIPv6Address": "",
110
+ "LinkLocalIPv6PrefixLen": 0,
111
+ "MacAddress": "",
112
+ "PortMapping": null,
113
+ "Ports": null
114
+ },
115
+ "Path": "/minimal",
116
+ "ProcessLabel": "",
117
+ "ResolvConfPath": "/data/docker/containers/8dac46bdb4c099f8a12c12e06b104e9ca86b839e241b386bf3ddb537c9923642/resolv.conf",
118
+ "RestartCount": 0,
119
+ "State": {
120
+ "Dead": false,
121
+ "Error": "",
122
+ "ExitCode": 0,
123
+ "FinishedAt": "2015-11-26T03:23:16.090077843Z",
124
+ "OOMKilled": false,
125
+ "Paused": false,
126
+ "Pid": 0,
127
+ "Restarting": false,
128
+ "Running": false,
129
+ "StartedAt": "2015-11-26T03:23:16.01885868Z"
130
+ },
131
+ "Volumes": {
132
+ "/mnt/mesos/sandbox": "/data/mesos-slave/slaves/20151115-193554-4078832138-5050-1872-S2/frameworks/20150821-174742-1460275722-5050-14777-0001/executors/ct:1448508194000:0:recurring-transaction2:task/runs/2e3e0f5c-ac02-40a0-804b-b4c7736935b5"
133
+ },
134
+ "VolumesRW": {
135
+ "/mnt/mesos/sandbox": true
136
+ }
137
+ }
@@ -118,8 +118,8 @@
118
118
  "StdinOnce": false,
119
119
  "Env": [
120
120
  "FLUENTD_OPT=-vv",
121
- "MESOS_TASK_ID=my-innovation-health-int-provider.14b0596d-93ea-11e5-a134-124eefe69197",
122
- "MARATHON_APP_ID=/my-innovation-health-int-provider",
121
+ "MESOS_TASK_ID=hello-world.14b0596d-93ea-11e5-a134-124eefe69197",
122
+ "MARATHON_APP_ID=/hello-world",
123
123
  "FLUENTD_CONF=test.conf",
124
124
  "PATH=/home/ubuntu/ruby/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
125
125
  "FLUENTD_OPT="
@@ -0,0 +1,187 @@
1
+ {
2
+ "Id": "somecontainer123",
3
+ "Created": "2015-11-26T06:57:56.560006039Z",
4
+ "Path": "/bin/sh",
5
+ "Args": [
6
+ "-c",
7
+ "exec fluentd -c /fluentd/etc/$FLUENTD_CONF -p /fluentd/plugins $FLUENTD_OPT"
8
+ ],
9
+ "State": {
10
+ "Status": "running",
11
+ "Running": true,
12
+ "Paused": false,
13
+ "Restarting": false,
14
+ "OOMKilled": false,
15
+ "Dead": false,
16
+ "Pid": 6563,
17
+ "ExitCode": 0,
18
+ "Error": "",
19
+ "StartedAt": "2015-11-26T07:00:45.707356204Z",
20
+ "FinishedAt": "2015-11-26T07:00:43.278299772Z"
21
+ },
22
+ "Image": "7cd5efd43c5097c8e6a9b6bdc050f887e0e1dee49a79b7eb920e2d741c36b3b5",
23
+ "ResolvConfPath": "/mnt/sda1/var/lib/docker/containers/110e3bd749e33e3b55bb55bec44a192457d722607afb1e6423e6a45a1c1e8b45/resolv.conf",
24
+ "HostnamePath": "/mnt/sda1/var/lib/docker/containers/110e3bd749e33e3b55bb55bec44a192457d722607afb1e6423e6a45a1c1e8b45/hostname",
25
+ "HostsPath": "/mnt/sda1/var/lib/docker/containers/110e3bd749e33e3b55bb55bec44a192457d722607afb1e6423e6a45a1c1e8b45/hosts",
26
+ "LogPath": "/mnt/sda1/var/lib/docker/containers/110e3bd749e33e3b55bb55bec44a192457d722607afb1e6423e6a45a1c1e8b45/110e3bd749e33e3b55bb55bec44a192457d722607afb1e6423e6a45a1c1e8b45-json.log",
27
+ "Name": "/fluentddockertest_fluentd_server_1",
28
+ "RestartCount": 0,
29
+ "Driver": "aufs",
30
+ "ExecDriver": "native-0.2",
31
+ "MountLabel": "",
32
+ "ProcessLabel": "",
33
+ "AppArmorProfile": "",
34
+ "ExecIDs": null,
35
+ "HostConfig": {
36
+ "Binds": [
37
+ "/Users/A658307/fluentd_docker_test/fluentd/etc:/fluentd/etc:rw",
38
+ "/Users/A658307/fluentd_docker_test/fluentd/logs:/fluentd/log:rw",
39
+ "/Users/A658307/fluentd_docker_test/fluentd/plugins:/fluentd/plugins:rw"
40
+ ],
41
+ "ContainerIDFile": "",
42
+ "LxcConf": null,
43
+ "Memory": 0,
44
+ "MemoryReservation": 0,
45
+ "MemorySwap": 0,
46
+ "KernelMemory": 0,
47
+ "CpuShares": 0,
48
+ "CpuPeriod": 0,
49
+ "CpusetCpus": "",
50
+ "CpusetMems": "",
51
+ "CpuQuota": 0,
52
+ "BlkioWeight": 0,
53
+ "OomKillDisable": false,
54
+ "MemorySwappiness": null,
55
+ "Privileged": false,
56
+ "PortBindings": {
57
+ "24224/tcp": [{
58
+ "HostIp": "",
59
+ "HostPort": "24224"
60
+ }]
61
+ },
62
+ "Links": null,
63
+ "PublishAllPorts": false,
64
+ "Dns": null,
65
+ "DnsOptions": null,
66
+ "DnsSearch": null,
67
+ "ExtraHosts": [],
68
+ "VolumesFrom": [],
69
+ "Devices": null,
70
+ "NetworkMode": "default",
71
+ "IpcMode": "",
72
+ "PidMode": "",
73
+ "UTSMode": "",
74
+ "CapAdd": null,
75
+ "CapDrop": null,
76
+ "GroupAdd": null,
77
+ "RestartPolicy": {
78
+ "Name": "",
79
+ "MaximumRetryCount": 0
80
+ },
81
+ "SecurityOpt": null,
82
+ "ReadonlyRootfs": false,
83
+ "Ulimits": null,
84
+ "LogConfig": {
85
+ "Type": "json-file",
86
+ "Config": {}
87
+ },
88
+ "CgroupParent": "",
89
+ "ConsoleSize": [
90
+ 0,
91
+ 0
92
+ ],
93
+ "VolumeDriver": ""
94
+ },
95
+ "GraphDriver": {
96
+ "Name": "aufs",
97
+ "Data": null
98
+ },
99
+ "Volumes": {
100
+ "/fluentd/etc": "/Users/A658307/fluentd_docker_test/fluentd/etc",
101
+ "/fluentd/log": "/Users/A658307/fluentd_docker_test/fluentd/logs",
102
+ "/fluentd/plugins": "/Users/A658307/fluentd_docker_test/fluentd/plugins"
103
+ },
104
+ "VolumesRW": {
105
+ "/fluentd/etc": true,
106
+ "/fluentd/log": true,
107
+ "/fluentd/plugins": true
108
+ },
109
+ "Config": {
110
+ "Hostname": "110e3bd749e3",
111
+ "Domainname": "",
112
+ "User": "ubuntu",
113
+ "AttachStdin": false,
114
+ "AttachStdout": false,
115
+ "AttachStderr": false,
116
+ "Tty": false,
117
+ "OpenStdin": false,
118
+ "StdinOnce": false,
119
+ "Env": [
120
+ "FLUENTD_OPT=-vv",
121
+ "MESOS_TASK_ID=hello-world.14b0596d-93ea-11e5-a134-124eefe69197",
122
+ "MARATHON_APP_ID=/hello-world",
123
+ "FLUENTD_CONF=test.conf",
124
+ "PATH=/home/ubuntu/ruby/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
125
+ "FLUENTD_OPT="
126
+ ],
127
+ "Cmd": [
128
+ "/bin/sh",
129
+ "-c",
130
+ "exec fluentd -c /fluentd/etc/$FLUENTD_CONF -p /fluentd/plugins $FLUENTD_OPT"
131
+ ],
132
+ "Image": "fluent/fluentd:latest",
133
+ "Volumes": {
134
+ "/fluentd/etc": {},
135
+ "/fluentd/log": {},
136
+ "/fluentd/plugins": {}
137
+ },
138
+ "WorkingDir": "/home/ubuntu",
139
+ "Entrypoint": null,
140
+ "OnBuild": null,
141
+ "Labels": {
142
+ "Description": "Fluentd docker image",
143
+ "Vendor": "Fluent Organization",
144
+ "Version": "1.0",
145
+ "com.docker.compose.config-hash": "7a03ce1e88388255c166fa62dcad89a43d43034ce605265094cf9c9951dcc9f0",
146
+ "com.docker.compose.container-number": "1",
147
+ "com.docker.compose.oneoff": "False",
148
+ "com.docker.compose.project": "fluentddockertest",
149
+ "com.docker.compose.service": "fluentd_server",
150
+ "com.docker.compose.version": "1.5.1"
151
+ },
152
+ "MacAddress": "",
153
+ "NetworkDisabled": false,
154
+ "ExposedPorts": {
155
+ "24224/tcp": {}
156
+ },
157
+ "VolumeDriver": "",
158
+ "Memory": 0,
159
+ "MemorySwap": 0,
160
+ "CpuShares": 0,
161
+ "Cpuset": ""
162
+ },
163
+ "NetworkSettings": {
164
+ "Bridge": "",
165
+ "SandboxID": "f304aa3b0e20fe1e0edb5984aedd03b7d5df999ff5ff9bd3ef8f831923e5f96a",
166
+ "HairpinMode": false,
167
+ "LinkLocalIPv6Address": "",
168
+ "LinkLocalIPv6PrefixLen": 0,
169
+ "Ports": {
170
+ "24224/tcp": [{
171
+ "HostIp": "0.0.0.0",
172
+ "HostPort": "24224"
173
+ }]
174
+ },
175
+ "SandboxKey": "/var/run/docker/netns/f304aa3b0e20",
176
+ "SecondaryIPAddresses": null,
177
+ "SecondaryIPv6Addresses": null,
178
+ "EndpointID": "7c6889977686db7c2b2042798b021a03df61c4273dcfd84fb2a9e5830fcb71a9",
179
+ "Gateway": "172.17.0.1",
180
+ "GlobalIPv6Address": "",
181
+ "GlobalIPv6PrefixLen": 0,
182
+ "IPAddress": "172.17.0.2",
183
+ "IPPrefixLen": 16,
184
+ "IPv6Gateway": "",
185
+ "MacAddress": "02:42:ac:11:00:02"
186
+ }
187
+ }
@@ -21,6 +21,10 @@ class AmplifierFilterTest < Test::Unit::TestCase
21
21
  cache_size 2000
22
22
  cache_ttl 300
23
23
  ]
24
+ CONFIG3 = %[
25
+ get_container_id_tag false
26
+ container_id_attr container_id
27
+ ]
24
28
 
25
29
  def create_driver(conf = CONFIG, tag = 'test')
26
30
  Fluent::Test::FilterTestDriver.new(Fluent::MesosphereFilter, tag).configure(conf)
@@ -31,9 +35,9 @@ class AmplifierFilterTest < Test::Unit::TestCase
31
35
  .to_return(status: 200, body: file, headers: {})
32
36
  end
33
37
 
34
- def setup_marathon_container
35
- docker_api_url = 'http://tcp//example.com:5422/v1.16/containers/foobar123/json'
36
- file = File.open('test/containers/marathon.json', 'rb')
38
+ def setup_marathon_container(container_id, file_name)
39
+ docker_api_url = "http://tcp//example.com:5422/v1.16/containers/#{container_id}/json"
40
+ file = File.open("test/containers/#{file_name}.json", 'rb')
37
41
  setup_docker_stub(file, docker_api_url)
38
42
  end
39
43
 
@@ -44,7 +48,7 @@ class AmplifierFilterTest < Test::Unit::TestCase
44
48
  end
45
49
 
46
50
  def test_marathon_filter
47
- setup_marathon_container
51
+ setup_marathon_container('foobar123', 'marathon')
48
52
 
49
53
  d1 = create_driver(CONFIG, 'docker.foobar123')
50
54
  d1.run do
@@ -52,17 +56,17 @@ class AmplifierFilterTest < Test::Unit::TestCase
52
56
  end
53
57
  filtered = d1.filtered_as_array
54
58
 
55
- task_id = 'my-innovation-health-int-provider.14b0596d-93ea-11e5-a134-124eefe69197'
59
+ task_id = 'hello-world.14b0596d-93ea-11e5-a134-124eefe69197'
56
60
 
57
61
  log_entry = filtered[0][2]
58
62
 
59
63
  assert_equal 'marathon', log_entry['mesos_framework']
60
- assert_equal '/my-innovation-health-int-provider', log_entry['app']
64
+ assert_equal 'hello-world', log_entry['app']
61
65
  assert_equal task_id, log_entry['mesos_task_id']
62
66
  end
63
67
 
64
68
  def test_container_cache
65
- setup_marathon_container
69
+ setup_marathon_container('foobar123', 'marathon')
66
70
 
67
71
  d1 = create_driver(CONFIG, 'docker.foobar123')
68
72
  d1.run do
@@ -77,7 +81,7 @@ class AmplifierFilterTest < Test::Unit::TestCase
77
81
  end
78
82
 
79
83
  def test_container_cache_expiration
80
- setup_marathon_container
84
+ setup_marathon_container('foobar123', 'marathon')
81
85
 
82
86
  d1 = create_driver(CONFIG2, 'docker.foobar123')
83
87
  d1.run do
@@ -115,6 +119,26 @@ class AmplifierFilterTest < Test::Unit::TestCase
115
119
  assert_equal 'deployTasks', log_entry['chronos_task_type']
116
120
  end
117
121
 
122
+ def test_chronos_bad_match
123
+ docker_api_url = 'http://tcp//example.com:5422/v1.16/containers/foobar124/json'
124
+ file = File.open('test/containers/chronos_bad.json', 'rb')
125
+ setup_docker_stub(file, docker_api_url)
126
+
127
+ d1 = create_driver(CONFIG, 'docker.foobar124')
128
+ d1.run do
129
+ d1.filter('log' => 'Hello World 1')
130
+ end
131
+ filtered = d1.filtered_as_array
132
+
133
+ task_id = 'ct:1448508194000:0:recurring-transaction3:task'
134
+ log_entry = filtered[0][2]
135
+
136
+ assert_equal 'chronos', log_entry['mesos_framework']
137
+ assert_equal task_id, log_entry['mesos_task_id']
138
+ refute log_entry['app']
139
+ refute log_entry['chronos_task_type']
140
+ end
141
+
118
142
  def test_merge_json
119
143
  setup_chronos_container
120
144
 
@@ -143,4 +167,21 @@ class AmplifierFilterTest < Test::Unit::TestCase
143
167
  assert_equal bad_json1, filtered[0][2]['log']
144
168
  assert_equal bad_json2, filtered[1][2]['log']
145
169
  end
170
+
171
+ def test_container_id_from_record
172
+ setup_marathon_container('somecontainer123', 'marathon2')
173
+
174
+ d1 = create_driver(CONFIG3, 'docker')
175
+ d1.run do
176
+ d1.filter('log' => 'hello_world', 'container_id' => 'somecontainer123')
177
+ end
178
+ filtered = d1.filtered_as_array
179
+
180
+ task_id = 'hello-world.14b0596d-93ea-11e5-a134-124eefe69197'
181
+ log_entry = filtered[0][2]
182
+
183
+ assert_equal 'marathon', log_entry['mesos_framework']
184
+ assert_equal 'hello-world', log_entry['app']
185
+ assert_equal task_id, log_entry['mesos_task_id']
186
+ end
146
187
  end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: fluent-plugin-mesosphere-filter
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.1.0
4
+ version: 0.1.1
5
5
  platform: ruby
6
6
  authors:
7
7
  - Joseph Hughes
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2015-11-26 00:00:00.000000000 Z
11
+ date: 2015-11-28 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: fluentd
@@ -195,7 +195,9 @@ files:
195
195
  - fluent-plugin-mesosphere-filter.gemspec
196
196
  - lib/fluent/plugin/mesosphere.rb
197
197
  - test/containers/chronos.json
198
+ - test/containers/chronos_bad.json
198
199
  - test/containers/marathon.json
200
+ - test/containers/marathon2.json
199
201
  - test/helper.rb
200
202
  - test/plugin/test_mesosphere.rb
201
203
  homepage: https://github.com/joshughes/fluent-plugin-mesosphere-filter
@@ -224,6 +226,8 @@ specification_version: 4
224
226
  summary: Filter plugin to add Mesosphere metadata to fluentd from Chronos and Marathon
225
227
  test_files:
226
228
  - test/containers/chronos.json
229
+ - test/containers/chronos_bad.json
227
230
  - test/containers/marathon.json
231
+ - test/containers/marathon2.json
228
232
  - test/helper.rb
229
233
  - test/plugin/test_mesosphere.rb