fluent-plugin-mesosphere-filter 0.1.0 → 0.1.1

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,15 +1,15 @@
1
1
  ---
2
2
  !binary "U0hBMQ==":
3
3
  metadata.gz: !binary |-
4
- ODM4MTU1YmFjNGUyYWZjZTIzYTI1ZDUxZTE3MjZiZDY3ZWNhMWViYQ==
4
+ NTcyNGFiYTRlM2JmMTE4NWU3MTRlNjVhNzUzM2Y0YTYxMjUwODQ4Mg==
5
5
  data.tar.gz: !binary |-
6
- YTZhNTVhZWRjZTE3ODMzNGYwMjM4YzU5ZGMyNDNjM2JmY2Q5ZWM3Mg==
6
+ NmRlODgyMDBhMWZlOTNhNTYyZmRjODA1OGI3YmJhMzUyYTkzMmExOQ==
7
7
  SHA512:
8
8
  metadata.gz: !binary |-
9
- MTdkNTFiMjVkZTBkMDdhZDczZjc3ZGZhNzkzZGEyMjFlNTExYTU4ODliNjhk
10
- ZGQ3NGU2NzdiYTZlZmY1NjYwM2MyOWY2ZDMwMTQ2ZDBjYzQyZGRmYzBkMjY2
11
- MmQxZTM0YzdhMWNjNTY2Y2VkYzkwZTBhYzNjNzEzYzcyMjZjOGU=
9
+ MTcyNjE1NGQxMzBkMzFlOGVkMWQ3ZDI2MjI4Yjk3MDU2MWY5ZGYxNzdjMDlh
10
+ YTU0NzM4ZmVjMWJkNGVmMjNkODk2ZjJiMTIxYTFjNTA0MzBkNzk5ZmY4YWU3
11
+ ZjA5ZTM2N2YzNTZkMDcyZDc0YTMyNzEwYWZlZTY3ZmZlOTk3OGM=
12
12
  data.tar.gz: !binary |-
13
- NTNkNTA4OTYwY2JkYzgwYjQyZWJkNmI1YmNmOTAwYTE2MDJjOTk3MTEyZmU3
14
- ZmJjOTA1ZDgyZTdjMWY5MDI5YTExNDhmM2FiNTY4ZmMxODJhMGNmMmNlMDRi
15
- MDgxNDY2MmE3MzQzNDA2ZTQ3NWIyYTBkZjA5NmMwYjU5YzM5MjY=
13
+ ZmY0ZDMxYjkxOTFlNGJhNzRlYTY5ODYxYmY4MjNhMTdhM2JiZjM4NGUzNWNl
14
+ MTBhMDUyMWRmOGY1YmFjZTcyYzQ1MzA3MDI5M2FmNzZhNjRkNjlmZWY4ZGNj
15
+ ZDc3YTA5NGU0ODRmNjdmNzhlMjZiZjU4NjIxOTM3YmFlYmY1NmY=
data/.gitignore CHANGED
@@ -17,3 +17,4 @@ test/version_tmp
17
17
  tmp
18
18
  .idea/
19
19
  *.iml
20
+ .DS_Store
data/README.md CHANGED
@@ -1,3 +1,109 @@
1
- #MEsos Stuff
1
+ #Mesosphere Fluentd Filter
2
2
  [![Code Climate](https://codeclimate.com/github/joshughes/fluent-plugin-mesosphere-filter/badges/gpa.svg)](https://codeclimate.com/github/joshughes/fluent-plugin-mesosphere-filter)
3
3
  [![Test Coverage](https://codeclimate.com/github/joshughes/fluent-plugin-mesosphere-filter/badges/coverage.svg)](https://codeclimate.com/github/joshughes/fluent-plugin-mesosphere-filter/coverage)
4
+ [![Gem Version](https://badge.fury.io/rb/fluent-plugin-mesosphere-filter.svg)](https://badge.fury.io/rb/fluent-plugin-mesosphere-filter)
5
+ [![Inline docs](http://inch-ci.org/github/joshughes/fluent-plugin-mesosphere-filter.svg?branch=master)](http://inch-ci.org/github/joshughes/fluent-plugin-mesosphere-filter)
6
+ [![Dependency Status](https://www.versioneye.com/user/projects/5658d10aaef3b5003e000000/badge.svg?style=flat)](https://www.versioneye.com/user/projects/5658d10aaef3b5003e000000)
7
+
8
+ Marathon, Chronos and Mesos combined can allow for teams to build a great solution for deploying and scaling containers. The issue is when you have a system like Kibana it can be hard to identify what container or task a log is coming from.
9
+
10
+ This filter aims to solve that issue by inspecting containers to inject Mesosphere metatdata into the Fluentd event stream.
11
+
12
+ ##What gets injected?
13
+ ###Marathon
14
+
15
+ |`key`| Description |
16
+ |---|---|
17
+ |`app`| The application name in marathon |
18
+ | `mesos_framework` | marathon |
19
+ | `mesos_task_id` | The unique Mesos task id running the docker container |
20
+
21
+ ###Chronos
22
+ Chronos does not have the idea of an 'application'. Just jobs. We run a lot of jobs that relate to our applications so we use a naming scheme that allows us to extract the marathon application that is running the Chronos task.
23
+
24
+ An example of a Chronos Job name is the following.
25
+
26
+ `some-task-app2-11182015-1718-deployTasks-1-144786721`
27
+
28
+ With the default regex in the plugin we extract the following data and inject it into the event stream.
29
+
30
+ |`key`| Description | Value |
31
+ |---|---|---|
32
+ |`app`| The application name running the task | some-task-app2 |
33
+ | `mesos_framework` | chronos | chronos |
34
+ | `mesos_task_id` | The unique Mesos task id running the docker container | |
35
+ | `chronos_task_type` | The task type our application is running. We run deployment and scheduled tasks. | deployTasks |
36
+
37
+ ##Configuration
38
+
39
+ If your using the docker fluentd logging plugin your configuration should look something like this.
40
+
41
+ ```
42
+ <source>
43
+ type forward
44
+ port 24224
45
+ bind 0.0.0.0
46
+ </source>
47
+
48
+ <filter docker.*>
49
+ type mesosphere-filter
50
+ cache_size 1000
51
+ cache_ttl 3600
52
+ merge_json_log true
53
+ cronos_task_regex (?<app>[a-z0-9]([-a-z0-9]*[a-z0-9]))-(?<date>[^-]+)-(?<time>[^-]+)-(?<task_type>[^-]+)-(?<run>[^-]+)-(?<epoc>[^-]+)
54
+ </filter>
55
+
56
+ <match docker.*>
57
+ type stdout
58
+ </match>
59
+ ```
60
+
61
+ |`key`| Description | Default |
62
+ |---|---|---|
63
+ |`cache_size`| This plugin will cache information from the docker daemon. This configuration determine how large that cache is. | 1000 |
64
+ | `cache_ttl ` | How long to keep items in the cache. | 3600 |
65
+ | `merge_json_log ` | If your application logs in a valid json format, this will merge that into the event stream. | true |
66
+ | `cronos_task_regex ` | If you don't provide a valid regex here then you will only get the mesos_task id from until you create a standard chronos job name that is parseable by a ruby regex. | **See example above** |
67
+
68
+
69
+ ##Example
70
+ We have an application called `hello-world`. That application is deployed via marathon and has the following environment variables.
71
+
72
+ |`key`| value |
73
+ |---|---|
74
+ | `MESOS_TASK_ID` | `hello-world.14b0596d-93ea-11e5-a134-124eefe69197`|
75
+ | `MARATHON_APP_ID` | `/hello-world`|
76
+
77
+ Like all great hello world applications. This docker container just execute:
78
+
79
+ ```bash
80
+ echo '{"say":"Hello World"}'
81
+ ```
82
+
83
+ Without this filter fluentd would process the docker log and output the following.
84
+
85
+ ```json
86
+ {
87
+ "container_id": "ce327cb0de115f7dbfcd2c6055ba945436dada26035a62587b951332a028a530",
88
+ "container_name": "/some_random_meaningless_name",
89
+ "source": "stdout",
90
+ "log": "{\"say\":\"Hello World\"}\r"
91
+ }
92
+ ```
93
+
94
+ With the filter in place that log will become the following.
95
+
96
+ ```json
97
+ {
98
+ "container_id": "ce327cb0de115f7dbfcd2c6055ba945436dada26035a62587b951332a028a530",
99
+ "container_name": "/some_random_meaningless_name",
100
+ "say": "Hello World",
101
+ "mesos_framework": "marathon",
102
+ "app": "hello-world",
103
+ "mesos_task_id": "unquie_task_id",
104
+ "source": "stdout",
105
+ "log": "{\"say\":\"Hello World\"}\r"
106
+ }
107
+ ```
108
+
109
+ So now in Kibana you can filter on many more fields and more easily track down issues when `hello-world` may be running in multiple containers that are associated with different Mesos tasks.
@@ -4,7 +4,7 @@ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
4
4
 
5
5
  Gem::Specification.new do |gem|
6
6
  gem.name = 'fluent-plugin-mesosphere-filter'
7
- gem.version = '0.1.0'
7
+ gem.version = '0.1.1'
8
8
  gem.authors = ['Joseph Hughes']
9
9
  gem.email = ['jjhughes57@gmail.com']
10
10
  gem.description = 'Filter plugin to add Mesosphere metadata'
@@ -16,21 +16,26 @@
16
16
  # limitations under the License.
17
17
  #
18
18
  module Fluent
19
+ # Parses Marathon and Chronos data from docker to make fluentd logs more
20
+ # useful.
19
21
  class MesosphereFilter < Fluent::Filter
20
22
  Fluent::Plugin.register_filter('mesosphere-filter', self)
21
23
 
22
24
  config_param :cache_size, :integer, default: 1000
23
25
  config_param :cache_ttl, :integer, default: 60 * 60
26
+ config_param :get_container_id_tag, :bool, default: true
27
+ config_param :container_id_attr, :string, default: 'container_id'
24
28
 
25
29
  config_param :merge_json_log, :bool, default: true
26
30
  config_param :cronos_task_regex,
27
31
  :string,
28
- default: '(?<app>[a-z0-9]([-a-z0-9]*[a-z0-9]))-(?<date>[^-]+)-(?<time>[^-]+)-(?<task_type>[^-]+)-(?<run>[^-]+)-(?<epoc>[^-]+)'
32
+ default: '(?<app>[a-z0-9]([-a-z0-9_]*[a-z0-9_]))-(?<date>[^-]+)-(?<time>[^-]+)-(?<task_type>[^-]+)-(?<run>[^-]+)-(?<epoc>[^-]+)'
29
33
 
30
34
  def initialize
31
35
  super
32
36
  end
33
37
 
38
+ # Get the configuration for the plugin
34
39
  def configure(conf)
35
40
  super
36
41
 
@@ -43,53 +48,137 @@ module Fluent
43
48
  @cache = LruRedux::TTL::ThreadSafeCache.new(@cache_size, @cache_ttl)
44
49
 
45
50
  @chronos_task_regex_compiled = Regexp.compile(@cronos_task_regex)
51
+
52
+ marathon_regex = '\/(?<app>[a-z0-9]([-a-z0-9_]*[a-z0-9_]))'
53
+ @marathon_app_regex_compiled = Regexp.compile(marathon_regex)
46
54
  end
47
55
 
56
+ # Gets the log event stream and moifies it. This is where the plugin hooks
57
+ # into the fluentd envent stream.
48
58
  def filter_stream(tag, es)
49
59
  new_es = MultiEventStream.new
60
+ container_id = ''
50
61
 
51
- container_id = tag.split('.').last
52
- mesos_data = @cache.getset(container_id) do
53
- get_container_metadata(container_id)
54
- end
62
+ container_id = get_container_id_from_tag(tag) if get_container_id_tag
55
63
 
56
64
  es.each do |time, record|
57
- record = record.merge(mesos_data)
58
- record = merge_json_log(record) if @merge_json_log
59
- new_es.add(time, record)
65
+ container_id =
66
+ get_container_id_from_record(record) if container_id.empty?
67
+ next unless container_id
68
+ new_es.add(time, modify_record(record, get_mesos_data(container_id)))
60
69
  end
61
-
62
70
  new_es
63
71
  end
64
72
 
73
+ # Injects the meso framework data into the record and also merges
74
+ # the json log if that configuration is enabled.
75
+ #
76
+ # ==== Attributes:
77
+ # * +record+ - The log record being processed
78
+ # * +mesos_data+ - The mesos data retrived from the docker container
79
+ #
80
+ # ==== Returns:
81
+ # * A record hash that has mesos data and optinally log data added
82
+ def modify_record(record, mesos_data)
83
+ modified_record = record.merge(mesos_data)
84
+ modified_record = merge_json_log(modified_record) if @merge_json_log
85
+ modified_record
86
+ end
87
+
88
+ # Gets the mesos data about a container from the cache or calls the Docker
89
+ # api to retrieve the data about the container and store it in the cache.
90
+ #
91
+ # ==== Attributes:
92
+ # * +container_id+ - The container_id where the log record originated from.
93
+ # ==== Returns:
94
+ # * A hash of data that describes a mesos task
95
+ def get_mesos_data(container_id)
96
+ @cache.getset(container_id) do
97
+ get_container_metadata(container_id)
98
+ end
99
+ end
100
+
101
+ # Goes out to docker to get environment variables for a container.
102
+ # Then we parse the environment varibles looking for known Marathon
103
+ # and Chronos environment variables
104
+ #
105
+ # ==== Attributes:
106
+ # * +id+ - The id of the container to look at for mesosphere metadata.
107
+ # ==== Returns:
108
+ # * A hash that describes a mesos task gathered from the Docker API
65
109
  def get_container_metadata(id)
66
110
  task_data = {}
67
111
  container = Docker::Container.get(id)
68
112
  if container
69
113
  environment = container.json['Config']['Env']
70
114
  environment.each do |env|
115
+ # Chronos puts task_id in lowercase, and Marathon does it with
116
+ # uppercase
71
117
  if env =~ /MESOS_TASK_ID/i
72
118
  task_data['mesos_task_id'] = parse_env(env)
73
119
  elsif env.include? 'MARATHON_APP_ID'
120
+ match_data = parse_env(env).match(@marathon_app_regex_compiled)
74
121
  task_data['mesos_framework'] = 'marathon'
75
- task_data['app'] = parse_env(env)
122
+ task_data['app'] = match_data['app'] if match_data
76
123
  elsif env.include? 'CHRONOS_JOB_NAME'
77
124
  match_data = parse_env(env).match(@chronos_task_regex_compiled)
78
125
  task_data['mesos_framework'] = 'chronos'
79
- task_data['app'] = match_data['app']
80
- task_data['chronos_task_type'] = match_data['task_type']
126
+ task_data['app'] = match_data['app'] if match_data
127
+ task_data['chronos_task_type'] = match_data['task_type'] if match_data
81
128
  end
82
129
  end
83
130
  end
84
131
  task_data
85
132
  end
86
133
 
134
+ # Gets the container id from the last element in the tag. If the user has
135
+ # configured container_id_attr the container id can be gathered from the
136
+ # record if it has been inserted there.
137
+ #
138
+ # ==== Attributes:
139
+ # * +tag+ - The tag of the log being processed
140
+ # ==== Returns:
141
+ # * A docker container id
142
+ def get_container_id_from_tag(tag)
143
+ tag.split('.').last
144
+ end
145
+
146
+ # If the user has configured container_id_attr the container id can be
147
+ # gathered from the record if it has been inserted there. If no container_id
148
+ # can be found, the record is not processed.
149
+ #
150
+ # ==== Attributes::
151
+ # * +record+ - The record that is being transformed by the filter
152
+ # ==== Returns:
153
+ # * A docker container id
154
+ def get_container_id_from_record(record)
155
+ record[@container_id_attr]
156
+ end
157
+
158
+ # Split the env var on = and return the value
159
+ # ==== Attributes:
160
+ # * +env+ - The docker environment variable to parse to get the value.
161
+ # ==== Examples
162
+ # # For the env value MARATHON_APP_ID the actual string value given to us
163
+ # # by docker is 'MARATHON_APP_ID=some-app'. We want to return 'some-app'.
164
+ # ==== Returns:
165
+ # * The value of an environment varaible
87
166
  def parse_env(env)
88
167
  env.split('=').last
89
168
  end
90
169
 
170
+ # Look at the log value and if it is valid json then we will parse the json
171
+ # and merge it into the log record.
172
+ # ==== Attributes:
173
+ # * +record+ - The record we are transforming in the fluentd event stream.
174
+ # ==== Examples
175
+ # # Docker captures stdout and passes it in the 'log' record attribute.
176
+ # # We try to discover is the value of 'log' is json, if it is then we
177
+ # # will parse the json and add the keys and values to the record.
178
+ # ==== Returns:
179
+ # * A record hash that has json log data merged into the record
91
180
  def merge_json_log(record)
92
- if record.has_key?('log')
181
+ if record.key?('log')
93
182
  log = record['log'].strip
94
183
  if log[0].eql?('{') && log[-1].eql?('}')
95
184
  begin
@@ -0,0 +1,137 @@
1
+ {
2
+ "AppArmorProfile": "",
3
+ "Args": [
4
+ " task",
5
+ "task"
6
+ ],
7
+ "Config": {
8
+ "AttachStderr": false,
9
+ "AttachStdin": false,
10
+ "AttachStdout": false,
11
+ "Cmd": [
12
+ " task",
13
+ "task"
14
+ ],
15
+ "CpuShares": 153,
16
+ "Cpuset": "",
17
+ "Domainname": "",
18
+ "Entrypoint": [
19
+ "/minimal"
20
+ ],
21
+ "Env": [
22
+ "mesos_task_id=ct:1448508194000:0:recurring-transaction3:task",
23
+ "CHRONOS_JOB_OWNER=chartreuse-team@itriagehealth.com",
24
+ "CHRONOS_JOB_NAME=foobarzasdf",
25
+ "HOST=ip-10-10-30-60.ec2.internal",
26
+ "CHRONOS_RESOURCE_MEM=1024.0",
27
+ "CHRONOS_RESOURCE_CPU=0.05",
28
+ "CHRONOS_RESOURCE_DISK=256.0",
29
+ "YEAR=2016",
30
+ "VAULT_HOSTS=https://dev-use1a-pr-01-vault-vault-0001.dev-utopia.com:8200,https://dev-use1b-pr-01-vault-vault-0001.dev-utopia.com:8200,https://dev-use1e-pr-01-vault-vault-0001.dev-utopia.com:8200",
31
+ "MESOS_SANDBOX=/mnt/mesos/sandbox",
32
+ "PORT=80"
33
+ ],
34
+ "ExposedPorts": {
35
+ "80/tcp": {}
36
+ },
37
+ "Hostname": "ip-10-10-30-60",
38
+ "Image": "docker-dev.itriagehealth.com/utopia/recurring-transaction2-task:564a03fd52f1800005000003",
39
+ "Labels": {},
40
+ "MacAddress": "",
41
+ "Memory": 1107296256,
42
+ "MemorySwap": -1,
43
+ "NetworkDisabled": false,
44
+ "OnBuild": null,
45
+ "OpenStdin": false,
46
+ "PortSpecs": null,
47
+ "StdinOnce": false,
48
+ "Tty": false,
49
+ "User": "",
50
+ "Volumes": null,
51
+ "WorkingDir": "/"
52
+ },
53
+ "Created": "2015-11-26T03:23:15.958740506Z",
54
+ "Driver": "overlay",
55
+ "ExecDriver": "native-0.2",
56
+ "ExecIDs": null,
57
+ "HostConfig": {
58
+ "Binds": [
59
+ "/data/mesos-slave/slaves/20151115-193554-4078832138-5050-1872-S2/docker/links/2e3e0f5c-ac02-40a0-804b-b4c7736935b5:/mnt/mesos/sandbox"
60
+ ],
61
+ "CapAdd": null,
62
+ "CapDrop": null,
63
+ "CgroupParent": "",
64
+ "ContainerIDFile": "",
65
+ "CpuShares": 153,
66
+ "CpusetCpus": "",
67
+ "Devices": [],
68
+ "Dns": null,
69
+ "DnsSearch": null,
70
+ "ExtraHosts": null,
71
+ "IpcMode": "",
72
+ "Links": null,
73
+ "LogConfig": {
74
+ "Config": null,
75
+ "Type": "json-file"
76
+ },
77
+ "LxcConf": [],
78
+ "Memory": 1107296256,
79
+ "MemorySwap": -1,
80
+ "NetworkMode": "host",
81
+ "PidMode": "",
82
+ "PortBindings": {},
83
+ "Privileged": false,
84
+ "PublishAllPorts": false,
85
+ "ReadonlyRootfs": false,
86
+ "RestartPolicy": {
87
+ "MaximumRetryCount": 0,
88
+ "Name": "no"
89
+ },
90
+ "SecurityOpt": null,
91
+ "Ulimits": null,
92
+ "VolumesFrom": null
93
+ },
94
+ "HostnamePath": "/data/docker/containers/8dac46bdb4c099f8a12c12e06b104e9ca86b839e241b386bf3ddb537c9923642/hostname",
95
+ "HostsPath": "/data/docker/containers/8dac46bdb4c099f8a12c12e06b104e9ca86b839e241b386bf3ddb537c9923642/hosts",
96
+ "Id": "foobar124",
97
+ "Image": "dc41b56ccebd1433c6be3a92b838cc64c8c5f7088237a3da08a380cf43d6d6ac",
98
+ "LogPath": "/data/docker/containers/8dac46bdb4c099f8a12c12e06b104e9ca86b839e241b386bf3ddb537c9923642/8dac46bdb4c099f8a12c12e06b104e9ca86b839e241b386bf3ddb537c9923642-json.log",
99
+ "MountLabel": "",
100
+ "Name": "/mesos-2e3e0f5c-ac02-40a0-804b-b4c7736935b5",
101
+ "NetworkSettings": {
102
+ "Bridge": "",
103
+ "Gateway": "",
104
+ "GlobalIPv6Address": "",
105
+ "GlobalIPv6PrefixLen": 0,
106
+ "IPAddress": "",
107
+ "IPPrefixLen": 0,
108
+ "IPv6Gateway": "",
109
+ "LinkLocalIPv6Address": "",
110
+ "LinkLocalIPv6PrefixLen": 0,
111
+ "MacAddress": "",
112
+ "PortMapping": null,
113
+ "Ports": null
114
+ },
115
+ "Path": "/minimal",
116
+ "ProcessLabel": "",
117
+ "ResolvConfPath": "/data/docker/containers/8dac46bdb4c099f8a12c12e06b104e9ca86b839e241b386bf3ddb537c9923642/resolv.conf",
118
+ "RestartCount": 0,
119
+ "State": {
120
+ "Dead": false,
121
+ "Error": "",
122
+ "ExitCode": 0,
123
+ "FinishedAt": "2015-11-26T03:23:16.090077843Z",
124
+ "OOMKilled": false,
125
+ "Paused": false,
126
+ "Pid": 0,
127
+ "Restarting": false,
128
+ "Running": false,
129
+ "StartedAt": "2015-11-26T03:23:16.01885868Z"
130
+ },
131
+ "Volumes": {
132
+ "/mnt/mesos/sandbox": "/data/mesos-slave/slaves/20151115-193554-4078832138-5050-1872-S2/frameworks/20150821-174742-1460275722-5050-14777-0001/executors/ct:1448508194000:0:recurring-transaction2:task/runs/2e3e0f5c-ac02-40a0-804b-b4c7736935b5"
133
+ },
134
+ "VolumesRW": {
135
+ "/mnt/mesos/sandbox": true
136
+ }
137
+ }
@@ -118,8 +118,8 @@
118
118
  "StdinOnce": false,
119
119
  "Env": [
120
120
  "FLUENTD_OPT=-vv",
121
- "MESOS_TASK_ID=my-innovation-health-int-provider.14b0596d-93ea-11e5-a134-124eefe69197",
122
- "MARATHON_APP_ID=/my-innovation-health-int-provider",
121
+ "MESOS_TASK_ID=hello-world.14b0596d-93ea-11e5-a134-124eefe69197",
122
+ "MARATHON_APP_ID=/hello-world",
123
123
  "FLUENTD_CONF=test.conf",
124
124
  "PATH=/home/ubuntu/ruby/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
125
125
  "FLUENTD_OPT="
@@ -0,0 +1,187 @@
1
+ {
2
+ "Id": "somecontainer123",
3
+ "Created": "2015-11-26T06:57:56.560006039Z",
4
+ "Path": "/bin/sh",
5
+ "Args": [
6
+ "-c",
7
+ "exec fluentd -c /fluentd/etc/$FLUENTD_CONF -p /fluentd/plugins $FLUENTD_OPT"
8
+ ],
9
+ "State": {
10
+ "Status": "running",
11
+ "Running": true,
12
+ "Paused": false,
13
+ "Restarting": false,
14
+ "OOMKilled": false,
15
+ "Dead": false,
16
+ "Pid": 6563,
17
+ "ExitCode": 0,
18
+ "Error": "",
19
+ "StartedAt": "2015-11-26T07:00:45.707356204Z",
20
+ "FinishedAt": "2015-11-26T07:00:43.278299772Z"
21
+ },
22
+ "Image": "7cd5efd43c5097c8e6a9b6bdc050f887e0e1dee49a79b7eb920e2d741c36b3b5",
23
+ "ResolvConfPath": "/mnt/sda1/var/lib/docker/containers/110e3bd749e33e3b55bb55bec44a192457d722607afb1e6423e6a45a1c1e8b45/resolv.conf",
24
+ "HostnamePath": "/mnt/sda1/var/lib/docker/containers/110e3bd749e33e3b55bb55bec44a192457d722607afb1e6423e6a45a1c1e8b45/hostname",
25
+ "HostsPath": "/mnt/sda1/var/lib/docker/containers/110e3bd749e33e3b55bb55bec44a192457d722607afb1e6423e6a45a1c1e8b45/hosts",
26
+ "LogPath": "/mnt/sda1/var/lib/docker/containers/110e3bd749e33e3b55bb55bec44a192457d722607afb1e6423e6a45a1c1e8b45/110e3bd749e33e3b55bb55bec44a192457d722607afb1e6423e6a45a1c1e8b45-json.log",
27
+ "Name": "/fluentddockertest_fluentd_server_1",
28
+ "RestartCount": 0,
29
+ "Driver": "aufs",
30
+ "ExecDriver": "native-0.2",
31
+ "MountLabel": "",
32
+ "ProcessLabel": "",
33
+ "AppArmorProfile": "",
34
+ "ExecIDs": null,
35
+ "HostConfig": {
36
+ "Binds": [
37
+ "/Users/A658307/fluentd_docker_test/fluentd/etc:/fluentd/etc:rw",
38
+ "/Users/A658307/fluentd_docker_test/fluentd/logs:/fluentd/log:rw",
39
+ "/Users/A658307/fluentd_docker_test/fluentd/plugins:/fluentd/plugins:rw"
40
+ ],
41
+ "ContainerIDFile": "",
42
+ "LxcConf": null,
43
+ "Memory": 0,
44
+ "MemoryReservation": 0,
45
+ "MemorySwap": 0,
46
+ "KernelMemory": 0,
47
+ "CpuShares": 0,
48
+ "CpuPeriod": 0,
49
+ "CpusetCpus": "",
50
+ "CpusetMems": "",
51
+ "CpuQuota": 0,
52
+ "BlkioWeight": 0,
53
+ "OomKillDisable": false,
54
+ "MemorySwappiness": null,
55
+ "Privileged": false,
56
+ "PortBindings": {
57
+ "24224/tcp": [{
58
+ "HostIp": "",
59
+ "HostPort": "24224"
60
+ }]
61
+ },
62
+ "Links": null,
63
+ "PublishAllPorts": false,
64
+ "Dns": null,
65
+ "DnsOptions": null,
66
+ "DnsSearch": null,
67
+ "ExtraHosts": [],
68
+ "VolumesFrom": [],
69
+ "Devices": null,
70
+ "NetworkMode": "default",
71
+ "IpcMode": "",
72
+ "PidMode": "",
73
+ "UTSMode": "",
74
+ "CapAdd": null,
75
+ "CapDrop": null,
76
+ "GroupAdd": null,
77
+ "RestartPolicy": {
78
+ "Name": "",
79
+ "MaximumRetryCount": 0
80
+ },
81
+ "SecurityOpt": null,
82
+ "ReadonlyRootfs": false,
83
+ "Ulimits": null,
84
+ "LogConfig": {
85
+ "Type": "json-file",
86
+ "Config": {}
87
+ },
88
+ "CgroupParent": "",
89
+ "ConsoleSize": [
90
+ 0,
91
+ 0
92
+ ],
93
+ "VolumeDriver": ""
94
+ },
95
+ "GraphDriver": {
96
+ "Name": "aufs",
97
+ "Data": null
98
+ },
99
+ "Volumes": {
100
+ "/fluentd/etc": "/Users/A658307/fluentd_docker_test/fluentd/etc",
101
+ "/fluentd/log": "/Users/A658307/fluentd_docker_test/fluentd/logs",
102
+ "/fluentd/plugins": "/Users/A658307/fluentd_docker_test/fluentd/plugins"
103
+ },
104
+ "VolumesRW": {
105
+ "/fluentd/etc": true,
106
+ "/fluentd/log": true,
107
+ "/fluentd/plugins": true
108
+ },
109
+ "Config": {
110
+ "Hostname": "110e3bd749e3",
111
+ "Domainname": "",
112
+ "User": "ubuntu",
113
+ "AttachStdin": false,
114
+ "AttachStdout": false,
115
+ "AttachStderr": false,
116
+ "Tty": false,
117
+ "OpenStdin": false,
118
+ "StdinOnce": false,
119
+ "Env": [
120
+ "FLUENTD_OPT=-vv",
121
+ "MESOS_TASK_ID=hello-world.14b0596d-93ea-11e5-a134-124eefe69197",
122
+ "MARATHON_APP_ID=/hello-world",
123
+ "FLUENTD_CONF=test.conf",
124
+ "PATH=/home/ubuntu/ruby/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin",
125
+ "FLUENTD_OPT="
126
+ ],
127
+ "Cmd": [
128
+ "/bin/sh",
129
+ "-c",
130
+ "exec fluentd -c /fluentd/etc/$FLUENTD_CONF -p /fluentd/plugins $FLUENTD_OPT"
131
+ ],
132
+ "Image": "fluent/fluentd:latest",
133
+ "Volumes": {
134
+ "/fluentd/etc": {},
135
+ "/fluentd/log": {},
136
+ "/fluentd/plugins": {}
137
+ },
138
+ "WorkingDir": "/home/ubuntu",
139
+ "Entrypoint": null,
140
+ "OnBuild": null,
141
+ "Labels": {
142
+ "Description": "Fluentd docker image",
143
+ "Vendor": "Fluent Organization",
144
+ "Version": "1.0",
145
+ "com.docker.compose.config-hash": "7a03ce1e88388255c166fa62dcad89a43d43034ce605265094cf9c9951dcc9f0",
146
+ "com.docker.compose.container-number": "1",
147
+ "com.docker.compose.oneoff": "False",
148
+ "com.docker.compose.project": "fluentddockertest",
149
+ "com.docker.compose.service": "fluentd_server",
150
+ "com.docker.compose.version": "1.5.1"
151
+ },
152
+ "MacAddress": "",
153
+ "NetworkDisabled": false,
154
+ "ExposedPorts": {
155
+ "24224/tcp": {}
156
+ },
157
+ "VolumeDriver": "",
158
+ "Memory": 0,
159
+ "MemorySwap": 0,
160
+ "CpuShares": 0,
161
+ "Cpuset": ""
162
+ },
163
+ "NetworkSettings": {
164
+ "Bridge": "",
165
+ "SandboxID": "f304aa3b0e20fe1e0edb5984aedd03b7d5df999ff5ff9bd3ef8f831923e5f96a",
166
+ "HairpinMode": false,
167
+ "LinkLocalIPv6Address": "",
168
+ "LinkLocalIPv6PrefixLen": 0,
169
+ "Ports": {
170
+ "24224/tcp": [{
171
+ "HostIp": "0.0.0.0",
172
+ "HostPort": "24224"
173
+ }]
174
+ },
175
+ "SandboxKey": "/var/run/docker/netns/f304aa3b0e20",
176
+ "SecondaryIPAddresses": null,
177
+ "SecondaryIPv6Addresses": null,
178
+ "EndpointID": "7c6889977686db7c2b2042798b021a03df61c4273dcfd84fb2a9e5830fcb71a9",
179
+ "Gateway": "172.17.0.1",
180
+ "GlobalIPv6Address": "",
181
+ "GlobalIPv6PrefixLen": 0,
182
+ "IPAddress": "172.17.0.2",
183
+ "IPPrefixLen": 16,
184
+ "IPv6Gateway": "",
185
+ "MacAddress": "02:42:ac:11:00:02"
186
+ }
187
+ }
@@ -21,6 +21,10 @@ class AmplifierFilterTest < Test::Unit::TestCase
21
21
  cache_size 2000
22
22
  cache_ttl 300
23
23
  ]
24
+ CONFIG3 = %[
25
+ get_container_id_tag false
26
+ container_id_attr container_id
27
+ ]
24
28
 
25
29
  def create_driver(conf = CONFIG, tag = 'test')
26
30
  Fluent::Test::FilterTestDriver.new(Fluent::MesosphereFilter, tag).configure(conf)
@@ -31,9 +35,9 @@ class AmplifierFilterTest < Test::Unit::TestCase
31
35
  .to_return(status: 200, body: file, headers: {})
32
36
  end
33
37
 
34
- def setup_marathon_container
35
- docker_api_url = 'http://tcp//example.com:5422/v1.16/containers/foobar123/json'
36
- file = File.open('test/containers/marathon.json', 'rb')
38
+ def setup_marathon_container(container_id, file_name)
39
+ docker_api_url = "http://tcp//example.com:5422/v1.16/containers/#{container_id}/json"
40
+ file = File.open("test/containers/#{file_name}.json", 'rb')
37
41
  setup_docker_stub(file, docker_api_url)
38
42
  end
39
43
 
@@ -44,7 +48,7 @@ class AmplifierFilterTest < Test::Unit::TestCase
44
48
  end
45
49
 
46
50
  def test_marathon_filter
47
- setup_marathon_container
51
+ setup_marathon_container('foobar123', 'marathon')
48
52
 
49
53
  d1 = create_driver(CONFIG, 'docker.foobar123')
50
54
  d1.run do
@@ -52,17 +56,17 @@ class AmplifierFilterTest < Test::Unit::TestCase
52
56
  end
53
57
  filtered = d1.filtered_as_array
54
58
 
55
- task_id = 'my-innovation-health-int-provider.14b0596d-93ea-11e5-a134-124eefe69197'
59
+ task_id = 'hello-world.14b0596d-93ea-11e5-a134-124eefe69197'
56
60
 
57
61
  log_entry = filtered[0][2]
58
62
 
59
63
  assert_equal 'marathon', log_entry['mesos_framework']
60
- assert_equal '/my-innovation-health-int-provider', log_entry['app']
64
+ assert_equal 'hello-world', log_entry['app']
61
65
  assert_equal task_id, log_entry['mesos_task_id']
62
66
  end
63
67
 
64
68
  def test_container_cache
65
- setup_marathon_container
69
+ setup_marathon_container('foobar123', 'marathon')
66
70
 
67
71
  d1 = create_driver(CONFIG, 'docker.foobar123')
68
72
  d1.run do
@@ -77,7 +81,7 @@ class AmplifierFilterTest < Test::Unit::TestCase
77
81
  end
78
82
 
79
83
  def test_container_cache_expiration
80
- setup_marathon_container
84
+ setup_marathon_container('foobar123', 'marathon')
81
85
 
82
86
  d1 = create_driver(CONFIG2, 'docker.foobar123')
83
87
  d1.run do
@@ -115,6 +119,26 @@ class AmplifierFilterTest < Test::Unit::TestCase
115
119
  assert_equal 'deployTasks', log_entry['chronos_task_type']
116
120
  end
117
121
 
122
+ def test_chronos_bad_match
123
+ docker_api_url = 'http://tcp//example.com:5422/v1.16/containers/foobar124/json'
124
+ file = File.open('test/containers/chronos_bad.json', 'rb')
125
+ setup_docker_stub(file, docker_api_url)
126
+
127
+ d1 = create_driver(CONFIG, 'docker.foobar124')
128
+ d1.run do
129
+ d1.filter('log' => 'Hello World 1')
130
+ end
131
+ filtered = d1.filtered_as_array
132
+
133
+ task_id = 'ct:1448508194000:0:recurring-transaction3:task'
134
+ log_entry = filtered[0][2]
135
+
136
+ assert_equal 'chronos', log_entry['mesos_framework']
137
+ assert_equal task_id, log_entry['mesos_task_id']
138
+ refute log_entry['app']
139
+ refute log_entry['chronos_task_type']
140
+ end
141
+
118
142
  def test_merge_json
119
143
  setup_chronos_container
120
144
 
@@ -143,4 +167,21 @@ class AmplifierFilterTest < Test::Unit::TestCase
143
167
  assert_equal bad_json1, filtered[0][2]['log']
144
168
  assert_equal bad_json2, filtered[1][2]['log']
145
169
  end
170
+
171
+ def test_container_id_from_record
172
+ setup_marathon_container('somecontainer123', 'marathon2')
173
+
174
+ d1 = create_driver(CONFIG3, 'docker')
175
+ d1.run do
176
+ d1.filter('log' => 'hello_world', 'container_id' => 'somecontainer123')
177
+ end
178
+ filtered = d1.filtered_as_array
179
+
180
+ task_id = 'hello-world.14b0596d-93ea-11e5-a134-124eefe69197'
181
+ log_entry = filtered[0][2]
182
+
183
+ assert_equal 'marathon', log_entry['mesos_framework']
184
+ assert_equal 'hello-world', log_entry['app']
185
+ assert_equal task_id, log_entry['mesos_task_id']
186
+ end
146
187
  end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: fluent-plugin-mesosphere-filter
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.1.0
4
+ version: 0.1.1
5
5
  platform: ruby
6
6
  authors:
7
7
  - Joseph Hughes
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2015-11-26 00:00:00.000000000 Z
11
+ date: 2015-11-28 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: fluentd
@@ -195,7 +195,9 @@ files:
195
195
  - fluent-plugin-mesosphere-filter.gemspec
196
196
  - lib/fluent/plugin/mesosphere.rb
197
197
  - test/containers/chronos.json
198
+ - test/containers/chronos_bad.json
198
199
  - test/containers/marathon.json
200
+ - test/containers/marathon2.json
199
201
  - test/helper.rb
200
202
  - test/plugin/test_mesosphere.rb
201
203
  homepage: https://github.com/joshughes/fluent-plugin-mesosphere-filter
@@ -224,6 +226,8 @@ specification_version: 4
224
226
  summary: Filter plugin to add Mesosphere metadata to fluentd from Chronos and Marathon
225
227
  test_files:
226
228
  - test/containers/chronos.json
229
+ - test/containers/chronos_bad.json
227
230
  - test/containers/marathon.json
231
+ - test/containers/marathon2.json
228
232
  - test/helper.rb
229
233
  - test/plugin/test_mesosphere.rb