fluent-plugin-record-modifier 0.4.1 → 0.5.0

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: 393309787a689b84bc9282849cd03b6d50bf68f4
4
- data.tar.gz: 24c92b2b9d47a616706701f4ec38c08ccbaccef7
3
+ metadata.gz: df0680e5b52d63630e68ce5c1179c10703c5c3d6
4
+ data.tar.gz: 3ee9b91266f11bbccd1958a768963089a11516e9
5
5
  SHA512:
6
- metadata.gz: 49ee18b86ca8e11cc803947c6dcc7f7beea0ce138d957ceaadb46f55053eb3ddf72d54b64bcbfd3b049bbc075823a54347acedb807cbb6b7ea972b8f6bce4eb2
7
- data.tar.gz: b41c41bc4a2a972cadf7b078125d8866ba76b5324f5990c3cf7a412707e7a56c43c934d4cca85479ce6b82ec2473a786cf350ce004f1264fe7b98dbc1b12499c
6
+ metadata.gz: cbcfd4bade6ee3779c2d37675a6f33ae10a6362a50d3449121f8fe4fcd3b3cf065f72007b88b19dd5fd4b97cf8e4063c8bb2ffb481962bee58d22092e5ebeea8
7
+ data.tar.gz: 053673498e8c4091ef80a83c3c5da0eebaf3fd29c35a03055ee36a70ebd9125ecd89a2ed8eafe1d7bc3775d9840b3b25e028a504be69d6965edd79705fdf1c75
data/.travis.yml CHANGED
@@ -3,8 +3,8 @@ language: ruby
3
3
  rvm:
4
4
  - 2.0.0
5
5
  - 2.1
6
- - 2.2.3
7
- - 2.3.0
6
+ - 2.2.5
7
+ - 2.3.1
8
8
  - ruby-head
9
9
  - rbx
10
10
 
@@ -19,6 +19,7 @@ matrix:
19
19
  allow_failures:
20
20
  - rvm: ruby-head
21
21
  - rvm: rbx
22
+ - rvm: 2.0.0
22
23
 
23
24
  before_install: gem update bundler
24
25
  script: bundle exec rake test
data/ChangeLog CHANGED
@@ -1,3 +1,8 @@
1
+ Release 0.5.0 - 2016/08/23
2
+
3
+ * Remove fluent-plugin-config-placeholder dependency
4
+ https://github.com/repeatedly/fluent-plugin-record-modifier/pull/13
5
+
1
6
  Release 0.4.1 - 2016/02/26
2
7
 
3
8
  * Set/Convert encoding recursively
data/README.md CHANGED
@@ -1,6 +1,6 @@
1
1
  # Filter plugin for modifying each event record for [Fluentd](http://fluentd.org)
2
2
 
3
- Adding arbitary field to event record without custmizing existence plugin.
3
+ Adding arbitary field to event record without customizing existence plugin.
4
4
 
5
5
  For example, generated event from *in_tail* doesn't contain "hostname" of running machine.
6
6
  In this case, you can use *record_modifier* to add "hostname" field to event record.
@@ -19,7 +19,7 @@ Use `record_modifier` filter.
19
19
  @type record_modifier
20
20
 
21
21
  <record>
22
- gen_host ${hostname}
22
+ gen_host "#{Socket.gethostname}"
23
23
  foo bar
24
24
  </record>
25
25
  </filter>
@@ -55,28 +55,16 @@ But unlike `record_transformer`, `record_modifier` doesn't support following fea
55
55
  - tag_suffix and tag_prefix
56
56
  - dynamic key placeholder
57
57
 
58
- ### record_modifier output
59
-
60
- In v0.10, you can use `record_modifier` output to emulate filter. `record_modifier` output doesn't support `<record>` way.
61
-
62
- <match pattern>
63
- type record_modifier
64
- tag foo.filtered
65
-
66
- gen_host ${hostname}
67
- foo bar
68
- </match>
69
-
70
58
  ### char_encoding
71
59
 
72
- Fluentd including some plugins treats the logs as a BINARY by default to forward.
73
- But an user sometimes processes the logs depends on their requirements, e.g. handling char encoding correctly.
60
+ Fluentd including some plugins treats logs as a BINARY by default to forward.
61
+ But a user sometimes wants to process logs depends on their requirements, e.g. handling char encoding correctly.
74
62
 
75
63
  `char_encoding` parameter is useful for this case.
76
64
 
77
65
  ```conf
78
66
  <filter pattern>
79
- type record_modifier
67
+ @type record_modifier
80
68
 
81
69
  # set UTF-8 encoding information to string.
82
70
  char_encoding utf-8
@@ -86,6 +74,8 @@ But an user sometimes processes the logs depends on their requirements, e.g. han
86
74
  </filter>
87
75
  ```
88
76
 
77
+ In `char_encoding from:to` case, it replaces invalid character with safe character.
78
+
89
79
  ### remove_keys
90
80
 
91
81
  The logs include needless record keys in some cases.
@@ -93,7 +83,7 @@ You can remove it by using `remove_keys` parameter.
93
83
 
94
84
  ```conf
95
85
  <filter pattern>
96
- type record_modifier
86
+ @type record_modifier
97
87
 
98
88
  # remove key1 and key2 keys from record
99
89
  remove_keys key1,key2
@@ -118,7 +108,7 @@ If you want to handle the set of explicitly specified keys, you can use `whiteli
118
108
 
119
109
  ```conf
120
110
  <filter pattern>
121
- type record_modifier
111
+ @type record_modifier
122
112
 
123
113
  # remove all keys except for key1 and key2
124
114
  whitelist_keys key1,key2
@@ -137,19 +127,24 @@ then you got new record like below:
137
127
  {"key1":"hoge", "key2":"foo"}
138
128
  ```
139
129
 
140
- ### Mixins
130
+ ### record_modifier output
131
+
132
+ In v0.10, you can use `record_modifier` output to emulate filter. `record_modifier` output doesn't support `<record>` way.
141
133
 
142
- * [fluent-mixin-config-placeholders](https://github.com/tagomoris/fluent-mixin-config-placeholders)
134
+ <match pattern>
135
+ type record_modifier
136
+ tag foo.filtered
137
+
138
+ gen_host "#{Socket.gethostname}"
139
+ foo bar
140
+ </match>
143
141
 
144
142
  ## TODO
145
143
 
146
144
  * Adding following features if needed
147
145
 
148
- * Use HandleTagNameMixin to keep original tag
149
-
150
146
  * Replace record value
151
147
 
152
-
153
148
  ## Copyright
154
149
 
155
150
  <table>
data/VERSION CHANGED
@@ -1 +1 @@
1
- 0.4.1
1
+ 0.5.0
@@ -18,7 +18,6 @@ Gem::Specification.new do |gem|
18
18
  gem.require_paths = ['lib']
19
19
 
20
20
  gem.add_dependency "fluentd", [">= 0.10.58", "< 2"]
21
- gem.add_dependency "fluent-mixin-config-placeholders", ">= 0.3.0"
22
21
  gem.add_development_dependency "rake", ">= 0.9.2"
23
22
  gem.add_development_dependency("test-unit", ["~> 3.1.4"])
24
23
  end
@@ -1,4 +1,4 @@
1
- require 'fluent/mixin/config_placeholders'
1
+ require 'fluent/filter'
2
2
 
3
3
  module Fluent
4
4
  class RecordModifierFilter < Filter
@@ -26,8 +26,6 @@ Modified events will have only specified keys (if exist in original events).
26
26
  This option is exclusive with `remove_keys`.
27
27
  DESC
28
28
 
29
- include Fluent::Mixin::ConfigPlaceholders
30
-
31
29
  BUILTIN_CONFIGURATIONS = %W(type @type log_level @log_level id @id char_encoding remove_keys whitelist_keys)
32
30
 
33
31
  def configure(conf)
@@ -40,6 +38,7 @@ DESC
40
38
  @map = {}
41
39
  conf.each_pair { |k, v|
42
40
  unless BUILTIN_CONFIGURATIONS.include?(k)
41
+ check_config_placeholders(k, v);
43
42
  conf.has_key?(k)
44
43
  $log.warn "top level definition is deprecated. Please put parameters inside <record>: '#{k} #{v}'"
45
44
  @map[k] = DynamicExpander.new(k, v)
@@ -65,6 +64,7 @@ DESC
65
64
  @has_tag_parts = false
66
65
  conf.elements.select { |element| element.name == 'record' }.each do |element|
67
66
  element.each_pair do |k, v|
67
+ check_config_placeholders(k, v)
68
68
  element.has_key?(k) # to suppress unread configuration warning
69
69
  @has_tag_parts = true if v.include?('tag_parts')
70
70
  @map[k] = DynamicExpander.new(k, v)
@@ -133,6 +133,16 @@ DESC
133
133
  end
134
134
  end
135
135
 
136
+ HOSTNAME_PLACEHOLDERS = %W(__HOSTNAME__ ${hostname})
137
+
138
+ def check_config_placeholders(k, v)
139
+ HOSTNAME_PLACEHOLDERS.each { |ph|
140
+ if v.include?(ph)
141
+ raise ConfigError, %!#{ph} placeholder in #{k} is removed. Use "\#{Socket.gethostname}" instead.!
142
+ end
143
+ }
144
+ end
145
+
136
146
  class DynamicExpander
137
147
  def initialize(param_key, param_value)
138
148
  if param_value.include?('${')
@@ -1,4 +1,4 @@
1
- require 'fluent/mixin/config_placeholders'
1
+ require 'fluent/output'
2
2
 
3
3
  module Fluent
4
4
  class RecordModifierOutput < Output
@@ -30,7 +30,6 @@ This option is exclusive with `remove_keys`.
30
30
  DESC
31
31
 
32
32
  include SetTagKeyMixin
33
- include Fluent::Mixin::ConfigPlaceholders
34
33
 
35
34
  BUILTIN_CONFIGURATIONS = %W(type tag include_tag_key tag_key char_encoding remove_keys whitelist_keys)
36
35
 
@@ -40,6 +39,7 @@ DESC
40
39
  @map = {}
41
40
  conf.each_pair { |k, v|
42
41
  unless BUILTIN_CONFIGURATIONS.include?(k)
42
+ check_config_placeholders(k, v)
43
43
  conf.has_key?(k)
44
44
  @map[k] = v
45
45
  end
@@ -83,6 +83,16 @@ DESC
83
83
 
84
84
  private
85
85
 
86
+ HOSTNAME_PLACEHOLDERS = %W(__HOSTNAME__ ${hostname})
87
+
88
+ def check_config_placeholders(k, v)
89
+ HOSTNAME_PLACEHOLDERS.each { |ph|
90
+ if v.include?(ph)
91
+ raise ConfigError, %!#{ph} placeholder in #{k} is removed. Use "\#{Socket.gethostname}" instead.!
92
+ end
93
+ }
94
+ end
95
+
86
96
  def modify_record(record)
87
97
  @map.each_pair { |k, v|
88
98
  record[k] = v
@@ -10,20 +10,20 @@ class RecordModifierFilterTest < Test::Unit::TestCase
10
10
  @tag = 'test.tag'
11
11
  end
12
12
 
13
- CONFIG = %[
13
+ CONFIG = %q!
14
14
  type record_modifier
15
15
  remove_keys hoge
16
16
 
17
17
  <record>
18
- gen_host ${hostname}
18
+ gen_host "#{Socket.gethostname}"
19
19
  foo bar
20
20
  included_tag ${tag}
21
21
  tag_wrap -${tag_parts[0]}-${tag_parts[1]}-
22
22
  </record>
23
- ]
23
+ !
24
24
 
25
25
  def create_driver(conf = CONFIG)
26
- Fluent::Test::FilterTestDriver.new(Fluent::RecordModifierFilter, @tag).configure(conf)
26
+ Fluent::Test::FilterTestDriver.new(Fluent::RecordModifierFilter, @tag).configure(conf, true)
27
27
  end
28
28
 
29
29
  def get_hostname
@@ -7,19 +7,19 @@ class RecordModifierOutputTest < Test::Unit::TestCase
7
7
  Fluent::Test.setup
8
8
  end
9
9
 
10
- CONFIG = %[
10
+ CONFIG = %q!
11
11
  type record_modifier
12
12
  tag foo.filtered
13
13
 
14
- gen_host ${hostname}
14
+ gen_host "#{Socket.gethostname}"
15
15
  foo bar
16
16
  include_tag_key
17
17
  tag_key included_tag
18
18
  remove_keys hoge
19
- ]
19
+ !
20
20
 
21
21
  def create_driver(conf = CONFIG)
22
- Fluent::Test::OutputTestDriver.new(Fluent::RecordModifierOutput, tag='test_tag').configure(conf)
22
+ Fluent::Test::OutputTestDriver.new(Fluent::RecordModifierOutput, tag='test_tag').configure(conf, true)
23
23
  end
24
24
 
25
25
  def get_hostname
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: fluent-plugin-record-modifier
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.4.1
4
+ version: 0.5.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Masahiro Nakagawa
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2016-02-26 00:00:00.000000000 Z
11
+ date: 2016-08-23 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: fluentd
@@ -30,20 +30,6 @@ dependencies:
30
30
  - - "<"
31
31
  - !ruby/object:Gem::Version
32
32
  version: '2'
33
- - !ruby/object:Gem::Dependency
34
- name: fluent-mixin-config-placeholders
35
- requirement: !ruby/object:Gem::Requirement
36
- requirements:
37
- - - ">="
38
- - !ruby/object:Gem::Version
39
- version: 0.3.0
40
- type: :runtime
41
- prerelease: false
42
- version_requirements: !ruby/object:Gem::Requirement
43
- requirements:
44
- - - ">="
45
- - !ruby/object:Gem::Version
46
- version: 0.3.0
47
33
  - !ruby/object:Gem::Dependency
48
34
  name: rake
49
35
  requirement: !ruby/object:Gem::Requirement
@@ -109,7 +95,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
109
95
  version: '0'
110
96
  requirements: []
111
97
  rubyforge_project:
112
- rubygems_version: 2.2.2
98
+ rubygems_version: 2.5.1
113
99
  signing_key:
114
100
  specification_version: 4
115
101
  summary: Filter plugin for modifying event record