fluent-plugin-memcached 0.1.1 → 0.2.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (6) hide show
  1. checksums.yaml +4 -4
  2. data/README.md +130 -11
  3. data/fluent-plugin-memcached.gemspec +1 -1
  4. data/lib/fluent/plugin/out_memcached.rb +54 -82
  5. data/test/plugin/test_out_memcached.rb +72 -85
  6. metadata +2 -2
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: 369bfe5ecc11048b67015f77bcfe16e78a033aae
4
- data.tar.gz: 00a867ad20989e5ef7c1c1134a87813de1bd52ea
3
+ metadata.gz: 76053cd2a49aafb9f108310579ddd20c6b75cc12
4
+ data.tar.gz: 391f5f6286dab6655e725ded9c7da27cdac57fe9
5
5
  SHA512:
6
- metadata.gz: 916fe9ccbfca3577f64599d285b28a4be8ed1e2efdf6054d427a6dee870ae1a052c94a2e6fd35eda25a2302bf69a119a3ffb01fd58ab804d9304e865dbe7c386
7
- data.tar.gz: 949d12468856097b688f26e07cc9131171bccce332d4bfd0a436909ce0a8bb990e95c22b92e8c2f995a3062b571a3f49528e585073124374fa510cdf0c7195ea
6
+ metadata.gz: c872f7ca9bb0b5aa884f0de84a99cd55c2124a61b5e9a7598e01be5f7489549f38a3fd272569f0c55d5e31ff48168dfac94d4cdf208e6e931bb553e5c7ec7658
7
+ data.tar.gz: e338d70a15e5ca4e0588d1f4395f9736d384e5551dec16bb1638878d7a71b24b81f3df59303074152d3610bfdeeb44a23f76e1e264c30509828ecdd255c46723
data/README.md CHANGED
@@ -5,32 +5,151 @@
5
5
 
6
6
  Send your logs to Memcached.
7
7
 
8
+ ## Requirements
9
+
10
+ | fluent-plugin-memcached | fluentd | ruby |
11
+ |-------------------------|---------|------|
12
+ | >= 0.1.0 | >= v0.14.0 | >= 2.1 |
13
+ | < 0.1.0 | >= v0.12.0 | >= 1.9 |
14
+
8
15
  ## Installation
9
16
 
10
- ```sh
17
+ ```console
11
18
  $ gem install fluent-plugin-memcached
12
19
  ```
13
20
 
14
- ## Usage
21
+ ## Configuration
15
22
 
16
- In your Fluentd configuration, use `type memcached`.
17
- Default values would look like this:
23
+ **NOTE: The version 0.2.0 includes breaking changes for configuration.** Please see [here](#for-previous-versions) if you use v0.1.1 or earlier.
24
+
25
+ In your Fluentd configuration, use `@type memcached`.
18
26
 
19
27
  ```
20
28
  <match dummy>
21
- type memcached
22
- host localhost
23
- port 11211
24
- increment false
25
- # value_separater " "
29
+ @type memcached
30
+ host localhost # Optional, default:localhost
31
+ port 11211 # Optional, default:11211
32
+
33
+ key id # Required, set a key name, the value of which is used as memcached key
34
+ include_key false # Optional, default: false
35
+ increment false # Optional, default: false
36
+
37
+ format csv # Optional, default: csv
38
+ fields field1,field2 # Required, set field names, the value of which is stored in memcached
39
+ delimiter " " # Optional, default: " "
40
+ force_quotes false # Optional, default: false
41
+ </match>
42
+ ```
43
+
44
+ ### Use cases
45
+
46
+ There are some results when the following input is coming.
47
+
48
+ input: `{"id" => "key1", "field1" => "value1", "field2" => "value2", "field_incr" => "1"}`
49
+
50
+ #### To store a data as CSV
51
+
52
+ ```
53
+ <match dummy>
54
+ @type memcached
55
+ key id
56
+ fields field1,field2
57
+ delimiter ,
26
58
  </match>
27
59
  ```
28
60
 
29
- To store values as json, like this:
61
+ The result of stored a data is as below:
62
+
63
+ - key: `key1`
64
+ - value: `value1,value2`
65
+
66
+ #### To store a data as JSON
67
+
68
+ ```
69
+ <match dummy>
70
+ @type memcached
71
+ key id
72
+ fields field1,field2
73
+ format json
74
+ </match>
75
+ ```
76
+
77
+ The result of stored a data is as below:
78
+
79
+ - key: `key1`
80
+ - value: `{"field1":"value1","field2":"value2"}`
81
+
82
+ #### To store a data as single incremental value
83
+
84
+ ```
85
+ <match dummy>
86
+ @type memcached
87
+ key id
88
+ increment true
89
+ format single_value
90
+ message_key field_incr
91
+ </match>
92
+ ```
93
+
94
+ The result of stored a data is as below:
95
+
96
+ - key: `key1`
97
+ - value: `1`
98
+
99
+ Then the following input is also coming,
100
+
101
+ input: `{"id" => "key1", "field1" => "value3", "field2" => "value4", "field_incr" => "2"}`
102
+
103
+ The result of stored a data will be as below:
104
+
105
+ - key: `key1`
106
+ - value: `3`
107
+
108
+ ### Fluentd v0.14 style
109
+
110
+ When using v0.14 style configuration, you can choose three different types of buffer behavior:
111
+
112
+ #### Simple Buffered Output
113
+
114
+ ```
115
+ <match dummy>
116
+ ...
117
+ <buffer>
118
+ @type memory
119
+ </buffer>
120
+ </match>
121
+ ```
122
+
123
+ #### Tag Separated Buffered Output
124
+
125
+ ```
126
+ <match dummy>
127
+ ...
128
+ <buffer tag>
129
+ @type memory
130
+ </buffer>
131
+ </match>
132
+ ```
133
+
134
+ #### Time Sliced Buffered Output
135
+
136
+ ```
137
+ <match dummy>
138
+ ...
139
+ <buffer tag, time>
140
+ @type memory
141
+ timekey 3600 # for 1 hour
142
+ </buffer>
143
+ </match>
144
+ ```
145
+
146
+ ### For previous versions
147
+
148
+ In v0.1.1 or earlier, to store a data as JSON, like this:
30
149
 
31
150
  ```
32
151
  <match dummy>
33
- type memcached
152
+ @type memcached
34
153
  host localhost
35
154
  port 11211
36
155
  value_format json
data/fluent-plugin-memcached.gemspec CHANGED
@@ -4,7 +4,7 @@ $LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
4
4
 
5
5
  Gem::Specification.new do |gem|
6
6
  gem.name = "fluent-plugin-memcached"
7
- gem.version = "0.1.1"
7
+ gem.version = "0.2.0"
8
8
  gem.authors = ["innossh"]
9
9
  gem.email = ["innossh@users.noreply.github.com"]
10
10
 
data/lib/fluent/plugin/out_memcached.rb CHANGED
@@ -1,109 +1,81 @@
1
1
  require 'dalli'
2
2
  require 'fluent/plugin/output'
3
3
 
4
- class Fluent::Plugin::MemcachedOutput < Fluent::Plugin::Output
5
- Fluent::Plugin.register_output('memcached', self)
4
+ module Fluent::Plugin
5
+ class MemcachedOutput < Output
6
+ Fluent::Plugin.register_output('memcached', self)
6
7
 
7
- helpers :compat_parameters
8
+ include Fluent::SetTimeKeyMixin
9
+ include Fluent::SetTagKeyMixin
8
10
 
9
- DEFAULT_BUFFER_TYPE = "memory"
11
+ helpers :inject, :formatter, :compat_parameters
10
12
 
11
- config_param :host, :string, :default => 'localhost'
12
- config_param :port, :integer, :default => 11211
13
+ DEFAULT_BUFFER_TYPE = 'memory'
14
+ DEFAULT_FORMAT_TYPE = 'csv'
13
15
 
14
- config_param :increment, :bool, :default => false
15
- config_param :value_separater, :string, :default => ' '
16
+ config_param :host, :string, :default => 'localhost'
17
+ config_param :port, :integer, :default => 11211
18
+ config_param :key, :string
19
+ config_param :include_key, :bool, :default => false
20
+ config_param :increment, :bool, :default => false
16
21
 
17
- config_param :value_format, :string, :default => 'raw'
18
- config_param :param_names, :string, :default => nil # nil doesn't allowed for json
22
+ config_section :buffer do
23
+ config_set_default :@type, DEFAULT_BUFFER_TYPE
24
+ end
19
25
 
20
- config_section :buffer do
21
- config_set_default :@type, DEFAULT_BUFFER_TYPE
22
- end
26
+ config_section :format do
27
+ config_set_default :@type, DEFAULT_FORMAT_TYPE
28
+ config_set_default :delimiter, ' '
29
+ config_set_default :force_quotes, false
30
+ end
23
31
 
24
- attr_accessor :memcached
25
- attr_accessor :formatter
32
+ attr_accessor :memcached
33
+ attr_accessor :formatter
26
34
 
27
- def configure(conf)
28
- compat_parameters_convert(conf, :buffer)
29
- super
30
- if @value_format == 'json' and @param_names.nil?
31
- raise Fluent::ConfigError, "param_names MUST be specified in the case of json format"
32
- end
33
- @formatter = RecordValueFormatter.new(@increment, @value_separater, @value_format, @param_names)
34
- end
35
+ def configure(conf)
36
+ compat_parameters_convert(conf, :buffer, :inject, :formatter)
37
+ super
35
38
 
36
- def start
37
- super
38
- @memcached = Dalli::Client.new("#{@host}:#{@port}")
39
- end
39
+ @formatter = formatter_create
40
+ end
40
41
 
41
- def shutdown
42
- @memcached.close
43
- super
44
- end
42
+ def start
43
+ super
45
44
 
46
- def format(tag, time, record)
47
- [time, record].to_msgpack
48
- end
45
+ @memcached = Dalli::Client.new("#{@host}:#{@port}")
46
+ end
49
47
 
50
- def formatted_to_msgpack_binary?
51
- true
52
- end
48
+ def shutdown
49
+ @memcached.close
53
50
 
54
- def multi_workers_ready?
55
- true
56
- end
51
+ super
52
+ end
57
53
 
58
- def write(chunk)
59
- chunk.msgpack_each { |time, record|
60
- key = @formatter.key(record)
61
- value = @formatter.value(record)
62
- if @increment
63
- if @memcached.get(key) == nil
64
- # initialize increment value
65
- @memcached.incr(key, 1, nil, 0)
66
- end
67
- @memcached.incr(key, amt=value)
54
+ def format(tag, time, record)
55
+ record = inject_values_to_record(tag, time, record)
68
56
 
69
- else
70
- @memcached.set(key, value)
71
- end
72
- }
73
- end
57
+ key = @include_key ? record[@key] : record.delete(@key)
58
+ [time, key, @formatter.format(tag, time, record).chomp].to_msgpack
59
+ end
74
60
 
75
- class RecordValueFormatter
76
- attr_reader :increment
77
- attr_reader :value_separater
78
- attr_reader :value_format
79
- attr_reader :param_names
80
-
81
- def initialize(increment, value_separater, value_format, param_names)
82
- @increment = increment
83
- @value_separater = value_separater
84
- @value_format = value_format
85
- @param_names = param_names
61
+ def formatted_to_msgpack_binary?
62
+ true
86
63
  end
87
64
 
88
- def key(record)
89
- record.values.first
65
+ def multi_workers_ready?
66
+ true
90
67
  end
91
68
 
92
- def value(record)
93
- values = record.values.drop(1)
94
- case @value_format
95
- when 'json'
96
- hash = {}
97
- @param_names.split(/\s*,\s*/).each_with_index { |param_name, i|
98
- hash[param_name] = (i > values.size - 1) ? nil : values[i]
99
- }
100
- hash.to_json
101
- else
102
- return values.first.to_i if @increment
103
-
104
- values.join(@value_separater)
69
+ def write(chunk)
70
+ chunk.msgpack_each do |time, key, value|
71
+ unless @increment
72
+ @memcached.set(key, value)
73
+ next
74
+ end
75
+
76
+ @memcached.incr(key, value.to_i, nil, value.to_i)
105
77
  end
106
78
  end
107
- end
108
79
 
80
+ end
109
81
  end
data/test/plugin/test_out_memcached.rb CHANGED
@@ -1,4 +1,3 @@
1
- require 'test/unit'
2
1
  require 'fluent/test'
3
2
  require 'fluent/test/helpers'
4
3
  require 'fluent/test/driver/output'
@@ -9,30 +8,38 @@ class MemcachedOutputTest < Test::Unit::TestCase
9
8
 
10
9
  def setup
11
10
  Fluent::Test.setup
11
+
12
+ @d = create_driver
13
+ # Invalidate all existing cache items before testing
14
+ Dalli::Client.new("#{@d.instance.host}:#{@d.instance.port}").flush_all
15
+ @time = event_time('2011-01-02 13:14:15 UTC')
12
16
  end
13
17
 
14
18
  CONFIG = %[
15
- host 127.0.0.1
16
- port 11211
19
+ key id
20
+ fields field1,field2
17
21
  ]
18
22
 
19
23
  CONFIG_JSON = %[
20
- host 127.0.0.1
21
- port 11211
22
- value_format json
23
- param_names param1,param2
24
+ key id
25
+ include_key true
26
+ format json
27
+ include_tag_key true
24
28
  ]
25
29
 
26
30
  CONFIG_INCREMENT = %[
27
- host 127.0.0.1
28
- port 11211
31
+ key id
32
+ format single_value
33
+ message_key field_incr
29
34
  increment true
30
35
  ]
31
36
 
32
37
  CONFIG_MYSQL = %[
33
- host 127.0.0.1
34
- port 11211
35
- value_separater |
38
+ include_time_key true
39
+ time_format %s
40
+ key time
41
+ fields metrics_name,metrics_value
42
+ delimiter |
36
43
  ]
37
44
 
38
45
  def create_driver(conf = CONFIG)
@@ -40,114 +47,94 @@ class MemcachedOutputTest < Test::Unit::TestCase
40
47
  end
41
48
 
42
49
  def test_configure
43
- d = create_driver('')
44
- assert_equal 'localhost', d.instance.host
45
- assert_equal 11211, d.instance.port
46
- assert_equal false, d.instance.increment
47
- assert_equal ' ', d.instance.value_separater
50
+ assert_raise(Fluent::ConfigError) {
51
+ create_driver('')
52
+ }
48
53
 
49
- d = create_driver
50
- assert_equal '127.0.0.1', d.instance.host
51
- assert_equal 11211, d.instance.port
54
+ assert_equal 'localhost', @d.instance.host
55
+ assert_equal 11211, @d.instance.port
56
+ assert_equal false, @d.instance.increment
57
+ assert_equal ' ', @d.instance.formatter.delimiter
58
+ assert_equal 'id', @d.instance.key
59
+ assert_equal ['field1', 'field2'], @d.instance.formatter.fields
52
60
 
53
61
  d = create_driver(CONFIG_JSON)
54
- assert_equal '127.0.0.1', d.instance.host
55
- assert_equal 11211, d.instance.port
56
- assert_equal 'json', d.instance.value_format
57
- assert_equal 'param1,param2', d.instance.param_names
58
-
59
- assert_raise(Fluent::ConfigError) {
60
- create_driver %[
61
- host 127.0.0.1
62
- port 11211
63
- value_format json
64
- ]
65
- }
62
+ assert_equal 'json', d.instance.formatter_configs.first[:@type]
66
63
 
67
64
  d = create_driver(CONFIG_INCREMENT)
68
- assert_equal '127.0.0.1', d.instance.host
69
- assert_equal 11211, d.instance.port
70
65
  assert_equal true, d.instance.increment
66
+ assert_equal 'single_value', d.instance.formatter_configs.first[:@type]
67
+ assert_equal 'field_incr', d.instance.formatter.message_key
71
68
 
72
69
  d = create_driver(CONFIG_MYSQL)
73
- assert_equal '127.0.0.1', d.instance.host
74
- assert_equal 11211, d.instance.port
75
- assert_equal '|', d.instance.value_separater
70
+ assert_equal 'time', d.instance.key
71
+ assert_equal true, d.instance.include_time_key
72
+ assert_equal '%s', d.instance.inject_config.time_format
73
+ assert_equal ['metrics_name', 'metrics_value'], d.instance.formatter.fields
74
+ assert_equal '|', d.instance.formatter.delimiter
76
75
  end
77
76
 
78
77
  def test_format
79
- d = create_driver
80
- time = Time.parse('2011-01-02 13:14:15 UTC').to_i
81
- record = {'key' => 'key', 'param1' => 'value'}
82
- d.run(default_tag: 'test') do
83
- d.feed(time, record)
78
+ record = {'id' => 'key', 'field1' => 'value1', 'field2' => 'value2'}
79
+ @d.run(default_tag: 'test') do
80
+ @d.feed(@time, record)
84
81
  end
85
- assert_equal [[time, record].to_msgpack], d.formatted
82
+ assert_equal [[@time.to_i, 'key', 'value1 value2'].to_msgpack], @d.formatted
86
83
  end
87
84
 
88
85
  def test_write
89
- d = create_driver
90
- time = event_time('2011-01-02 13:14:15 UTC')
91
- record1 = {'key' => 'a', 'param1' => '1'}
92
- record2 = {'key' => 'b', 'param1' => '2', 'param2' => '3'}
93
- d.run(default_tag: 'test') do
94
- d.feed(time, record1)
95
- d.feed(time, record2)
86
+ @d = create_driver
87
+ record1 = {'id' => 'a', 'field1' => '1'}
88
+ record2 = {'id' => 'b', 'field1' => '2', 'field2' => '3'}
89
+ @d.run(default_tag: 'test') do
90
+ @d.feed(@time, record1)
91
+ @d.feed(@time, record2)
96
92
  end
97
93
 
98
- assert_equal '1', d.instance.memcached.get('a')
99
- assert_equal '2 3', d.instance.memcached.get('b')
94
+ assert_equal '1 ', @d.instance.memcached.get('a')
95
+ assert_equal '2 3', @d.instance.memcached.get('b')
100
96
  end
101
97
 
102
98
  def test_write_json
103
99
  d = create_driver(CONFIG_JSON)
104
- time = event_time('2011-01-02 13:14:15 UTC')
105
- record1 = {'key' => 'c', 'param1' => '4'}
106
- record2 = {'key' => 'd', 'param1' => '5', 'param2' => '6'}
107
- record1_value_json = {'param1' => '4', 'param2' => nil}.to_json
108
- record2_value_json = {'param1' => '5', 'param2' => '6'}.to_json
100
+ record1 = {'id' => 'a', 'field1' => '4'}
101
+ record2 = {'id' => 'b', 'field1' => '5', 'field2' => '6'}
102
+ record1_value_json = {'id' => 'a', 'field1' => '4', 'tag' => 'test'}.to_json
103
+ record2_value_json = {'id' => 'b', 'field1' => '5', 'field2' => '6', 'tag' => 'test'}.to_json
109
104
  d.run(default_tag: 'test') do
110
- d.feed(time, record1)
111
- d.feed(time, record2)
105
+ d.feed(@time, record1)
106
+ d.feed(@time, record2)
112
107
  end
113
108
 
114
- assert_equal record1_value_json, d.instance.memcached.get('c')
115
- assert_equal record2_value_json, d.instance.memcached.get('d')
109
+ assert_equal record1_value_json, d.instance.memcached.get('a')
110
+ assert_equal record2_value_json, d.instance.memcached.get('b')
116
111
  end
117
112
 
118
- class IncrementTest < self
119
- def teardown
120
- @d.instance.memcached.flush_all
113
+ def test_write_increment
114
+ d = create_driver(CONFIG_INCREMENT)
115
+ record1 = {'id' => 'count1', 'field_incr' => 1}
116
+ record2 = {'id' => 'count2', 'field_incr' => 2}
117
+ record3 = {'id' => 'count1', 'field_incr' => 3}
118
+ record4 = {'id' => 'count2', 'field_incr' => 4}
119
+ d.run(default_tag: 'test') do
120
+ d.feed(@time, record1)
121
+ d.feed(@time, record2)
122
+ d.feed(@time, record3)
123
+ d.feed(@time, record4)
121
124
  end
122
125
 
123
- def test_write_increment
124
- @d = create_driver(CONFIG_INCREMENT)
125
- time = event_time('2011-01-02 13:14:15 UTC')
126
- record1 = {'key' => 'count1', 'param1' => 1}
127
- record2 = {'key' => 'count2', 'param1' => 2}
128
- record3 = {'key' => 'count1', 'param1' => 3}
129
- record4 = {'key' => 'count2', 'param1' => 4}
130
- @d.run(default_tag: 'test') do
131
- @d.feed(time, record1)
132
- @d.feed(time, record2)
133
- @d.feed(time, record3)
134
- @d.feed(time, record4)
135
- end
136
-
137
- assert_equal (1 + 3), @d.instance.memcached.get('count1').to_i
138
- assert_equal (2 + 4), @d.instance.memcached.get('count2').to_i
139
- end
126
+ assert_equal (1 + 3), d.instance.memcached.get('count1').to_i
127
+ assert_equal (2 + 4), d.instance.memcached.get('count2').to_i
140
128
  end
141
129
 
142
130
  def test_write_to_mysql
143
131
  d = create_driver(CONFIG_MYSQL)
144
- time = event_time('2011-01-02 13:14:15 UTC')
145
- record = {'key' => time, 'metrics_name' => 'count', 'metrics_value' => '100'}
132
+ record = {'metrics_name' => 'count', 'metrics_value' => '100'}
146
133
  d.run(default_tag: 'test') do
147
- d.feed(time, record)
134
+ d.feed(@time, record)
148
135
  end
149
136
 
150
- assert_equal 'count|100', d.instance.memcached.get(time)
137
+ assert_equal 'count|100', d.instance.memcached.get(@time.to_i)
151
138
  end
152
139
 
153
140
  end
metadata CHANGED
@@ -1,14 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: fluent-plugin-memcached
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.1.1
4
+ version: 0.2.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - innossh
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2017-06-07 00:00:00.000000000 Z
11
+ date: 2017-06-10 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: fluentd