fluent-plugin-buffer-lightening 0.0.1 → 0.0.2

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: 10e80754b85bcc5fa9bc83d47e75ee7bf7637b68
4
- data.tar.gz: 558822e85a83088e88268869ad13cbc121a7c39f
3
+ metadata.gz: 7d229e5d137896e042eb07b559b77b820d2986a3
4
+ data.tar.gz: 2d63ad73873268d9c126e9d8f98f50dda7c512bb
5
5
  SHA512:
6
- metadata.gz: ef5d463b78bf73975c8c5ddad3e01a09960bc6b115ea578308d4dbf6d0afd5e9c0022b8164ea6f5e6b08573cf342819cfe8245304b6ff6c885019b6f93e8b03b
7
- data.tar.gz: f6fdf47d488ecc61ce7efa12e392409686f2b1fbf94c2de8b2807f3e3eb5aa5eb1ca4f3303516ce1144f3e8740269a3c6b8e34994d89f12ac4ec395e2bc08bfa
6
+ metadata.gz: a56592eb36adf6ada002b03e8684bf39edb6b886c709f2d02e97078f2eecd0a610f5e88fdf85ee82df140c9bce3bc0f1afb98e99966896bc05ef975a7213203e
7
+ data.tar.gz: a98a9e1d136feed27f6422ebdf2b3c040428aafe30da29922769ab004d966116c15e2098daff266a88b3fc09cce4e7ea5ec30209e0a06184a99377b4f9e79f9d
@@ -0,0 +1,4 @@
1
+ language: ruby
2
+ rvm:
3
+ - 1.9.3
4
+ - 2.0.0
data/README.md CHANGED
@@ -1,14 +1,12 @@
1
1
  # fluent-plugin-buffer-lightening
2
2
 
3
- Fluentd buffer plugin on memory to flush with many types of chunk limit methods:
3
+ [Fluentd](http://fluentd.org) buffer plugin on memory to flush with many types of chunk limit methods:
4
4
  * events count limit in chunk
5
5
 
6
6
  These options are to decrease latency from emit to write, and to control chunk sizes and flush sizes.
7
7
 
8
8
  **NOTICE:** Lightening buffer plugin stores data on memory, so these data will be lost when process/server crashes.
9
9
 
10
- And current version of this plugin adds `try_flush_interval` option to BufferedOutput plugins, to flush buffer chunk with high frequency. For this option, run fluentd with `-r fluent/plugin/output_try_flush_interval_patch`.
11
-
12
10
  ## Installation
13
11
 
14
12
  Do `gem install fluent-plugin-buffer-lightening` or `fluent-gem ...`.
@@ -41,7 +39,7 @@ Options of `buffer_type memory` are also available:
41
39
 
42
40
  ### For less delay
43
41
 
44
- For more frequently flushing, use `flush_interval` and `try_flush_interval` with floating point values:
42
+ For more frequently flushing, use `flush_interval` and `try_flush_interval` with floating point values on Fluentd v0.10.42 or later:
45
43
  ```
46
44
  <match data.**>
47
45
  type any_buffered_output_plugin
@@ -53,11 +51,8 @@ For more frequently flushing, use `flush_interval` and `try_flush_interval` with
53
51
  </match>
54
52
  ```
55
53
 
56
- And, execute fluentd as `fluentd -r fluent/plugin/output_try_flush_interval_patch -c fluentd.conf`.
57
-
58
54
  ## TODO
59
55
 
60
- * remove `output_try_flush_interval_patch` with incoming fluentd dependency
61
56
  * more limit patterns
62
57
  * patches welcome!
63
58
 
@@ -2,7 +2,7 @@
2
2
 
3
3
  Gem::Specification.new do |spec|
4
4
  spec.name = "fluent-plugin-buffer-lightening"
5
- spec.version = "0.0.1"
5
+ spec.version = "0.0.2"
6
6
  spec.authors = ["TAGOMORI Satoshi"]
7
7
  spec.email = ["tagomoris@gmail.com"]
8
8
  spec.description = %q{Fluentd memory buffer plugin with many types of chunk limits}
@@ -17,5 +17,5 @@ Gem::Specification.new do |spec|
17
17
 
18
18
  spec.add_development_dependency "bundler", "~> 1.3"
19
19
  spec.add_development_dependency "rake"
20
- spec.add_runtime_dependency "fluentd"
20
+ spec.add_runtime_dependency "fluentd", ">= 0.10.42"
21
21
  end
@@ -1,5 +1,4 @@
1
1
  require 'fluent/plugin/buf_memory'
2
- require_relative 'output_try_flush_interval_patch'
3
2
 
4
3
  module Fluent
5
4
  class LighteningBufferChunk < MemoryBufferChunk
@@ -34,51 +33,5 @@ module Fluent
34
33
  return false if @buffer_chunk_records_limit && chunk.record_counter >= @buffer_chunk_records_limit
35
34
  true
36
35
  end
37
-
38
- # TODO: remove w/ fluentd v0.10.42 (or td-agent including fluentd v0.10.42)
39
- def emit(key, data, chain) # copy&paste from BasicBuffer, and fix to add hook point
40
- key = key.to_s
41
-
42
- synchronize do
43
- top = (@map[key] ||= new_chunk(key))
44
-
45
- if storable?(top, data) # hook point (FIXED THIS LINE ONLY)
46
- chain.next
47
- top << data
48
- return false
49
- elsif @queue.size >= @buffer_queue_limit
50
- raise BufferQueueLimitError, "queue size exceeds limit"
51
- end
52
-
53
- if data.bytesize > @buffer_chunk_limit
54
- $log.warn "Size of the emitted data exceeds buffer_chunk_limit."
55
- $log.warn "This may occur problems in the output plugins ``at this server.``"
56
- $log.warn "To avoid problems, set a smaller number to the buffer_chunk_limit"
57
- $log.warn "in the forward output ``at the log forwarding server.``"
58
- end
59
-
60
- nc = new_chunk(key)
61
- ok = false
62
-
63
- begin
64
- nc << data
65
- chain.next
66
-
67
- flush_trigger = false
68
- @queue.synchronize {
69
- enqueue(top)
70
- flush_trigger = @queue.empty?
71
- @queue << top
72
- @map[key] = nc
73
- }
74
-
75
- ok = true
76
- return flush_trigger
77
- ensure
78
- nc.purge unless ok
79
- end
80
-
81
- end # synchronize
82
- end
83
36
  end
84
37
  end
@@ -22,7 +22,6 @@ unless ENV.has_key?('VERBOSE')
22
22
  $log = nulllogger
23
23
  end
24
24
 
25
- require 'fluent/plugin/output_try_flush_interval_patch'
26
25
  require 'fluent/plugin/buf_lightening'
27
26
 
28
27
  class Test::Unit::TestCase
metadata CHANGED
@@ -1,57 +1,57 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: fluent-plugin-buffer-lightening
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.0.1
4
+ version: 0.0.2
5
5
  platform: ruby
6
6
  authors:
7
7
  - TAGOMORI Satoshi
8
8
  autorequire:
9
9
  bindir: bin
10
10
  cert_chain: []
11
- date: 2013-12-22 00:00:00.000000000 Z
11
+ date: 2014-05-15 00:00:00.000000000 Z
12
12
  dependencies:
13
13
  - !ruby/object:Gem::Dependency
14
14
  name: bundler
15
15
  requirement: !ruby/object:Gem::Requirement
16
16
  requirements:
17
- - - ~>
17
+ - - "~>"
18
18
  - !ruby/object:Gem::Version
19
19
  version: '1.3'
20
20
  type: :development
21
21
  prerelease: false
22
22
  version_requirements: !ruby/object:Gem::Requirement
23
23
  requirements:
24
- - - ~>
24
+ - - "~>"
25
25
  - !ruby/object:Gem::Version
26
26
  version: '1.3'
27
27
  - !ruby/object:Gem::Dependency
28
28
  name: rake
29
29
  requirement: !ruby/object:Gem::Requirement
30
30
  requirements:
31
- - - '>='
31
+ - - ">="
32
32
  - !ruby/object:Gem::Version
33
33
  version: '0'
34
34
  type: :development
35
35
  prerelease: false
36
36
  version_requirements: !ruby/object:Gem::Requirement
37
37
  requirements:
38
- - - '>='
38
+ - - ">="
39
39
  - !ruby/object:Gem::Version
40
40
  version: '0'
41
41
  - !ruby/object:Gem::Dependency
42
42
  name: fluentd
43
43
  requirement: !ruby/object:Gem::Requirement
44
44
  requirements:
45
- - - '>='
45
+ - - ">="
46
46
  - !ruby/object:Gem::Version
47
- version: '0'
47
+ version: 0.10.42
48
48
  type: :runtime
49
49
  prerelease: false
50
50
  version_requirements: !ruby/object:Gem::Requirement
51
51
  requirements:
52
- - - '>='
52
+ - - ">="
53
53
  - !ruby/object:Gem::Version
54
- version: '0'
54
+ version: 0.10.42
55
55
  description: Fluentd memory buffer plugin with many types of chunk limits
56
56
  email:
57
57
  - tagomoris@gmail.com
@@ -59,14 +59,14 @@ executables: []
59
59
  extensions: []
60
60
  extra_rdoc_files: []
61
61
  files:
62
- - .gitignore
62
+ - ".gitignore"
63
+ - ".travis.yml"
63
64
  - Gemfile
64
65
  - LICENSE.txt
65
66
  - README.md
66
67
  - Rakefile
67
68
  - fluent-plugin-buffer-lightening.gemspec
68
69
  - lib/fluent/plugin/buf_lightening.rb
69
- - lib/fluent/plugin/output_try_flush_interval_patch.rb
70
70
  - test/helper.rb
71
71
  - test/plugin/dummy_output.rb
72
72
  - test/plugin/test_buf_lightening.rb
@@ -80,17 +80,17 @@ require_paths:
80
80
  - lib
81
81
  required_ruby_version: !ruby/object:Gem::Requirement
82
82
  requirements:
83
- - - '>='
83
+ - - ">="
84
84
  - !ruby/object:Gem::Version
85
85
  version: '0'
86
86
  required_rubygems_version: !ruby/object:Gem::Requirement
87
87
  requirements:
88
- - - '>='
88
+ - - ">="
89
89
  - !ruby/object:Gem::Version
90
90
  version: '0'
91
91
  requirements: []
92
92
  rubyforge_project:
93
- rubygems_version: 2.0.3
93
+ rubygems_version: 2.2.2
94
94
  signing_key:
95
95
  specification_version: 4
96
96
  summary: Alternative memory buffer plugin for Fluentd to realize less delay
@@ -98,4 +98,3 @@ test_files:
98
98
  - test/helper.rb
99
99
  - test/plugin/dummy_output.rb
100
100
  - test/plugin/test_buf_lightening.rb
101
- has_rdoc:
@@ -1,113 +0,0 @@
1
- # MEMO: execute fluentd with "-r fluent/plugin/output_try_flush_interval_patch" switch
2
-
3
- # TODO: remove w/ fluentd v0.10.42 (or td-agent including fluentd v0.10.42)
4
- module Fluent
5
- class BufferedOutput < Output
6
- config_param :try_flush_interval, :float, :default => 1
7
-
8
- # override with @try_flush_interval
9
- def try_flush
10
- time = Engine.now
11
-
12
- empty = @buffer.queue_size == 0
13
- if empty && @next_flush_time < (now = Engine.now)
14
- @buffer.synchronize do
15
- if @next_flush_time < now
16
- enqueue_buffer
17
- @next_flush_time = now + @flush_interval
18
- empty = @buffer.queue_size == 0
19
- end
20
- end
21
- end
22
- if empty
23
- return time + @try_flush_interval
24
- end
25
-
26
- begin
27
- retrying = !@error_history.empty?
28
-
29
- if retrying
30
- @error_history.synchronize do
31
- if retrying = !@error_history.empty? # re-check in synchronize
32
- if @next_retry_time >= time
33
- # allow retrying for only one thread
34
- return time + @try_flush_interval
35
- end
36
- # assume next retry failes and
37
- # clear them if when it succeeds
38
- @last_retry_time = time
39
- @error_history << time
40
- @next_retry_time += calc_retry_wait
41
- end
42
- end
43
- end
44
-
45
- if @secondary && @error_history.size > @retry_limit
46
- has_next = flush_secondary(@secondary)
47
- else
48
- has_next = @buffer.pop(self)
49
- end
50
-
51
- # success
52
- if retrying
53
- @error_history.clear
54
- # Note: don't notify to other threads to prevent
55
- # burst to recovered server
56
- $log.warn "retry succeeded.", :instance=>object_id
57
- end
58
-
59
- if has_next
60
- return Engine.now + @queued_chunk_flush_interval
61
- else
62
- return time + @try_flush_interval
63
- end
64
-
65
- rescue => e
66
- if retrying
67
- error_count = @error_history.size
68
- else
69
- # first error
70
- error_count = 0
71
- @error_history.synchronize do
72
- if @error_history.empty?
73
- @last_retry_time = time
74
- @error_history << time
75
- @next_retry_time = time + calc_retry_wait
76
- end
77
- end
78
- end
79
-
80
- if error_count < @retry_limit
81
- $log.warn "temporarily failed to flush the buffer.", :next_retry=>Time.at(@next_retry_time), :error_class=>e.class.to_s, :error=>e.to_s, :instance=>object_id
82
- $log.warn_backtrace e.backtrace
83
-
84
- elsif @secondary
85
- if error_count == @retry_limit
86
- $log.warn "failed to flush the buffer.", :error_class=>e.class.to_s, :error=>e.to_s, :instance=>object_id
87
- $log.warn "retry count exceededs limit. falling back to secondary output."
88
- $log.warn_backtrace e.backtrace
89
- retry # retry immediately
90
- elsif error_count <= @retry_limit + @secondary_limit
91
- $log.warn "failed to flush the buffer, next retry will be with secondary output.", :next_retry=>Time.at(@next_retry_time), :error_class=>e.class.to_s, :error=>e.to_s, :instance=>object_id
92
- $log.warn_backtrace e.backtrace
93
- else
94
- $log.warn "failed to flush the buffer.", :error_class=>e.class, :error=>e.to_s, :instance=>object_id
95
- $log.warn "secondary retry count exceededs limit."
96
- $log.warn_backtrace e.backtrace
97
- write_abort
98
- @error_history.clear
99
- end
100
-
101
- else
102
- $log.warn "failed to flush the buffer.", :error_class=>e.class.to_s, :error=>e.to_s, :instance=>object_id
103
- $log.warn "retry count exceededs limit."
104
- $log.warn_backtrace e.backtrace
105
- write_abort
106
- @error_history.clear
107
- end
108
-
109
- return @next_retry_time
110
- end
111
- end
112
- end
113
- end