fluent-plugin-s3 0.4.3 → 0.5.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA1:
3
- metadata.gz: 6e12d64135d362d4053421a2e3d9ac0b90f01582
4
- data.tar.gz: cdf020286a2e57aa6cd21b25629db8d781a9284f
3
+ metadata.gz: 3db75bd6afaa816615c066329d862d2134aa63cb
4
+ data.tar.gz: 29a26bf2d9d7dc34d915a0f5444f5f510547df1e
5
5
  SHA512:
6
- metadata.gz: 33b2ea7ceffef945de20e6ea4602c01c750d3c7a4e01329401cf39ab549cc4a7d0c8f28e5ea51e727a3e227a5e13d1616cb1a9b3e2d0c6086451da0e054073dd
7
- data.tar.gz: 921e4017bd14f0fe3b17cde337a1608fd6aeb20e33f2e2b1b0b5adeee0cecd66141221652ee947fc44bea3ac209cbe3e17523b2ff30707bbc1b0a78cfcd60067
6
+ metadata.gz: c10a4b62d9ebab53a1fd3535001d972f46df95ce0df20432df9b73a81aac5a379d1e8967132bb557386030078ec95991658e651cb28963a294c2ae36c8b52899
7
+ data.tar.gz: e46f3df727e359413cb8ec6b4bc8697346ccc2f58ebf5e70a363f9c3b463052ce974be574c7ffc3ccee0f2174e060fd0b2b49c7d8f2da0f63306d68fd28dde1a
data/ChangeLog CHANGED
@@ -1,3 +1,11 @@
1
+ Release 0.5.0 - 2014/12/07
2
+
3
+ * Make compression algorithm pluggable
4
+ * Remove format_json parameter
5
+ * Remove s3_endpoint parameter
6
+ * Relax fluentd version restriction to support 0.12 or later
7
+
8
+
1
9
  Release 0.4.3 - 2014/11/10
2
10
 
3
11
  * Change API check message to consider region mismatch
data/README.rdoc CHANGED
@@ -21,7 +21,7 @@ Simply use RubyGems:
21
21
  aws_key_id YOUR_AWS_KEY_ID
22
22
  aws_sec_key YOUR_AWS_SECRET/KEY
23
23
  s3_bucket YOUR_S3_BUCKET_NAME
24
- s3_endpoint s3-ap-northeast-1.amazonaws.com
24
+ s3_region ap-northeast-1
25
25
  s3_object_key_format %{path}%{time_slice}_%{index}.%{file_extension}
26
26
  path logs/
27
27
  buffer_path /var/log/fluent/s3
@@ -39,8 +39,6 @@ Simply use RubyGems:
39
39
 
40
40
  [s3_region] s3 region name. For example, US West (Oregon) Region is "us-west-2". The full list of regions are available here. > http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region. We recommend using `s3_region` instead of `s3_endpoint`.
41
41
 
42
- [s3_endpoint] s3 endpoint name. For example, US West (Oregon) Region is "s3-us-west-2.amazonaws.com". The full list of endpoints are available here. > http://docs.aws.amazon.com/general/latest/gr/rande.html#s3_region
43
-
44
42
  [s3_object_key_format] The format of S3 object keys. You can use several built-in variables:
45
43
 
46
44
  - %{path}
@@ -88,6 +86,8 @@ The {fluent-mixin-config-placeholders}[https://github.com/tagomoris/fluent-mixin
88
86
  - text
89
87
  - lzo (Need lzop command)
90
88
 
89
+ See 'Use your compression algorithm' section for adding another format.
90
+
91
91
  [format] Change one line format in the S3 object. Supported formats are "out_file", "json", "ltsv" and "single_value".
92
92
 
93
93
  - out_file (default).
@@ -106,7 +106,7 @@ At this format, "time" and "tag" are omitted.
106
106
  But you can set these information to the record by setting "include_tag_key" / "tag_key" and "include_time_key" / "time_key" option.
107
107
  If you set following configuration in S3 output:
108
108
 
109
- format_json true
109
+ format json
110
110
  include_time_key true
111
111
  time_key log_time # default is time
112
112
 
@@ -165,6 +165,37 @@ Refer to the {AWS documentation}[http://docs.aws.amazon.com/IAM/latest/UserGuide
165
165
 
166
166
  Using {IAM roles}[http://docs.aws.amazon.com/IAM/latest/UserGuide/WorkingWithRoles.html] with a properly configured IAM policy are preferred over embedding access keys on EC2 instances.
167
167
 
168
+ == Use your compression algorithm
169
+
170
+ s3 plugin has plugabble compression mechanizm like Fleuntd\'s input / output plugin.
171
+ If you set 'store_as xxx', s3 plugin searches `fluent/plugin/s3_compressor_xxx.rb`.
172
+ You can define your compression with 'S3Output::Compressor' class. Compressor API is here:
173
+
174
+ module Fluent
175
+ class S3Output
176
+ class XXXCompressor < Compressor
177
+ S3Output.register_compressor('xxx', self)
178
+
179
+ # Used to file extension
180
+ def ext
181
+ 'xxx'
182
+ end
183
+
184
+ # Used to file content type
185
+ def content_type
186
+ 'application/x-xxx'
187
+ end
188
+
189
+ # chunk is buffer chunk. tmp is destination file for upload
190
+ def compress(chunk, tmp)
191
+ # call command or something
192
+ end
193
+ end
194
+ end
195
+ end
196
+
197
+ See bundled Compressor classes for more detail.
198
+
168
199
  == Website, license, et. al.
169
200
 
170
201
  Web site:: http://fluentd.org/
data/VERSION CHANGED
@@ -1 +1 @@
1
- 0.4.3
1
+ 0.5.0
@@ -16,7 +16,7 @@ Gem::Specification.new do |gem|
16
16
  gem.executables = `git ls-files -- bin/*`.split("\n").map{ |f| File.basename(f) }
17
17
  gem.require_paths = ['lib']
18
18
 
19
- gem.add_dependency "fluentd", "~> 0.10.49"
19
+ gem.add_dependency "fluentd", [">= 0.10.49", "< 2"]
20
20
  gem.add_dependency "aws-sdk", "~> 1.38"
21
21
  gem.add_dependency "yajl-ruby", "~> 1.0"
22
22
  gem.add_dependency "fluent-mixin-config-placeholders", ">= 0.3.0"
@@ -4,31 +4,24 @@ module Fluent
4
4
  class S3Output < Fluent::TimeSlicedOutput
5
5
  Fluent::Plugin.register_output('s3', self)
6
6
 
7
- unless method_defined?(:log)
8
- define_method(:log) { $log }
9
- end
10
-
11
7
  def initialize
12
8
  super
13
9
  require 'aws-sdk'
14
10
  require 'zlib'
15
11
  require 'time'
16
12
  require 'tempfile'
17
- require 'open3'
18
13
 
19
- @use_ssl = true
14
+ @compressor = nil
20
15
  end
21
16
 
22
17
  config_param :path, :string, :default => ""
23
-
18
+ config_param :use_ssl, :bool, :default => true
24
19
  config_param :aws_key_id, :string, :default => nil
25
20
  config_param :aws_sec_key, :string, :default => nil
26
21
  config_param :s3_bucket, :string
27
22
  config_param :s3_region, :string, :default => nil
28
- config_param :s3_endpoint, :string, :default => nil
29
23
  config_param :s3_object_key_format, :string, :default => "%{path}%{time_slice}_%{index}.%{file_extension}"
30
24
  config_param :store_as, :string, :default => "gzip"
31
- config_param :command_parameter, :string, :default => nil
32
25
  config_param :auto_create_bucket, :bool, :default => true
33
26
  config_param :check_apikey_on_start, :bool, :default => true
34
27
  config_param :proxy_uri, :string, :default => nil
@@ -46,40 +39,20 @@ module Fluent
46
39
  def configure(conf)
47
40
  super
48
41
 
49
- if use_ssl = conf['use_ssl']
50
- if use_ssl.empty?
51
- @use_ssl = true
52
- else
53
- @use_ssl = Config.bool_value(use_ssl)
54
- if @use_ssl.nil?
55
- raise ConfigError, "'true' or 'false' is required for use_ssl option on s3 output"
56
- end
57
- end
42
+ if conf.has_key?('s3_endpoint')
43
+ raise ConfigError, "s3_endpoint parameter is removed. Use s3_region instead"
58
44
  end
59
45
 
60
- @ext, @mime_type = case @store_as
61
- when 'gzip'
62
- ['gz', 'application/x-gzip']
63
- when 'lzo'
64
- check_command('lzop', 'LZO')
65
- @command_parameter = '-qf1' if @command_parameter.nil?
66
- ['lzo', 'application/x-lzop']
67
- when 'lzma2'
68
- check_command('xz', 'LZMA2')
69
- @command_parameter = '-qf0' if @command_parameter.nil?
70
- ['xz', 'application/x-xz']
71
- when 'json'
72
- ['json', 'application/json']
73
- else
74
- ['txt', 'text/plain']
75
- end
76
-
77
- if format_json = conf['format_json']
78
- $log.warn "format_json is deprecated. Use 'format json' instead"
79
- conf['format'] = 'json'
80
- else
81
- conf['format'] = @format
46
+ begin
47
+ @compressor = COMPRESSOR_REGISTRY.lookup(@store_as).new
48
+ rescue => e
49
+ $log.warn "#{@store_as} not found. Use 'text' instead"
50
+ @compressor = TextCompressor.new
82
51
  end
52
+ @compressor.configure(conf)
53
+
54
+ # TODO: use Plugin.new_formatter instead of TextFormatter.create
55
+ conf['format'] = @format
83
56
  @formatter = TextFormatter.create(conf)
84
57
 
85
58
  if @localtime
@@ -101,7 +74,6 @@ module Fluent
101
74
  options[:secret_access_key] = @aws_sec_key
102
75
  end
103
76
  options[:region] = @s3_region if @s3_region
104
- options[:endpoint] = @s3_endpoint if @s3_endpoint
105
77
  options[:proxy_uri] = @proxy_uri if @proxy_uri
106
78
  options[:use_ssl] = @use_ssl
107
79
 
@@ -125,7 +97,7 @@ module Fluent
125
97
  values_for_s3_object_key = {
126
98
  "path" => path,
127
99
  "time_slice" => chunk.key,
128
- "file_extension" => @ext,
100
+ "file_extension" => @compressor.ext,
129
101
  "index" => i
130
102
  }
131
103
  s3path = @s3_object_key_format.gsub(%r(%{[^}]+})) { |expr|
@@ -141,33 +113,11 @@ module Fluent
141
113
 
142
114
  tmp = Tempfile.new("s3-")
143
115
  begin
144
- if @store_as == "gzip"
145
- w = Zlib::GzipWriter.new(tmp)
146
- chunk.write_to(w)
147
- w.close
148
- elsif @store_as == "lzo"
149
- w = Tempfile.new("chunk-tmp")
150
- chunk.write_to(w)
151
- w.close
152
- tmp.close
153
- # We don't check the return code because we can't recover lzop failure.
154
- system "lzop #{@command_parameter} -o #{tmp.path} #{w.path}"
155
- elsif @store_as == "lzma2"
156
- w = Tempfile.new("chunk-xz-tmp")
157
- chunk.write_to(w)
158
- w.close
159
- tmp.close
160
- system "xz #{@command_parameter} -c #{w.path} > #{tmp.path}"
161
- else
162
- chunk.write_to(tmp)
163
- tmp.close
164
- end
165
- @bucket.objects[s3path].write(Pathname.new(tmp.path), {:content_type => @mime_type,
116
+ @compressor.compress(chunk, tmp)
117
+ @bucket.objects[s3path].write(Pathname.new(tmp.path), {:content_type => @compressor.content_type,
166
118
  :reduced_redundancy => @reduced_redundancy})
167
119
  ensure
168
120
  tmp.close(true) rescue nil
169
- w.close rescue nil
170
- w.unlink rescue nil
171
121
  end
172
122
  end
173
123
 
@@ -190,12 +140,91 @@ module Fluent
190
140
  raise "can't call S3 API. Please check your aws_key_id / aws_sec_key or s3_region configuration"
191
141
  end
192
142
 
193
- def check_command(command, algo)
194
- begin
195
- Open3.capture3("#{command} -V")
196
- rescue Errno::ENOENT
197
- raise ConfigError, "'#{command}' utility must be in PATH for #{algo} compression"
143
+ class Compressor
144
+ include Configurable
145
+
146
+ def configure(conf)
147
+ super
148
+ end
149
+
150
+ def ext
151
+ end
152
+
153
+ def content_type
154
+ end
155
+
156
+ def compress(chunk, tmp)
157
+ end
158
+
159
+ private
160
+
161
+ def check_command(command, algo = nil)
162
+ require 'open3'
163
+
164
+ algo = command if algo.nil?
165
+ begin
166
+ Open3.capture3("#{command} -V")
167
+ rescue Errno::ENOENT
168
+ raise ConfigError, "'#{command}' utility must be in PATH for #{algo} compression"
169
+ end
198
170
  end
199
171
  end
172
+
173
+ class GzipCompressor < Compressor
174
+ def ext
175
+ 'gz'.freeze
176
+ end
177
+
178
+ def content_type
179
+ 'application/x-gzip'.freeze
180
+ end
181
+
182
+ def compress(chunk, tmp)
183
+ w = Zlib::GzipWriter.new(tmp)
184
+ chunk.write_to(w)
185
+ w.close
186
+ ensure
187
+ w.close rescue nil
188
+ w.unlink rescue nil
189
+ end
190
+ end
191
+
192
+ class TextCompressor < Compressor
193
+ def ext
194
+ 'txt'.freeze
195
+ end
196
+
197
+ def content_type
198
+ 'text/plain'.freeze
199
+ end
200
+
201
+ def compress(chunk, tmp)
202
+ chunk.write_to(tmp)
203
+ tmp.close
204
+ end
205
+ end
206
+
207
+ class JsonCompressor < TextCompressor
208
+ def ext
209
+ 'json'.freeze
210
+ end
211
+
212
+ def content_type
213
+ 'application/json'.freeze
214
+ end
215
+ end
216
+
217
+ COMPRESSOR_REGISTRY = Registry.new(:s3_compressor_type, 'fluent/plugin/s3_compressor_')
218
+ {
219
+ 'gzip' => GzipCompressor,
220
+ 'json' => JsonCompressor,
221
+ 'text' => TextCompressor
222
+ }.each { |name, compressor|
223
+ COMPRESSOR_REGISTRY.register(name, compressor)
224
+ }
225
+
226
+ def self.register_compressor(name, compressor)
227
+ COMPRESSOR_REGISTRY.register(name, compressor)
228
+ end
200
229
  end
201
230
  end
@@ -0,0 +1,34 @@
1
+ module Fluent
2
+ class S3Output
3
+ class LZMA2Compressor < Compressor
4
+ S3Output.register_compressor('lzma2', self)
5
+
6
+ config_param :command_parameter, :string, :default => '-qf0'
7
+
8
+ def configure(conf)
9
+ super
10
+ check_command('xz', 'LZMA2')
11
+ end
12
+
13
+ def ext
14
+ 'xz'.freeze
15
+ end
16
+
17
+ def content_type
18
+ 'application/x-xz'.freeze
19
+ end
20
+
21
+ def compress(chunk, tmp)
22
+ w = Tempfile.new("chunk-xz-tmp")
23
+ chunk.write_to(w)
24
+ w.close
25
+ tmp.close
26
+ # We don't check the return code because we can't recover lzop failure.
27
+ system "xz #{@command_parameter} -c #{w.path} > #{tmp.path}"
28
+ ensure
29
+ w.close rescue nil
30
+ w.unlink rescue nil
31
+ end
32
+ end
33
+ end
34
+ end
@@ -0,0 +1,34 @@
1
+ module Fluent
2
+ class S3Output
3
+ class LZOCompressor < Compressor
4
+ S3Output.register_compressor('lzo', self)
5
+
6
+ config_param :command_parameter, :string, :default => '-qf1'
7
+
8
+ def configure(conf)
9
+ super
10
+ check_command('lzop', 'LZO')
11
+ end
12
+
13
+ def ext
14
+ 'lzo'.freeze
15
+ end
16
+
17
+ def content_type
18
+ 'application/x-lzop'.freeze
19
+ end
20
+
21
+ def compress(chunk, tmp)
22
+ w = Tempfile.new("chunk-tmp")
23
+ chunk.write_to(w)
24
+ w.close
25
+ tmp.close
26
+ # We don't check the return code because we can't recover lzop failure.
27
+ system "lzop #{@command_parameter} -o #{tmp.path} #{w.path}"
28
+ ensure
29
+ w.close rescue nil
30
+ w.unlink rescue nil
31
+ end
32
+ end
33
+ end
34
+ end
data/test/test_out_s3.rb CHANGED
@@ -42,32 +42,32 @@ class S3OutputTest < Test::Unit::TestCase
42
42
  assert_equal 'test_bucket', d.instance.s3_bucket
43
43
  assert_equal 'log', d.instance.path
44
44
  assert d.instance.instance_variable_get(:@use_ssl)
45
- assert_equal 'gz', d.instance.instance_variable_get(:@ext)
46
- assert_equal 'application/x-gzip', d.instance.instance_variable_get(:@mime_type)
45
+ assert_equal 'gz', d.instance.instance_variable_get(:@compressor).ext
46
+ assert_equal 'application/x-gzip', d.instance.instance_variable_get(:@compressor).content_type
47
47
  end
48
48
 
49
49
  def test_configure_with_mime_type_json
50
50
  conf = CONFIG.clone
51
51
  conf << "\nstore_as json\n"
52
52
  d = create_driver(conf)
53
- assert_equal 'json', d.instance.instance_variable_get(:@ext)
54
- assert_equal 'application/json', d.instance.instance_variable_get(:@mime_type)
53
+ assert_equal 'json', d.instance.instance_variable_get(:@compressor).ext
54
+ assert_equal 'application/json', d.instance.instance_variable_get(:@compressor).content_type
55
55
  end
56
56
 
57
57
  def test_configure_with_mime_type_text
58
58
  conf = CONFIG.clone
59
59
  conf << "\nstore_as text\n"
60
60
  d = create_driver(conf)
61
- assert_equal 'txt', d.instance.instance_variable_get(:@ext)
62
- assert_equal 'text/plain', d.instance.instance_variable_get(:@mime_type)
61
+ assert_equal 'txt', d.instance.instance_variable_get(:@compressor).ext
62
+ assert_equal 'text/plain', d.instance.instance_variable_get(:@compressor).content_type
63
63
  end
64
64
 
65
65
  def test_configure_with_mime_type_lzo
66
66
  conf = CONFIG.clone
67
67
  conf << "\nstore_as lzo\n"
68
68
  d = create_driver(conf)
69
- assert_equal 'lzo', d.instance.instance_variable_get(:@ext)
70
- assert_equal 'application/x-lzop', d.instance.instance_variable_get(:@mime_type)
69
+ assert_equal 'lzo', d.instance.instance_variable_get(:@compressor).ext
70
+ assert_equal 'application/x-lzop', d.instance.instance_variable_get(:@compressor).content_type
71
71
  rescue => e
72
72
  # TODO: replace code with disable lzop command
73
73
  assert(e.is_a?(Fluent::ConfigError))
@@ -147,22 +147,8 @@ class S3OutputTest < Test::Unit::TestCase
147
147
  d.run
148
148
  end
149
149
 
150
- def test_format_with_format_json_deprecated
151
- config = [CONFIG, 'format_json true'].join("\n")
152
- d = create_driver(config)
153
-
154
- time = Time.parse("2011-01-02 13:14:15 UTC").to_i
155
- d.emit({"a"=>1}, time)
156
- d.emit({"a"=>2}, time)
157
-
158
- d.expect_format %[{"a":1}\n]
159
- d.expect_format %[{"a":2}\n]
160
-
161
- d.run
162
- end
163
-
164
150
  def test_format_with_format_json_included_tag
165
- config = [CONFIG, 'format_json true', 'include_tag_key true'].join("\n")
151
+ config = [CONFIG, 'format json', 'include_tag_key true'].join("\n")
166
152
  d = create_driver(config)
167
153
 
168
154
  time = Time.parse("2011-01-02 13:14:15 UTC").to_i
metadata CHANGED
@@ -1,7 +1,7 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: fluent-plugin-s3
3
3
  version: !ruby/object:Gem::Version
4
- version: 0.4.3
4
+ version: 0.5.0
5
5
  platform: ruby
6
6
  authors:
7
7
  - Sadayuki Furuhashi
@@ -9,22 +9,28 @@ authors:
9
9
  autorequire:
10
10
  bindir: bin
11
11
  cert_chain: []
12
- date: 2014-11-11 00:00:00.000000000 Z
12
+ date: 2014-12-07 00:00:00.000000000 Z
13
13
  dependencies:
14
14
  - !ruby/object:Gem::Dependency
15
15
  name: fluentd
16
16
  requirement: !ruby/object:Gem::Requirement
17
17
  requirements:
18
- - - "~>"
18
+ - - ">="
19
19
  - !ruby/object:Gem::Version
20
20
  version: 0.10.49
21
+ - - "<"
22
+ - !ruby/object:Gem::Version
23
+ version: '2'
21
24
  type: :runtime
22
25
  prerelease: false
23
26
  version_requirements: !ruby/object:Gem::Requirement
24
27
  requirements:
25
- - - "~>"
28
+ - - ">="
26
29
  - !ruby/object:Gem::Version
27
30
  version: 0.10.49
31
+ - - "<"
32
+ - !ruby/object:Gem::Version
33
+ version: '2'
28
34
  - !ruby/object:Gem::Dependency
29
35
  name: aws-sdk
30
36
  requirement: !ruby/object:Gem::Requirement
@@ -110,6 +116,8 @@ files:
110
116
  - VERSION
111
117
  - fluent-plugin-s3.gemspec
112
118
  - lib/fluent/plugin/out_s3.rb
119
+ - lib/fluent/plugin/s3_compressor_lzma2.rb
120
+ - lib/fluent/plugin/s3_compressor_lzo.rb
113
121
  - test/test_out_s3.rb
114
122
  homepage: https://github.com/fluent/fluent-plugin-s3
115
123
  licenses: []