fluent-plugin-aliyun-oss 0.0.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: e97dcc54cfbae27991f9c36fb3cfba27eea1b34dbb4e042c59373581dff3dcd9
+  data.tar.gz: c20c73e48b3acd11e18f8f37b5f07617fbe7839b46e9293f5a373c54cf1778fc
+SHA512:
+  metadata.gz: 13b8298032dc9c39fe28cbc7e79f56868df3b022166cf1b3a304c36ea815a86343e73eb4bfbac3bf1604d28a9cde11e197e83e8cb62b0ac8d3469f528c5ee9bc
+  data.tar.gz: 4597eeabd1865a7291c60cc5caffd9125cb1d215b697696649fc5613329d57c5d4bb6389a208bf9e688f92ab0c34aff650a8edc23c5f9df8791a222b02db2bef

data/.gitignore

@@ -0,0 +1,14 @@
+~*
+#*
+*~
+[._]*.s[a-w][a-z]
+.DS_Store
+
+*.gem
+.bundle
+Gemfile.lock
+vendor
+.ruby-version
+.idea/*
+
+test/tmp/

data/AUTHORS

@@ -0,0 +1 @@
+Jinhu Wu <jinhu.wu.nju _at_ gmail.com>

data/ChangeLog

@@ -0,0 +1,2 @@
+Release 0.0.1 - 2019/04/23
+  * First release

data/Gemfile

@@ -0,0 +1,6 @@
+source 'http://rubygems.org'
+
+gem 'rspec'
+gem 'uuidtools'
+
+gemspec

data/README.md

@@ -0,0 +1,342 @@
+# Aliyun OSS plugin for [Fluentd](http://github.com/fluent/fluentd)
+
+It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it whatever you want.
+
+## Overview
+**Fluent OSS output plugin** buffers event logs in local files and uploads them to OSS periodically in background threads.
+
+This plugin splits events by using the timestamp of event logs. For example,  a log '2019-04-09 message Hello' is reached, and then another log '2019-04-10 message World' is reached in this order, the former is stored in "20190409.gz" file, and latter in "20190410.gz" file.
+
+**Fluent OSS input plugin** reads data from OSS periodically.
+
+This plugin uses MNS on the same region of the OSS bucket. We must setup MNS and OSS event notification before using this plugin.
+
+[This document](https://help.aliyun.com/document_detail/52656.html) shows how to setup MNS and OSS event notification.
+
+This plugin will poll events from MNS queue and extract object keys from these events, and then will read those objects from OSS.
+
+## Installation
+
+Simply use RubyGems(Run command in td-agent installation directory):
+```bash
+[root@master td-agent]# ./embedded/bin/fluent-gem install fluent-plugin-oss
+```
+Then, you can check installed plugin
+```bash
+[root@master td-agent]# ./embedded/bin/fluent-gem list fluent-plugin-oss
+
+*** LOCAL GEMS ***
+
+fluent-plugin-oss (0.0.1)
+```
+
+## Development
+
+### 1. Plugin Developement and Testing
+
+#### Code
+- Install dependencies
+
+```sh
+bundle install
+```
+
+#### Test
+
+- Update your dependencies
+
+```sh
+bundle install
+```
+
+- Run tests
+You should set environment variables like below:
+
+test_out_oss.rb
+```sh
+STORE_AS="" OSS_ENDPOINT="" ACCESS_KEY_ID="" ACCESS_KEY_SECRET="" OSS_BUCKET="" OSS_PATH=""  bundle exec rspec test/plugin/test_out_oss.rb
+```
+
+test_in_oss.rb
+```sh
+STORE_AS=""  OSS_ENDPOINT="" ACCESS_KEY_ID="" ACCESS_KEY_SECRET="" OSS_BUCKET="" MNS_ENDPOINT="" MNS_QUEUE=""  bundle exec rspec test/plugin/test_in_oss.rb
+
+```
+
+## Usage
+This is an example of fluent config.
+
+It will read data posted by HTTP and buffer data to local directory before writing to OSS.
+You can try it by running curl command:
+```bash
+[root@master td-agent]# while [[ 1 ]]; do curl -X POST -d 'json={"json":"message"}' http://localhost:8888/debug.test; done
+```
+	<match debug.*>
+	  @type oss
+	  endpoint <OSS endpoint to connect to>
+	  bucket <Your Bucket>
+	  access_key_id <Your Access Key>
+	  access_key_secret <Your Secret Key>
+	  path fluent-oss/logs
+	  auto_create_bucket true
+	  key_format %{path}/%{time_slice}_%{index}_%{thread_id}.%{file_extension}
+	  store_as gzip
+	  <buffer tag,time>
+	    @type file
+	    path /var/log/fluent/oss
+	    timekey 60 # 1 min partition
+	    timekey_wait 20s
+	    #timekey_use_utc true
+	  </buffer>
+	  <format>
+	    @type json
+	  </format>
+	</match>
+	
+	# HTTP input
+	# POST http://localhost:8888/<tag>?json=<json>
+	# POST http://localhost:8888/td.myapp.login?json={"user"%3A"me"}
+	# @see http://docs.fluentd.org/articles/in_http
+	<source>
+	  @type http
+	  @id input_http
+	  port 8888
+	</source>
+
+
+## Configuration: Output Plugin
+This plugin supports the following configuration options
+
+|Configuration|Type|Required|Comments|Default|
+|:---:|:---:|:---:|:---:|:---|
+|endpoint|string|Yes|OSS endpoint to connect|
+|bucket|string|Yes|Your OSS bucket name|
+|access_key_id|string|Yes|Your access key id|
+|access_key_secret|string|Yes|Your access secret key|
+|path|string|No|Prefix that added to the generated file name|fluent/logs|
+|oss_sdk_log_dir|string|No|OSS SDK log directory|/var/log/td-agent|
+|upload_crc_enable|bool|No|Enable upload crc check|true|
+|download_crc_enable|bool|No|Enable download crc check|true|
+|open_timeout|integer|No|Timeout seconds for open connections|10|
+|read_timeout|integer|No|Timeout seconds for read response|120|
+|key_format|string|No|The format of OSS object keys|%{path}/%{time_slice}\_%{index}\_%{thread_id}.%{file_extension}|
+|store_as|string|No|Archive format on OSS|gzip|
+|auto_create_bucket|bool|No|Create OSS bucket if it does not exists|true|
+|overwrite|bool|No|Overwrite already existing OSS path|false|
+|check_bucket|bool|No|Check bucket if exists or not|true|
+|check_object|bool|No|Check object before creation|true|
+|hex_random_length|integer|No|The length of `%{hex_random}` placeholder(4-16)|4|
+|index_format|string|No|`sprintf` format for `%{index}`|%d|
+|warn_for_delay|time|No|Set a threshold of events latency and mark these slow events as delayed, output warning logs if delayed events were put into OSS|nil|
+
+### Some configuration details
+**key_format**
+
+The format of OSS object keys. You can use the following built-in variables to generate keys dynamically:
+*   %{path}
+*   %{time_slice}
+*   %{index}
+*   %{file_extension}
+*   %{hex_random}
+*   %{uuid_flush}
+*   %{thread_id}
+
+* %{path} is exactly the value of **path** configured in the configuration file.
+E.g., "fluent/logs" in the example configuration above.
+* %{time_slice} is the time-slice in text that are formatted with **time_slice_format**.
+* %{index} is the sequential number starts from 0, increments when multiple files are uploaded to OSS in the same time slice.
+* %{file_extension} depends on **store_as** parameter.
+* %{thread_id} is the unique ids of flush threads(flush thread number is define by `flush_thread_count`). You can use %{thread_id} with other built-in variables to make unique object names. 
+* %{uuid_flush} a uuid that is renewed everytime the buffer is flushed. If you want to use this placeholder, install `uuidtools` gem first.
+* %{hex_random} a random hex string that is renewed for each buffer chunk, not
+guaranteed to be unique. This is used for performance tuning as the article below described,
+[OSS performance best practice](https://help.aliyun.com/document_detail/64945.html).
+You can configure the length of string with a
+`hex_random_length` parameter (Default is 4).
+
+The default format is `%{path}/%{time_slice}_%{index}_%{thread_id}.%{file_extension}`.
+For instance, using the example configuration above, actual object keys on OSS
+will be something like(flush_thread_count is 1):
+
+    "fluent-oss/logs_20190410-10_15_0_69928273148640.gz"
+    "fluent-oss/logs_20190410-10_16_0_69928273148640.gz"
+    "fluent-oss/logs_20190410-10_17_0_69928273148640.gz"
+    
+With the configuration(flush_thread_count is 2):
+
+    key_format %{path}/events/ts=%{time_slice}/events_%{index}_%{thread_id}.%{file_extension}
+    time_slice_format %Y%m%d-%H
+    path fluent-oss/logs
+
+You get:
+
+    fluent-oss/logs/events/ts=20190410-10/events_0_69997953090220.gz
+    fluent-oss/logs/events/ts=20190410-10/events_0_69997953090620.gz
+    fluent-oss/logs/events/ts=20190410-10/events_1_69997953090220.gz
+    fluent-oss/logs/events/ts=20190410-10/events_1_69997953090620.gz
+    fluent-oss/logs/events/ts=20190410-10/events_2_69997953090220.gz
+    fluent-oss/logs/events/ts=20190410-10/events_2_69997953090620.gz
+    
+This plugin also supports add hostname to the final object keys, with the configuration:
+
+**Note:** You should add double quotes to value of `key_format` if use this feature
+
+    key_format "%{path}/events/ts=%{time_slice}/#{Socket.gethostname}/events_%{index}_%{thread_id}.%{file_extension}"
+    time_slice_format %Y%m%d-%H
+    path fluent-oss/logs
+    
+You get(flush_thread_count is 1):
+
+    fluent-oss/logs/events/ts=20190410-10/master/events_0_70186087552680.gz
+    fluent-oss/logs/events/ts=20190410-10/master/events_1_70186087552680.gz
+    fluent-oss/logs/events/ts=20190410-10/master/events_2_70186087552680.gz
+    
+**store_as**
+
+archive format on OSS. You can use several format:
+*   gzip (default)
+*   json
+*   text
+*   lzo (Need lzop command)
+*   lzma2 (Need xz command)
+*   gzip_command (Need gzip command)
+    *   This compressor uses an external gzip command, hence would result in
+        utilizing CPU cores well compared with `gzip`
+            
+**auto_create_bucket**
+
+Create OSS bucket if it does not exists. Default is true.
+
+**check_bucket**
+
+Check configured bucket if it exists or not. Default is true.
+When it is false, fluentd will not check the existence of the configured bucket.
+This is the case where bucket will be pre-created before running fluentd.
+
+**check_object**
+
+Check object before creation if it exists or not. Default is true.
+
+When it is false, key_format will be %{path}/%{time_slice}\_%{hms_slice}\_%{thread_id}.%{file_extension} by default where,
+hms_slice will be time-slice in hhmmss format. With hms_slice and thread_id, each object is unique.
+Example object name, assuming it is created on 2019/04/10 10:30:54 AM 20190410_103054_70186087552260.txt (extension can be anything as per user's choice)
+
+**path**
+
+Path prefix of the files on OSS. Default is "fluent-oss/logs".
+
+**time_slice_format**
+
+Format of the time used as the file name. Default is '%Y%m%d'. Use
+'%Y%m%d%H' to split files hourly.
+
+**utc**
+
+Use UTC instead of local time.
+
+**hex_random_length**
+
+The length of `%{hex_random}` placeholder. Default is 4.
+
+**index_format**
+
+`%{index}` is formatted by [sprintf](http://ruby-doc.org/core-2.2.0/Kernel.html#method-i-sprintf) using this format_string. Default is '%d'. Zero padding is supported e.g. `%04d` to ensure minimum length four digits. `%{index}` can be in lowercase or uppercase hex using '%x' or '%X'
+
+**overwrite**
+
+Overwrite already existing path. Default is false, which raises an error
+if an OSS object of the same path already exists, or increment the
+`%{index}` placeholder until finding an absent path.
+
+**warn_for_delay**
+
+Set a threshold to treat events as delay, output warning logs if delayed events were put into OSS.
+
+## Configuration: Input Plugin
+
+|Configuration|Type|Required|Comments|Default|
+|:---:|:---:|:---:|:---:|:---|
+|endpoint|string|Yes|OSS endpoint to connect|
+|bucket|string|Yes|Your OSS bucket name|
+|access_key_id|string|Yes|Your access key id|
+|access_key_secret|string|Yes|Your access secret key|
+|oss_sdk_log_dir|string|No|OSS SDK log directory|/var/log/td-agent|
+|upload_crc_enable|bool|No|Enable upload crc check|true|
+|download_crc_enable|bool|No|Enable download crc check|true|
+|open_timeout|integer|No|Timeout seconds for open connections|10|
+|read_timeout|integer|No|Timeout seconds for read response|120|
+|store_as|string|No|Archive format on OSS|gzip|
+|flush_batch_lines|integer|No|Flush to down streams every `flush_batch_lines` lines.|10000|
+|flush_pause_milliseconds|integer|No|Sleep interval between two flushes to downstream.|1|
+|store_local|bool|No|Store OSS Objects to local or memory before parsing(Used for objects with `text`/`json`/`gzip` formats)|true|
+|mns|configuration section|Yes|MNS configurations|
+
+### Usage
+This is an example of fluent config.
+
+    <source>
+      @type oss
+      endpoint <OSS endpoint to connect to>
+      bucket <Your Bucket>
+      access_key_id <Your Access Key>
+      access_key_secret <Your Secret Key>
+      flush_batch_lines 1000
+      <mns>
+        endpoint <MNS endpoint to connect to, E.g.,{account-id}.mns.cn-zhangjiakou-internal.aliyuncs.com>
+        queue <MNS queue>
+        wait_seconds 10
+        poll_interval_seconds 10
+      </mns>
+    </source>
+
+### Some configuration details
+
+**store_as**
+archive format on OSS. You can use several format:
+*   gzip (default)
+*   json
+*   text
+*   lzo (Need lzop command)
+*   lzma2 (Need xz command)
+*   gzip_command (Need gzip command)
+    *   This compressor uses an external gzip command, hence would result in
+        utilizing CPU cores well compared with `gzip`
+
+**flush_batch_lines**
+
+Flush to down streams every `flush_batch_lines` lines.
+
+**flush_pause_milliseconds**
+
+Sleep interval between two flushes to downstream. Default is 1ms, and wil not sleep if `flush_pause_milliseconds` is less than or equal to 0.
+
+**store_local(default is true)**
+
+Store OSS Objects to local or memory before parsing(Used for objects with `text`/`json`/`gzip` formats).
+
+Objects with `lzo`/`lzma2`/`gzip_command` formats are always stored to local directory before parsing.
+
+**format**
+
+Parse a line as this format in the OSS object. Supported formats are "apache_error", "apache2", "syslog", "json", "tsv", "ltsv", "csv", "nginx" and "none".
+
+**mns**
+
+[MNS consume messages](https://help.aliyun.com/document_detail/35136.html)
+
+*   endpoint
+*   queue
+*   wait_seconds
+*   poll_interval_seconds  Poll messages interval from MNS
+
+For more details about mns configurations, please view MNS documentation in the link above.
+
+## Website, license, et. al.
+
+| Web site          | http://fluentd.org/                       |
+|-------------------|-------------------------------------------|
+| Documents         | http://docs.fluentd.org/                  |
+| Source repository | http://github.com/aliyun/fluent-plugin-oss |
+| Discussion        | http://groups.google.com/group/fluentd    |
+| Author            | Jinhu Wu                        |
+| License           | Apache License, Version 2.0               |

data/VERSION

@@ -0,0 +1 @@
+0.0.1

data/fluent-plugin-oss.gemspec

@@ -0,0 +1,24 @@
+$LOAD_PATH.push File.expand_path('lib', __dir__)
+
+Gem::Specification.new do |gem|
+  gem.name        = 'fluent-plugin-aliyun-oss'
+  gem.description = 'Aliyun OSS plugin for Fluentd event collector'
+  gem.license     = 'Apache-2.0'
+  gem.homepage    = 'https://github.com/aliyun/fluent-plugin-oss'
+  gem.summary     = gem.description
+  gem.version     = File.read('VERSION').strip
+  gem.authors     = ['Jinhu Wu']
+  gem.email       = 'jinhu.wu.nju@gmail.com'
+  gem.files       = `git ls-files`.split("\n")
+  gem.test_files  = `git ls-files -- {test,spec,features}/*`.split("\n")
+  gem.executables = `git ls-files -- bin/*`.split("\n").map do |f|
+    File.basename(f)
+  end
+  gem.require_paths = ['lib']
+
+  gem.add_dependency 'aliyun-sdk', ['0.7.0']
+  gem.add_dependency 'fluentd', ['>= 0.14.22', '< 2']
+  gem.add_development_dependency 'rake', '~> 0.9', '>= 0.9.2'
+  gem.add_development_dependency 'test-unit', '~> 3.0', '>= 3.0.8'
+  gem.add_development_dependency 'test-unit-rr', '~> 1.0', '>= 1.0.3'
+end

data/lib/fluent/plugin/in_oss.rb

@@ -0,0 +1,348 @@
+require 'fluent/plugin/input'
+require 'aliyun/oss'
+require 'aliyun/sts'
+require 'base64'
+require 'fluent/plugin/mns/request'
+require 'fluent/plugin/mns/message'
+require 'json'
+require 'zlib'
+
+# This is Fluent OSS Input Plugin
+module Fluent
+  # Fluent OSS Plugin
+  module Plugin
+    # OSSInput class implementation
+    class OSSInput < Input
+      Fluent::Plugin.register_input('oss', self)
+
+      helpers :compat_parameters, :parser, :thread
+
+      DEFAULT_PARSE_TYPE = 'none'.freeze
+
+      desc 'OSS endpoint to connect to'
+      config_param :endpoint, :string
+      desc 'Your bucket name'
+      config_param :bucket, :string
+      desc 'Your access key id'
+      config_param :access_key_id, :string
+      desc 'Your access secret key'
+      config_param :access_key_secret, :string
+      config_param :upload_crc_enable, :bool, default: true
+      config_param :download_crc_enable, :bool, default: true
+      desc 'Timeout for open connections'
+      config_param :open_timeout, :integer, default: 10
+      desc 'Timeout for read response'
+      config_param :read_timeout, :integer, default: 120
+
+      desc 'OSS SDK log directory'
+      config_param :oss_sdk_log_dir, :string, default: '/var/log/td-agent'
+
+      desc 'Archive format on OSS'
+      config_param :store_as, :string, default: 'gzip'
+
+      desc 'Flush to down streams every `flush_batch_lines` lines'
+      config_param :flush_batch_lines, :integer, default: 1000
+
+      desc 'Sleep interval between two flushes to downstream'
+      config_param :flush_pause_milliseconds, :integer, default: 1
+
+      desc 'Store OSS Objects to local or memory before parsing'
+      config_param :store_local, :bool, default: true
+
+      config_section :mns, required: true, multi: false do
+        desc 'MNS endpoint to connect to'
+        config_param :endpoint, :string
+        desc 'MNS queue to poll messages'
+        config_param :queue, :string
+        desc 'MNS max waiting time to receive messages'
+        config_param :wait_seconds, :integer, default: nil
+        desc 'Poll messages interval from MNS'
+        config_param :poll_interval_seconds, :integer, default: 30
+      end
+
+      def initialize
+        super
+        @decompressor = nil
+      end
+
+      desc 'Tag string'
+      config_param :tag, :string, default: 'input.oss'
+
+      config_section :parse do
+        config_set_default :@type, DEFAULT_PARSE_TYPE
+      end
+
+      def configure(conf)
+        super
+
+        raise Fluent::ConfigError, 'Invalid oss endpoint' if @endpoint.nil?
+
+        raise Fluent::ConfigError, 'Invalid mns endpoint' if @mns.endpoint.nil?
+
+        raise Fluent::ConfigError, 'Invalid mns queue' if @mns.queue.nil?
+
+        @decompressor = DECOMPRESSOR_REGISTRY.lookup(@store_as).new(log: log)
+        @decompressor.configure(conf)
+
+        parser_config = conf.elements('parse').first
+        @parser = parser_create(conf: parser_config, default_type: DEFAULT_PARSE_TYPE)
+
+        @flush_pause_milliseconds *= 0.001
+      end
+
+      def multi_workers_ready?
+        true
+      end
+
+      def start
+        @oss_sdk_log_dir += '/' unless @oss_sdk_log_dir.end_with?('/')
+        Aliyun::Common::Logging.set_log_file(@oss_sdk_log_dir + Aliyun::Common::Logging::DEFAULT_LOG_FILE)
+        create_oss_client unless @oss
+
+        check_bucket
+        super
+
+        @running = true
+        thread_create(:in_oss, &method(:run))
+      end
+
+      def check_bucket
+        unless @oss.bucket_exist?(@bucket)
+          raise "The specified bucket does not exist: bucket = #{@bucket}"
+        end
+
+        @bucket_handler = @oss.get_bucket(@bucket)
+      end
+
+      def create_oss_client
+        @oss = Aliyun::OSS::Client.new(
+          endpoint: @endpoint,
+          access_key_id: @access_key_id,
+          access_key_secret: @access_key_secret,
+          download_crc_enable: @download_crc_enable,
+          upload_crc_enable: @upload_crc_enable,
+          open_timeout: @open_timeout,
+          read_timeout: @read_timeout
+        )
+      end
+
+      def shutdown
+        @running = false
+        super
+      end
+
+      private
+
+      def run
+        while @running
+          log.info "start to poll message from MNS queue #{@mns.queue}"
+          message = receive_message(@mns.queue, @mns.wait_seconds)
+          process(Fluent::Plugin::MNS::Message.new(@mns.queue, message)) unless message.nil?
+          sleep(@mns.poll_interval_seconds)
+        end
+      end
+
+      def receive_message(queue, wait_seconds)
+        request_opts = {}
+        request_opts = { waitseconds: wait_seconds } if wait_seconds
+        opts = {
+          log: log,
+          method: 'GET',
+          endpoint: @mns.endpoint,
+          path: "/queues/#{queue}/messages",
+          access_key_id: @access_key_id,
+          access_key_secret: @access_key_secret
+        }
+        Fluent::Plugin::MNS::Request.new(opts, {}, request_opts).execute
+      end
+
+      def process(message)
+        objects = get_objects(message)
+        objects.each do |object|
+          key = object.key
+          log.info "read object #{key}, size #{object.size} from OSS"
+
+          if @bucket_handler.object_exists?(key)
+            if @decompressor.save_to_local?
+              io = Tempfile.new('chunk-' + @store_as + '-in-')
+              io.binmode
+              @bucket_handler.get_object(key) do |chunk|
+                io.write(chunk)
+              end
+            else
+              io = StringIO.new
+              @bucket_handler.get_object(key) do |chunk|
+                io << chunk
+              end
+            end
+
+            io.rewind
+
+            begin
+              content = @decompressor.decompress(io)
+            rescue StandardError => ex
+              log.warn "#{ex}, skip object #{key}"
+              next
+            end
+
+            es = Fluent::MultiEventStream.new
+            content.each_line do |line|
+              @parser.parse(line) do |time, record|
+                es.add(time, record)
+              end
+
+              if es.size >= @flush_batch_lines
+                router.emit_stream(@tag, es)
+                es = Fluent::MultiEventStream.new
+                if @flush_pause_milliseconds > 0
+                  sleep(@flush_pause_milliseconds)
+                end
+              end
+            end
+            router.emit_stream(@tag, es)
+            io.close(true) rescue nil if @decompressor.save_to_local?
+          else
+            log.warn "in_oss: object #{key} does not exist!"
+          end
+        end
+        delete_message(@mns.queue, message)
+      end
+
+      def get_objects(message)
+        objects = []
+        events = JSON.parse(Base64.decode64(message.body))['events']
+        events.each do |event|
+          objects.push(OSSObject.new(event['eventName'],
+                                     @bucket,
+                                     event['oss']['object']['key'],
+                                     event['oss']['object']['size'],
+                                     event['oss']['object']['eTag']))
+        end
+        objects
+      end
+
+      def delete_message(queue, message)
+        request_opts = { ReceiptHandle: message.receipt_handle }
+        opts = {
+          log: log,
+          method: 'DELETE',
+          endpoint: @mns.endpoint,
+          path: "/queues/#{queue}/messages",
+          access_key_id: @access_key_id,
+          access_key_secret: @access_key_secret
+        }
+        Fluent::Plugin::MNS::Request.new(opts, {}, request_opts).execute
+      end
+
+      # OSS Object class from MNS events
+      class OSSObject
+        attr_reader :event_name, :bucket, :key, :size, :etag
+        def initialize(event_name, bucket, key, size, etag)
+          @event_name = event_name
+          @bucket = bucket
+          @key = key
+          @size = size
+          @etag = etag
+        end
+      end
+
+      # Decompression base class.
+      class Decompressor
+        include Fluent::Configurable
+
+        attr_reader :log
+
+        def initialize(opts = {})
+          super()
+          @log = opts[:log]
+        end
+
+        def ext; end
+
+        def save_to_local?
+          true
+        end
+
+        def content_type; end
+
+        def decompress(io); end
+
+        private
+
+        def check_command(command, encode = nil)
+          require 'open3'
+
+          encode = command if encode.nil?
+          begin
+            Open3.capture3("#{command} -V")
+          rescue Errno::ENOENT
+            raise Fluent::ConfigError,
+                  "'#{command}' utility must be in PATH for #{encode} decompression"
+          end
+        end
+      end
+
+      # Gzip decompression.
+      class GZipDecompressor < Decompressor
+        def ext
+          'gz'.freeze
+        end
+
+        def content_type
+          'application/x-gzip'.freeze
+        end
+
+        def decompress(io)
+          Zlib::GzipReader.wrap(io)
+        end
+
+        def save_to_local?
+          config['store_local']
+        end
+      end
+
+      # Text decompression.
+      class TextDecompressor < Decompressor
+        def ext
+          'txt'.freeze
+        end
+
+        def content_type
+          'text/plain'.freeze
+        end
+
+        def decompress(io)
+          io
+        end
+
+        def save_to_local?
+          config['store_local']
+        end
+      end
+
+      # Json decompression.
+      class JsonDecompressor < TextDecompressor
+        def ext
+          'json'.freeze
+        end
+
+        def content_type
+          'application/json'.freeze
+        end
+      end
+
+      DECOMPRESSOR_REGISTRY = Fluent::Registry.new(:oss_decompressor_type, 'fluent/plugin/oss_decompressor_')
+      {
+        'gzip' => GZipDecompressor,
+        'text' => TextDecompressor,
+        'json' => JsonDecompressor
+      }.each do |name, decompressor|
+        DECOMPRESSOR_REGISTRY.register(name, decompressor)
+      end
+
+      def self.register_decompressor(name, decompressor)
+        DECOMPRESSOR_REGISTRY.register(name, decompressor)
+      end
+    end
+  end
+end