logstash-input-qingstor 0.1.3 → 0.1.5

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 2c3ef515c2bf42830ada2c07360ee1444a239470
-  data.tar.gz: 24375bc480c7560c2fb358cd3d6b65635d0b18bc
+  metadata.gz: 6bcfff3cc30d80ea28ba519311f4fa572ae10bdb
+  data.tar.gz: 0775ebfb4fe462564e9430b1553bf32a47279a85
 SHA512:
-  metadata.gz: 924eb9d4eb958184a9efa290195a5728fcae475be0f948d71bffa204dfabea6df7bef25dd21124abefa4b0f0276ef490fce0438c88eb027a3cd7bad1e0aaa18f
-  data.tar.gz: fc3332dab20489769f43e5cf0e26939bfdbd4978ebfea9e5e8d10caaf76d012c02489108d0a5899ddce3abee38e2f5913182e662ad281fbb25a9c61d709c5f81
+  metadata.gz: 15985ef6e93965b02752c7de12f158269e648b3447ec517abd9fa2224dcec0f03bedb8177e1552460a4d8e6234bb5d01bcc0b9c6b3c1db0a21fe48e3cee851eb
+  data.tar.gz: bbca219710ce7f3357bbbbfa6eb1b5c9ef5d1b185ca64d0224513942010b12d43a1a4bca7d866c9639fcbfd95a7025bd9027ee905c1dc4f52c99db056617b7df

data/Gemfile

@@ -1,3 +1,2 @@
 source 'https://rubygems.org'
 gemspec
-

data/README.md

@@ -1,124 +1,7 @@
-# Logstash Input Plugin for Qingstor 
+# Logstash Output Plugin for QingStor 
 
-这是一个适配了[Qingstor](https://www.qingcloud.com/products/storage#qingstor), 工作在logstash中的input插件.  Qingstor是[Qingcloud](https://www.qingcloud.com/)推出的对象存储服务.  
-作为一个input插件, 它能下载存储在Qingstor上的日志文件, 读入到logstash中进行进一步处理.  
-详细功能参考下面配置说明.  
-
-目前插件已经提交至rubygems.org, 使用以下命令安装:
-```sh
-    bin/logstash-plugin install logstash-input-qingstor
-```
- 手动安装本地代码, 安装方法参考下文.
-
-## 1. 配置说明
-
-#### 1.1 最小运行配置
-- 使用'-f' 接受一个*.conf文件或者使用'-e'参数最小运行配置时, 至少需要以下三项
-```sh
-input {
-    qingstor {
-        access_key_id => 'your_access_key_id'           #required 
-        secret_access_key => 'your_secret_access_key'   #required  
-        bucket => 'bucket_name'                         #required 
-        # region => "pek3a"                             #optional, default value "pek3a"                                
-    }
-}
-
-```
-
-#### 1.2 其他可选参数说明
-```sh
-input {
-    qingstor {
-        ......
-        # 指定下载文件的前缀. 
-        # 默认nil, 
-        prefix => 'aprefix'
-
-        # 本地保存临时文件的目录. 
-        # 默认: 系统临时文件目录下的qingstor2logstash文件夹, 例如linux下 "/tmp/qingstor2logstash".
-        tmpdir => '/local/temporary/directory' 
-
-        # 是否在处理之后, 删除远程bucket中的文件.
-        # 默认: false
-        delete_remote_files => true
-
-        # 重新配置qingstor的地址
-        # 默认: nil
-        host => "new.qingstor.net"
-
-        # 重新配置qingstor地址的端口号
-        # 默认: 443
-        port => 443
-
-        # 如果指定一个本地目录, 那么在处理完之后将文件备份至该位置.
-        # 默认:　nil 
-        local_dir => 'your/local/directory'
-
-        # 如果指定了该值, 那么在处理完之后将文件上传到Qingstor指定的bucket中.
-        # 默认: nil
-        backup_bucket => 'backupbucket'
-
-        # 配合上一项使用, 指定备份bucket所在的region.
-        # 默认: "pek3a", 可选枚举值: ["pek3a", "sh1a"]
-        backup_region => "sh1a"
-
-        # 备份文件的前缀
-        # 默认: nil 
-        backup_prefix => "logstash/backup"
-
-        # 指定一个sincedb的保存位置, sincedb用于记录上一次抓取文件的时间
-        # 没有指定时默认在用户HOME目录下创建
-        # 默认: nil
-        sincedb_path => "~/qingstor/.sincedb" 
-
-        # 每次抓取的时间间隔, 单位秒
-        # 默认: 10(s)
-        interval => 30
-                                       
-    }
-}
-
-```
-
-## 2. 安装插件
-
-#### 2.1 直接运行本地的插件
-
-- 编辑Logstash目录下的Genfile, 添加插件的路径, 例如
-```ruby
-gem "logstash-input-qingstor", :path => "/your/local/logstash-input-qingstor"
-```
-- 安装插件
-```sh
-bin/logstash-plugin install --no-verify
-```
-- 使用插件运行
-```sh
-bin/logstash -e "input {qingstor {  access_key_id => 'your_access_key_id'            
-        secret_access_key => 'your_secret_access_key'     
-        bucket => 'bucket_name'                          }}'
-```
-此时你对插件所做的任意的代码上的修改都会直接生效.
-
-#### 2.2 安装一个本地插件然后运行
-
-这一步你需要生成一个插件的gem包, 然后通过logstash来安装到logstash的插件目录下
-- 在logstash-input-qingstor项目目录下生成gem
-```sh
-gem build logstash-input-qingstor.gemspec
-```
-- 在Logstash的目录下使用logstash-plugin安装
-```sh
-bin/logstash-plugin install /your/local/plugin/logstash-input-qingstor.gem
-```
-- 安装完毕之后, 就可以使用Logstash运行开始测试了.
-
-
-# Logstash Output Plugin for Qingstor 
-
-As an input plugin, it can download the log files from [Qingstor](https://www.qingcloud.com/products/storage#qingstor), and read them into logstash.  
-Qingstor is a remarkable object storage service provided by [Qingcloud](https://www.qingcloud.com/).
+As an input plugin, it can download the log files from [QingStor](https://www.qingcloud.com/products/storage#qingstor), and read them into logstash.  
+QingStor is a remarkable object storage service provided by [QingCloud](https://www.qingcloud.com/).
 
 For now, We've submitted this plugin to rubygems.org. Use the following command to install.
 ```sh
@@ -126,6 +9,7 @@ For now, We've submitted this plugin to rubygems.org. Use the following command
 ```
 If want to install the local code, please refer to the following guide to install it manually.
 
+Simplified Chinese version README can be found [here](/README_zh_CN.md)
 ## 1. Configuration Guide
 
 #### 1.1 Run in minimal Configuration Settings
@@ -141,7 +25,7 @@ input {
 
 ```
 
-More configuration details please refer to [code part](lib/logstash/inputs/qingstor.rb).
+More configuration details please refer to [common options](/docs/index.asciidoc).
 
 ## 2. Running your unpublished Plugin in Logstash
 

data/README_zh_CN.md

@@ -0,0 +1,116 @@
+# Logstash Input Plugin for QingStor 
+
+这是一个适配了[QingStor](https://www.qingcloud.com/products/storage#qingstor), 工作在logstash中的input插件.  QingStor是[QingCloud](https://www.qingcloud.com/)推出的对象存储服务.  
+作为一个input插件, 它能下载存储在QingStor上的日志文件, 读入到logstash中进行进一步处理.  
+详细功能参考下面配置说明.  
+
+目前插件已经提交至rubygems.org, 使用以下命令安装:
+```sh
+    bin/logstash-plugin install logstash-input-qingstor
+```
+ 手动安装本地代码, 安装方法参考下文.
+
+## 1. 配置说明
+
+#### 1.1 最小运行配置
+- 使用'-f' 接受一个*.conf文件或者使用'-e'参数最小运行配置时, 至少需要以下三项
+```sh
+input {
+    qingstor {
+        access_key_id => 'your_access_key_id'           #required 
+        secret_access_key => 'your_secret_access_key'   #required  
+        bucket => 'bucket_name'                         #required 
+        # region => "pek3a"                             #optional, default value "pek3a"                                
+    }
+}
+
+```
+
+#### 1.2 其他可选参数说明
+```sh
+input {
+    qingstor {
+        ......
+        # 指定下载文件的前缀. 
+        # 默认nil, 
+        prefix => 'aprefix'
+
+        # 本地保存临时文件的目录. 
+        # 默认: 系统临时文件目录下的qingstor2logstash文件夹, 例如linux下 "/tmp/qingstor2logstash".
+        tmpdir => '/local/temporary/directory' 
+
+        # 是否在处理之后, 删除远程bucket中的文件.
+        # 默认: false
+        delete_remote_files => true
+
+        # 重新配置QingStor的地址
+        # 默认: nil
+        host => "new.qingstor.net"
+
+        # 重新配置QingStor地址的端口号
+        # 默认: 443
+        port => 443
+
+        # 如果指定一个本地目录, 那么在处理完之后将文件备份至该位置.
+        # 默认:　nil 
+        local_dir => 'your/local/directory'
+
+        # 如果指定了该值, 那么在处理完之后将文件上传到QingStor指定的bucket中.
+        # 默认: nil
+        backup_bucket => 'backupbucket'
+
+        # 配合上一项使用, 指定备份bucket所在的region.
+        # 默认: "pek3a", 可选枚举值: ["pek3a", "sh1a"]
+        backup_region => "sh1a"
+
+        # 备份文件的前缀
+        # 默认: nil 
+        backup_prefix => "logstash/backup"
+
+        # 指定一个sincedb的保存位置, sincedb用于记录上一次抓取文件的时间
+        # 没有指定时默认在用户HOME目录下创建
+        # 默认: nil
+        sincedb_path => "~/qingstor/.sincedb" 
+
+        # 每次抓取的时间间隔, 单位秒
+        # 默认: 10(s)
+        interval => 30
+                                       
+    }
+}
+
+```
+
+## 2. 安装插件
+
+#### 2.1 直接运行本地的插件
+
+- 编辑Logstash目录下的Genfile, 添加插件的路径, 例如
+```ruby
+gem "logstash-input-qingstor", :path => "/your/local/logstash-input-qingstor"
+```
+- 安装插件
+```sh
+bin/logstash-plugin install --no-verify
+```
+- 使用插件运行
+```sh
+bin/logstash -e "input {qingstor {  access_key_id => 'your_access_key_id'            
+        secret_access_key => 'your_secret_access_key'     
+        bucket => 'bucket_name'                          }}'
+```
+此时你对插件所做的任意的代码上的修改都会直接生效.
+
+#### 2.2 安装一个本地插件然后运行
+
+这一步你需要生成一个插件的gem包, 然后通过logstash来安装到logstash的插件目录下
+- 在logstash-input-qingstor项目目录下生成gem
+```sh
+gem build logstash-input-qingstor.gemspec
+```
+- 在Logstash的目录下使用logstash-plugin安装
+```sh
+bin/logstash-plugin install /your/local/plugin/logstash-input-qingstor.gem
+```
+- 安装完毕之后, 就可以使用Logstash运行开始测试了.
+

data/lib/logstash/inputs/qingstor.rb

@@ -1,16 +1,22 @@
-# encoding: utf-8
-require "logstash/inputs/base"
-require "logstash/namespace"
-require "stud/interval"
-require "qingstor/sdk"
-require "fileutils"
-require "tmpdir"
-
+require 'logstash/inputs/base'
+require 'logstash/namespace'
+require 'stud/interval'
+require 'qingstor/sdk'
+require 'fileutils'
+require 'tmpdir'
+
+# Logstash Inputs Plugin for QingStor
 class LogStash::Inputs::Qingstor < LogStash::Inputs::Base
-  require "logstash/inputs/qingstor/qingstor_validator"
-  config_name "qingstor"
+  require 'logstash/inputs/qingstor/qingstor_validator'
+  require 'logstash/inputs/qingstor/sincedb'
+  require 'logstash/inputs/qingstor/uploader'
+  require 'logstash/inputs/qingstor/log_reader'
+
+  include QingstorValidator
+
+  config_name 'qingstor'
 
-  default :codec, "plain"
+  default :codec, 'plain'
 
   # The key id to access your QingStor
   config :access_key_id, :validate => :string, :required => true
@@ -24,246 +30,166 @@ class LogStash::Inputs::Qingstor < LogStash::Inputs::Base
   # It specifies the host port, please coordinate with config 'host'.
   config :port, :validate => :number, :default => 443
 
-  # The name of the qingstor bucket
+  # The name of the QingStor bucket
   config :bucket, :validate => :string, :required => true
 
   # The region of the QingStor bucket
-  config :region, :validate => :string, :default => "pek3a"
+  config :region, :validate => :string, :default => 'pek3a'
 
   # If specified, it would only download the files with the specified prefix.
   config :prefix, :validate => :string, :default => nil
 
-  # Set the directory where logstash store the tmp files before 
-  # sending it to logstash, default directory in linux /tmp/logstash2qingstor
-  config :tmpdir, :validate => :string, :default => File.join(Dir.tmpdir, "qingstor2logstash")
+  # Set the directory where logstash store the tmp files before
+  # sending it to logstash, default directory in linux /tmp/qingstor2logstash
+  config :tmpdir, :validate => :string,
+                  :default => File.join(Dir.tmpdir, 'qingstor2logstash')
 
   # If this set to true, the remote file will be deleted after processed
   config :delete_remote_files, :validate => :boolean, :default => false
-  
-  # If this set to true, the file will backup to a local dir,
+
+  # If this set to a valid path, the file will be backup under a local path,
   # please make sure you can access to this dir.
-  config :backup_local_dir, :validate => :string, :default => File.expand_path("~/")
+  config :backup_local_dir, :validate => :string, :default => nil
 
   # If specified, the file will be upload to this bucket of the given region
   config :backup_bucket, :validate => :string, :default => nil
 
-  # Specified the backup region in Qingstor.
-  config :backup_region, :validate => ["pek3a", "sh1a"], :default => "pek3a"
+  # Specified the backup region in QingStor.
+  config :backup_region, :validate => :string, :default => 'pek3a'
+
+  # This prefix, specially in QingStor, will add before the backup filename
+  config :backup_prefix, :validate => :string, :default => ''
+
+  # Use sincedb to record the last download time
+  config :sincedb_path, :validate => :string, :default => nil
 
-  # This prefix, specially in qingstor, will add before the backup filename
-  config :backup_prefix, :validate => :string, :default => ""
-  
-  # Use sincedb to record the last download time 
-  config :sincedb_path, :validate => :string, :default => nil 
-  
   # Set how frequently messages should be sent.
   # The default, `10`, means send a message every 10 seconds.
   config :interval, :validate => :number, :default => 10
 
-  public
   def register
-
-    if !@tmpdir.nil? && !directory_valid?(@tmpdir)
-      raise LogStash::ConfigurationError, "Logstash must have the permissions to write to the temporary directory: #{@tmpdir}"
-    end
-
-    if !@backup_local_dir.nil? && !directory_valid?(@backup_local_dir)
-      raise LogStash::ConfigurationError, "Logstash must have the permissions to write to the temporary directory: #{@backup_local_dir}"
+    check_settings
+    @logger.info('Registering QingStor plugin', :bucket => @bucket,
+                                                :region => @region)
+    @qs_bucket = qs_bucket(@bucket, @region)
+    bucket_valid?(@qs_bucket)
+    unless @backup_bucket.nil?
+      backup_bucket = qs_bucket(@backup_bucket, @backup_region)
+      prefix_valid?(@backup_prefix) unless @backup_prefix.nil?
+      @uploader = Uploader.new(backup_bucket, @backup_prefix, @logger)
     end
-
-
-    @logger.info "Registering QingStor plugin", :bucket => @bucket, :region => @region
-    
-    @qs_config = QingStor::SDK::Config.init @access_key_id, @secret_access_key
-    @qs_config.update({ host: @host, port: @port }) unless @host.nil?
-    @qs_service = QingStor::SDK::Service.new @qs_config
-    @qs_bucket = @qs_service.bucket @bucket, @region
-
-    QingstorValidator.bucket_valid?(@qs_bucket)
-    QingstorValidator.prefix_valid?(@backup_prefix) unless @backup_prefix.nil?
-
+    @log_reader = LogReader.new('/a/tmp/path')
   end # def register
 
   def run(queue)
-    @logger.info "starting processing"
+    @logger.info('starting processing')
     @current_thread = Thread.current
     Stud.interval(@interval) do
       process_files(queue)
     end
   end # def run
 
-   
   def process_files(queue)
     objects = list_new_files
 
-    # For each record in objects, it will be download and read 
-    # into logstash. Then it will do deletion or backup job 
+    # For each record in objects, it will be download and read
+    # into logstash. Then it will do deletion or backup job
     # according to preset flags.
-    objects.each do |key, time|
+    objects.each do |key, _|
       process_log(queue, key)
-      backup_to_bucket key unless @backup_bucket.nil?
+      @uploader.upload_async(key, @tmp_file_path) unless @backup_bucket.nil?
       backup_to_local_dir unless @backup_local_dir.nil?
-      @qs_bucket.delete_object key if @delete_remote_files
+      @qs_bucket.delete_object(key) if @delete_remote_files
     end
 
     # if fetched nothing, just return
     return if objects.empty?
-    _time = objects.values.max 
-    sincedb.write(Time.at(_time))
+    tmp_time = objects.values.max
+    sincedb.write(Time.at(tmp_time))
+  end
+
+  def check_settings
+    unless directory_valid?(@tmpdir)
+      raise(LogStash::ConfigurationError, 'ERROR: no permissions to access'\
+          + '#{@tmpdir}')
+    end
+    unless @backup_local_dir.nil? ||
+           directory_valid?(@backup_local_dir)
+      raise(LogStash::ConfigurationError, 'ERROR: no permissions to access'\
+          + '#{@backup_local_dir}')
+    end
+  end
+
+  def stop
+    Stud.stop!(@current_thread) unless @current_thread.nil?
+  end
+
+  def directory_valid?(path)
+    FileUtils.mkdir_p(path) unless Dir.exist?(path)
+    ::File.writable?(path)
+  rescue
+    false
+  end
+
+  private
+
+  def qs_bucket(bucket, region)
+    config = QingStor::SDK::Config.init(@access_key_id, @secret_access_key)
+    config.update(:host => @host, :port => @port) unless @host.nil?
+    service = QingStor::SDK::Service.new(config)
+    service.bucket(bucket, region)
   end
 
-  private 
   def list_new_files
     objects = {}
-    @logger.info "starting fetching objects"
-    res = @qs_bucket.list_objects({'prefix' => @prefix})
+    @logger.info('starting fetching objects')
+    res = @qs_bucket.list_objects('prefix' => @prefix)
     res[:keys].each do |log|
-      next unless is_desired?(log[:key])
+      next unless @log_reader.valid_format?(log[:key])
       next unless sincedb.newer?(log[:modified])
       objects[log[:key]] = log[:modified]
-      @logger.info("QingStor input: Adding to object:", :key => log[:key])
+      @logger.info('QingStor input: Adding to objects:', :key => log[:key])
     end
-    return objects
+    objects
   end
 
   def process_log(queue, key)
-    # a global var, for the next possible upload and copy job 
+    # a global var, for the next possible upload and copy job
     @tmp_file_path = File.join(@tmpdir, File.basename(key))
-
     File.open(@tmp_file_path, 'wb') do |logfile|
-      logfile.write @qs_bucket.get_object(key)[:body]
+      logfile.write(@qs_bucket.get_object(key)[:body])
     end
     process_local_log(queue, @tmp_file_path)
   end
- 
-  def backup_to_bucket(key)
-    properties = {'bucket-name' => @backup_bucket, 'zone' => @backup_region}
-    bucket = QingStor::SDK::Bucket.new @qs_config, properties
-    
-    if bucket.head[:status_code] != 200
-      res = bucket.put
-      if res[:status_code] != 201
-        @logger.error("ERROR : cannot create the bucket ", res[:message])
-        raise LogStash::ConfigurationError, "cannot create the bucket"
-      end 
-    end 
-    
-    md5_string = Digest::MD5.file(@tmp_file_path).to_s
-
-    new_key = if @backup_prefix.end_with?('/') || @backup_prefix.empty?
-                @backup_prefix + key 
-              else 
-                @backup_prefix + '/' + key 
-              end 
-
-    bucket.put_object new_key, { 
-      'content_md5' => md5_string,
-      'body' => File.open(@tmp_file_path)
-    }
-  end 
- 
+
   def backup_to_local_dir
     FileUtils.mkdir_p @backup_local_dir unless File.exist? @backup_local_dir
     FileUtils.cp @tmp_file_path, @backup_local_dir
   end
- 
+
   def process_local_log(queue, filename)
-    read_file(filename) do |line|
+    @log_reader.filepath = filename
+    @log_reader.read_file do |line|
       @codec.decode(line) do |event|
         decorate(event)
         queue << event
       end
     end
   end
-   
-  def read_file(filename, &block)
-    if gzip?(filename)
-      read_gzip_file(filename, block)
-    else
-      read_plain_file(filename, block)
-    end
-  end
-
-  def read_gzip_file(filename, block)
-    begin
-      Zlib::GzipReader.open(filename) do |decoder|
-        decoder.each_line { |line| block.call(line) }
-      end
-    rescue Zlib::Error, Zlib::GzipFile::Error => e 
-      @logger.error("Gzip codec: Cannot uncompress the file", :filename => filename)
-      raise e 
-    end 
-  end 
-   
-  def read_plain_file(filename, block)
-    File.open(filename, 'rb') do |file|
-      file.each(&block)
-    end
-  end
 
-  def is_desired?(filename)
-    return logger?(filename) || gzip?(filename)
-  end
- 
-  def logger?(filename)
-    return filename.end_with?('.log')
-  end
- 
-  def gzip?(filename)
-    return filename.end_with?('.gz')
-  end
- 
   def sincedb
     @sincedb ||= if @sincedb_path.nil?
-                    @logger.info("Using default path for the sincedb", :filename => sincedb_file)
-                    SinceDB::File.new(sincedb_file)
+                   @logger.info('Using default path for the sincedb',
+                                :filename => sincedb_file)
+                   SinceDB.new(sincedb_file)
                  else
-                    @logger.info("Using the provided sincedb_path")
-                    SinceDB::File.new(@sincedb_path)
+                   @logger.info('Using the provided sincedb_path')
+                   SinceDB.new(@sincedb_path)
                  end
-  end 
- 
-  def sincedb_file
-    File.join(ENV["HOME"], ".sincedb_" + Digest::MD5.hexdigest("#{@bucket}+#{@prefix}"))
   end
 
-  module SinceDB
-    class File
-      def initialize(file)
-        @sincedb_path = file
-      end
-
-      def newer?(date)
-        Time.at(date) > read 
-      end 
-
-      def read 
-        if ::File.exist?(@sincedb_path)
-          content = ::File.read(@sincedb_path).chomp.strip
-          return content.empty? ? Time.new(0) : Time.parse(content)
-        else 
-          return Time.new(0)
-        end
-      end 
-
-      def write(since = nil)
-        since = Time.now() if since.nil?
-        ::File.open(@sincedb_path, 'w') { |file| file.write(since.to_s) }
-      end 
-    end 
-  end  
-
-  public 
-  def stop
-   Stud.stop!(@current_thread) unless @current_thread.nil?
+  def sincedb_file
+    File.join(ENV['HOME'],
+              '.sincedb_' + Digest::MD5.hexdigest('#{@bucket}+#{@prefix}'))
   end
-
-  def directory_valid?(path)
-    begin 
-      FileUtils.mkdir_p(path) unless Dir.exist?(path)
-      ::File.writable?(path)
-    rescue 
-      false 
-    end 
-  end 
 end # class LogStash::Inputs::Qingstor