logstash-output-qingstor 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: ff8942eb9e9d52efdd58c06bfdeb77abc8698d2c
+  data.tar.gz: 34704f434b5c411eb7ab5bd3cbf4a553b9e41176
+SHA512:
+  metadata.gz: 80a48c0492352fe8d1b486eccb9e4dc3848bb3aafefc8cc0229a0db3ffb63f7346cce08e8dc4efad7fcfa09f9c3a78681f2d0b81d8e705afc8d1f902068ede28
+  data.tar.gz: cc45d155ba152691d7972e4895f838ea4f3486fccf59bea0dd60480bbb1c6d154b2d5d3a84b356a741c1632d8fb02553c338fa7dd5e38c5966cf2888a7508b3e

data/CHANGELOG.md

@@ -0,0 +1,2 @@
+## 0.1.0
+  - Plugin created with the logstash plugin generator

data/CODE_OF_CONDUCT.md

@@ -0,0 +1,44 @@
+# Contributor Covenant Code of Conduct
+
+## Our Pledge
+
+In the interest of fostering an open and welcoming environment, we as contributors and maintainers pledge to making participation in our project and our community a harassment-free experience for everyone, regardless of age, body size, disability, ethnicity, gender identity and expression, level of experience, nationality, personal appearance, race, religion, or sexual identity and orientation.
+
+## Our Standards
+
+Examples of behavior that contributes to creating a positive environment
+include:
+
+* Using welcoming and inclusive language
+* Being respectful of differing viewpoints and experiences
+* Gracefully accepting constructive criticism
+* Focusing on what is best for the community
+* Showing empathy towards other community members
+
+Examples of unacceptable behavior by participants include:
+
+* The use of sexualized language or imagery and unwelcome sexual attention or advances
+* Trolling, insulting/derogatory comments, and personal or political attacks
+* Public or private harassment
+* Publishing others' private information, such as a physical or electronic address, without explicit permission
+* Other conduct which could reasonably be considered inappropriate in a professional setting
+
+## Our Responsibilities
+
+Project maintainers are responsible for clarifying the standards of acceptable behavior and are expected to take appropriate and fair corrective action in response to any instances of unacceptable behavior.
+
+Project maintainers have the right and responsibility to remove, edit, or reject comments, commits, code, wiki edits, issues, and other contributions that are not aligned to this Code of Conduct, or to ban temporarily or permanently any contributor for other behaviors that they deem inappropriate, threatening, offensive, or harmful.
+
+## Scope
+
+This Code of Conduct applies both within project spaces and in public spaces when an individual is representing the project or its community. Examples of representing a project or community include using an official project e-mail address, posting via an official social media account, or acting as an appointed representative at an online or offline event. Representation of a project may be further defined and clarified by project maintainers.
+
+## Enforcement
+
+Instances of abusive, harassing, or otherwise unacceptable behavior may be reported by contacting the project team at pengsrc@yunify.com. All complaints will be reviewed and investigated and will result in a response that is deemed necessary and appropriate to the circumstances. The project team is obligated to maintain confidentiality with regard to the reporter of an incident. Further details of specific enforcement policies may be posted separately.
+
+Project maintainers who do not follow or enforce the Code of Conduct in good faith may face temporary or permanent repercussions as determined by other members of the project's leadership.
+
+## Attribution
+
+This Code of Conduct is adapted from the [Contributor Covenant][http://contributor-covenant.org], version 1.4, available at [http://contributor-covenant.org/version/1/4][http://contributor-covenant.org/version/1/4].

data/CONTRIBUTING.md

@@ -0,0 +1,43 @@
+# Contributing Guidelines
+
+We love contributions from everyone. By participating in this project, you agree to abide by the [_`Contributor Covenant Code of Conduct`_](./CODE_OF_CONDUCT.md) and the conventions described in this documentation.
+
+## Getting Started
+
+### Reporting Issues
+
+A great way to contribute to the project is to send a detailed report when you encounter an issue. We always appreciate a well-written, thorough bug report, and will thank you for it!
+
+Check that [our issues list](https://github.com/Tacinight/logstash-output-qingstor/issues) doesn't already include that problem or suggestion before submitting an issue. If you find a match, you can use the "subscribe" button to get notified on updates. Do not leave random "+1" or "I have this too" comments, as they only clutter the discussion, and don't help resolving it. However, if you have ways to reproduce the issue or have additional information that may help resolving the issue, please leave a comment.
+
+When reporting issues, please describe what happened clearly, and include the steps required to reproduce the problem if possible and applicable. This information will help us review and fix your issue faster. When sending lengthy log-files, consider posting them as a gist (https://gist.github.com). Don't forget to remove sensitive data from your log files before posting (you can replace those parts with "CONFIDENTIAL").
+
+### Sending Pull Request
+
+Pull requests are always welcome, we will appreciate it.
+
+Any significant improvement should be documented as a GitHub [issue](https://github.com/Tacinight/logstash-output-qingstor/issues) before anybody starts working on it.
+
+1. Fork it ( https://github.com/Tacinight/logstash-output-qingstor/fork )
+2. Create your feature branch (`git checkout -b new-feature`)
+3. Commit your changes (`git commit -asm 'Add some feature'`)
+4. Push to the branch (`git push origin new-feature`)
+5. Create a new Pull Request
+
+## Conventions
+
+### Patch Content
+
+* Always remove the trailing whitespace
+* Keep a newline at the end of text files
+* Exclude irrelevant files, such as `*.a`, `.bundle`, and `.DS_Store`
+* Always pass all the tests before submitting
+* Let the propose of each patch simple and clear
+
+### Commit Message
+
+* Capitalize the subject line
+* Limit the subject line to 50 characters
+* Do not end the subject line with a period
+* Wrap the body at 72 characters
+* Use the body to explain what happened

data/CONTRIBUTORS

@@ -0,0 +1,10 @@
+The following is a list of people who have contributed ideas, code, bug
+reports, or in general have helped logstash along its way.
+
+Contributors:
+* Evan Zhao - tacingiht@gmail.com
+
+Note: If you've sent us patches, bug reports, or otherwise contributed to
+Logstash, and you aren't on the list above and want to be, please let us know
+and we'll make sure you're here. Contributions from folks like you are what make
+open source awesome.

data/DEVELOPER.md

@@ -0,0 +1,2 @@
+# logstash-output-qinstor
+Example output plugin. This should help bootstrap your effort to write your own output plugin!

data/Gemfile

@@ -0,0 +1,3 @@
+source 'https://rubygems.org'
+gemspec
+

data/LICENSE

@@ -0,0 +1,11 @@
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

data/README.md

@@ -0,0 +1,181 @@
+# Logstash Output Plugin for Qingstor
+
+这是一个适配了[Qingstor](https://www.qingcloud.com/products/storage#qingstor), 工作在logstash中的output插件.   
+Qingstor是[Qingcloud](https://www.qingcloud.com/)推出的对象存储服务.  
+作为一个output插件, 可以将logstash输出的结果打包上传至Qingstor中.  
+详细功能参考下面配置说明.  
+
+目前插件已经提交至rubygems.org, 使用以下命令安装:
+```sh
+    bin/logstash-plugin install logstash-output-qingstor
+```
+ 手动安装本地代码, 安装方法参考下文.
+
+## 1. 配置说明
+
+#### 1.1 最小运行配置
+- 使用'-f'接受一个*.conf文件或者使用'-e'参数时, 最小运行配置至少需要以下三项
+```sh
+output {
+    qingstor {
+        access_key_id => 'your_access_key_id'           #required 
+        secret_access_key => 'your_secret_access_key'   #required  
+        bucket => 'bucket_name'                         #required 
+        # region => "pek3a"                             #optional, default value "pek3a"                                
+    }
+}
+
+```
+
+#### 1.2 其他可选参数说明
+```sh
+output {
+    qingstor {
+        ......
+        # 前缀, 生成本地子目录, 并且添加到上传文件名的前半部分, 形成Qingstor的目录.
+        # 默认空, 
+        prefix => 'aprefix'
+
+        # 本地保存临时文件的目录. 
+        # 默认: 系统临时文件目录下的logstash2qingstor文件夹, 例如linux下 "/tmp/logstash2qingstor"
+        tmpdir => '/local/temporary/directory' 
+
+        # 字符串数组, 添加到文件名中, 例如["a", "b", "c"], 文件名会形成如 xxx-a-b-c-xxx.xx 
+        # 默认: 空
+        tags => ["tag1", "tag2", "tag3"]
+
+        # 上传文件的格式. 可选"gzip", 后缀为".gz".
+        # 默认: "none", 后缀".log".
+        encoding => "gzip"
+
+        # 文件上传的策略.
+        # 分别表示结合文件大小和时间的综合策略, 基于文件大小的策略, 以及基于时间的策略.
+        # 基于文件大小的策略表示: 当文件大小满足预设值之后, 将文件上传.
+        # 基于时间的策略表示: 每经过预设时间之后, 将文件上传.
+        # 默认: "size_and_time". 可选枚举值["size_and_time", "size", "time"].
+        rotation_strategy => "size_and_time"
+
+        # 配合"size_and_time", "size"的可选配置型, 单位byte
+        # 默认: 1024*1024*5 (byte) = 5 (MB)
+        size_file => 1024*1024*5
+
+        # 配合"size_and_time", "time"的可选配置型, 单位minute
+        # 默认: 15 (minutes)
+        time_file => 15 
+
+        # 服务端文件加密, 支持AES256. 
+        # 默认: "none". 可选枚举值: ["AES256", "none"]
+        server_side_encryption_algorithm => "AES256"
+
+        # 选用服务端文件加密时提供的秘钥, 秘钥要求32位/256bit
+        customer_key => "your_encryption_key"
+
+        # 宕机恢复, 启动logstash时, 自动上传目录下的遗留文件
+        # 默认: false
+        restore => true                     
+                                       
+    }
+}
+
+```
+
+## 2. 安装插件
+
+#### 2.1 直接运行本地的插件
+
+- 编辑Logstash目录下的Genfile, 添加插件的路径, 例如
+```ruby
+gem "logstash-output-qingstor", :path => "/your/local/logstash-output-qingstor"
+```
+- 安装插件
+```sh
+bin/logstash-plugin install --no-verify
+```
+- 使用插件运行
+```sh
+bin/logstash -e "output {qingstor {access_key_id => 'your_access_key_id'            
+        secret_access_key => 'your_secret_access_key'     
+        bucket => 'bucket_name'                          }}"
+```
+此时你对插件所做的任意的代码上的修改都会直接生效.
+
+#### 2.2 安装一个本地插件然后运行
+
+这一步你需要生成一个插件的gem包, 然后通过logstash来安装到logstash的插件目录下
+- 在项目目录下生成gem
+```sh
+gem build logstash-output-qingstor.gemspec
+```
+- 在Logstash的目录下使用logstash-plugin安装
+```sh
+bin/logstash-plugin install /your/local/plugin/logstash-output-qingstor.gem
+```
+- 安装完毕之后, 就可以使用Logstash运行开始测试了.
+
+
+# Logstash Output Plugin for Qingstor 
+
+As an output plugin, it can collect the outputs from logstash, and store them in [Qingstor](https://www.qingcloud.com/products/storage#qingstor), which is a remarkable object storage service provided by [Qingcloud](https://www.qingcloud.com/).
+
+For now, We've submitted this plugin to rubygems.org. Use the following command to install.
+```sh
+    bin/logstash-plugin install logstash-output-qingstor
+```
+If want to install the local code, please refer to the following guide to install it manually.
+
+## 1. Configuration Guide
+
+#### 1.1 Run in minimal Configuration Items
+```sh
+output {
+    qingstor {
+        access_key_id => 'your_access_key_id'           #required 
+        secret_access_key => 'your_secret_access_key'   #required  
+        bucket => 'bucket_name'                         #required 
+        # region => "pek3a"                             #optional, default value "pek3a"                                
+    }
+}
+
+```
+
+More configuration details please refer to [code part](lib/logstash/outputs/qingstor.rb).
+
+## 2. Running your unpublished Plugin in Logstash
+
+#### 2.1 Run in a local Logstash clone
+
+- Edit Logstash `Gemfile` and add the local plugin path, for example:
+```ruby
+gem "logstash-output-qingstor", :path => "/your/local/logstash-output-qingstor"
+```
+- Install plugin
+```sh
+bin/logstash-plugin install --no-verify
+```
+- Run Logstash with your plugin
+```sh
+bin/logstash -e "output {qingstor {access_key_id => 'your_access_key_id'            
+        secret_access_key => 'your_secret_access_key'     
+        bucket => 'bucket_name'                          }}"
+```
+At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.
+
+#### 2.2 Run in an installed Logstash
+
+You can use the same **2.1** method to run your plugin in an installed Logstash by editing its `Gemfile` and pointing the `:path` to your local plugin development directory or you can build the gem and install it using:
+
+- Build your plugin gem
+```sh
+gem build logstash-output-qingstor.gemspec
+```
+- Install the plugin from the Logstash home
+```sh
+bin/logstash-plugin install /your/local/plugin/logstash-output-qingstor.gem
+```
+- Start Logstash and proceed to test the plugin
+
+## Contributing
+Please see [Contributing Guidelines](./CONTRIBUTING.md) of this project before submitting patches.
+
+## LICENSE
+The Apache License (Version 2.0, January 2004).

data/lib/logstash/outputs/qingstor/file_repository.rb

@@ -0,0 +1,112 @@
+# encoding: utf-8
+require "java"
+require "concurrent"
+require "concurrent/timer_task"
+require "logstash/util"
+
+ConcurrentHashMap = java.util.concurrent.ConcurrentHashMap
+
+module LogStash 
+  module Outputs
+    class Qingstor
+      class FileRepository
+        DEFAULT_STATE_SWEEPER_INTERVAL_SECS = 60
+        DEFAULT_STATE_TIME_SECS = 15 * 60
+        class PrefixedValue
+          def initialize(file_factory, stale_time)
+            @file_factory = file_factory
+            @lock = Mutex.new
+            @stale_time = stale_time
+          end
+
+          def with_lock
+            @lock.synchronize {
+              yield @file_factory
+            }          
+          end 
+
+          def stale?
+            with_lock { |factory| factory.current.size == 0 && (Time.now - factory.current.ctime > @stale_time) }
+          end 
+
+          def apply(prefix)
+            return self
+          end 
+
+          def delete! 
+            with_lock { |factory| factory.current.delete! }
+          end 
+        end
+
+        class FactoryInitializer
+          def initialize(tags, encoding, temporary_directory, stale_time)
+            @tags = tags
+            @encoding = encoding
+            @temporary_directory = temporary_directory
+            @stale_time = stale_time
+          end 
+
+          def apply(prefix_key)
+            PrefixedValue.new(TemporaryFileFactory.new(prefix_key, @tags, @encoding, @temporary_directory), @stale_time) 
+          end 
+        end 
+
+        def initialize(tags, encoding, temporary_directory,
+                       stale_time = DEFAULT_STATE_TIME_SECS,
+                       sweeper_interval = DEFAULT_STATE_SWEEPER_INTERVAL_SECS)
+          @prefixed_factories = ConcurrentHashMap.new 
+          @sweeper_interval = sweeper_interval
+          @factoryinitializer = FactoryInitializer.new(tags, encoding, temporary_directory, stale_time)
+
+          start_stale_sweeper
+        end
+
+        def keys
+          @prefixed_factories.keySet
+        end 
+
+        def each_files 
+          @prefixed_factories.elements.each do |prefixed_file|
+            prefixed_file.with_lock { |factory| yield factory.current }
+          end 
+        end 
+
+        def get_factory(prefix_key)
+          @prefixed_factories.computeIfAbsent(prefix_key, @factoryinitializer).with_lock { |factory| yield factory }
+        end 
+
+        def get_file(prefix_key)
+          get_factory(prefix_key) { |factory| yield factory.current }
+        end 
+
+        def shutdown 
+          stop_stale_sweeper 
+        end 
+
+        def size 
+          @prefixed_factories.size 
+        end 
+        
+        def remove_stale(k, v)
+          if v.stale? 
+            @prefixed_factories.remove(k, v)
+            v.delete!
+          end 
+        end 
+
+        def start_stale_sweeper 
+          @stale_sweeper = Concurrent::TimerTask.new(:execution_interval => @sweeper_interval) do 
+            LogStash::Util.set_thread_name("Qingstor, stale factory sweeper")
+            @prefixed_factories.forEach { |k, v| remove_stale(k, v) }
+          end 
+
+          @stale_sweeper.execute
+        end 
+
+        def stop_stale_sweeper 
+          @stale_sweeper.shutdown
+        end 
+      end 
+    end 
+  end 
+end 

data/lib/logstash/outputs/qingstor/qingstor_validator.rb

@@ -0,0 +1,30 @@
+# encoding: utf-8
+require "qingstor/sdk"
+require "fileutils"
+
+module LogStash
+  module Outputs
+    class Qingstor
+      class QingstorValidator
+
+        def self.bucket_valid?(bucket)
+          res = bucket.head
+          case res[:status_code] 
+          when 401
+            raise LogStash::ConfigurationError, "Incorrect key id or access key."
+          when 404
+            raise LogStash::ConfigurationError, "Incorrect bucket/region name."
+          end 
+          true 
+        end 
+
+        def self.prefix_valid?(prefix)
+          if prefix.start_with?("/") || prefix.length >= 1024
+            raise LogStash::ConfigurationError, "Prefix must not start with '/' with length less than 1024 "
+          end 
+          true
+        end 
+      end 
+    end 
+  end 
+end 

data/lib/logstash/outputs/qingstor/size_and_time_rotation_policy.rb

@@ -0,0 +1,24 @@
+# encoding: utf-8
+require "logstash/outputs/qingstor/size_rotation_policy"
+require "logstash/outputs/qingstor/time_rotation_policy"
+
+module LogStash 
+  module Outputs 
+    class Qingstor
+      class SizeAndTimeRotationPolicy
+        def initialize(size_file, time_file)
+          @size_strategy = SizeRotationPolicy.new(size_file)
+          @time_strategy = TimeRotationPolicy.new(time_file)
+        end 
+
+        def rotate?(file)
+          @size_strategy.rotate?(file) || @time_strategy.rotate?(file)
+        end 
+
+        def needs_periodic?
+          true
+        end 
+      end 
+    end 
+  end 
+end 

data/lib/logstash/outputs/qingstor/size_rotation_policy.rb

@@ -0,0 +1,26 @@
+# encoding: utf-8
+module LogStash
+  module Outputs
+    class Qingstor
+      class SizeRotationPolicy
+        attr_reader :size_file
+
+        def initialize(size_file)
+          if size_file <= 0
+            raise LogStash::ConfigurationError, "'size_file' need to be greater than 0"
+          end 
+
+          @size_file = size_file
+        end 
+
+        def rotate?(file)
+          file.size >= size_file
+        end 
+
+        def needs_periodic?
+          false 
+        end 
+      end 
+    end 
+  end 
+end 

data/lib/logstash/outputs/qingstor/temporary_file.rb

@@ -0,0 +1,63 @@
+# encoding: utf-8
+require "thread"
+require "forwardable"
+require "fileutils"
+require "pathname"
+
+module LogStash
+  module Outputs
+    class Qingstor
+      class TemporaryFile
+        extend Forwardable
+
+        def_delegators :@fd, :path, :write, :close, :fsync
+
+        attr_reader :fd
+
+        def initialize(key, fd, tmp_path)
+          @key = key 
+          @fd = fd
+          @tmp_path = tmp_path
+          @created_at = Time.now
+        end 
+
+        def ctime 
+          @created_at 
+        end 
+        
+        def tmp_path 
+          @tmp_path 
+        end 
+
+        def size 
+          begin
+            @fd.size
+          rescue IOError
+            ::File.size(path)
+          end 
+        end 
+
+        def key 
+          @key.gsub(/^\//, "")
+        end 
+
+        def delete!
+          @fd.close rescue IOError 
+          FileUtils.rm_r(@tmp_path, :secure => true)
+        end 
+
+        def empty?
+          size == 0
+        end 
+
+        def self.create_from_existing_file(file_path, tmp_folder)
+          key_parts = Pathname.new(file_path).relative_path_from(tmp_folder).to_s.split(::File::SEPARATOR)
+          TemporaryFile.new(key_parts.slice(1, key_parts.size).join("/"),
+                            ::File.open(file_path, "r"),
+                            ::File.join(tmp_folder, key_parts.slice(0, 1)))
+                    
+        end 
+      end 
+    end 
+  end 
+end 

data/lib/logstash/outputs/qingstor/temporary_file_factory.rb

@@ -0,0 +1,110 @@
+# encoding: utf-8
+require "socket"
+require "securerandom"
+require "fileutils"
+require "zlib"
+require "forwardable"
+
+module LogStash
+  module Outputs
+    class Qingstor
+      class TemporaryFileFactory
+        FILE_MODE = "a"
+        GZIP_ENCODING = "gzip"
+        GZIP_EXTENSION = "log.gz"
+        TXT_EXTENSION = "log"
+        STRFTIME = "%Y-%m-%dT%H.%M"
+
+        attr_accessor :counter, :tags, :prefix, :encoding, :tmpdir, :current
+
+        def initialize(prefix, tags, encoding, tmpdir)
+          @counter = 0
+          @prefix = prefix
+          @tags = tags
+          @encoding = encoding
+          @tmpdir = tmpdir
+          @lock = Mutex.new 
+
+          rotate!
+        end 
+
+        def rotate!
+          @lock.synchronize {
+            @current = new_file
+            increment_counter
+            @current 
+          }
+        end 
+
+        private 
+        def extension
+          gzip? ? GZIP_EXTENSION : TXT_EXTENSION
+        end 
+        
+        def gzip?
+          encoding == GZIP_ENCODING
+        end 
+
+        def increment_counter
+          @counter += 1
+        end 
+
+        def current_time 
+          Time.new.strftime(STRFTIME)
+        end 
+
+        def generate_name 
+          filename = "ls.qingstor.#{SecureRandom.uuid}.#{current_time}"
+
+          if tags.size > 0
+            "#{filename}.tag_#{tags.join('.')}.part#{counter}.#{extension}"
+          else 
+            "#{filename}.part#{counter}.#{extension}"
+          end 
+        end 
+
+        def new_file 
+          uuid = SecureRandom.uuid 
+          name = generate_name 
+          path = ::File.join(@tmpdir, uuid)
+          key = ::File.join(@prefix, name)
+
+          FileUtils.mkdir_p(::File.join(path, @prefix))
+
+          io = if gzip? 
+                 IOWrappedGzip.new(::File.open(::File.join(path, key), FILE_MODE))
+               else 
+                 ::File.open(::File.join(path, key), FILE_MODE)
+               end 
+
+          TemporaryFile.new(key, io, path)
+        end
+
+        class IOWrappedGzip 
+          extend Forwardable 
+
+          def_delegators :@gzip_writer, :write, :close
+          attr_accessor :file_io, :gzip_writer
+        
+          def initialize(file_io)
+            @file_io = file_io
+            @gzip_writer = Zlib::GzipWriter.open(file_io)
+          end 
+
+          def path 
+            @gzip_writer.to_io.path 
+          end 
+          
+          def size 
+            @gzip_writer.flush 
+            @gzip_writer.to_io.size 
+          end 
+
+          def fsync 
+            @gzip_writer.to_io.fsync 
+          end 
+        end 
+      end 
+    end 
+  end 
+end