logstash-input-sls 0.0.2

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 9d5043bd9ad3a1cc123d446408dda36912b2dddd57b385c1c759923d66ee60ef
+  data.tar.gz: d721fa819bda7140a27c5ca9971058060591d7f497b6319b274500b7796377d2
+SHA512:
+  metadata.gz: ea69bedf30527c0bc0e7eb39fe9fd6539759429564563ae4eb645c81f046644371d244356c2ef3ac93aca476a0462cb09a8d12f3a34e7aea2c31a70bfa2aff6e
+  data.tar.gz: 5be4fdf65df4f3bc48a0b2216f57a1d6a046e33819c0127a2e4695192b2bd36a855e0a881a679eec4e9f1880a5f00e25ce078ec052f1b91ae56b8b02ee4708ff

data/Gemfile

@@ -0,0 +1,2 @@
+source 'https://rubygems.org'
+gemspec

data/LICENSE

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2019 Aliyun
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE.

data/README.md

@@ -0,0 +1,90 @@
+# Logstash Logservice Input Plugin
+
+This is a plugin for [Logstash](https://github.com/elastic/logstash).
+
+## Documentation
+This plugin provides infrastructure to automatically consume logs from Aliyun Log Service .
+
+
+### Usage:
+This is an example of logstash config:
+```
+input {
+  logservice{
+  endpoint => "your project endpoint"
+  access_id => "your access id"
+  access_key => "your access key"
+  project => "your project name"
+  logstore => "your logstore name"
+  consumer_group => "consumer group name"
+  consumer_name => "consumer name"
+  position => "end"
+  checkpoint_second => 30
+  include_meta => true
+  consumer_name_with_ip => true
+  }
+}
+
+output {
+  stdout {}
+}
+```
+
+### Logstash Logservice Input Configuration Options
+This plugin supports the following configuration options
+
+|Configuration|Type|Required|Comments|
+|:---:|:---:|:---:|:---|
+|endpoint|string|true|Your project endpoint|
+|access_id|string|true|Your access id|
+|access_key|string|true|Your access key|
+|project|string|true|Your project name|
+|logstore|string|true|Your consumer_name name|
+|consumer_group|string|true|Consumer group name|
+|consumer_name|string|true|Consumer name,The consumer name in the same consumer group must not be repeated, otherwise undefined behavior will occur.|
+|position|string|true|Position to consume. Options are `begin`, `end` and `yyyy-MM-dd HH:mm:ss`|
+|checkpoint_second|number| false|Time to checkpoint,default is 30|
+|include_meta|boolean| false|Whether the meta is included,default is true|
+|consumer_name_with_ip|boolean| false|Whether the consumer name has ip,default is true,Must be set to true under distributed collaborative consumption|
+
+
+## Install the plugin
+
+[logstash-input-sls-0.0.1.gem](https://github.com/aliyun/logstash-input-logservice/raw/master/logstash-input-sls-0.0.1.gem)
+
+you can build the gem and install it using:
+
+- Build your plugin gem
+
+```sh
+gem build logstash-input-sls.gemspec
+```
+
+- Install the plugin from the Logstash home
+
+```sh
+logstash-plugin install --local logstash-input-sls-0.0.1.gem
+```
+
+- Start Logstash
+
+```bash
+logstash -f logstash-sample.conf
+```
+
+## The performance test
+
+### The test environment
+
+- cpu : Intel(R) Xeon(R) Platinum 8163 CPU @ 2.50GHz,4 Core
+- memory : 8GB 
+- env : Linux
+
+### Test result
+| Data flow(MB/S) |CPU(%) | MEM(GB) |
+| :---: | :---: | :---: |
+|32|170.3|1.3|
+|16|83.3|1.3|
+|8|41.5|1.3|
+|4|21.0|1.3|
+|2|11.3|1.3|

data/README_CN.md

@@ -0,0 +1,135 @@
+# Logstash 日志服务 Input 插件
+
+本插件作为[Logstash](https://github.com/elastic/logstash)的input插件，提供从日志服务拉取（消费）日志的功能。
+
+### 功能特性
+* 支持分布式协同消费：可配置多台服务器同时消费某一Logstore。
+* 高性能：基于Java ConsumerGroup实现，单核消费速度可达20MB/s。
+* 高可靠：消费进度保存到服务端，宕机恢复后会从上一次checkpoint处自动恢复。
+* 自动负载均衡：根据消费者数量自动分配Shard，消费者增加/退出后会自动Rebalance。
+
+
+### 使用方式:
+以下示例为配置Logstash消费某一个Logstore并将日志打印到标准输出:
+```
+input {
+  logservice{
+  endpoint => "your project endpoint"
+  access_id => "your access id"
+  access_key => "your access key"
+  project => "your project name"
+  logstore => "your logstore name"
+  consumer_group => "consumer group name"
+  consumer_name => "consumer name"
+  position => "end"
+  checkpoint_second => 30
+  include_meta => true
+  consumer_name_with_ip => true
+  }
+}
+
+output {
+  stdout {}
+}
+```
+分布式并发消费的配置：
+````
+例如某Logstore有10个shard，
+每个shard数据流量1M/s，
+每台机器处理的能力为3M/s，
+可分配5台服务器，
+每个服务器设置相同的consumer_group和consumer_name，consumer_name_with_ip字段设置为true。
+这种情况每台服务器会分配到2个Shard，分别处理2M/s的数据。
+````
+
+
+### Logstash 日志服务 Input 配置参数
+本插件提供以下配置参数
+
+|参数名|参数类型|是否必填|备注|
+|:---:|:---:|:---:|:---|
+|endpoint|string|是|日志服务项目所在的endpoint，详情请参考[endpoint列表](https://help.aliyun.com/document_detail/29008.html)|
+|access_id|string|是|阿里云Access Key ID，需要具备ConsumerGroup相关权限，详情请参考[consumer group](https://help.aliyun.com/document_detail/28998.html)|
+|access_key|string|是|阿里云Access Key Secret，需要具备ConsumerGroup相关权限，详情请参考[consumer group](https://help.aliyun.com/document_detail/28998.html)|
+|project|string|是|日志服务项目名|
+|logstore|string|是|日志服务日志库名|
+|consumer_group|string|是|消费组名|
+|consumer_name|string|是|消费者名，同一个消费组内消费者名，必须不能重复，否则会出现未定义行为|
+|position|string|是|消费位置，可选项为 `begin`（从日志库写入的第一条数据开始消费）、`end`（从当前时间点开始消费） 和 `yyyy-MM-dd HH:mm:ss`（从指定时间点开始消费）|
+|checkpoint_second|number| 否|每隔几秒 checkpoint 一次，建议10-60秒，不能低于10秒，默认30秒|
+|include_meta|boolean| 否|传入日志是否包含meta，Meta包括日志source、time、tag、topic，默认为 true|
+|consumer_name_with_ip|boolean| 否|消费者名是否包含ip地址，默认为 true，分布式协同消费下必须设置为true|
+
+
+## 安装插件
+
+* 注意：如果没有修改源码可直接安装
+
+下载地址：[logstash-input-sls-0.0.1.gem](https://github.com/aliyun/logstash-input-logservice/raw/master/logstash-input-sls-0.0.1.gem)
+
+- 构建 插件
+
+```sh
+gem build logstash-input-sls.gemspec
+```
+
+- 安装 日志服务 input 插件
+
+```sh
+logstash-plugin install --local logstash-input-sls-0.0.1.gem
+```
+
+- 启动 Logstash
+
+```bash
+logstash -f logstash.conf
+```
+
+## 性能基准测试
+
+### 测试环境
+
+- 处理器 : Intel(R) Xeon(R) Platinum 8163 CPU @ 2.50GHz,4 Core
+- 内存 : 8GB 
+- 环境 : Linux
+
+### Logstash配置
+
+```
+input {
+  logservice{
+  endpoint => "cn-hangzhou.log.aliyuncs.com"
+  access_id => "***"
+  access_key => "***"
+  project => "test-project"
+  logstore => "logstore1"
+  consumer_group => "consumer_group1"
+  consumer => "consumer1"
+  position => "end"
+  checkpoint_second => 30
+  include_meta => true
+  consumer_name_with_ip => true
+  }
+}
+
+output {
+  file {
+  path => "/dev/null"
+  }
+}
+```
+
+### 测试过程
+
+- 使用Java Producer向logstore发送数据，分别达到每秒发送2MB、4MB、8MB、16MB、32MB数据。
+- 每条日志约500字节，包括10个Key&Value对。
+- 启动Logstash，消费logstore中的数据并确保消费延迟没有上涨（消费速度能够跟上生产的速度）。
+
+### 测试结果
+| 流量(MB/S) |处理器占用(%) | 内存占用(GB) |
+| :---: | :---: | :---: |
+|32|170.3|1.3|
+|16|83.3|1.3|
+|8|41.5|1.3|
+|4|21.0|1.3|
+|2|11.3|1.3|

data/lib/logstash/inputs/logservice.rb

@@ -0,0 +1,71 @@
+require 'java'
+require "logstash/namespace"
+require "logstash/event"
+require "logstash/environment"
+require "logstash/inputs/base"
+
+root_dir = File.expand_path(File.join(File.dirname(__FILE__), ".."))
+LogStash::Environment.load_runtime_jars! File.join(root_dir, "vendor")
+
+java_import 'com.aliyun.log.logstash.LogHubStarter'
+java_import 'com.aliyun.log.logstash.LogstashLogHubProcessor'
+
+class LogStash::Inputs::LogService < LogStash::Inputs::Base
+  attr_accessor :worker
+  def initialize(*args)
+    super(*args)
+  end
+
+  class LogHubProcessor < LogstashLogHubProcessor
+    attr_accessor :queue
+    def showContent(logmap)
+        event = LogStash::Event.new(logmap)
+        # this will block if output_queue is full. output_queue size is 20
+        @queue << event
+    end
+  end
+
+  config_name "logservice"
+
+  config :endpoint, :validate => :string, :required => true
+  config :access_id, :validate => :string, :required => true
+  config :access_key, :validate => :string, :required => true
+  config :project, :validate => :string, :required => true
+  config :logstore, :validate => :string, :required => true
+  config :consumer_group, :validate => :string, :required => true
+  config :consumer_name, :validate => :string, :required => true
+  config :position, :validate => :string, :required => true
+  config :checkpoint_second, :validate => :number, :default => 30
+  config :include_meta, :validate => :boolean, :default => true
+  config :consumer_name_with_ip, :validate => :boolean, :default => true
+  public
+  def register
+    @logger.info("Init logstash-input-logservice", :endpoint => @endpoint, :project => @project, :logstore => @logstore,
+        :consumer_group => @consumer_group, :consumer_name => @consumer_name, :position => @position,
+        :checkpoint_second => @checkpoint_second, :include_meta => @include_meta ,:consumer_name_with_ip => @consumer_name_with_ip)
+  end
+
+  def run(queue)
+    @local_address = java.net.InetAddress.getLocalHost().getHostAddress();
+    @ip_suffix = ''
+    if @consumer_name_with_ip
+        @ip_suffix = '_' + @local_address
+    end
+    @logger.info("Running logstash-input-logservice",:local_address => @local_address)
+    @handler = LogHubProcessor.new()
+    @handler.setCheckpointSecond(@checkpoint_second)
+    @handler.setIncludeMeta(@include_meta)
+    @handler.queue = queue
+    LogHubStarter.startWorker(@handler, @endpoint, @access_id, @access_key, @project, @logstore, @consumer_group, @consumer_name + @ip_suffix, @position)
+    rescue Exception => e
+        @logger.error("Start logstash-input-logservice", :endpoint => @endpoint, :project => @project, :logstore => @logstore,
+            :consumer_group => @consumer_group, :consumer_name => @consumer_name, :position => @position,
+            :checkpoint_second => @checkpoint_second, :include_meta => @include_meta, :consumer_name_with_ip => @consumer_name_with_ip, :exception => e)
+  end
+
+  def teardown
+    @interrupted = true
+    finished
+  end # def teardown
+end 
+

data/logstash-input-sls.gemspec

@@ -0,0 +1,25 @@
+Gem::Specification.new do |s|
+  s.name = 'logstash-input-sls'
+  s.version         = '0.0.2'
+  s.licenses = ['MIT']
+  s.summary = "This sls input streams a string at a definable interval."
+  s.description     = "This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program"
+  s.authors = ["lichao"]
+  s.email = 'info@elastic.co'
+  s.homepage = "https://www.aliyun.com/product/sls"
+  s.require_paths = ["lib"]
+
+  # Files
+  s.files = Dir['lib/**/*','spec/**/*','vendor/**/*','*.gemspec','*.md','CONTRIBUTORS','Gemfile','LICENSE','NOTICE.TXT']
+   # Tests
+  s.test_files = s.files.grep(%r{^(test|spec|features)/})
+
+  # Special flag to let us know this is actually a logstash plugin
+  s.metadata = { "logstash_plugin" => "true", "logstash_group" => "input" }
+
+  # Gem dependencies
+  s.add_runtime_dependency "logstash-core-plugin-api", ">= 1.60", "<= 2.99"
+  s.add_runtime_dependency 'logstash-codec-plain'
+  s.add_runtime_dependency 'stud', '>= 0.0.22'
+  s.add_development_dependency 'logstash-devutils'
+end

metadata

@@ -0,0 +1,116 @@
+--- !ruby/object:Gem::Specification
+name: logstash-input-sls
+version: !ruby/object:Gem::Version
+  version: 0.0.2
+platform: ruby
+authors:
+- lichao
+autorequire:
+bindir: bin
+cert_chain: []
+date: 2019-07-25 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '1.60'
+    - - "<="
+      - !ruby/object:Gem::Version
+        version: '2.99'
+  name: logstash-core-plugin-api
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '1.60'
+    - - "<="
+      - !ruby/object:Gem::Version
+        version: '2.99'
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  name: logstash-codec-plain
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 0.0.22
+  name: stud
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 0.0.22
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  name: logstash-devutils
+  prerelease: false
+  type: :development
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+description: This gem is a Logstash plugin required to be installed on top of the
+  Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This
+  gem is not a stand-alone program
+email: info@elastic.co
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- Gemfile
+- LICENSE
+- README.md
+- README_CN.md
+- lib/logstash/inputs/logservice.rb
+- lib/logstash/vendor/jar-dependencies/runtime-jars/logstash-input-sls-1.0-SNAPSHOT.jar
+- logstash-input-sls.gemspec
+homepage: https://www.aliyun.com/product/sls
+licenses:
+- MIT
+metadata:
+  logstash_plugin: 'true'
+  logstash_group: input
+post_install_message:
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project:
+rubygems_version: 2.7.6
+signing_key:
+specification_version: 4
+summary: This sls input streams a string at a definable interval.
+test_files: []