logstash-output-oss 0.1.1-java

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 1d607407845973f61ed5a1db2e22fbf3d99f160131ce6d5b82b769ed2befc137
+  data.tar.gz: 3735dd4691fd97ee642bab10479ee08a574659c3658f99f620851d64d5e0710d
+SHA512:
+  metadata.gz: 8bb4fd199bf14b90c6091256b2ce517ba9ebf41ebd9bee90d167489f4d80bfd54d9c94e13207b2391f0edb42d7cfc04067b27c6a190eda7d17976a56274e0c4e
+  data.tar.gz: d2113870411fe783374e66d20c5eb43ddd91f0bfdb94cda80bd329d767140c229a221a951f4be5dc0ba658ce3ac7bccaf22d1b977d802bf40c85f56f1dee1777

data/CHANGELOG.md

@@ -0,0 +1,2 @@
+## 0.1.1
+  - First release to support OSS

data/CONTRIBUTORS

@@ -0,0 +1,10 @@
+The following is a list of people who have contributed ideas, code, bug
+reports, or in general have helped logstash along its way.
+
+Contributors:
+* jinhu.wu - jinhu.wu.nju@gmail.com
+
+Note: If you've sent us patches, bug reports, or otherwise contributed to
+Logstash, and you aren't on the list above and want to be, please let us know
+and we'll make sure you're here. Contributions from folks like you are what make
+open source awesome.

data/DEVELOPER.md

@@ -0,0 +1,10 @@
+# Running the tests
+```bash
+bundle install
+bundle exec rspec
+```
+
+If you want to run the integration tests against a real bucket you need to pass your Aliyun credentials to the test runner or declare it in your environment.
+```bash
+OSS_ENDPOINT="Aliyun OSS endpoint to connect to" OSS_ACCESS_KEY="Your access key id" OSS_SECRET_KEY="Your access secret" OSS_BUCKET="Your bucket" bundle exec rspec spec/integration --tag integration
+```

data/Gemfile

@@ -0,0 +1,2 @@
+source 'https://rubygems.org'
+gemspec

data/LICENSE

@@ -0,0 +1,11 @@
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

data/README.md

@@ -0,0 +1,149 @@
+# Logstash OSS Output Plugin
+
+This is a plugin for [Logstash](https://github.com/elastic/logstash).
+
+It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
+
+## Documentation
+This plugin batches and uploads logstash events into Aliyun Object Storage Service (Aliyun OSS).
+
+First, you should have a writable bucket and OSS access permissions((Typically access_key_id and access_key_secret)).
+
+OSS output plugin creates temporary files into the OS' temporary directory(You can set this configuration by **temporary_directory** option) before uploading them to OSS.
+
+OSS output plugin output files have the following format
+```bash
+/tmp/logstash/oss/eaced620-e972-0136-2a14-02b7449ba0a9/logstash/1/ls.oss.eaced620-e972-0136-2a14-02b7449ba0a9.2018-12-24T14.27.part-0.data
+```
+
+|||
+|---|---|
+|/tmp/logstash/oss| OS' temporary directory specified by **temporary_directory** option |
+|eaced620-e972-0136-2a14-02b7449ba0a9 | random uuid |
+|logstash/1|OSS object prefix|
+|ls.oss|indicate Logstash OSS output plugin|
+|eaced620-e972-0136-2a14-02b7449ba0a9 | random uuid |
+|2018-12-24T14.27 | represents the file created time |
+|part-0|This is the nth file of this prefix|
+|.data|output suffix, if you set `encoding` to gzip , it will ends with .gz, else ends with .data|
+
+
+This plugin also supports crash recovery, you can set configuration **recover** to true if you want to recover from abnormal crash.
+
+### Usage:
+This is an example of logstash config:
+```ruby
+input {
+  file {
+    path => "/etc/logstash-6.5.3/sample.data"
+    codec => json {
+      charset => "UTF-8"
+    }
+  }
+}
+
+output {
+  oss {
+    "endpoint" => "oss-cn-zhangjiakou.aliyuncs.com"         (required)
+    "bucket" => "Your bucket name"                          (required)
+    "access_key_id" => "Your access key id"                 (required)
+    "access_key_secret" => "Your access secret key"         (required)
+    "prefix" => "logstash/%{index}"                         (optional, default = "")
+    "recover" => true                                       (optional, default = true)
+    "rotation_strategy" => "size_and_time"                  (optional, default = "size_and_time")
+    "time_rotate" => 15                                     (optional, default = 15) - Minutes
+    "size_rotate" => 31457280                               (optional, default = 31457280) - Bytes
+    "encoding" => "gzip"                                    (optional, default = "none")
+    "additional_oss_settings" => {
+      "max_connections_to_oss" => 1024                      (optional, default = 1024)
+      "secure_connection_enabled" => false                  (optional, default = true)
+    }
+    codec => json {
+      charset => "UTF-8"
+    }
+  }
+}
+```
+
+### Logstash OSS Output Configuration Options
+This plugin supports the following configuration options
+
+|Configuration|Type|Required|Comments|
+|:---:|:---:|:---:|:---|
+|endpoint|string|Yes|OSS endpoint to connect|
+|bucket|string|Yes|Your OSS bucket name|
+|access_key_id|string|Yes|Your access key id|
+|access_key_secret|string|Yes|Your access secret key|
+|prefix|string|No|Prefix that added to the generated file name(WARNING: this option supports string interpolation, so it may create a lot of temporary local files)|
+|additional_oss_settings|hash|No|Additional oss client configurations, valid keys are: `server_side_encryption_algorithm`, `secure_connection_enabled` and `max_connections_to_oss`|
+|temporary_directory|string|No|Temporary directory that used to cache events before uploading to OSS, default is /{OS' tmp dir}/logstash/oss|
+|rotation_strategy|string|No|File rotation strategy. Options are `size`, `time` and `size_and_time`|
+|size_rotate|number|No|Rotate this file if its size greater than or equal to `size_rotate`(depends on `rotation_strategy`)|
+|time_rotate|number|No|Rotate this file if its life time greater than or equal to `time_rotate`(depends on `rotation_strategy`)|
+|upload_workers_count|number|No|Concurrent number of upload threads|
+|upload_queue_size|number|No|Upload queue size|
+|encoding|string|No|Support plain and gzip compression before uploading files to OSS. Options are `gzip` and `none`|
+
+## Need Help?
+
+Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/logstash discussion forum.
+
+## Developing
+
+### 1. Plugin Developement and Testing
+
+#### Code
+- To get started, you'll need JRuby with the Bundler gem installed.
+
+- Install dependencies
+```sh
+bundle install
+```
+
+#### Test
+
+- Update your dependencies
+
+```sh
+bundle install
+```
+
+- Run tests
+
+```sh
+bundle exec rspec
+```
+
+### 2. Running your unpublished Plugin in Logstash
+
+#### 2.1 Run in an installed Logstash
+
+you can build the gem and install it using:
+
+- Build your plugin gem
+
+```sh
+gem build logstash-output-oss.gemspec
+```
+
+- Install the plugin from the Logstash home
+
+```sh
+bin/logstash-plugin install /path/to/logstash-output-oss-0.1.0-java.gem
+```
+
+- Start Logstash and proceed to test the plugin
+
+```bash
+./bin/logstash -f config/logstash-sample.conf
+```
+
+## Contributing
+
+All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.
+
+Programming is not a required skill. Whatever you've seen about open source and maintainers or community members  saying "send patches or die" - you will not see that here.
+
+It is more important to the community that you are able to contribute.
+
+For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file.

data/lib/logstash-output-oss_jars.rb

@@ -0,0 +1,52 @@
+# this is a generated file, to avoid over-writing it just delete this comment
+begin
+  require 'jar_dependencies'
+rescue LoadError
+  require 'org/apache/httpcomponents/httpcore/4.4.1/httpcore-4.4.1.jar'
+  require 'com/aliyun/oss/aliyun-sdk-oss/3.4.0/aliyun-sdk-oss-3.4.0.jar'
+  require 'commons-codec/commons-codec/1.9/commons-codec-1.9.jar'
+  require 'org/codehaus/jackson/jackson-xc/1.8.3/jackson-xc-1.8.3.jar'
+  require 'com/aliyun/aliyun-java-sdk-ram/3.0.0/aliyun-java-sdk-ram-3.0.0.jar'
+  require 'org/apache/httpcomponents/httpclient/4.4.1/httpclient-4.4.1.jar'
+  require 'javax/xml/bind/jaxb-api/2.2.2/jaxb-api-2.2.2.jar'
+  require 'com/sun/jersey/jersey-json/1.9/jersey-json-1.9.jar'
+  require 'org/codehaus/jettison/jettison/1.1/jettison-1.1.jar'
+  require 'com/sun/jersey/jersey-core/1.9/jersey-core-1.9.jar'
+  require 'javax/xml/stream/stax-api/1.0-2/stax-api-1.0-2.jar'
+  require 'com/aliyun/aliyun-java-sdk-core/3.4.0/aliyun-java-sdk-core-3.4.0.jar'
+  require 'commons-logging/commons-logging/1.2/commons-logging-1.2.jar'
+  require 'org/codehaus/jackson/jackson-jaxrs/1.8.3/jackson-jaxrs-1.8.3.jar'
+  require 'javax/activation/activation/1.1/activation-1.1.jar'
+  require 'com/sun/xml/bind/jaxb-impl/2.2.3-1/jaxb-impl-2.2.3-1.jar'
+  require 'org/jdom/jdom/1.1/jdom-1.1.jar'
+  require 'org/codehaus/jackson/jackson-mapper-asl/1.8.3/jackson-mapper-asl-1.8.3.jar'
+  require 'org/codehaus/jackson/jackson-core-asl/1.8.3/jackson-core-asl-1.8.3.jar'
+  require 'com/aliyun/aliyun-java-sdk-ecs/4.2.0/aliyun-java-sdk-ecs-4.2.0.jar'
+  require 'com/aliyun/aliyun-java-sdk-sts/3.0.0/aliyun-java-sdk-sts-3.0.0.jar'
+  require 'stax/stax-api/1.0.1/stax-api-1.0.1.jar'
+end
+
+if defined? Jars
+  require_jar 'org.apache.httpcomponents', 'httpcore', '4.4.1'
+  require_jar 'com.aliyun.oss', 'aliyun-sdk-oss', '3.4.0'
+  require_jar 'commons-codec', 'commons-codec', '1.9'
+  require_jar 'org.codehaus.jackson', 'jackson-xc', '1.8.3'
+  require_jar 'com.aliyun', 'aliyun-java-sdk-ram', '3.0.0'
+  require_jar 'org.apache.httpcomponents', 'httpclient', '4.4.1'
+  require_jar 'javax.xml.bind', 'jaxb-api', '2.2.2'
+  require_jar 'com.sun.jersey', 'jersey-json', '1.9'
+  require_jar 'org.codehaus.jettison', 'jettison', '1.1'
+  require_jar 'com.sun.jersey', 'jersey-core', '1.9'
+  require_jar 'javax.xml.stream', 'stax-api', '1.0-2'
+  require_jar 'com.aliyun', 'aliyun-java-sdk-core', '3.4.0'
+  require_jar 'commons-logging', 'commons-logging', '1.2'
+  require_jar 'org.codehaus.jackson', 'jackson-jaxrs', '1.8.3'
+  require_jar 'javax.activation', 'activation', '1.1'
+  require_jar 'com.sun.xml.bind', 'jaxb-impl', '2.2.3-1'
+  require_jar 'org.jdom', 'jdom', '1.1'
+  require_jar 'org.codehaus.jackson', 'jackson-mapper-asl', '1.8.3'
+  require_jar 'org.codehaus.jackson', 'jackson-core-asl', '1.8.3'
+  require_jar 'com.aliyun', 'aliyun-java-sdk-ecs', '4.2.0'
+  require_jar 'com.aliyun', 'aliyun-java-sdk-sts', '3.0.0'
+  require_jar 'stax', 'stax-api', '1.0.1'
+end

data/lib/logstash/outputs/oss.rb

@@ -0,0 +1,288 @@
+# encoding: utf-8
+require "logstash/outputs/base"
+require 'logstash-output-oss_jars'
+require 'concurrent'
+
+java_import "com.aliyun.oss.OSS"
+java_import "com.aliyun.oss.OSSClientBuilder"
+java_import "com.aliyun.oss.ClientBuilderConfiguration"
+
+#
+#  Logstash OSS Output Plugin
+#
+# Usage:
+# In order to write output data to OSS, you should add configurations like below to logstash
+# output {
+#   oss {
+#     "endpoint" => "oss-cn-zhangjiakou.aliyuncs.com"         (required)
+#     "bucket" => "Your bucket name"                          (required)
+#     "access_key_id" => "Your access key id"                 (required)
+#     "access_key_secret" => "Your access secret key"         (required)
+#     "prefix" => "logstash/%{index}"                         (optional, default = "")
+#     "recover" => true                                       (optional, default = true)
+#     "rotation_strategy" => "size_and_time"                  (optional, default = "size_and_time")
+#     "time_rotate" => 15                                     (optional, default = 15) - Minutes
+#     "size_rotate" => 31457280                               (optional, default = 31457280) - Bytes
+#     "encoding" => "none"                                    (optional, default = "none")
+#     "additional_oss_settings" => {
+#       "max_connections_to_oss" => 1024                      (optional, default = 1024)
+#       "secure_connection_enabled" => false                  (optional, default = true)
+#     }
+#   }
+# }
+#
+class LogStash::Outputs::OSS < LogStash::Outputs::Base
+  require 'logstash/outputs/oss/rotations/hybrid_rotation'
+  require 'logstash/outputs/oss/file_uploader'
+  require 'logstash/outputs/oss/file_manager'
+  require 'logstash/outputs/oss/version'
+
+  ROTATE_CHECK_INTERVAL_IN_SECONDS = 15
+
+  MAX_CONNECTIONS_TO_OSS_KEY = "max_connections_to_oss"
+
+  SERVER_SIDE_ENCRYPTION_ALGORITHM_KEY = "server_side_encryption_algorithm"
+
+  SECURE_CONNECTION_ENABLED_KEY = "secure_connection_enabled"
+
+  config_name "oss"
+
+  concurrency :shared
+
+  default :codec, "line"
+
+  # OSS bucket name
+  config :bucket, :validate => :string, :required => true
+
+  # OSS endpoint to connect
+  config :endpoint, :validate => :string, :required => true
+
+  # access key id
+  config :access_key_id, :validate => :string, :required => true
+
+  # access secret key
+  config :access_key_secret, :validate => :string, :required => true
+
+  # additional oss client configurations, valid keys are:
+  # server_side_encryption_algorithm(server side encryption, only support AES256 now)
+  # secure_connection_enabled(enable https or not)
+  # max_connections_to_oss(max connections to oss)
+  # TODO: add other oss configurations
+  config :additional_oss_settings, :validate => :hash, :required => false
+
+  # rotate this file if its size greater or equal than `size_rotate`
+  config :size_rotate, :validate => :number, :default => 30 * 1024 * 1024
+
+  # rotate this file if its life time greater or equal than `time_rotate`
+  config :time_rotate, :validate => :number, :default => 15
+
+  # if true, OSS plugin will recover files since last crash
+  config :recover, :validate => :boolean, :default => true
+
+  # temporary directory that used to cache event before upload to OSS
+  config :temporary_directory, :validate => :string, :default => File.join(Dir.tmpdir, "logstash/oss")
+
+  # prefix that added to generated file name
+  # sample file name looks like
+  # `prefix`/logstash.oss.{random-uuid}.{%Y-%m-%dT%H.%M}.part-{index}.{extension}
+  # WARNING: this option support string interpolation, so it may create a lot of temporary local files
+  config :prefix, :validate => :string, :default => ''
+
+  # file rotation strategy
+  config :rotation_strategy, :validate => %w(size time size_and_time), :default => "size_and_time"
+
+  # concurrent number of upload threads
+  config :upload_workers_count, :validate => :number, :default => (Concurrent.processor_count * 0.5).ceil
+
+  # upload queue size
+  config :upload_queue_size, :validate => :number, :default => 2 * (Concurrent.processor_count * 0.25).ceil
+
+  # support plain and gzip compression before upload to OSS
+  config :encoding, :validate => %w(none gzip), :default => "none"
+
+  public
+  def register
+    # TODO: check if prefix is valid
+
+    # check if temporary_directory is writable
+    begin
+      FileUtils.mkdir_p(@temporary_directory) unless Dir.exist?(@temporary_directory)
+      ::File.writable?(@temporary_directory)
+    rescue
+      raise LogStash::ConfigurationError, "Logstash OSS Output Plugin can not write data to " + @temporary_directory
+    end
+
+    # check rotation configuration
+    if @size_rotate.nil? and @time_rotate.nil? || @size_rotate <= 0 && @time_rotate <= 0
+      raise LogStash::ConfigurationError, "Logstash OSS Output Plugin must have at least one of time_file or size_file set to a value greater than 0"
+    end
+
+    if @upload_workers_count <= 0 || @upload_queue_size <= 0
+      raise LogStash::ConfigurationError,  "Logstash OSS Output Plugin must have both upload_workers_count and upload_queue_size are positive"
+    end
+
+    # create upload thread pool
+    executor = Concurrent::ThreadPoolExecutor.new({ :min_threads => 1,
+                                                    :max_threads => @upload_workers_count,
+                                                    :max_queue => @upload_queue_size,
+                                                    :fallback_policy => :caller_runs })
+
+    # get file rotation strategy
+    @rotation = rotation
+
+    # initialize oss client
+    @oss = initialize_oss_client
+
+    # initialize file uploader
+    @file_uploader = FileUploader.new(@oss, @bucket, @additional_oss_settings, @logger, executor)
+
+    # initialize file manager
+    @file_manager = FileManager.new(@logger, @encoding, @temporary_directory)
+
+    # recover from crash
+    recover_from_crash if @recover
+
+    # start rotate check
+    start_rotate_check if @rotation.needs_periodic_check?
+  end # def register
+
+  public
+  def multi_receive_encoded(events_and_encoded)
+    prefixes = Set.new
+    events_and_encoded.each do |event, encoded|
+      prefix = event.sprintf(@prefix)
+      prefixes << prefix
+
+      begin
+        @file_manager.get_file_generator(prefix) { |generator| generator.current_file.write(encoded) }
+      rescue Errno::ENOSPC => e
+        @logger.error("Logstash OSS Output Plugin: No space left in temporary directory", :temporary_directory => @temporary_directory)
+        raise e
+      end
+    end
+    rotate(prefixes)
+  end
+
+  def close
+    @logger.info("Logstash OSS Output Plugin is shutting down...")
+
+    # stop rotate check
+    stop_rotate_check if @rotation.needs_periodic_check?
+
+    prefixes = @file_manager.prefixes
+    prefixes.each do |prefix|
+      @file_manager.get_file_generator(prefix) do |generator|
+        file = generator.current_file
+        file.close
+        if file.size > 0
+          # upload async
+          @file_uploader.upload_async(file, :on_complete => method(:clean_temporary_file))
+        end
+      end
+    end
+
+    @file_manager.close
+
+    # stop file uploader
+    @file_uploader.close
+
+    # stop oss client
+    @oss.shutdown
+  end
+
+  private
+  def initialize_oss_client
+    clientConf = ClientBuilderConfiguration.new
+    unless @additional_oss_settings.nil?
+      if @additional_oss_settings.include?(SECURE_CONNECTION_ENABLED_KEY)
+        clientConf.setProtocol(@additional_oss_settings[SECURE_CONNECTION_ENABLED_KEY] ?
+          com.aliyun.oss.common.comm.Protocol::HTTPS : com.aliyun.oss.common.comm.Protocol::HTTP)
+      end
+
+      if @additional_oss_settings.include?(MAX_CONNECTIONS_TO_OSS_KEY)
+        if @additional_oss_settings[MAX_CONNECTIONS_TO_OSS_KEY] <= 0
+          raise LogStash::ConfigurationError, "Logstash OSS output plugin must have positive " + MAX_CONNECTIONS_TO_OSS_KEY
+        end
+        clientConf.setMaxConnections(@additional_oss_settings[MAX_CONNECTIONS_TO_OSS_KEY])
+      else
+        clientConf.setMaxConnections(1024)
+      end
+    end
+
+    clientConf.setUserAgent(clientConf.getUserAgent() + ", Logstash-oss/" + Version.version)
+    OSSClientBuilder.new().build(@endpoint, @access_key_id, @access_key_secret, clientConf)
+  end
+
+  private
+  def rotation
+    case @rotation_strategy
+    when "size"
+      SizeBasedRotation.new(size_rotate)
+    when "time"
+      TimeBasedRotation.new(time_rotate)
+    when "size_and_time"
+      HybridRotation.new(size_rotate, time_rotate)
+    end
+  end
+
+  private
+  def recover_from_crash
+    @logger.info("Logstash OSS Output Plugin starts to recover from crash and uploading...")
+    Dir.glob(::File.join(@temporary_directory, "**/*"))
+      .select {|file| ::File.file?(file) }
+      .each do |file|
+        temporary_file = TemporaryFile.create_existing_file(file, @temporary_directory)
+        if temporary_file.size > 0
+          @file_uploader.upload_async(temporary_file, :on_complete => method(:clean_temporary_file))
+        else
+          clean_temporary_file(temporary_file)
+        end
+      end
+  end
+
+  private
+  def rotate(prefixes)
+    prefixes.each do |prefix|
+      @file_manager.get_file_generator(prefix) do |file_generator|
+        file = file_generator.current_file
+        if @rotation.rotate?(file)
+          @logger.info("Logstash OSS Output Plugin starts to rotate file",
+                       :strategy => @rotation.class.name,
+                       :key => file.key,
+                       :path => file.path,
+                       :size => file.size,
+                       :thread => Thread.current.to_s)
+          file.close
+          if file.size > 0
+            # upload async
+            @file_uploader.upload_async(file, :on_complete => method(:clean_temporary_file))
+          end
+          file_generator.rotate
+        end
+      end
+    end
+  end
+
+  private
+  def clean_temporary_file(file)
+    @logger.debug("Logstash OSS Output Plugin: starts to remove temporary file",
+                 :file => file.path)
+    file.delete!
+  end
+
+  private
+  def start_rotate_check
+    @rotate_check = Concurrent::TimerTask.new(:execution_interval => ROTATE_CHECK_INTERVAL_IN_SECONDS) do
+      @logger.debug("Logstash OSS Output Plugin: starts rotation check")
+
+      rotate(@file_manager.prefixes)
+    end
+
+    @rotate_check.execute
+  end
+
+  private
+  def stop_rotate_check
+    @rotate_check.shutdown
+  end
+end # class LogStash::Outputs::OSS