logstash-output-azureblob 0.9.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 3b36f0e9cef45f58a3ce40f93e9d8808f4acb554cf3eeea2fc20a77b6a62e25a
+  data.tar.gz: 372d3ac19b637d675df14fc9733eb8da90117c41a6aab4b5fa745d31ea46c81c
+SHA512:
+  metadata.gz: 0bf72cc5ba7923f1c9ba8714c33ca1b226ef40c04fe611ece979f254e65f44a9cee95c332734e05678ad3dc6c13c59be1278b987145cfd3d6bf633c2a2e58863
+  data.tar.gz: 9aad5f4b182892d3700a09b3e58a95846ca235f14075aef507898bff50cc36965d7e0a73c3b21c90641da8dab06051143cd949b130eeeb857d8de762c38d5e4f

data/CHANGELOG.md

@@ -0,0 +1,2 @@
+## 0.90
+  - Initial release to support latest libraries and jruby versions

data/CONTRIBUTORS

@@ -0,0 +1,16 @@
+The following is a list of people who have contributed ideas, code, bug
+reports, or in general have helped logstash along its way.
+
+Current Contributors:
+* Sean Stark -sean.stark@microsoft.com
+
+Original Contributors - no longer maintainers
+* Tuffk - tuffkmulhall@gmail.com
+* BrunoLerner - bru.lerner@gmail.com
+* Alex-Tsyganok -
+* Charlie Zha - zysimplelife@gmail.com
+
+Note: If you've sent us patches, bug reports, or otherwise contributed to
+Logstash, and you aren't on the list above and want to be, please let us know
+and we'll make sure you're here. Contributions from folks like you are what make
+open source awesome.

data/DEVELOPER.md

@@ -0,0 +1,2 @@
+# logstash-output-Logstash_Azure_Blob_Output
+Example output plugin. This should help bootstrap your effort to write your own output plugin!

data/Gemfile

@@ -0,0 +1,3 @@
+source 'https://rubygems.org'
+gemspec
+# gem "logstash", :github => "elastic/logstash", :branch => "main"

data/LICENSE

@@ -0,0 +1,11 @@
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

data/README.md

@@ -0,0 +1,102 @@
+
+# Logstash Output Plugin for Azure Blob Storage
+
+This is an output plugin for [Logstash](https://github.com/elastic/logstash). It is fully free and open source. The license is Apache 2.0. This plugin was forked from https://github.com/tuffk/Logstash-output-to-Azure-Blob and updated to use the latest Azure Storage Ruby SDK
+
+- This plugin uses the https://github.com/Azure/azure-storage-ruby library
+- The class documentation is here: https://www.rubydoc.info/gems/azure-storage-blob
+
+## Disclaimers
+
+I am not a Ruby developer and may not be able to respond efficently to issues or bugs. Please take this into consideration when using this plugin
+
+- Azure Data Lake Storage Gen2 accounts are not currently supported. 
+- Managed Identities and Service Principles are currently not supported for auth. 
+
+## Requirements
+- Logstash version 8.6+ [Installation instructions](https://www.elastic.co/guide/en/logstash/current/installing-logstash.html). 
+    - Tested on 8.6.2
+- Azure Storage Account
+    - Azure Storage Account Access Key(s)
+
+## Installation
+```sh
+bin/logstash-plugin install logstash-output-azureblob
+```
+> On Ubuntu the default path is /usr/share/logstash/bin/
+
+## Configuration
+
+Information about configuring Logstash can be found in the [Logstash configuration guide](https://www.elastic.co/guide/en/logstash/current/configuration.html).
+
+You will need to configure this plugin before sending events from Logstash to an Azure Storage Account. The following example shows the minimum you need to provide:
+
+```yaml
+output {
+    azure_blob {
+        storage_account_name => "my-azure-account"    # required
+        storage_access_key => "my-super-secret-key"   # required
+        container_name => "my-container"              # required
+        size_file => 1024*1024*5                      # optional
+        time_file => 10                               # optional
+        restore => true                               # optional
+        temporary_directory => "path/to/directory"    # optional
+        prefix => "a_prefix"                          # optional
+        upload_queue_size => 2                        # optional
+        upload_workers_count => 1                     # optional
+        rotation_strategy_val => "size_and_time"      # optional
+        tags => []                                    # optional
+        encoding => "none"                            # optional
+    }
+}
+```
+
+### Example with syslog
+
+```yaml
+input {
+    syslog {
+        port => "5514"
+        type => "syslog"
+        codec => cef
+    }
+}
+
+output {
+    azure_blob {
+        storage_account_name => "<account-name>"
+        storage_access_key => "<access-key>"
+        container_name => "<container-name>"
+    }
+}
+```
+
+## Development
+
+- Docker Image - [cameronkerrnz/logstash-plugin-dev:7.17](https://hub.docker.com/r/cameronkerrnz/logstash-plugin-dev)
+    - https://github.com/cameronkerrnz/logstash-plugin-dev
+- jruby 9.2.20.1 (2.5.8)
+- Logstash Version 8.6.2+
+
+1. Install Dependencies
+
+    ```shell
+    rake vendor
+    bundle install
+    ```
+2. Build the plugin
+    ```shell
+    gem build logstash-output-azureblob.gemspec
+    ```
+3. Install Locally
+    ```shell
+    /usr/share/logstash/bin/logstash-plugin install /usr/share/logstash/logstash-output-azureblob-0.9.0.gem
+    ```
+4. Test with configuration file
+    ```shell
+    /usr/share/logstash/bin/logstash -f blob.conf
+    ```
+    
+## Contributing
+
+All contributions are welcome: ideas, patches, documentation, bug reports, and complaints. For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file.

data/lib/logstash/outputs/azure_blob.rb

@@ -0,0 +1,235 @@
+require 'logstash/outputs/base'
+require 'logstash/namespace'
+require 'azure/storage/blob'
+require 'azure/storage/common'
+require 'tmpdir'
+
+class LogStash::Outputs::LogstashAzureBlobOutput < LogStash::Outputs::Base
+  # name for the namespace under output for logstash configuration
+  config_name 'azure_blob'
+  default :codec, "line"
+
+  require 'logstash/outputs/blob/writable_directory_validator'
+  require 'logstash/outputs/blob/path_validator'
+  require 'logstash/outputs/blob/size_rotation_policy'
+  require 'logstash/outputs/blob/time_rotation_policy'
+  require 'logstash/outputs/blob/size_and_time_rotation_policy'
+  require 'logstash/outputs/blob/temporary_file'
+  require 'logstash/outputs/blob/temporary_file_factory'
+  require 'logstash/outputs/blob/uploader'
+  require 'logstash/outputs/blob/file_repository'
+
+  PREFIX_KEY_NORMALIZE_CHARACTER = '_'.freeze
+  PERIODIC_CHECK_INTERVAL_IN_SECONDS = 15
+  CRASH_RECOVERY_THREADPOOL = Concurrent::ThreadPoolExecutor.new(min_threads: 1,
+                                                                 max_threads: 2,
+                                                                 fallback_policy: :caller_runs)
+
+  # azure container
+  config :storage_account_name, validate: :string, required: false
+
+  # azure key
+  config :storage_access_key, validate: :string, required: false
+
+  # conatainer name
+  config :container_name, validate: :string, required: false
+
+  # mamadas
+  config :size_file, validate: :number, default: 1024 * 1024 * 5
+  config :time_file, validate: :number, default: 15
+  config :restore, validate: :boolean, default: true
+  config :temporary_directory, validate: :string, default: File.join(Dir.tmpdir, 'logstash')
+  config :prefix, validate: :string, default: ''
+  config :upload_queue_size, validate: :number, default: 2 * (Concurrent.processor_count * 0.25).ceil
+  config :upload_workers_count, validate: :number, default: (Concurrent.processor_count * 0.5).ceil
+  config :rotation_strategy_val, validate: %w[size_and_time size time], default: 'size_and_time'
+  config :tags, validate: :array, default: []
+  config :encoding, validate: %w[none gzip], default: 'none'
+
+  attr_accessor :storage_account_name, :storage_access_key, :container_name,
+                :size_file, :time_file, :restore, :temporary_directory, :prefix, :upload_queue_size,
+                :upload_workers_count, :rotation_strategy_val, :tags, :encoding
+
+  # initializes the +LogstashAzureBlobOutput+ instances
+  # validates all config parameters
+  # initializes the uploader
+  def register
+    unless @prefix.empty?
+      unless PathValidator.valid?(prefix)
+        raise LogStash::ConfigurationError.new("Prefix must not contains: #{PathValidator::INVALID_CHARACTERS}")
+      end
+    end
+
+    unless WritableDirectoryValidator.valid?(@temporary_directory)
+      raise LogStash::ConfigurationError.new("Logstash must have the permissions to write to the temporary directory: #{@temporary_directory}")
+    end
+
+    if @time_file.nil? && @size_file.nil? || @size_file.zero? && @time_file.zero?
+      raise LogStash::ConfigurationError.new('at least one of time_file or size_file set to a value greater than 0')
+    end
+
+    @file_repository = FileRepository.new(@tags, @encoding, @temporary_directory)
+
+    @rotation = rotation_strategy
+
+    executor = Concurrent::ThreadPoolExecutor.new(min_threads: 1,
+                                                  max_threads: @upload_workers_count,
+                                                  max_queue: @upload_queue_size,
+                                                  fallback_policy: :caller_runs)
+
+    @uploader = Uploader.new(blob_container_resource, container_name, @logger, executor)
+
+    restore_from_crash if @restore
+    start_periodic_check if @rotation.needs_periodic?
+  end
+
+  # Receives multiple events and check if there is space in temporary directory
+  # @param events_and_encoded [Object]
+  def multi_receive_encoded(events_and_encoded)
+    prefix_written_to = Set.new
+
+    events_and_encoded.each do |event, encoded|
+      prefix_key = normalize_key(event.sprintf(@prefix))
+      prefix_written_to << prefix_key
+
+      begin
+        @file_repository.get_file(prefix_key) { |file| file.write(encoded) }
+        # The output should stop accepting new events coming in, since it cannot do anything with them anymore.
+        # Log the error and rethrow it.
+      rescue Errno::ENOSPC => e
+        @logger.error('Azure: No space left in temporary directory', temporary_directory: @temporary_directory)
+        raise e
+      end
+    end
+
+    # Groups IO calls to optimize fstat checks
+    rotate_if_needed(prefix_written_to)
+  end
+
+  # close the temporary file and uploads the content to Azure
+  def close
+    stop_periodic_check if @rotation.needs_periodic?
+
+    @logger.debug('Uploading current workspace')
+
+    # The plugin has stopped receiving new events, but we still have
+    # data on disk, lets make sure it get to Azure blob.
+    # If Logstash get interrupted, the `restore_from_crash` (when set to true) method will pickup
+    # the content in the temporary directly and upload it.
+    # This will block the shutdown until all upload are done or the use force quit.
+    @file_repository.each_files do |file|
+      upload_file(file)
+    end
+
+    @file_repository.shutdown
+
+    @uploader.stop # wait until all the current upload are complete
+    @crash_uploader.stop if @restore # we might have still work to do for recovery so wait until we are done
+  end
+
+  # Validates and normalize prefix key
+  # @param prefix_key [String]
+  def normalize_key(prefix_key)
+    prefix_key.gsub(PathValidator.matches_re, PREFIX_KEY_NORMALIZE_CHARACTER)
+  end
+
+  # checks periodically the tmeporary file if it needs to be rotated
+  def start_periodic_check
+    @logger.debug('Start periodic rotation check')
+
+    @periodic_check = Concurrent::TimerTask.new(execution_interval: PERIODIC_CHECK_INTERVAL_IN_SECONDS) do
+      @logger.debug('Periodic check for stale files')
+
+      rotate_if_needed(@file_repository.keys)
+    end
+
+    @periodic_check.execute
+  end
+
+  def stop_periodic_check
+    @periodic_check.shutdown
+  end
+
+  # login to azure cloud using azure storage blob client and create the container if it doesn't exist
+  # @return [Object] the azure_blob_service object, which is the endpoint to azure gem
+  def blob_container_resource
+    blob_client = Azure::Storage::Blob::BlobService.create(
+        storage_account_name: storage_account_name, 
+        storage_access_key: storage_access_key
+    )
+    list = blob_client.list_containers()
+    list.each do |item|
+      @container = item if item.name == container_name
+    end
+
+    blob_client.create_container(container_name) unless @container
+    blob_client
+  end
+
+  # check if it needs to rotate according to rotation policy and rotates it if it needs
+  # @param prefixes [String]
+  def rotate_if_needed(prefixes)
+    prefixes.each do |prefix|
+      # Each file access is thread safe,
+      # until the rotation is done then only
+      # one thread has access to the resource.
+      @file_repository.get_factory(prefix) do |factory|
+        temp_file = factory.current
+
+        if @rotation.rotate?(temp_file)
+          @logger.debug('Rotate file',
+                        strategy: @rotation.class.name,
+                        key: temp_file.key,
+                        path: temp_file.path)
+
+          upload_file(temp_file)
+          factory.rotate!
+        end
+      end
+    end
+  end
+
+  # uploads the file using the +Uploader+
+  def upload_file(temp_file)
+    @logger.debug('Queue for upload', path: temp_file.path)
+
+    # if the queue is full the calling thread will be used to upload
+    temp_file.close # make sure the content is on disk
+    unless temp_file.empty? # rubocop:disable GuardClause
+      @uploader.upload_async(temp_file,
+                             on_complete: method(:clean_temporary_file))
+    end
+  end
+
+  # creates an instance for the rotation strategy
+  def rotation_strategy
+    case @rotation_strategy_val
+    when 'size'
+      SizeRotationPolicy.new(size_file)
+    when 'time'
+      TimeRotationPolicy.new(time_file)
+    when 'size_and_time'
+      SizeAndTimeRotationPolicy.new(size_file, time_file)
+    end
+  end
+
+  # Cleans the temporary files after it is uploaded to azure blob
+  def clean_temporary_file(file)
+    @logger.debug('Removing temporary file', file: file.path)
+    file.delete!
+  end
+
+  # uploads files if there was a crash before
+  def restore_from_crash
+    @crash_uploader = Uploader.new(blob_container_resource, container_name, @logger, CRASH_RECOVERY_THREADPOOL)
+
+    temp_folder_path = Pathname.new(@temporary_directory)
+    Dir.glob(::File.join(@temporary_directory, '**/*'))
+       .select { |file| ::File.file?(file) }
+       .each do |file|
+      temp_file = TemporaryFile.create_from_existing_file(file, temp_folder_path)
+      @logger.debug('Recovering from crash and uploading', file: temp_file.path)
+      @crash_uploader.upload_async(temp_file, on_complete: method(:clean_temporary_file))
+    end
+  end
+end

data/lib/logstash/outputs/blob/file_repository.rb

@@ -0,0 +1,138 @@
+
+require 'java'
+require 'concurrent'
+require 'concurrent/timer_task'
+require 'logstash/util'
+
+ConcurrentHashMap = java.util.concurrent.ConcurrentHashMap
+
+module LogStash
+  module Outputs
+    class LogstashAzureBlobOutput
+      # sub class for +LogstashAzureBlobOutput+
+      # this class manages the temporary directory for the temporary files
+      class FileRepository
+        DEFAULT_STATE_SWEEPER_INTERVAL_SECS = 60
+        DEFAULT_STALE_TIME_SECS = 15 * 60
+        # Ensure that all access or work done
+        # on a factory is threadsafe
+        class PrefixedValue
+          # initialize the factory
+          def initialize(file_factory, stale_time)
+            @file_factory = file_factory
+            @lock = Mutex.new
+            @stale_time = stale_time
+          end
+
+          # activate the lock
+          def with_lock
+            @lock.synchronize do
+              yield @file_factory
+            end
+          end
+
+          # boolean method
+          def stale?
+            with_lock { |factory| factory.current.size.zero? && (Time.now - factory.current.ctime > @stale_time) }
+          end
+
+          # return this class
+          def apply(_prefix)
+            self
+          end
+
+          # delete the current factory
+          def delete!
+            with_lock { |factory| factory.current.delete! }
+          end
+        end
+
+        # class for initializing the repo manager
+        class FactoryInitializer
+          # initializes the class
+          def initialize(tags, encoding, temporary_directory, stale_time)
+            @tags = tags
+            @encoding = encoding
+            @temporary_directory = temporary_directory
+            @stale_time = stale_time
+          end
+
+          # applies the prefix key
+          def apply(prefix_key)
+            PrefixedValue.new(TemporaryFileFactory.new(prefix_key, @tags, @encoding, @temporary_directory), @stale_time)
+          end
+        end
+        # initializes the class with more variables
+        def initialize(tags, encoding, temporary_directory,
+                       stale_time = DEFAULT_STALE_TIME_SECS,
+                       sweeper_interval = DEFAULT_STATE_SWEEPER_INTERVAL_SECS)
+          # The path need to contains the prefix so when we start
+          # logtash after a crash we keep the remote structure
+          @prefixed_factories = ConcurrentHashMap.new
+
+          @sweeper_interval = sweeper_interval
+
+          @factory_initializer = FactoryInitializer.new(tags, encoding, temporary_directory, stale_time)
+
+          start_stale_sweeper
+        end
+
+        # gets the key set
+        def keys
+          @prefixed_factories.keySet
+        end
+
+        # with lock for each file
+        def each_files
+          @prefixed_factories.elements.each do |prefixed_file|
+            prefixed_file.with_lock { |factory| yield factory.current }
+          end
+        end
+
+        # Return the file factory
+        def get_factory(prefix_key)
+          @prefixed_factories.computeIfAbsent(prefix_key, @factory_initializer).with_lock { |factory| yield factory }
+        end
+
+        # gets file from prefix_key
+        def get_file(prefix_key)
+          get_factory(prefix_key) { |factory| yield factory.current }
+        end
+
+        # stops. shutdown
+        def shutdown
+          stop_stale_sweeper
+        end
+
+        # gets factory's size
+        def size
+          @prefixed_factories.size
+        end
+
+        # remove the stale given key and value
+        def remove_stale(k, v)
+          if v.stale? # rubocop:disable Style/GuardClause
+            @prefixed_factories.remove(k, v)
+            v.delete!
+          end
+        end
+
+        # starts the stale sweeper
+        def start_stale_sweeper
+          @stale_sweeper = Concurrent::TimerTask.new(execution_interval: @sweeper_interval) do
+            LogStash::Util.set_thread_name('LogstashAzureBlobOutput, Stale factory sweeper')
+
+            @prefixed_factories.forEach { |k, v| remove_stale(k, v) }
+          end
+
+          @stale_sweeper.execute
+        end
+
+        # stops the stale sweeper
+        def stop_stale_sweeper
+          @stale_sweeper.shutdown
+        end
+      end
+    end
+  end
+end

data/lib/logstash/outputs/blob/path_validator.rb

@@ -0,0 +1,20 @@
+module LogStash
+  module Outputs
+    class LogstashAzureBlobOutput
+      # a sub class of +LogstashAzureBlobOutput+
+      # valdiates the path for the temporary directory
+      class PathValidator
+        INVALID_CHARACTERS = "\^`><".freeze
+        # boolean method to check if a name is valid
+        def self.valid?(name)
+          name.match(matches_re).nil?
+        end
+
+        # define the invalid characters that shouldn't be in the path name
+        def self.matches_re
+          /[#{Regexp.escape(INVALID_CHARACTERS)}]/
+        end
+      end
+    end
+  end
+end

data/lib/logstash/outputs/blob/size_and_time_rotation_policy.rb

@@ -0,0 +1,28 @@
+require 'logstash/outputs/blob/size_rotation_policy'
+require 'logstash/outputs/blob/time_rotation_policy'
+
+module LogStash
+  module Outputs
+    class LogstashAzureBlobOutput
+      # a sub class of +LogstashAzureBlobOutput+
+      # sets the rotation policy
+      class SizeAndTimeRotationPolicy
+        # initialize the class
+        def initialize(file_size, time_file)
+          @size_strategy = SizeRotationPolicy.new(file_size)
+          @time_strategy = TimeRotationPolicy.new(time_file)
+        end
+
+        # check if it is time to rotate
+        def rotate?(file)
+          @size_strategy.rotate?(file) || @time_strategy.rotate?(file)
+        end
+
+        # boolean method
+        def needs_periodic?
+          true
+        end
+      end
+    end
+  end
+end

data/lib/logstash/outputs/blob/size_rotation_policy.rb

@@ -0,0 +1,29 @@
+module LogStash
+  module Outputs
+    class LogstashAzureBlobOutput
+      # a sub class of +LogstashAzureBlobOutput+
+      # sets the rotation policy by size
+      class SizeRotationPolicy
+        attr_reader :size_file
+        # initialize the class
+        def initialize(size_file)
+          if size_file <= 0
+            raise LogStash::ConfigurationError.new('`size_file` need to be greather than 0')
+          end
+
+          @size_file = size_file
+        end
+
+        # boolean method to check if it is time to rotate
+        def rotate?(file)
+          file.size >= size_file
+        end
+
+        # boolean method
+        def needs_periodic?
+          false
+        end
+      end
+    end
+  end
+end

data/lib/logstash/outputs/blob/temporary_file.rb

@@ -0,0 +1,81 @@
+require 'thread'
+require 'forwardable'
+require 'fileutils'
+
+module LogStash
+  module Outputs
+    class LogstashAzureBlobOutput
+      # a sub class of +LogstashAzureBlobOutput+
+      # Wrap the actual file descriptor into an utility classe
+      # It make it more OOP and easier to reason with the paths.
+      class TemporaryFile
+        extend Forwardable
+
+        def_delegators :@fd, :path, :write, :close, :fsync
+
+        attr_reader :fd
+
+        # initialize the class
+        def initialize(key, fd, temp_path)
+          @fd = fd
+          @key = key
+          @temp_path = temp_path
+          @created_at = Time.now
+        end
+
+        # gets the created at time
+        def ctime
+          @created_at
+        end
+
+        # gets path to temporary directory
+        attr_reader :temp_path
+
+        # gets the size of file
+        def size
+          # Use the fd size to get the accurate result,
+          # so we dont have to deal with fsync
+          # if the file is close we will use the File::size
+
+          @fd.size
+        rescue IOError
+          ::File.size(path)
+        end
+
+        # gets the key
+        def key
+          @key.gsub(/^\//, '')
+        end
+
+        # Each temporary file is made inside a directory named with an UUID,
+        # instead of deleting the file directly and having the risk of deleting other files
+        # we delete the root of the UUID, using a UUID also remove the risk of deleting unwanted file, it acts as
+        # a sandbox.
+        def delete!
+          begin
+            @fd.close
+          rescue
+            IOError
+          end
+          FileUtils.rm_r(@temp_path, secure: true)
+        end
+
+        # boolean method to determine if the file is empty
+        def empty?
+          size.zero?
+        end
+
+        # creates the temporary file in an existing temporary directory from existing file
+        # @param file_path [String] path to the file
+        # @param temporary_folder [String] path to the temporary folder
+        def self.create_from_existing_file(file_path, temporary_folder)
+          key_parts = Pathname.new(file_path).relative_path_from(temporary_folder).to_s.split(::File::SEPARATOR)
+
+          TemporaryFile.new(key_parts.slice(1, key_parts.size).join('/'),
+                            ::File.open(file_path, 'r'),
+                            ::File.join(temporary_folder, key_parts.slice(0, 1)))
+        end
+      end
+    end
+  end
+end