logstash-output-kusto 1.0.6-java → 2.0.1-java

data/README.md

@@ -1,94 +1,103 @@
-# Logstash Output Plugin for Azure Data Explorer (Kusto)
-
-![build](https://github.com/Azure/logstash-output-kusto/workflows/build/badge.svg)
-![build](https://github.com/Azure/logstash-output-kusto/workflows/build/badge.svg?branch=master)
-[![Gem](https://img.shields.io/gem/v/logstash-output-kusto.svg)](https://rubygems.org/gems/logstash-output-kusto)
-[![Gem](https://img.shields.io/gem/dt/logstash-output-kusto.svg)](https://rubygems.org/gems/logstash-output-kusto)
-
-This is a plugin for [Logstash](https://github.com/elastic/logstash).
-
-It is fully free and open source. The license is Apache 2.0.
-
-This Azure Data Explorer (ADX) Logstash plugin enables you to process events from Logstash into an **Azure Data Explorer** database for later analysis. 
-
-## Requirements
-
-- Logstash version 6+. [Installation instructions](https://www.elastic.co/guide/en/logstash/current/installing-logstash.html) 
-- Azure Data Explorer cluster with a database. Read [Create a cluster and database](https://docs.microsoft.com/en-us/azure/data-explorer/create-cluster-database-portal) for more information.
-- AAD Application credentials with permission to ingest data into Azure Data Explorer. Read [Creating an AAD Application](https://docs.microsoft.com/en-us/azure/kusto/management/access-control/how-to-provision-aad-app) for more information.
-
-## Installation
-
-To make the Azure Data Explorer plugin available in your Logstash environment, run the following command:
-```sh
-bin/logstash-plugin install logstash-output-kusto
-```
-
-## Configuration
-
-Perform configuration before sending events from Logstash to Azure Data Explorer. The following example shows the minimum you need to provide. It should be enough for most use-cases:
-
-```ruby
-output {
-	kusto {
-            path => "/tmp/kusto/%{+YYYY-MM-dd-HH-mm}.txt"
-            ingest_url => "https://ingest-<cluster-name>.kusto.windows.net/"
-            app_id => "<application id>"
-            app_key => "<application key/secret>"
-            app_tenant => "<tenant id>"
-            database => "<database name>"
-            table => "<target table>"
-            json_mapping => "<mapping name>"
-            proxy_host => "<proxy host>"
-            proxy_port => <proxy port>
-            proxy_protocol => <"http"|"https">              
-	}
-}
-```
-More information about configuring Logstash can be found in the [logstash configuration guide](https://www.elastic.co/guide/en/logstash/current/configuration.html)
-
-### Available Configuration Keys
-
-| Parameter Name | Description | Notes |
-| --- | --- | --- |
-| **path** | The plugin writes events to temporary files before sending them to ADX. This parameter includes a path where files should be written and a time expression for file rotation to trigger an upload to the ADX service. The example above shows how to rotate the files every minute and check the Logstash docs for more information on time expressions. | Required
-| **ingest_url** | The Kusto endpoint for ingestion-related communication. See it on the Azure Portal.| Required|
-| **app_id, app_key, app_tenant**| Credentials required to connect to the ADX service. Be sure to use an application with 'ingest' privileges. | Required|
-| **database**| Database name to place events | Required |
-| **table** | Target table name to place events | Required
-| **json_mapping** | Maps each attribute from incoming event JSON strings to the appropriate column in the table. Note that this must be in JSON format, as this is the interface between Logstash and Kusto | Required |
-| **recovery** | If set to true (default), plugin will attempt to resend pre-existing temp files found in the path upon startup | |
-| **delete_temp_files** | Determines if temp files will be deleted after a successful upload (true is default; set false for debug purposes only)| |
-| **flush_interval** | The time (in seconds) for flushing writes to temporary files. Default is 2 seconds, 0 will flush on every event. Increase this value to reduce IO calls but keep in mind that events in the buffer will be lost in case of abrupt failure.| |
-| **proxy_host** | The proxy hostname for redirecting traffic to Kusto.| |
-| **proxy_port** | The proxy port for the proxy. Defaults to 80.| |
-| **proxy_protocol** | The proxy server protocol , is one of http or https.| |
-
-> Note : LS_JAVA_OPTS can be used to set proxy parameters as well (using export or SET options)
-
-```bash
-export  LS_JAVA_OPTS="-Dhttp.proxyHost=1.2.34 -Dhttp.proxyPort=8989 -Dhttps.proxyHost=1.2.3.4 -Dhttps.proxyPort=8989"
-```
-
-
-## Development Requirements
-
-- Openjdk **8 64bit** (https://www.openlogic.com/openjdk-downloads)
-- JRuby 9.2 or higher, defined with openjdk 8 64bit
-- Logstash, defined with openjdk 8 64bit
-
-*It is reccomened to use the bundled jdk and jruby with logstash to avoid compatibility issues.*
-
-To fully build the gem, run: 
-
-```shell
-bundle install
-lock_jars
-gem build
-```
-
-## Contributing
-
-All contributions are welcome: ideas, patches, documentation, bug reports, and complaints.
-Programming is not a required skill. It is more important to the community that you are able to contribute.
-For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file.
+# Logstash Output Plugin for Azure Data Explorer (Kusto)
+
+![build](https://github.com/Azure/logstash-output-kusto/workflows/build/badge.svg)
+![build](https://github.com/Azure/logstash-output-kusto/workflows/build/badge.svg?branch=master)
+[![Gem](https://img.shields.io/gem/v/logstash-output-kusto.svg)](https://rubygems.org/gems/logstash-output-kusto)
+[![Gem](https://img.shields.io/gem/dt/logstash-output-kusto.svg)](https://rubygems.org/gems/logstash-output-kusto)
+
+This is a plugin for [Logstash](https://github.com/elastic/logstash).
+
+It is fully free and open source. The license is Apache 2.0.
+
+This Azure Data Explorer (ADX) Logstash plugin enables you to process events from Logstash into an **Azure Data Explorer** database for later analysis. 
+
+## Requirements
+
+- Logstash version 6+. [Installation instructions](https://www.elastic.co/guide/en/logstash/current/installing-logstash.html) 
+- Azure Data Explorer cluster with a database. Read [Create a cluster and database](https://docs.microsoft.com/en-us/azure/data-explorer/create-cluster-database-portal) for more information.
+- AAD Application credentials with permission to ingest data into Azure Data Explorer. Read [Creating an AAD Application](https://docs.microsoft.com/en-us/azure/kusto/management/access-control/how-to-provision-aad-app) for more information.
+
+## Installation
+
+To make the Azure Data Explorer plugin available in your Logstash environment, run the following command:
+```sh
+bin/logstash-plugin install logstash-output-kusto
+```
+
+## Configuration
+
+Perform configuration before sending events from Logstash to Azure Data Explorer. The following example shows the minimum you need to provide. It should be enough for most use-cases:
+
+```ruby
+output {
+	kusto {
+            path => "/tmp/kusto/%{+YYYY-MM-dd-HH-mm}.txt"
+            ingest_url => "https://ingest-<cluster-name>.kusto.windows.net/"
+            app_id => "<application id>"
+            app_key => "<application key/secret>"
+            app_tenant => "<tenant id>"
+            database => "<database name>"
+            table => "<target table>"
+            json_mapping => "<mapping name>"
+            proxy_host => "<proxy host>"
+            proxy_port => <proxy port>
+            proxy_protocol => <"http"|"https">              
+	}
+}
+```
+More information about configuring Logstash can be found in the [logstash configuration guide](https://www.elastic.co/guide/en/logstash/current/configuration.html)
+
+### Available Configuration Keys
+
+| Parameter Name | Description | Notes |
+| --- | --- | --- |
+| **path** | The plugin writes events to temporary files before sending them to ADX. This parameter includes a path where files should be written and a time expression for file rotation to trigger an upload to the ADX service. The example above shows how to rotate the files every minute and check the Logstash docs for more information on time expressions. | Required
+| **ingest_url** | The Kusto endpoint for ingestion-related communication. See it on the Azure Portal.| Required|
+| **app_id, app_key, app_tenant**| Credentials required to connect to the ADX service. Be sure to use an application with 'ingest' privileges. | Optional|
+| **managed_identity**| Managed Identity to authenticate. For user-based managed ID, use the Client ID GUID. For system-based, use the value `system`. The ID needs to have 'ingest' privileges on the cluster. | Optional|
+| **database**| Database name to place events | Required |
+| **table** | Target table name to place events | Required
+| **json_mapping** | Maps each attribute from incoming event JSON strings to the appropriate column in the table. Note that this must be in JSON format, as this is the interface between Logstash and Kusto | Required |
+| **recovery** | If set to true (default), plugin will attempt to resend pre-existing temp files found in the path upon startup | |
+| **delete_temp_files** | Determines if temp files will be deleted after a successful upload (true is default; set false for debug purposes only)| |
+| **flush_interval** | The time (in seconds) for flushing writes to temporary files. Default is 2 seconds, 0 will flush on every event. Increase this value to reduce IO calls but keep in mind that events in the buffer will be lost in case of abrupt failure.| |
+| **proxy_host** | The proxy hostname for redirecting traffic to Kusto.| |
+| **proxy_port** | The proxy port for the proxy. Defaults to 80.| |
+| **proxy_protocol** | The proxy server protocol , is one of http or https.| |
+
+> Note : LS_JAVA_OPTS can be used to set proxy parameters as well (using export or SET options)
+
+```bash
+export  LS_JAVA_OPTS="-Dhttp.proxyHost=1.2.34 -Dhttp.proxyPort=8989 -Dhttps.proxyHost=1.2.3.4 -Dhttps.proxyPort=8989"
+```
+
+
+### Release Notes and versions
+
+| Version | Release Date | Notes |
+| --- | --- | --- |
+| 2.0.0 | 2023-09-19 | - Upgrade to the latest Java SDK version [5.0.2](https://github.com/Azure/azure-kusto-java/releases/tag/v5.0.2). Tests have been performed on **__Logstash 8.5__**  and up (Does not work with 6.x or 7.x versions of Logstash - For these versions use 1.x.x versions of logstash-output-kusto gem) - Fixes CVE's in common-text & outdated Jackson libraries  |
+| 1.0.6 | 2022-11-29 | - Upgrade to the latest Java SDK [3.2.1](https://github.com/Azure/azure-kusto-java/releases/tag/v3.2.1) version. Tests have been performed on Logstash 6.x and up.|
+
+
+## Development Requirements
+
+- Openjdk **8 64bit** (https://www.openlogic.com/openjdk-downloads)
+- JRuby 9.2 or higher, defined with openjdk 8 64bit
+- Logstash, defined with openjdk 8 64bit
+
+*It is reccomened to use the bundled jdk and jruby with logstash to avoid compatibility issues.*
+
+To fully build the gem, run: 
+
+```shell
+bundle install
+lock_jars
+gem build
+```
+
+## Contributing
+
+All contributions are welcome: ideas, patches, documentation, bug reports, and complaints.
+Programming is not a required skill. It is more important to the community that you are able to contribute.
+For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file.

data/SECURITY.md

@@ -1,41 +1,41 @@
-<!-- BEGIN MICROSOFT SECURITY.MD V0.0.7 BLOCK -->
-
-## Security
-
-Microsoft takes the security of our software products and services seriously, which includes all source code repositories managed through our GitHub organizations, which include [Microsoft](https://github.com/Microsoft), [Azure](https://github.com/Azure), [DotNet](https://github.com/dotnet), [AspNet](https://github.com/aspnet), [Xamarin](https://github.com/xamarin), and [our GitHub organizations](https://opensource.microsoft.com/).
-
-If you believe you have found a security vulnerability in any Microsoft-owned repository that meets [Microsoft's definition of a security vulnerability](https://aka.ms/opensource/security/definition), please report it to us as described below.
-
-## Reporting Security Issues
-
-**Please do not report security vulnerabilities through public GitHub issues.**
-
-Instead, please report them to the Microsoft Security Response Center (MSRC) at [https://msrc.microsoft.com/create-report](https://aka.ms/opensource/security/create-report).
-
-If you prefer to submit without logging in, send email to [secure@microsoft.com](mailto:secure@microsoft.com).  If possible, encrypt your message with our PGP key; please download it from the [Microsoft Security Response Center PGP Key page](https://aka.ms/opensource/security/pgpkey).
-
-You should receive a response within 24 hours. If for some reason you do not, please follow up via email to ensure we received your original message. Additional information can be found at [microsoft.com/msrc](https://aka.ms/opensource/security/msrc). 
-
-Please include the requested information listed below (as much as you can provide) to help us better understand the nature and scope of the possible issue:
-
-  * Type of issue (e.g. buffer overflow, SQL injection, cross-site scripting, etc.)
-  * Full paths of source file(s) related to the manifestation of the issue
-  * The location of the affected source code (tag/branch/commit or direct URL)
-  * Any special configuration required to reproduce the issue
-  * Step-by-step instructions to reproduce the issue
-  * Proof-of-concept or exploit code (if possible)
-  * Impact of the issue, including how an attacker might exploit the issue
-
-This information will help us triage your report more quickly.
-
-If you are reporting for a bug bounty, more complete reports can contribute to a higher bounty award. Please visit our [Microsoft Bug Bounty Program](https://aka.ms/opensource/security/bounty) page for more details about our active programs.
-
-## Preferred Languages
-
-We prefer all communications to be in English.
-
-## Policy
-
-Microsoft follows the principle of [Coordinated Vulnerability Disclosure](https://aka.ms/opensource/security/cvd).
-
-<!-- END MICROSOFT SECURITY.MD BLOCK -->
+<!-- BEGIN MICROSOFT SECURITY.MD V0.0.7 BLOCK -->
+
+## Security
+
+Microsoft takes the security of our software products and services seriously, which includes all source code repositories managed through our GitHub organizations, which include [Microsoft](https://github.com/Microsoft), [Azure](https://github.com/Azure), [DotNet](https://github.com/dotnet), [AspNet](https://github.com/aspnet), [Xamarin](https://github.com/xamarin), and [our GitHub organizations](https://opensource.microsoft.com/).
+
+If you believe you have found a security vulnerability in any Microsoft-owned repository that meets [Microsoft's definition of a security vulnerability](https://aka.ms/opensource/security/definition), please report it to us as described below.
+
+## Reporting Security Issues
+
+**Please do not report security vulnerabilities through public GitHub issues.**
+
+Instead, please report them to the Microsoft Security Response Center (MSRC) at [https://msrc.microsoft.com/create-report](https://aka.ms/opensource/security/create-report).
+
+If you prefer to submit without logging in, send email to [secure@microsoft.com](mailto:secure@microsoft.com).  If possible, encrypt your message with our PGP key; please download it from the [Microsoft Security Response Center PGP Key page](https://aka.ms/opensource/security/pgpkey).
+
+You should receive a response within 24 hours. If for some reason you do not, please follow up via email to ensure we received your original message. Additional information can be found at [microsoft.com/msrc](https://aka.ms/opensource/security/msrc). 
+
+Please include the requested information listed below (as much as you can provide) to help us better understand the nature and scope of the possible issue:
+
+  * Type of issue (e.g. buffer overflow, SQL injection, cross-site scripting, etc.)
+  * Full paths of source file(s) related to the manifestation of the issue
+  * The location of the affected source code (tag/branch/commit or direct URL)
+  * Any special configuration required to reproduce the issue
+  * Step-by-step instructions to reproduce the issue
+  * Proof-of-concept or exploit code (if possible)
+  * Impact of the issue, including how an attacker might exploit the issue
+
+This information will help us triage your report more quickly.
+
+If you are reporting for a bug bounty, more complete reports can contribute to a higher bounty award. Please visit our [Microsoft Bug Bounty Program](https://aka.ms/opensource/security/bounty) page for more details about our active programs.
+
+## Preferred Languages
+
+We prefer all communications to be in English.
+
+## Policy
+
+Microsoft follows the principle of [Coordinated Vulnerability Disclosure](https://aka.ms/opensource/security/cvd).
+
+<!-- END MICROSOFT SECURITY.MD BLOCK -->

data/lib/logstash/outputs/kusto/ingestor.rb

@@ -1,138 +1,166 @@
-# encoding: utf-8
-
-require 'logstash/outputs/base'
-require 'logstash/namespace'
-require 'logstash/errors'
-
-class LogStash::Outputs::Kusto < LogStash::Outputs::Base
-  ##
-  # This handles the overall logic and communication with Kusto
-  #
-  class Ingestor
-    require 'logstash-output-kusto_jars'
-    RETRY_DELAY_SECONDS = 3
-    DEFAULT_THREADPOOL = Concurrent::ThreadPoolExecutor.new(
-      min_threads: 1,
-      max_threads: 8,
-      max_queue: 1,
-      fallback_policy: :caller_runs
-    )
-    LOW_QUEUE_LENGTH = 3
-    FIELD_REF = /%\{[^}]+\}/
-
-    def initialize(ingest_url, app_id, app_key, app_tenant, database, table, json_mapping, delete_local, proxy_host , proxy_port , proxy_protocol,logger, threadpool = DEFAULT_THREADPOOL)
-      @workers_pool = threadpool
-      @logger = logger
-      validate_config(database, table, json_mapping,proxy_protocol)
-      @logger.info('Preparing Kusto resources.')
-
-      kusto_java = Java::com.microsoft.azure.kusto
-      apache_http = Java::org.apache.http
-      kusto_connection_string = kusto_java.data.auth.ConnectionStringBuilder.createWithAadApplicationCredentials(ingest_url, app_id, app_key.value, app_tenant)
-      #
-      @logger.debug(Gem.loaded_specs.to_s)
-      # Unfortunately there's no way to avoid using the gem/plugin name directly...
-      name_for_tracing = "logstash-output-kusto:#{Gem.loaded_specs['logstash-output-kusto']&.version || "unknown"}"
-      @logger.debug("Client name for tracing: #{name_for_tracing}")
-      kusto_connection_string.setClientVersionForTracing(name_for_tracing)
-      
-      @kusto_client = begin
-        if proxy_host.nil? || proxy_host.empty?
-          kusto_java.ingest.IngestClientFactory.createClient(kusto_connection_string)
-        else
-          kusto_http_client_properties = kusto_java.data.HttpClientProperties.builder().proxy(apache_http.HttpHost.new(proxy_host,proxy_port,proxy_protocol)).build()
-          kusto_java.ingest.IngestClientFactory.createClient(kusto_connection_string, kusto_http_client_properties)
-        end
-      end
-
-      @ingestion_properties = kusto_java.ingest.IngestionProperties.new(database, table)
-      @ingestion_properties.setIngestionMapping(json_mapping, kusto_java.ingest.IngestionMapping::IngestionMappingKind::JSON)
-      @ingestion_properties.setDataFormat(kusto_java.ingest.IngestionProperties::DataFormat::JSON)
-      @delete_local = delete_local
-
-      @logger.debug('Kusto resources are ready.')
-    end
-
-    def validate_config(database, table, json_mapping,proxy_protocol)
-      if database =~ FIELD_REF
-        @logger.error('database config value should not be dynamic.', database)
-        raise LogStash::ConfigurationError.new('database config value should not be dynamic.')
-      end
-
-      if table =~ FIELD_REF
-        @logger.error('table config value should not be dynamic.', table)
-        raise LogStash::ConfigurationError.new('table config value should not be dynamic.')
-      end
-
-      if json_mapping =~ FIELD_REF
-        @logger.error('json_mapping config value should not be dynamic.', json_mapping)
-        raise LogStash::ConfigurationError.new('json_mapping config value should not be dynamic.')
-      end
-
-      if not(["https", "http"].include? proxy_protocol)
-        @logger.error('proxy_protocol has to be http or https.', proxy_protocol)
-        raise LogStash::ConfigurationError.new('proxy_protocol has to be http or https.')
-      end
-
-    end
-
-    def upload_async(path, delete_on_success)
-      if @workers_pool.remaining_capacity <= LOW_QUEUE_LENGTH
-        @logger.warn("Ingestor queue capacity is running low with #{@workers_pool.remaining_capacity} free slots.")
-      end
-
-      @workers_pool.post do
-        LogStash::Util.set_thread_name("Kusto to ingest file: #{path}")
-        upload(path, delete_on_success)
-      end
-    rescue Exception => e
-      @logger.error('StandardError.', exception: e.class, message: e.message, path: path, backtrace: e.backtrace)
-      raise e
-    end
-
-    def upload(path, delete_on_success)
-      file_size = File.size(path)
-      @logger.debug("Sending file to kusto: #{path}. size: #{file_size}")
-
-      # TODO: dynamic routing
-      # file_metadata = path.partition('.kusto.').last
-      # file_metadata_parts = file_metadata.split('.')
-
-      # if file_metadata_parts.length == 3
-      #   # this is the number we expect - database, table, json_mapping
-      #   database = file_metadata_parts[0]
-      #   table = file_metadata_parts[1]
-      #   json_mapping = file_metadata_parts[2]
-
-      #   local_ingestion_properties = Java::KustoIngestionProperties.new(database, table)
-      #   local_ingestion_properties.addJsonMappingName(json_mapping)
-      # end
-
-      if file_size > 0
-        file_source_info = Java::com.microsoft.azure.kusto.ingest.source.FileSourceInfo.new(path, 0); # 0 - let the sdk figure out the size of the file
-        @kusto_client.ingestFromFile(file_source_info, @ingestion_properties)
-      else
-        @logger.warn("File #{path} is an empty file and is not ingested.")
-      end
-      File.delete(path) if delete_on_success
-      @logger.debug("File #{path} sent to kusto.")
-    rescue Errno::ENOENT => e
-      @logger.error("File doesn't exist! Unrecoverable error.", exception: e.class, message: e.message, path: path, backtrace: e.backtrace)
-    rescue Java::JavaNioFile::NoSuchFileException => e
-      @logger.error("File doesn't exist! Unrecoverable error.", exception: e.class, message: e.message, path: path, backtrace: e.backtrace)
-    rescue => e
-      # When the retry limit is reached or another error happen we will wait and retry.
-      #
-      # Thread might be stuck here, but I think its better than losing anything
-      # its either a transient errors or something bad really happened.
-      @logger.error('Uploading failed, retrying.', exception: e.class, message: e.message, path: path, backtrace: e.backtrace)
-      sleep RETRY_DELAY_SECONDS
-      retry
-    end
-
-    def stop
-      @workers_pool.shutdown
-      @workers_pool.wait_for_termination(nil) # block until its done
-    end
-  end
-end
+# encoding: utf-8
+
+require 'logstash/outputs/base'
+require 'logstash/namespace'
+require 'logstash/errors'
+
+class LogStash::Outputs::Kusto < LogStash::Outputs::Base
+  ##
+  # This handles the overall logic and communication with Kusto
+  #
+  class Ingestor
+    require 'logstash-output-kusto_jars'
+    RETRY_DELAY_SECONDS = 3
+    DEFAULT_THREADPOOL = Concurrent::ThreadPoolExecutor.new(
+      min_threads: 1,
+      max_threads: 8,
+      max_queue: 1,
+      fallback_policy: :caller_runs
+    )
+    LOW_QUEUE_LENGTH = 3
+    FIELD_REF = /%\{[^}]+\}/
+
+    def initialize(ingest_url, app_id, app_key, app_tenant, managed_identity_id, database, table, json_mapping, delete_local, proxy_host , proxy_port , proxy_protocol,logger, threadpool = DEFAULT_THREADPOOL)
+      @workers_pool = threadpool
+      @logger = logger
+      validate_config(database, table, json_mapping,proxy_protocol,app_id, app_key, managed_identity_id)
+      @logger.info('Preparing Kusto resources.')
+
+      kusto_java = Java::com.microsoft.azure.kusto
+      apache_http = Java::org.apache.http
+      # kusto_connection_string = kusto_java.data.auth.ConnectionStringBuilder.createWithAadApplicationCredentials(ingest_url, app_id, app_key.value, app_tenant)
+      # If there is managed identity, use it. This means the AppId and AppKey are empty/nil
+      is_managed_identity = (app_id.nil? && app_key.empty?)
+      # If it is system managed identity, propagate the system identity
+      is_system_assigned_managed_identity = is_managed_identity && 0 == "system".casecmp(managed_identity_id)
+      # Is it direct connection
+      is_direct_conn = (proxy_host.nil? || proxy_host.empty?)
+      # Create a connection string
+      kusto_connection_string = if is_managed_identity
+          if is_system_assigned_managed_identity
+            @logger.info('Using system managed identity.')
+            kusto_java.data.auth.ConnectionStringBuilder.createWithAadManagedIdentity(ingest_url)  
+          else
+            @logger.info('Using user managed identity.')
+            kusto_java.data.auth.ConnectionStringBuilder.createWithAadManagedIdentity(ingest_url, managed_identity_id)
+          end
+        else
+          kusto_java.data.auth.ConnectionStringBuilder.createWithAadApplicationCredentials(ingest_url, app_id, app_key.value, app_tenant)
+        end
+
+      #
+      @logger.debug(Gem.loaded_specs.to_s)
+      # Unfortunately there's no way to avoid using the gem/plugin name directly...
+      name_for_tracing = "logstash-output-kusto:#{Gem.loaded_specs['logstash-output-kusto']&.version || "unknown"}"
+      @logger.debug("Client name for tracing: #{name_for_tracing}")
+
+      tuple_utils = Java::org.apache.commons.lang3.tuple
+      # kusto_connection_string.setClientVersionForTracing(name_for_tracing)
+      version_for_tracing=Gem.loaded_specs['logstash-output-kusto']&.version || "unknown"
+      kusto_connection_string.setConnectorDetails("Logstash",version_for_tracing.to_s,"","",false,"", tuple_utils.Pair.emptyArray());
+      
+      @kusto_client = begin
+        if is_direct_conn
+          kusto_java.ingest.IngestClientFactory.createClient(kusto_connection_string)
+        else
+          kusto_http_client_properties = kusto_java.data.HttpClientProperties.builder().proxy(apache_http.HttpHost.new(proxy_host,proxy_port,proxy_protocol)).build()
+          kusto_java.ingest.IngestClientFactory.createClient(kusto_connection_string, kusto_http_client_properties)
+        end
+      end
+
+      @ingestion_properties = kusto_java.ingest.IngestionProperties.new(database, table)
+      @ingestion_properties.setIngestionMapping(json_mapping, kusto_java.ingest.IngestionMapping::IngestionMappingKind::JSON)
+      @ingestion_properties.setDataFormat(kusto_java.ingest.IngestionProperties::DataFormat::JSON)
+      @delete_local = delete_local
+
+      @logger.debug('Kusto resources are ready.')
+    end
+
+    def validate_config(database, table, json_mapping, proxy_protocol, app_id, app_key, managed_identity_id)
+      # Add an additional validation and fail this upfront
+      if app_id.nil? && app_key.empty? && managed_identity_id.empty?
+        @logger.error('managed_identity_id is not provided and app_id/app_key is empty.')
+        raise LogStash::ConfigurationError.new('managed_identity_id is not provided and app_id/app_key is empty.')
+      end      
+      if database =~ FIELD_REF
+        @logger.error('database config value should not be dynamic.', database)
+        raise LogStash::ConfigurationError.new('database config value should not be dynamic.')
+      end
+
+      if table =~ FIELD_REF
+        @logger.error('table config value should not be dynamic.', table)
+        raise LogStash::ConfigurationError.new('table config value should not be dynamic.')
+      end
+
+      if json_mapping =~ FIELD_REF
+        @logger.error('json_mapping config value should not be dynamic.', json_mapping)
+        raise LogStash::ConfigurationError.new('json_mapping config value should not be dynamic.')
+      end
+
+      if not(["https", "http"].include? proxy_protocol)
+        @logger.error('proxy_protocol has to be http or https.', proxy_protocol)
+        raise LogStash::ConfigurationError.new('proxy_protocol has to be http or https.')
+      end
+
+    end
+
+    def upload_async(path, delete_on_success)
+      if @workers_pool.remaining_capacity <= LOW_QUEUE_LENGTH
+        @logger.warn("Ingestor queue capacity is running low with #{@workers_pool.remaining_capacity} free slots.")
+      end
+
+      @workers_pool.post do
+        LogStash::Util.set_thread_name("Kusto to ingest file: #{path}")
+        upload(path, delete_on_success)
+      end
+    rescue Exception => e
+      @logger.error('StandardError.', exception: e.class, message: e.message, path: path, backtrace: e.backtrace)
+      raise e
+    end
+
+    def upload(path, delete_on_success)
+      file_size = File.size(path)
+      @logger.debug("Sending file to kusto: #{path}. size: #{file_size}")
+
+      # TODO: dynamic routing
+      # file_metadata = path.partition('.kusto.').last
+      # file_metadata_parts = file_metadata.split('.')
+
+      # if file_metadata_parts.length == 3
+      #   # this is the number we expect - database, table, json_mapping
+      #   database = file_metadata_parts[0]
+      #   table = file_metadata_parts[1]
+      #   json_mapping = file_metadata_parts[2]
+
+      #   local_ingestion_properties = Java::KustoIngestionProperties.new(database, table)
+      #   local_ingestion_properties.addJsonMappingName(json_mapping)
+      # end
+
+      if file_size > 0
+        file_source_info = Java::com.microsoft.azure.kusto.ingest.source.FileSourceInfo.new(path, 0); # 0 - let the sdk figure out the size of the file
+        @kusto_client.ingestFromFile(file_source_info, @ingestion_properties)
+      else
+        @logger.warn("File #{path} is an empty file and is not ingested.")
+      end
+      File.delete(path) if delete_on_success
+      @logger.debug("File #{path} sent to kusto.")
+    rescue Errno::ENOENT => e
+      @logger.error("File doesn't exist! Unrecoverable error.", exception: e.class, message: e.message, path: path, backtrace: e.backtrace)
+    rescue Java::JavaNioFile::NoSuchFileException => e
+      @logger.error("File doesn't exist! Unrecoverable error.", exception: e.class, message: e.message, path: path, backtrace: e.backtrace)
+    rescue => e
+      # When the retry limit is reached or another error happen we will wait and retry.
+      #
+      # Thread might be stuck here, but I think its better than losing anything
+      # its either a transient errors or something bad really happened.
+      @logger.error('Uploading failed, retrying.', exception: e.class, message: e.message, path: path, backtrace: e.backtrace)
+      sleep RETRY_DELAY_SECONDS
+      retry
+    end
+
+    def stop
+      @workers_pool.shutdown
+      @workers_pool.wait_for_termination(nil) # block until its done
+    end
+  end
+end