logstash-output-kusto 0.1.6 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 89aa0d2c0229ca92f8d1dbcc5ba749ae8b6afbbfff8aab57ab862ddc5ea2d329
-  data.tar.gz: 7023796abee3ebae86b8987406917beeb5ccfa9c8e2d26d8d56c77091626c411
+  metadata.gz: 606d298b80da2887ad2223db406fd23e38351b46ef7ec87861468828f1b3ec24
+  data.tar.gz: b7f420f9f84c886460c145874625400198acbcde1aa05cafac9ac4a03e7da008
 SHA512:
-  metadata.gz: 4bfacbb9d6965366a2f1ce4e0bf031f3d100225da8d7da76d431a28cb82ff40c9e33ffce9a54a69bc45bc89bae86951f03ed0fb3eed7021df5534e0c234f52b2
-  data.tar.gz: e670720f4b5c39a5ac574ca760141009544475c74b3fa5cd7faf8625db9d951592e4673f9a8c6c973fbe5b41a2a47f3cae81fac99b5799d2b42eb6c9900ef27d
+  metadata.gz: a4e96bcb287915d805ef9fddc71f63399f4d9299888ac4da8f06d5baf6d7594825ec503b657ddae2812daa581cdc7465c2882575a236f0b9e28cd339dae2702a
+  data.tar.gz: 17041bdad667f37658a9246bc2068f5dcf53ec3cd41c6a665e9b100727194e7cf81baaa6d2927b942c1592cb6b3f15b7382f43e17669ccdbd039a1ad2c85b132

data/CHANGELOG.md

@@ -1,4 +1,8 @@
 ## 0.1.0
   - Plugin created with the logstash plugin generator
 ## 0.1.6
-  - plugin published to the public. supports json events without dynamic routing meaning one output defined per target kusto table
+  - plugin published to the public. supports ingestion json events into a specific table-database (without dynamic routing currently)
+## 0.1.7
+  - fixed app_key (password) bug, include 0.1.7 of the kusto-java-sdk to allow working through a proxy
+## 0.2.0
+  - move to version 1.0.0-BETA-01 of azure-kusto-kava sdk

data/README.md

@@ -1,86 +1,62 @@
-# Logstash Plugin
+# Logstash Output Plugin for Azure Data Explorer (Kusto)
 
-This is a plugin for [Logstash](https://github.com/elastic/logstash).
-
-It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
-
-## Documentation
-
-Logstash provides infrastructure to automatically generate documentation for this plugin. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html. All plugin documentation are placed under one [central location](http://www.elastic.co/guide/en/logstash/current/).
-
-- For formatting code or config example, you can use the asciidoc `[source,ruby]` directive
-- For more asciidoc formatting tips, see the excellent reference here https://github.com/elastic/docs#asciidoc-guide
-
-## Need Help?
-
-Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/logstash discussion forum.
-
-## Developing
+master: [![Build Status](https://travis-ci.org/Azure/logstash-output-kusto.svg)](https://travis-ci.org/Azure/logstash-output-kusto) 
+dev: [![Build Status](https://travis-ci.org/Azure/logstash-output-kusto.svg?branch=dev)](https://travis-ci.org/Azure/logstash-output-kusto)
 
-### 1. Plugin Developement and Testing
-
-#### Code
-- To get started, you'll need JRuby with the Bundler gem installed.
+This is a plugin for [Logstash](https://github.com/elastic/logstash).
 
-- Create a new plugin or clone and existing from the GitHub [logstash-plugins](https://github.com/logstash-plugins) organization. We also provide [example plugins](https://github.com/logstash-plugins?query=example).
+It is fully free and open source. The license is Apache 2.0.
 
-- Install dependencies
-```sh
-bundle install
-```
+This Azure Data Explorer (ADX) Logstash plugin enables you to process events from Logstash into an **Azure Data Explorer** database for later analysis. 
 
-#### Test
+## Requirements
 
-- Update your dependencies
-
-```sh
-bundle install
-```
+- Logstash version 6+. [Installation instructions](https://www.elastic.co/guide/en/logstash/current/installing-logstash.html) 
+- Azure Data Explorer cluster with a database. Read [Create a cluster and database](https://docs.microsoft.com/en-us/azure/data-explorer/create-cluster-database-portal) for more information.
+- AAD Application credentials with permission to ingest data into Azure Data Explorer. Read [Creating an AAD Application](https://docs.microsoft.com/en-us/azure/kusto/management/access-control/how-to-provision-aad-app) for more information.
 
-- Run tests
+## Installation
 
+To make the Azure Data Explorer plugin available in your Logstash environment, run the following command:
 ```sh
-bundle exec rspec
+bin/logstash-plugin install logstash-output-kusto
 ```
 
-### 2. Running your unpublished Plugin in Logstash
+## Configuration
 
-#### 2.1 Run in a local Logstash clone
+Perform configuration before sending events from Logstash to Azure Data Explorer. The following example shows the minimum you need to provide. It should be enough for most use-cases:
 
-- Edit Logstash `Gemfile` and add the local plugin path, for example:
 ```ruby
-gem "logstash-filter-awesome", :path => "/your/local/logstash-filter-awesome"
-```
-- Install plugin
-```sh
-bin/logstash-plugin install --no-verify
-```
-- Run Logstash with your plugin
-```sh
-bin/logstash -e 'filter {awesome {}}'
-```
-At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.
-
-#### 2.2 Run in an installed Logstash
-
-You can use the same **2.1** method to run your plugin in an installed Logstash by editing its `Gemfile` and pointing the `:path` to your local plugin development directory or you can build the gem and install it using:
-
-- Build your plugin gem
-```sh
-gem build logstash-filter-awesome.gemspec
-```
-- Install the plugin from the Logstash home
-```sh
-bin/logstash-plugin install /your/local/plugin/logstash-filter-awesome.gem
-```
-- Start Logstash and proceed to test the plugin
+output {
+	kusto {
+            path => "/tmp/kusto/%{+YYYY-MM-dd-HH-mm}.txt"
+            ingest_url => "https://ingest-<cluster-name>.kusto.windows.net/"
+            app_id => "<application id>"
+            app_key => "<application key/secret>"
+            app_tenant => "<tenant id>"
+            database => "<database name>"
+            table => "<target table>"
+            mapping => "<mapping name>"
+	}
+}
+```
+
+### Available Configuration Keys
+
+| Parameter Name | Description | Notes |
+| --- | --- | --- |
+| **path** | The plugin writes events to temporary files before sending them to ADX. This parameter includes a path where files should be written and a time expression for file rotation to trigger an upload to the ADX service. The example above shows how to rotate the files every minute and check the Logstash docs for more information on time expressions. | Required
+| **ingest_url** | The Kusto endpoint for ingestion-related communication. See it on the Azure Portal.| Required|
+| **app_id, app_key, app_tenant**| Credentials required to connect to the ADX service. Be sure to use an application with 'ingest' priviledges. | Required|
+| **database**| Database name to place events | Required |
+| **table** | Target table name to place events | Required
+| **mapping** | Mapping is used to map an incoming event json string into the correct row format (which property goes into which column) | Required |
+| **recovery** | If set to true (default), plugin will attempt to resend pre-existing temp files found in the path upon startup | |
+| **delete_temp_files** | Determines if temp files will be deleted after a successful upload (true is default; set false for debug purposes only)| |
+| **flush_interval** | The time (in seconds) for flushing writes to temporary files. Default is 2 seconds, 0 will flush on every event. Increase this value to reduce IO calls but keep in mind that events in the buffer will be lost in case of abrupt failure.| |
 
 ## Contributing
 
-All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.
-
-Programming is not a required skill. Whatever you've seen about open source and maintainers or community members  saying "send patches or die" - you will not see that here.
-
-It is more important to the community that you are able to contribute.
-
+All contributions are welcome: ideas, patches, documentation, bug reports, and complaints.
+Programming is not a required skill. It is more important to the community that you are able to contribute.
 For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file.

data/lib/logstash/outputs/kusto.rb

@@ -8,7 +8,7 @@ require 'logstash/outputs/kusto/ingestor'
 require 'logstash/outputs/kusto/interval'
 
 ##
-# This plugin send messages to Azure Kusto in batches.
+# This plugin sends messages to Azure Kusto in batches.
 #
 class LogStash::Outputs::Kusto < LogStash::Outputs::Base
   config_name 'kusto'
@@ -30,8 +30,9 @@ class LogStash::Outputs::Kusto < LogStash::Outputs::Base
   # E.g: `/%{myfield}/`, `/test-%{myfield}/` are not valid paths
   config :path, validate: :string, required: true
 
-  # Flush interval (in seconds) for flushing writes to log files.
-  # 0 will flush on every message.
+  # Flush interval (in seconds) for flushing writes to files.
+  # 0 will flush on every message. Increase this value to recude IO calls but keep 
+  # in mind that events buffered before flush can be lost in case of abrupt failure.
   config :flush_interval, validate: :number, default: 2
 
   # If the generated path is invalid, the events will be saved
@@ -54,15 +55,6 @@ class LogStash::Outputs::Kusto < LogStash::Outputs::Base
   # Example: `"file_mode" => 0640`
   config :file_mode, validate: :number, default: -1
 
-  # How should the file be written?
-  #
-  # If `append`, the file will be opened for appending and each new event will
-  # be written at the end of the file.
-  #
-  # If `overwrite`, the file will be truncated before writing and only the most
-  # recent event will appear in the file.
-  config :write_behavior, validate: %w[overwrite append], default: 'append'
-
   # TODO: fix the interval type...
   config :stale_cleanup_interval, validate: :number, default: 10
   config :stale_cleanup_type, validate: %w[events interval], default: 'events'
@@ -75,16 +67,27 @@ class LogStash::Outputs::Kusto < LogStash::Outputs::Base
   # If `false`, the plugin will disregard temp files found
   config :recovery, validate: :boolean, default: true
 
-  # Kusto configuration
+  
+  # The Kusto endpoint for ingestion related communication. You can see it on the Azure Portal.
   config :ingest_url, validate: :string, required: true
+
+  # The following are the credentails used to connect to the Kusto service
+  # application id 
   config :app_id, validate: :string, required: true
-  config :app_key, validate: :string, required: true
+  # application key (secret)
+  config :app_key, validate: :password, required: true
+  # aad tenant id
   config :app_tenant, validate: :string, default: nil
 
+  # The following are the data settings that impact where events are written to
+  # Database name
   config :database, validate: :string, required: true
+  # Target table name
   config :table, validate: :string, required: true
+  # Mapping name - used by kusto to map an incoming event to the right row format (what value goes into which column)
   config :mapping, validate: :string
 
+
   # Determines if local files used for temporary storage will be deleted
   # after upload is successful
   config :delete_temp_files, validate: :boolean, default: true
@@ -149,9 +152,14 @@ class LogStash::Outputs::Kusto < LogStash::Outputs::Base
   private
   def validate_path
     if (root_directory =~ FIELD_REF) != nil
-      @logger.error('File: The starting part of the path should not be dynamic.', path: @path)
+      @logger.error('The starting part of the path should not be dynamic.', path: @path)
       raise LogStash::ConfigurationError.new('The starting part of the path should not be dynamic.')
     end
+
+    if !path_with_field_ref?
+      @logger.error('Path should include some time related fields to allow for file rotation.', path: @path)
+      raise LogStash::ConfigurationError.new('Path should include some time related fields to allow for file rotation.')
+    end
   end
 
   private 
@@ -177,14 +185,8 @@ class LogStash::Outputs::Kusto < LogStash::Outputs::Base
     @io_mutex.synchronize do
       encoded_by_path.each do |path, chunks|
         fd = open(path)
-        if @write_behavior == 'overwrite'
-          fd.truncate(0)
-          fd.seek(0, IO::SEEK_SET)
-          fd.write(chunks.last)
-        else
-          # append to the file
-          chunks.each { |chunk| fd.write(chunk) }
-        end
+        # append to the file
+        chunks.each { |chunk| fd.write(chunk) }
         fd.flush unless @flusher && @flusher.alive?
       end
 
@@ -210,7 +212,7 @@ class LogStash::Outputs::Kusto < LogStash::Outputs::Base
       end
     end
 
-    @ingestor.stop
+    @ingestor.stop unless @ingestor.nil?
   end
 
   private

data/lib/logstash/outputs/kusto/ingestor.rb

@@ -1,94 +1,116 @@
-# encoding: utf-8
-
-require 'logstash/outputs/base' 
-require 'logstash/namespace'
-require 'logstash/errors'
-
-class LogStash::Outputs::Kusto < LogStash::Outputs::Base
-  ##
-  # This handles the overall logic and communication with Kusto
-  #
-  class Ingestor
-    require 'kusto/KustoClient-0.1.6.jar'
-
-    RETRY_DELAY_SECONDS = 3
-    DEFAULT_THREADPOOL = Concurrent::ThreadPoolExecutor.new(
-      min_threads: 1,
-      max_threads: 8,
-      max_queue: 1,
-      fallback_policy: :caller_runs
-    )
-    LOW_QUEUE_LENGTH = 3
-
-    def initialize(ingest_url, app_id, app_key, app_tenant, database, table, mapping, delete_local, logger, threadpool = DEFAULT_THREADPOOL)
-      @workers_pool = threadpool
-      @logger = logger
-
-      @logger.debug('Preparing Kusto resources.')
-      kusto_connection_string = Java::KustoConnectionStringBuilder.createWithAadApplicationCredentials(ingest_url, app_id, app_key, app_tenant)
-
-      @kusto_client = Java::KustoIngestClient.new(kusto_connection_string)
-
-      @ingestion_properties = Java::KustoIngestionProperties.new(database, table)
-      @ingestion_properties.setJsonMappingName(mapping)
-
-      @delete_local = delete_local
-
-      @logger.debug('Kusto resources are ready.')
-    end
-
-    def upload_async(path, delete_on_success)
-      if @workers_pool.remaining_capacity <= LOW_QUEUE_LENGTH
-        @logger.warn("Ingestor queue capacity is running low with #{@workers_pool.remaining_capacity} free slots.")
-      end
-
-      @workers_pool.post do
-        LogStash::Util.set_thread_name("Kusto to ingest file: #{path}")
-        upload(path, delete_on_success)
-      end
-    rescue Exception => e
-      @logger.error('StandardError.', exception: e.class, message: e.message, path: path, backtrace: e.backtrace)
-      raise e
-    end
-
-    def upload(path, delete_on_success)
-      file_size = File.size(path)
-      @logger.debug("Sending file to kusto: #{path}. size: #{file_size}")
-
-      # TODO: dynamic routing
-      # file_metadata = path.partition('.kusto.').last
-      # file_metadata_parts = file_metadata.split('.')
-
-      # if file_metadata_parts.length == 3
-      #   # this is the number we expect - database, table, mapping
-      #   database = file_metadata_parts[0]
-      #   table = file_metadata_parts[1]
-      #   mapping = file_metadata_parts[2]
-
-      #   local_ingestion_properties = Java::KustoIngestionProperties.new(database, table)
-      #   local_ingestion_properties.addJsonMappingName(mapping)
-      # end
-
-      @kusto_client.ingestFromSingleFile(path, @ingestion_properties)
-
-      File.delete(path) if delete_on_success
-
-      @logger.debug("File #{path} sent to kusto.")
-    rescue Java::JavaNioFile::NoSuchFileException => e
-      @logger.error("File doesn't exist! Unrecoverable error.", exception: e.class, message: e.message, path: path, backtrace: e.backtrace)
-    rescue => e
-      # When the retry limit is reached or another error happen we will wait and retry.
-      #
-      # Thread might be stuck here, but I think its better than losing anything
-      # its either a transient errors or something bad really happened.
-      @logger.error('Uploading failed, retrying.', exception: e.class, message: e.message, path: path, backtrace: e.backtrace)
-      sleep RETRY_DELAY_SECONDS
-      retry
-    end
-
-    def stop
-      @workers_pool.shutdown
-      @workers_pool.wait_for_termination(nil) # block until its done
-    end
-  end
-end
+# encoding: utf-8
+
+require 'logstash/outputs/base' 
+require 'logstash/namespace'
+require 'logstash/errors'
+
+class LogStash::Outputs::Kusto < LogStash::Outputs::Base
+  ##
+  # This handles the overall logic and communication with Kusto
+  #
+  class Ingestor
+    require 'kusto/kusto-ingest-1.0.0-BETA-01-jar-with-dependencies.jar'
+
+    RETRY_DELAY_SECONDS = 3
+    DEFAULT_THREADPOOL = Concurrent::ThreadPoolExecutor.new(
+      min_threads: 1,
+      max_threads: 8,
+      max_queue: 1,
+      fallback_policy: :caller_runs
+    )
+    LOW_QUEUE_LENGTH = 3
+    FIELD_REF = /%\{[^}]+\}/
+
+    def initialize(ingest_url, app_id, app_key, app_tenant, database, table, mapping, delete_local, logger, threadpool = DEFAULT_THREADPOOL)
+      @workers_pool = threadpool
+      @logger = logger
+
+      validate_config(database, table, mapping)
+
+      @logger.debug('Preparing Kusto resources.')
+
+      kusto_connection_string = Java::com.microsoft.azure.kusto.data.ConnectionStringBuilder.createWithAadApplicationCredentials(ingest_url, app_id, app_key.value, app_tenant)
+
+      @kusto_client = Java::com.microsoft.azure.kusto.ingest.IngestClientFactory.createClient(kusto_connection_string)
+
+      @ingestion_properties = Java::com.microsoft.azure.kusto.ingest.IngestionProperties.new(database, table)
+      @ingestion_properties.setJsonMappingName(mapping)
+
+      @delete_local = delete_local
+
+      @logger.debug('Kusto resources are ready.')
+    end
+
+    def validate_config(database, table, mapping)
+      if database =~ FIELD_REF
+        @logger.error('database config value should not be dynamic.', database)
+        raise LogStash::ConfigurationError.new('database config value should not be dynamic.')
+      end
+
+      if table =~ FIELD_REF
+        @logger.error('table config value should not be dynamic.', table)
+        raise LogStash::ConfigurationError.new('table config value should not be dynamic.')
+      end
+
+      if mapping =~ FIELD_REF
+        @logger.error('mapping config value should not be dynamic.', mapping)
+        raise LogStash::ConfigurationError.new('mapping config value should not be dynamic.')
+      end
+    end
+
+    def upload_async(path, delete_on_success)
+      if @workers_pool.remaining_capacity <= LOW_QUEUE_LENGTH
+        @logger.warn("Ingestor queue capacity is running low with #{@workers_pool.remaining_capacity} free slots.")
+      end
+
+      @workers_pool.post do
+        LogStash::Util.set_thread_name("Kusto to ingest file: #{path}")
+        upload(path, delete_on_success)
+      end
+    rescue Exception => e
+      @logger.error('StandardError.', exception: e.class, message: e.message, path: path, backtrace: e.backtrace)
+      raise e
+    end
+
+    def upload(path, delete_on_success)
+      file_size = File.size(path)
+      @logger.debug("Sending file to kusto: #{path}. size: #{file_size}")
+
+      # TODO: dynamic routing
+      # file_metadata = path.partition('.kusto.').last
+      # file_metadata_parts = file_metadata.split('.')
+
+      # if file_metadata_parts.length == 3
+      #   # this is the number we expect - database, table, mapping
+      #   database = file_metadata_parts[0]
+      #   table = file_metadata_parts[1]
+      #   mapping = file_metadata_parts[2]
+
+      #   local_ingestion_properties = Java::KustoIngestionProperties.new(database, table)
+      #   local_ingestion_properties.addJsonMappingName(mapping)
+      # end
+
+      file_source_info = Java::com.microsoft.azure.kusto.ingest.source.FileSourceInfo.new(path, 0); # 0 - let the sdk figure out the size of the file
+      @kusto_client.ingestFromFile(file_source_info, @ingestion_properties)
+
+      File.delete(path) if delete_on_success
+
+      @logger.debug("File #{path} sent to kusto.")
+    rescue Java::JavaNioFile::NoSuchFileException => e
+      @logger.error("File doesn't exist! Unrecoverable error.", exception: e.class, message: e.message, path: path, backtrace: e.backtrace)
+    rescue => e
+      # When the retry limit is reached or another error happen we will wait and retry.
+      #
+      # Thread might be stuck here, but I think its better than losing anything
+      # its either a transient errors or something bad really happened.
+      @logger.error('Uploading failed, retrying.', exception: e.class, message: e.message, path: path, backtrace: e.backtrace)
+      sleep RETRY_DELAY_SECONDS
+      retry
+    end
+
+    def stop
+      @workers_pool.shutdown
+      @workers_pool.wait_for_termination(nil) # block until its done
+    end
+  end
+end

data/lib/logstash/outputs/kusto/interval.rb

@@ -1,81 +1,81 @@
-# encoding: utf-8
-
-require 'logstash/outputs/base'
-require 'logstash/namespace'
-require 'logstash/errors'
-
-class LogStash::Outputs::Kusto < LogStash::Outputs::Base
-  ##
-  # Bare-bones utility for running a block of code at an interval.
-  #
-  class Interval
-    ##
-    # Initializes a new Interval with the given arguments and starts it
-    # before returning it.
-    #
-    # @param interval [Integer] (see: Interval#initialize)
-    # @param procsy [#call] (see: Interval#initialize)
-    #
-    # @return [Interval]
-    #
-    def self.start(interval, procsy)
-      new(interval, procsy).tap(&:start)
-    end
-
-    ##
-    # @param interval [Integer]: time in seconds to wait between calling the given proc
-    # @param procsy [#call]: proc or lambda to call periodically; must not raise exceptions.
-    def initialize(interval, procsy)
-      @interval = interval
-      @procsy = procsy
-
-      # Mutex, ConditionVariable, etc.
-      @mutex = Mutex.new
-      @sleeper = ConditionVariable.new
-    end
-
-    ##
-    # Starts the interval, or returns if it has already been started.
-    #
-    # @return [void]
-    def start
-      @mutex.synchronize do
-        return if @thread && @thread.alive?
-
-        @thread = Thread.new { run }
-      end
-    end
-
-    ##
-    # Stop the interval.
-    # Does not interrupt if execution is in-progress.
-    def stop
-      @mutex.synchronize do
-        @stopped = true
-      end
-
-      @thread && @thread.join
-    end
-
-    ##
-    # @return [Boolean]
-    def alive?
-      @thread && @thread.alive?
-    end
-
-    private
-
-    def run
-      @mutex.synchronize do
-        loop do
-          @sleeper.wait(@mutex, @interval)
-          break if @stopped
-
-          @procsy.call
-        end
-      end
-    ensure
-      @sleeper.broadcast
-    end
-  end
-end
+# encoding: utf-8
+
+require 'logstash/outputs/base'
+require 'logstash/namespace'
+require 'logstash/errors'
+
+class LogStash::Outputs::Kusto < LogStash::Outputs::Base
+  ##
+  # Bare-bones utility for running a block of code at an interval.
+  #
+  class Interval
+    ##
+    # Initializes a new Interval with the given arguments and starts it
+    # before returning it.
+    #
+    # @param interval [Integer] (see: Interval#initialize)
+    # @param procsy [#call] (see: Interval#initialize)
+    #
+    # @return [Interval]
+    #
+    def self.start(interval, procsy)
+      new(interval, procsy).tap(&:start)
+    end
+
+    ##
+    # @param interval [Integer]: time in seconds to wait between calling the given proc
+    # @param procsy [#call]: proc or lambda to call periodically; must not raise exceptions.
+    def initialize(interval, procsy)
+      @interval = interval
+      @procsy = procsy
+
+      # Mutex, ConditionVariable, etc.
+      @mutex = Mutex.new
+      @sleeper = ConditionVariable.new
+    end
+
+    ##
+    # Starts the interval, or returns if it has already been started.
+    #
+    # @return [void]
+    def start
+      @mutex.synchronize do
+        return if @thread && @thread.alive?
+
+        @thread = Thread.new { run }
+      end
+    end
+
+    ##
+    # Stop the interval.
+    # Does not interrupt if execution is in-progress.
+    def stop
+      @mutex.synchronize do
+        @stopped = true
+      end
+
+      @thread && @thread.join
+    end
+
+    ##
+    # @return [Boolean]
+    def alive?
+      @thread && @thread.alive?
+    end
+
+    private
+
+    def run
+      @mutex.synchronize do
+        loop do
+          @sleeper.wait(@mutex, @interval)
+          break if @stopped
+
+          @procsy.call
+        end
+      end
+    ensure
+      @sleeper.broadcast
+    end
+  end
+end

data/logstash-output-kusto.gemspec

@@ -1,17 +1,17 @@
 Gem::Specification.new do |s|
   s.name          = 'logstash-output-kusto'
-  s.version       = '0.1.6'
+  s.version       = '0.2.0'
   s.licenses      = ['Apache-2.0']
-  s.summary       = 'Writes events to Azure KustoDB'
-  s.description   = 'This is a logstash output plugin used to write events to an Azure KustoDB instance'
-  s.homepage      = 'https://github.com/Azure/azure-diagnostics-tools/tree/master/Logstash'
+  s.summary       = 'Writes events to Azure Data Explorer (Kusto)'
+  s.description   = 'This is a logstash output plugin used to write events to an Azure Data Explorer (a.k.a Kusto)'
+  s.homepage      = 'https://github.com/Azure/logstash-output-kusto'
   s.authors       = ['Tamir Kamara']
-  s.email         = 'tamir.kamara@microsoft.com'
+  s.email         = 'nugetkusto@microsoft.com'
   s.require_paths = ['lib']
 
   # Files
-  s.files = Dir['lib/**/*','spec/**/*','vendor/**/*','*.gemspec','*.md','CONTRIBUTORS','Gemfile','LICENSE','NOTICE.TXT']
-   
+  s.files = Dir['lib/**/*', 'spec/**/*', 'vendor/**/*', '*.gemspec', '*.md', 'CONTRIBUTORS', 'Gemfile', 'LICENSE', 'NOTICE.TXT']
+
   # Tests
   s.test_files = s.files.grep(%r{^(test|spec|features)/})
 
@@ -19,7 +19,7 @@ Gem::Specification.new do |s|
   s.metadata = { "logstash_plugin" => "true", "logstash_group" => "output" }
 
   # Gem dependencies
-  s.add_runtime_dependency "logstash-core-plugin-api", "~> 2.0"
+  s.add_runtime_dependency 'logstash-core-plugin-api', '~> 2.0'
   s.add_runtime_dependency 'logstash-codec-json_lines'
   s.add_runtime_dependency 'logstash-codec-line'
 

data/spec/outputs/kusto/ingestor_spec.rb

@@ -0,0 +1,109 @@
+# encoding: utf-8
+require_relative "../../spec_helpers.rb"
+require 'logstash/outputs/kusto'
+require 'logstash/outputs/kusto/ingestor'
+
+describe LogStash::Outputs::Kusto::Ingestor do
+
+  let(:ingest_url) { "mycluster" }
+  let(:app_id) { "myid" }
+  let(:app_key) { LogStash::Util::Password.new("mykey") }
+  let(:app_tenant) { "mytenant" }
+  let(:database) { "mydatabase" }
+  let(:table) { "mytable" }
+  let(:mapping) { "mymapping" }
+  let(:delete_local) { false }
+  let(:logger) { spy('logger') }
+
+  describe '#initialize' do
+
+    it 'does not throw an error when initializing' do
+      # note that this will cause an internal error since connection is being tried.
+      # however we still want to test that all the java stuff is working as expected
+      expect { 
+        ingestor = described_class.new(ingest_url, app_id, app_key, app_tenant, database, table, mapping, delete_local, logger)
+        ingestor.stop
+      }.not_to raise_error
+    end
+    
+    dynamic_name_array = ['/a%{name}/', '/a %{name}/', '/a- %{name}/', '/a- %{name}']
+
+    context 'doesnt allow database to have some dynamic part' do
+      dynamic_name_array.each do |test_database|
+        it "with database: #{test_database}" do
+          expect {
+            ingestor = described_class.new(ingest_url, app_id, app_key, app_tenant, test_database, table, mapping, delete_local, logger)            
+            ingestor.stop
+          }.to raise_error(LogStash::ConfigurationError)          
+        end
+      end
+    end
+
+    context 'doesnt allow table to have some dynamic part' do
+      dynamic_name_array.each do |test_table|
+        it "with database: #{test_table}" do
+          expect {
+            ingestor = described_class.new(ingest_url, app_id, app_key, app_tenant, database, test_table, mapping, delete_local, logger)
+            ingestor.stop
+          }.to raise_error(LogStash::ConfigurationError)          
+        end
+      end
+    end
+
+    context 'doesnt allow mapping to have some dynamic part' do
+      dynamic_name_array.each do |test_mapping|
+        it "with database: #{test_mapping}" do
+          expect {
+            ingestor = described_class.new(ingest_url, app_id, app_key, app_tenant, database, table, test_mapping, delete_local, logger)
+            ingestor.stop
+          }.to raise_error(LogStash::ConfigurationError)          
+        end
+      end
+    end
+
+  end
+
+  # describe 'receiving events' do
+
+  #   context 'with non-zero flush interval' do
+  #     let(:temporary_output_file) { Stud::Temporary.pathname }
+
+  #     let(:event_count) { 100 }
+  #     let(:flush_interval) { 5 }
+
+  #     let(:events) do
+  #       event_count.times.map do |idx|
+  #         LogStash::Event.new('subject' => idx)
+  #       end
+  #     end
+
+  #     let(:output) { described_class.new(options.merge( {'path' => temporary_output_file, 'flush_interval' => flush_interval, 'delete_temp_files' => false } )) }
+
+  #     before(:each) { output.register }
+      
+  #     after(:each) do
+  #       output.close
+  #       File.exist?(temporary_output_file) && File.unlink(temporary_output_file)
+  #       File.exist?(temporary_output_file + '.kusto') && File.unlink(temporary_output_file + '.kusto')
+  #     end
+
+  #     it 'eventually flushes without receiving additional events' do
+  #       output.multi_receive_encoded(events)
+
+  #       # events should not all be flushed just yet...
+  #       expect(File.read(temporary_output_file)).to satisfy("have less than #{event_count} lines") do |contents|
+  #         contents && contents.lines.count < event_count
+  #       end
+
+  #       # wait for the flusher to run...
+  #       sleep(flush_interval + 1)
+
+  #       # events should all be flushed
+  #       expect(File.read(temporary_output_file)).to satisfy("have exactly #{event_count} lines") do |contents|
+  #         contents && contents.lines.count == event_count
+  #       end
+  #     end
+  #   end
+
+  # end
+end

data/spec/outputs/kusto_spec.rb

@@ -1,22 +1,54 @@
 # encoding: utf-8
-require "logstash/devutils/rspec/spec_helper"
-require "logstash/outputs/kusto"
-require "logstash/codecs/plain"
-require "logstash/event"
+require 'logstash/devutils/rspec/spec_helper'
+require 'logstash/outputs/kusto'
+require 'logstash/codecs/plain'
+require 'logstash/event'
 
 describe LogStash::Outputs::Kusto do
-  let(:sample_event) { LogStash::Event.new }
-  let(:output) { LogStash::Outputs::Kusto.new }
 
-  before do
-    output.register
-  end
+  let(:options) { { "path" => "./kusto_tst/%{+YYYY-MM-dd-HH-mm}",
+    "ingest_url" => "mycluster",
+    "app_id" => "myid",
+    "app_key" => "mykey",
+    "app_tenant" => "mytenant", 
+    "database" => "mydatabase",
+    "table" => "mytable",
+    "mapping" => "mymapping"
+  } }
+
+  describe '#register' do
+
+    it 'doesnt allow the path to start with a dynamic string' do
+      kusto = described_class.new(options.merge( {'path' => '/%{name}'} ))
+      expect { kusto.register }.to raise_error(LogStash::ConfigurationError)
+      kusto.close
+    end
+
+    it 'path must include a dynamic string to allow file rotation' do
+      kusto = described_class.new(options.merge( {'path' => '/{name}'} ))
+      expect { kusto.register }.to raise_error(LogStash::ConfigurationError)
+      kusto.close
+    end
+
 
-  describe "receive message" do
-    subject { output.receive(sample_event) }
+    dynamic_name_array = ['/a%{name}/', '/a %{name}/', '/a- %{name}/', '/a- %{name}']
 
-    it "returns a string" do
-      expect(subject).to eq("Event received")
+    context 'doesnt allow the root directory to have some dynamic part' do
+      dynamic_name_array.each do |test_path|
+         it "with path: #{test_path}" do
+           kusto = described_class.new(options.merge( {'path' => test_path} ))
+           expect { kusto.register }.to raise_error(LogStash::ConfigurationError)
+           kusto.close
+         end
+       end
     end
+
+    it 'allow to have dynamic part after the file root' do
+      kusto = described_class.new(options.merge({'path' => '/tmp/%{name}'}))
+      expect { kusto.register }.not_to raise_error
+      kusto.close
+    end
+
   end
+  
 end

data/spec/spec_helpers.rb

@@ -0,0 +1,5 @@
+# encoding: utf-8
+require "logstash/devutils/rspec/spec_helper"
+require "logstash/logging/logger"
+
+LogStash::Logging::Logger::configure_logging("debug")

metadata

@@ -1,116 +1,116 @@
 --- !ruby/object:Gem::Specification
 name: logstash-output-kusto
 version: !ruby/object:Gem::Version
-  version: 0.1.6
+  version: 0.2.0
 platform: ruby
 authors:
 - Tamir Kamara
-autorequire:
+autorequire: 
 bindir: bin
 cert_chain: []
-date: 2018-08-09 00:00:00.000000000 Z
+date: 2019-01-16 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
+  name: logstash-core-plugin-api
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '2.0'
-  name: logstash-core-plugin-api
-  prerelease: false
   type: :runtime
+  prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - "~>"
       - !ruby/object:Gem::Version
         version: '2.0'
 - !ruby/object:Gem::Dependency
+  name: logstash-codec-json_lines
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
-  name: logstash-codec-json_lines
-  prerelease: false
   type: :runtime
+  prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
+  name: logstash-codec-line
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
-  name: logstash-codec-line
-  prerelease: false
   type: :runtime
+  prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
+  name: logstash-devutils
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
-  name: logstash-devutils
-  prerelease: false
   type: :development
+  prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
+  name: flores
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
-  name: flores
-  prerelease: false
   type: :development
+  prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
+  name: logstash-input-generator
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
-  name: logstash-input-generator
-  prerelease: false
   type: :development
+  prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
+  name: jar-dependencies
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
-  name: jar-dependencies
-  prerelease: false
   type: :runtime
+  prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
-description: This is a logstash output plugin used to write events to an Azure KustoDB
-  instance
-email: tamir.kamara@microsoft.com
+description: This is a logstash output plugin used to write events to an Azure Data
+  Explorer (a.k.a Kusto)
+email: nugetkusto@microsoft.com
 executables: []
 extensions: []
 extra_rdoc_files: []
@@ -120,19 +120,21 @@ files:
 - Gemfile
 - LICENSE
 - README.md
-- lib/kusto/KustoClient-0.1.6.jar
+- lib/kusto/kusto-ingest-1.0.0-BETA-01-jar-with-dependencies.jar
 - lib/logstash/outputs/kusto.rb
 - lib/logstash/outputs/kusto/ingestor.rb
 - lib/logstash/outputs/kusto/interval.rb
 - logstash-output-kusto.gemspec
+- spec/outputs/kusto/ingestor_spec.rb
 - spec/outputs/kusto_spec.rb
-homepage: https://github.com/Azure/azure-diagnostics-tools/tree/master/Logstash
+- spec/spec_helpers.rb
+homepage: https://github.com/Azure/logstash-output-kusto
 licenses:
 - Apache-2.0
 metadata:
   logstash_plugin: 'true'
   logstash_group: output
-post_install_message:
+post_install_message: 
 rdoc_options: []
 require_paths:
 - lib
@@ -147,10 +149,11 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubyforge_project:
-rubygems_version: 2.7.6
-signing_key:
+rubygems_version: 3.0.2
+signing_key: 
 specification_version: 4
-summary: Writes events to Azure KustoDB
+summary: Writes events to Azure Data Explorer (Kusto)
 test_files:
+- spec/outputs/kusto/ingestor_spec.rb
 - spec/outputs/kusto_spec.rb
+- spec/spec_helpers.rb