microsoft-sentinel-log-analytics-logstash-output-plugin 1.1.1 → 1.1.3

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: bc9cee055d5aa8f90acde6ed20eacccf0d9e9a36fa9a5ebd1c916d85a410e628
-  data.tar.gz: '081cd8f53b716eeed931c417ce6fc7a13e3d2d1fe9acf0050858e5b87918d4ff'
+  metadata.gz: bf8c57d14129f064f4d2256d5578d7aacd8ab97e3b263748bc703da757cfeb58
+  data.tar.gz: bcbf850de17a394e10702c613354b1638d963df818fbbb8e680ac54642571297
 SHA512:
-  metadata.gz: 77753b09f6c4631fb2e2eb1e0ba5dd4eb4c33840901226f1caa08043b39caa77497cdfd42adb7ac6a857b22e3bd98512fff8e3e1ba4d6b7916eb16e822f752ce
-  data.tar.gz: 9ae33cb6bb9c96c19011b4a21f9ac943f75df18b99761023cb4ba00c84584a6ad0c05d8b0d5c0819993b434e2f6d14da1e62f92ba2f44a11aa6eeed22cbe0b9c
+  metadata.gz: edf7141d94c4da2a3518197e74cf976c60b664c1f8118ba817aef9bf26d4c7b04b0812dc21be0549ebbc8246fa2f3bb6f828281e7db11ed661556720538e792b
+  data.tar.gz: b256cb4d7d78684c3626dcfe6723094f61ea522b008f38c802278fbf0cb19dea18d3061842108f832f12af25757936f4f450726b2fdb9abf8aa81c58b1d11cce

data/CHANGELOG.md

@@ -1,14 +1,13 @@
-## 1.0.0
-* Initial release for output plugin for logstash to Microsoft Sentinel. This is done with the Log Analytics DCR based API.
-
-## 1.1.0 
-* Increase timeout for read/open connections to 120 seconds.
-* Add error handling for when connection timeout occurs.
-* Upgrade the rest-client dependency minimum version to 2.1.0.
-* Allow setting different proxy values for api connections.
-* Upgrade version for ingestion api to 2023-01-01.
-* Rename the plugin to microsoft-sentinel-log-analytics-logstash-output-plugin.
-
+## 1.1.3
+-  Replaces the `rest-client` library used for connecting to Azure with the `excon` library.
+ 
 ## 1.1.1
-* Support China and US Government Azure sovereign clouds.
-* Increase timeout for read/open connections to 240 seconds.
+- Adds support for Azure US Government cloud and Microsoft Azure operated by 21Vianet in China.
+ 
+## 1.1.0 
+- Allows setting different proxy values for API connections.
+- Upgrades version for logs ingestion API to 2023-01-01.
+- Renames the plugin to microsoft-sentinel-log-analytics-logstash-output-plugin.
+ 
+## 1.0.0
+- The initial release for the Logstash output plugin for Microsoft Sentinel. This plugin uses Data Collection Rules (DCRs) with Azure Monitor's Logs Ingestion API.

data/README.md

@@ -3,13 +3,12 @@
 Microsoft Sentinel provides a new output plugin for Logstash. Use this output plugin to send any log via Logstash to the Microsoft Sentinel/Log Analytics workspace. This is done with the Log Analytics DCR-based API.
 You may send logs to custom or standard tables.
 
-Plugin version: v1.1.0  
-Released on: 2023-07-23
+Plugin version: v1.1.3  
+Released on: 2024-10-10
 
 This plugin is currently in development and is free to use. We welcome contributions from the open source community on this project, and we request and appreciate feedback from users.
 
-
-## Steps to implement the output plugin
+## Installation Instructions
 1) Install the plugin
 2) Create a sample file
 3) Create the required DCR-related resources
@@ -19,13 +18,16 @@ This plugin is currently in development and is free to use. We welcome contribut
 
 ## 1. Install the plugin
 
-Microsoft Sentinel provides Logstash output plugin to Log analytics workspace using DCR based logs API. 
-Install the microsoft-sentinel-log-analytics-logstash-output-plugin, use [Logstash Offline Plugin Management instruction](<https://www.elastic.co/guide/en/logstash/current/offline-plugins.html>). 
+Microsoft Sentinel provides Logstash output plugin to Log analytics workspace using DCR based logs API.
+
+The plugin is published on [RubyGems](https://rubygems.org/gems/microsoft-sentinel-log-analytics-logstash-output-plugin). To install to an existing logstash installation, run `logstash-plugin install microsoft-sentinel-log-analytics-logstash-output-plugin`.
+
+If you do not have a direct internet connection, you can install the plugin to another logstash installation, and then export and import a plugin bundle to the offline host. For more information, see [Logstash Offline Plugin Management instruction](<https://www.elastic.co/guide/en/logstash/current/offline-plugins.html>).  
 
 Microsoft Sentinel's Logstash output plugin supports the following versions
 - 7.0 - 7.17.13
 - 8.0 - 8.9
-- 8.11
+- 8.11 - 8.15
 
 Please note that when using Logstash 8, it is recommended to disable ECS in the pipeline. For more information refer to [Logstash documentation.](<https://www.elastic.co/guide/en/logstash/8.4/ecs-ls.html>)
 
@@ -234,3 +236,23 @@ Which will produce this content in the sample file:
 	}
 ]
 ```
+
+
+## Known issues
+ 
+When using Logstash installed on a Docker image of Lite Ubuntu, the following warning may appear:
+
+```
+java.lang.RuntimeException: getprotobyname_r failed
+```
+
+To resolve it, use the following commands to install the *netbase* package within your Dockerfile:
+```bash
+USER root
+RUN apt install netbase -y
+```
+For more information, see [JNR regression in Logstash 7.17.0 (Docker)](https://github.com/elastic/logstash/issues/13703).
+
+If your environment's event rate is low considering the number of allocated Logstash workers, we recommend increasing the value of *plugin_flush_interval* to 60 or more. This change will allow each worker to batch more events before uploading to the Data Collection Endpoint (DCE).  You can monitor the ingestion payload using [DCR metrics](https://learn.microsoft.com/azure/azure-monitor/essentials/data-collection-monitor#dcr-metrics).
+For more information on *plugin_flush_interval*, see the [Optional Configuration table](https://learn.microsoft.com/azure/sentinel/connect-logstash-data-connection-rules#optional-configuration) mentioned earlier.
+

data/lib/logstash/sentinel_la/logAnalyticsAadTokenProvider.rb

@@ -1,10 +1,10 @@
 # encoding: utf-8
 require "logstash/sentinel_la/logstashLoganalyticsConfiguration"
-require 'rest-client'
 require 'json'
 require 'openssl'
 require 'base64'
 require 'time'
+require 'excon'
 
 module LogStash; module Outputs; class MicrosoftSentinelOutputInternal
 class LogAnalyticsAadTokenProvider
@@ -64,14 +64,13 @@ class LogAnalyticsAadTokenProvider
     while true
       begin
         # Post REST request 
-        response = RestClient::Request.execute(method: :post, url: @token_request_uri, payload: @token_request_body, headers: headers,
-                                              proxy: @logstashLoganalyticsConfiguration.proxy_aad)
-                    
-        if (response.code == 200 || response.code == 201)
+        response = Excon.post(@token_request_uri, :body => @token_request_body, :headers => headers, :proxy => @logstashLoganalyticsConfiguration.proxy_aad, expects: [200, 201])
+
+        if (response.status == 200 || response.status == 201)
           return JSON.parse(response.body)
         end
-      rescue RestClient::ExceptionWithResponse => ewr
-        @logger.error("Exception while authenticating with AAD API ['#{ewr.response}']")          
+      rescue Excon::Error::HTTPStatus => ex
+        @logger.error("Error while authenticating with AAD [#{ex.class}: '#{ex.response.status}', Response: '#{ex.response.body}']")
       rescue Exception => ex          
         @logger.trace("Exception while authenticating with AAD API ['#{ex}']")
       end

data/lib/logstash/sentinel_la/logAnalyticsClient.rb

@@ -1,11 +1,11 @@
 # encoding: utf-8
 require "logstash/sentinel_la/version"
-require 'rest-client'
 require 'json'
 require 'openssl'
 require 'base64'
 require 'time'
 require 'rbconfig'
+require 'excon'
 
 module LogStash; module Outputs; class MicrosoftSentinelOutputInternal 
 class LogAnalyticsClient
@@ -22,28 +22,78 @@ require "logstash/sentinel_la/logAnalyticsAadTokenProvider"
     @uri = sprintf("%s/dataCollectionRules/%s/streams/%s?api-version=%s",@logstashLoganalyticsConfiguration.data_collection_endpoint, @logstashLoganalyticsConfiguration.dcr_immutable_id, logstashLoganalyticsConfiguration.dcr_stream_name, la_api_version)
     @aadTokenProvider=LogAnalyticsAadTokenProvider::new(logstashLoganalyticsConfiguration)
     @userAgent = getUserAgent()
+    
+    # Auto close connection after 60 seconds of inactivity
+    @connectionAutoClose = {
+      :last_use => Time.now,
+      :lock => Mutex.new,
+      :max_idel_time => 60,
+      :is_closed => true
+    }
+
+    @timer = Thread.new do
+      loop do
+        sleep @connectionAutoClose[:max_idel_time] / 2
+        if is_connection_stale?
+          @connectionAutoClose[:lock].synchronize do
+            if is_connection_stale?
+              reset_connection
+            end
+          end
+        end
+      end
+    end
+
+   
   end # def initialize
 
   # Post the given json to Azure Loganalytics
   def post_data(body)
     raise ConfigError, 'no json_records' if body.empty?
+    response = nil
+    
+    @connectionAutoClose[:lock].synchronize do 
+      #close connection if its stale
+      if is_connection_stale?
+        reset_connection
+      end
+      if @connectionAutoClose[:is_closed]
+        open_connection
+      end
+      
+      headers = get_header()
+      # Post REST request
+      response = @connection.request(method: :post, body: body, headers: headers)
+      @connectionAutoClose[:is_closed] = false
+      @connectionAutoClose[:last_use] = Time.now
+    end
+    return response
 
-    # Create REST request header
-    headers = get_header()
-
-    # Post REST request
-
-    return RestClient::Request.execute(method: :post, url: @uri, payload: body, headers: headers,
-                                        proxy: @logstashLoganalyticsConfiguration.proxy_endpoint, timeout: 240)
   end # def post_data
 
   # Static function to return if the response is OK or else
   def self.is_successfully_posted(response)
-    return (response.code >= 200 && response.code < 300 ) ? true : false
+    return (response.status >= 200 && response.status < 300 ) ? true : false
   end # def self.is_successfully_posted
 
   private 
 
+  def open_connection
+    @connection = Excon.new(@uri, :persistent => true, :proxy => @logstashLoganalyticsConfiguration.proxy_endpoint, 
+          expects: [200, 201, 202, 204, 206, 207, 208, 226, 300, 301, 302, 303, 304, 305, 306, 307, 308],
+          read_timeout: 240, write_timeout: 240, connect_timeout: 240)
+    @logger.trace("Connection to Azure LogAnalytics was opened.");
+  end
+
+  def reset_connection
+    @connection.reset
+    @connectionAutoClose[:is_closed] = true    
+    @logger.trace("Connection to Azure LogAnalytics was closed due to inactivity.");
+  end
+
+  def is_connection_stale?
+    return Time.now - @connectionAutoClose[:last_use] > @connectionAutoClose[:max_idel_time] && !@connectionAutoClose[:is_closed]
+  end
   # Create a header for the given length 
   def get_header()
     # Getting an authorization token bearer (if the token is expired, the method will post a request to get a new authorization token)

data/lib/logstash/sentinel_la/logStashEventsBatcher.rb

@@ -2,7 +2,7 @@
 
 require "logstash/sentinel_la/logAnalyticsClient"
 require "logstash/sentinel_la/logstashLoganalyticsConfiguration"
-
+require "excon"
 # LogStashAutoResizeBuffer class setting a resizable buffer which is flushed periodically
 # The buffer resize itself according to Azure Loganalytics  and configuration limitations
 module LogStash; module Outputs; class MicrosoftSentinelOutputInternal
@@ -59,34 +59,32 @@ class LogStashEventsBatcher
                     return
                 else
                     @logger.trace("Rest client response ['#{response}']")
-                    @logger.error("#{api_name} request failed. Error code: #{response.code} #{try_get_info_from_error_response(response)}")
+                    @logger.error("#{api_name} request failed. Error code: #{response.pree} #{try_get_info_from_error_response(response)}")
                 end
-            rescue RestClient::Exceptions::Timeout => eto
-                @logger.trace("Timeout exception ['#{eto.display}'] when posting data to #{api_name}. Rest client response ['#{eto.response.display}']. [amount_of_documents=#{amount_of_documents}]")
-                @logger.error("Timeout exception while posting data to #{api_name}. [Exception: '#{eto}'] [amount of documents=#{amount_of_documents}]'")
-                force_retry = true
-
-            rescue RestClient::ExceptionWithResponse => ewr
-                response = ewr.response
-                @logger.trace("Exception in posting data to #{api_name}. Rest client response ['#{ewr.response}']. [amount_of_documents=#{amount_of_documents} request payload=#{call_payload}]")
-                @logger.error("Exception when posting data to #{api_name}. [Exception: '#{ewr}'] #{try_get_info_from_error_response(ewr.response)} [amount of documents=#{amount_of_documents}]'")
+                rescue Excon::Error::HTTPStatus => ewr
+                    response = ewr.response
+                    @logger.trace("Exception in posting data to #{api_name}. Rest client response ['#{response}']. [amount_of_documents=#{amount_of_documents} request payload=#{call_payload}]")
+                    @logger.error("Exception when posting data to #{api_name}. [Exception: '#{ewr.class}'] #{try_get_info_from_error_response(ewr.response)} [amount of documents=#{amount_of_documents}]'")
 
-                if ewr.http_code.to_f == 400
-                    @logger.info("Not trying to resend since exception http code is #{ewr.http_code}")
-                    return                
-                elsif ewr.http_code.to_f == 408
-                    force_retry = true
-                elsif ewr.http_code.to_f == 429
-                    # thrutteling detected, backoff before resending
-                    parsed_retry_after = response.headers.include?(:retry_after) ? response.headers[:retry_after].to_i : 0
-                    seconds_to_sleep = parsed_retry_after > 0 ? parsed_retry_after : 30
+                    if ewr.class ==  Excon::Error::BadRequest
+                        @logger.info("Not trying to resend since exception http code is 400")
+                        return                
+                    elsif ewr.class ==  Excon::Error::RequestTimeout
+                        force_retry = true
+                    elsif ewr.class ==  Excon::Error::TooManyRequests
+                        # thrutteling detected, backoff before resending
+                        parsed_retry_after = response.data[:headers].include?('Retry-After') ? response.data[:headers]['Retry-After'].to_i : 0
+                        seconds_to_sleep = parsed_retry_after > 0 ? parsed_retry_after : 30
 
-                    #force another retry even if the next iteration of the loop will be after the retransmission_timeout
+                        #force another retry even if the next iteration of the loop will be after the retransmission_timeout
+                        force_retry = true
+                    end               
+                rescue Excon::Error::Socket => ex
+                    @logger.trace("Exception: '#{ex.class.name}]#{ex} in posting data to #{api_name}. [amount_of_documents=#{amount_of_documents}]'")
                     force_retry = true
-                end
-            rescue Exception => ex
-                @logger.trace("Exception in posting data to #{api_name}.[amount_of_documents=#{amount_of_documents} request payload=#{call_payload}]")       
-                @logger.error("Exception in posting data to #{api_name}. [Exception: '#{ex}, amount of documents=#{amount_of_documents}]'")
+                rescue Exception => ex
+                    @logger.trace("Exception in posting data to #{api_name}.[amount_of_documents=#{amount_of_documents} request payload=#{call_payload}]")       
+                    @logger.error("Exception in posting data to #{api_name}. [Exception: '[#{ex.class.name}]#{ex}, amount of documents=#{amount_of_documents}]'")
             end
             is_retry = true
             @logger.info("Retrying transmission to #{api_name} in #{seconds_to_sleep} seconds.")
@@ -110,8 +108,8 @@ class LogStashEventsBatcher
     def get_request_id_from_response(response)
         output =""
         begin
-            if !response.nil? && response.headers.include?(:x_ms_request_id)
-                output += response.headers[:x_ms_request_id]
+            if !response.nil? && response.data[:headers].include?("x-ms-request-id")
+                output += response.data[:headers]["x-ms-request-id"]
             end
         rescue Exception => ex
             @logger.debug("Error while getting reqeust id from success response headers: #{ex.display}")
@@ -124,12 +122,13 @@ class LogStashEventsBatcher
         begin
             output = ""
             if !response.nil?                
-                if response.headers.include?(:x_ms_error_code)
-                    output += " [ms-error-code header: #{response.headers[:x_ms_error_code]}]"
+                if response.data[:headers].include?("x-ms-error-code")
+                    output += " [ms-error-code header: #{response.data[:headers]["x-ms-error-code"]}]"
             end
-            if response.headers.include?(:x_ms_request_id)
-                output += " [x-ms-request-id header: #{response.headers[:x_ms_request_id]}]"
+            if response.data[:headers].include?("x-ms-request-id")
+                output += " [x-ms-request-id header: #{response.data[:headers]["x-ms-request-id"]}]"
             end
+            output += " [response body: #{response.data[:body]}]"            
         end        
             return output
         rescue Exception => ex

data/lib/logstash/sentinel_la/version.rb

@@ -1,6 +1,6 @@
 module LogStash; module Outputs;
 class MicrosoftSentinelOutputInternal
-  VERSION_INFO = [1, 1, 1].freeze
+  VERSION_INFO = [1, 1, 3].freeze
   VERSION = VERSION_INFO.map(&:to_s).join('.').freeze
 
   def self.version

data/microsoft-sentinel-log-analytics-logstash-output-plugin.gemspec

@@ -20,8 +20,8 @@ Gem::Specification.new do |s|
   s.metadata = { "logstash_plugin" => "true", "logstash_group" => "output" }
 
   # Gem dependencies
-  s.add_runtime_dependency "rest-client", ">= 2.1.0"
   s.add_runtime_dependency "logstash-core-plugin-api", ">= 1.60", "<= 2.99"
   s.add_runtime_dependency "logstash-codec-plain"
+  s.add_runtime_dependency "excon", ">= 0.88.0"
   s.add_development_dependency "logstash-devutils"
 end

metadata

@@ -1,29 +1,15 @@
 --- !ruby/object:Gem::Specification
 name: microsoft-sentinel-log-analytics-logstash-output-plugin
 version: !ruby/object:Gem::Version
-  version: 1.1.1
+  version: 1.1.3
 platform: ruby
 authors:
 - Microsoft Sentinel
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2024-01-17 00:00:00.000000000 Z
+date: 2024-10-10 00:00:00.000000000 Z
 dependencies:
-- !ruby/object:Gem::Dependency
-  name: rest-client
-  requirement: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: 2.1.0
-  type: :runtime
-  prerelease: false
-  version_requirements: !ruby/object:Gem::Requirement
-    requirements:
-    - - ">="
-      - !ruby/object:Gem::Version
-        version: 2.1.0
 - !ruby/object:Gem::Dependency
   name: logstash-core-plugin-api
   requirement: !ruby/object:Gem::Requirement
@@ -58,6 +44,20 @@ dependencies:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+- !ruby/object:Gem::Dependency
+  name: excon
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 0.88.0
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 0.88.0
 - !ruby/object:Gem::Dependency
   name: logstash-devutils
   requirement: !ruby/object:Gem::Requirement