RubyGems - microsoft-sentinel-log-analytics-logstash-output-plugin - Versions diffs - 1.1.1 → 1.1.4 - Mend

microsoft-sentinel-log-analytics-logstash-output-plugin 1.1.1 → 1.1.4

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.

Files changed (12) hide show

checksums.yaml +4 -4
data/CHANGELOG.md +16 -14
data/Gemfile +2 -2
data/LICENSE +21 -21
data/README.md +258 -236
data/lib/logstash/outputs/microsoft-sentinel-log-analytics-logstash-output-plugin.rb +116 -116
data/lib/logstash/sentinel_la/logAnalyticsAadTokenProvider.rb +6 -7
data/lib/logstash/sentinel_la/logAnalyticsClient.rb +59 -9
data/lib/logstash/sentinel_la/logStashEventsBatcher.rb +32 -30
data/lib/logstash/sentinel_la/version.rb +2 -2
data/microsoft-sentinel-log-analytics-logstash-output-plugin.gemspec +27 -27
metadata +24 -18

checksums.yaml CHANGED Viewed

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: bc9cee055d5aa8f90acde6ed20eacccf0d9e9a36fa9a5ebd1c916d85a410e628
-  data.tar.gz: '081cd8f53b716eeed931c417ce6fc7a13e3d2d1fe9acf0050858e5b87918d4ff'
+  metadata.gz: a1b93ef7e00ec54fd6aea1712b10a16947a3bf4a226a7992897ae5e6fb2f7708
+  data.tar.gz: 712e6f247d574d5b48bb87693af44098401246c192ab886d0b3960c318861491
 SHA512:
-  metadata.gz: 77753b09f6c4631fb2e2eb1e0ba5dd4eb4c33840901226f1caa08043b39caa77497cdfd42adb7ac6a857b22e3bd98512fff8e3e1ba4d6b7916eb16e822f752ce
-  data.tar.gz: 9ae33cb6bb9c96c19011b4a21f9ac943f75df18b99761023cb4ba00c84584a6ad0c05d8b0d5c0819993b434e2f6d14da1e62f92ba2f44a11aa6eeed22cbe0b9c
+  metadata.gz: 66758c4c92a88a28a81b19161dc1e55d727675ad5f788d6abfce1a495603de7a601a0f4fd9ee35c8ad746a3f782b019cf7343c3c9c472c3f2ddf03a8a1b348f1
+  data.tar.gz: 234aee955f0a3ce3a1dbb5c84f620d87ef350053491a716e57feb15282fddf2cce69003add06ef73bf017fa682d9c5e165cfe1b7ad3bc158e1f6a9bf21ec55a8

data/CHANGELOG.md CHANGED Viewed

@@ -1,14 +1,16 @@
-## 1.0.0
-* Initial release for output plugin for logstash to Microsoft Sentinel. This is done with the Log Analytics DCR based API.
-## 1.1.0
-* Increase timeout for read/open connections to 120 seconds.
-* Add error handling for when connection timeout occurs.
-* Upgrade the rest-client dependency minimum version to 2.1.0.
-* Allow setting different proxy values for api connections.
-* Upgrade version for ingestion api to 2023-01-01.
-* Rename the plugin to microsoft-sentinel-log-analytics-logstash-output-plugin.
-## 1.1.1
-* Support China and US Government Azure sovereign clouds.
-* Increase timeout for read/open connections to 240 seconds.
+## 1.1.4
+-  Limit `excon` library version to lower than 1.0.0 to make sure port is always used when using a proxy.
+## 1.1.3
+-  Replaces the `rest-client` library used for connecting to Azure with the `excon` library.
+## 1.1.1
+- Adds support for Azure US Government cloud and Microsoft Azure operated by 21Vianet in China.
+## 1.1.0
+- Allows setting different proxy values for API connections.
+- Upgrades version for logs ingestion API to 2023-01-01.
+- Renames the plugin to microsoft-sentinel-log-analytics-logstash-output-plugin.
+## 1.0.0
+- The initial release for the Logstash output plugin for Microsoft Sentinel. This plugin uses Data Collection Rules (DCRs) with Azure Monitor's Logs Ingestion API.

data/Gemfile CHANGED Viewed

@@ -1,2 +1,2 @@
-source 'https://rubygems.org'
-gemspec
+source 'https://rubygems.org'
+gemspec

data/LICENSE CHANGED Viewed

@@ -1,21 +1,21 @@
-MIT License
-Copyright (c) Microsoft Corporation. All rights reserved.
-Permission is hereby granted, free of charge, to any person obtaining a copy
-of this software and associated documentation files (the "Software"), to deal
-in the Software without restriction, including without limitation the rights
-to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
-copies of the Software, and to permit persons to whom the Software is
-furnished to do so, subject to the following conditions:
-The above copyright notice and this permission notice shall be included in all
-copies or substantial portions of the Software.
-THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
-IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
-FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
-AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
-LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
-OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
-SOFTWARE
+MIT License
+Copyright (c) Microsoft Corporation. All rights reserved.
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+The above copyright notice and this permission notice shall be included in all
+copies or substantial portions of the Software.
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
+SOFTWARE

data/README.md CHANGED Viewed

@@ -1,236 +1,258 @@
-# Microsoft Sentinel output plugin for Logstash
-Microsoft Sentinel provides a new output plugin for Logstash. Use this output plugin to send any log via Logstash to the Microsoft Sentinel/Log Analytics workspace. This is done with the Log Analytics DCR-based API.
-You may send logs to custom or standard tables.
-Plugin version: v1.1.0
-Released on: 2023-07-23
-This plugin is currently in development and is free to use. We welcome contributions from the open source community on this project, and we request and appreciate feedback from users.
-## Steps to implement the output plugin
-1) Install the plugin
-2) Create a sample file
-3) Create the required DCR-related resources
-4) Configure Logstash configuration file
-5) Basic logs transmission
-## 1. Install the plugin
-Microsoft Sentinel provides Logstash output plugin to Log analytics workspace using DCR based logs API.
-Install the microsoft-sentinel-log-analytics-logstash-output-plugin, use [Logstash Offline Plugin Management instruction](<https://www.elastic.co/guide/en/logstash/current/offline-plugins.html>).
-Microsoft Sentinel's Logstash output plugin supports the following versions
-- 7.0 - 7.17.13
-- 8.0 - 8.9
-- 8.11
-Please note that when using Logstash 8, it is recommended to disable ECS in the pipeline. For more information refer to [Logstash documentation.](<https://www.elastic.co/guide/en/logstash/8.4/ecs-ls.html>)
-## 2. Create a sample file
-To create a sample file, follow the following steps:
-1)	Copy the output plugin configuration below to your Logstash configuration file:
-```
-output {
-    microsoft-sentinel-log-analytics-logstash-output-plugin {
-        create_sample_file => true
-        sample_file_path => "<enter the path to the file in which the sample data will be written>" #for example: "c:\\temp" (for windows) or "/var/log" for Linux.
-    }
-}
-```
-Note: make sure that the path exists before creating the sample file.
-2) Start Logstash. The plugin will collect up to 10 records to a sample.
-3) The file named "sampleFile<epoch seconds>.json" in the configured path will be created once there are 10 events to sample or when the Logstash process exited gracefully. (for example: "c:\temp\sampleFile1648453501.json").
-### Configurations:
-The following parameters are optional and should be used to create a sample file.
-- **create_sample_file** - Boolean, False by default. When enabled, up to 10 events will be written to a sample json file.
-- **sample_file_path** - Number, Empty by default. Required when create_sample_file is enabled. Should include a valid path in which to place the sample file generated.
-### Complete example
-1. set the pipeline.conf with the following configuration:
-```
-input {
-      generator {
-        lines => [ "This is a test log message"]
-        count => 10
-      }
-}
-output {
-    microsoft-sentinel-log-analytics-logstash-output-plugin {
-        create_sample_file => true
-        sample_file_path => "<enter the path to the file in which the sample data will be written>" #for example: "c:\\temp" (for windows) or "/var/log" for Linux.
-    }
-}
-```
-2. the following sample file will be generated:
-```
-[
-	{
-		"host": "logstashMachine",
-		"sequence": 0,
-		"message": "This is a test log message",
-		"ls_timestamp": "2022-10-29T13:19:28.116Z",
-		"ls_version": "1"
-	},
-	...
-]
-```
-## 3. Create the required DCR-related resources
-To configure Microsoft Sentinel Logstash plugin you first need to create the DCR-related resources. To create these resources, follow one of the following tutorials:
-1) To ingest the data to a custom table use [Tutorial - Send custom logs to Azure Monitor Logs (preview) - Azure Monitor | Microsoft Docs](<https://docs.microsoft.com/azure/azure-monitor/logs/tutorial-custom-logs>) tutorial. Note that as part of creating the table and the DCR you will need to provide the sample file that you've created in the previous section.
-2) To ingest the data to a standard table like Syslog or CommonSecurityLog use [Tutorial - Send custom logs to Azure Monitor Logs using resource manager templates - Azure Monitor | Microsoft Docs](<https://docs.microsoft.com/azure/azure-monitor/logs/tutorial-custom-logs-api>).
-## 4. Configure Logstash configuration file
-Use the tutorial from the previous section to retrieve the following attributes:
-- **client_app_Id** - String, The 'Application (client) ID' value created in step #3 of the "Configure Application" section of the tutorial you used in the previous step.
-- **client_app_secret** -String, The value of the client secret created in step #5 of the "Configure Application" section of the tutorial you used in the previous step.
-- **tenant_id** - String, Your subscription's tenant id. You can find in the following path: Home -> Azure Active Directory -> Overview Under 'Basic Information'.
-- **data_collection_endpoint** - String, - The value of the logsIngestion URI (see step #3 of the "Create data collection endpoint" section in Tutorial [Tutorial - Send custom logs to Azure Monitor Logs using resource manager templates - Azure Monitor | Microsoft Docs](<https://docs.microsoft.com/azure/azure-monitor/logs/tutorial-custom-logs-api#create-data-collection-endpoint>).
-- **dcr_immutable_id** - String, The value of the DCR immutableId (see the "Collect information from DCR" section in [Tutorial - Send custom logs to Azure Monitor Logs (preview) - Azure Monitor | Microsoft Docs](<https://docs.microsoft.com/azure/azure-monitor/logs/tutorial-custom-logs#collect-information-from-dcr>).
-- **dcr_stream_name** - String, The name of the data stream (Go to the json view of the DCR as explained in the "Collect information from DCR" section in [Tutorial - Send custom logs to Azure Monitor Logs (preview) - Azure Monitor | Microsoft Docs](<https://docs.microsoft.com/azure/azure-monitor/logs/tutorial-custom-logs#collect-information-from-dcr>) and copy the value of the "dataFlows -> streams" property (see circled in red in the below example).
-After retrieving the required values replace the output section of the Logstash configuration file created in the previous steps with the example below. Then, replace the strings in the brackets below with the corresponding values. Make sure you change the "create_sample_file" attribute to false.
-Here is an example for the output plugin configuration section:
-```
-output {
-    microsoft-sentinel-log-analytics-logstash-output-plugin {
-        client_app_Id => "<enter your client_app_id value here>"
-        client_app_secret => "<enter your client_app_secret value here>"
-        tenant_id => "<enter your tenant id here>"
-        data_collection_endpoint => "<enter your DCE logsIngestion URI here>"
-        dcr_immutable_id => "<enter your DCR immutableId here>"
-        dcr_stream_name => "<enter your stream name here>"
-        create_sample_file=> false
-        sample_file_path => "c:\\temp"
-    }
-}
-```
-### Optional configuration
-- **key_names** – Array of strings, if you wish to send a subset of the columns to Log Analytics.
-- **plugin_flush_interval** – Number, 5 by default. Defines the maximal time difference (in seconds) between sending two messages to Log Analytics.
-- **retransmission_time** - Number, 10 by default. This will set the amount of time in seconds given for retransmitting messages once sending has failed.
-- **compress_data** - Boolean, false by default. When this field is true, the event data is compressed before using the API. Recommended for high throughput pipelines
-- **proxy** - String, Empty by default. Specify which proxy URL to use for API calls for all of the communications with Azure.
-- **proxy_aad** - String, Empty by default. Specify which proxy URL to use for API calls for the Azure Active Directory service. Overrides the proxy setting.
-- **proxy_endpoint** - String, Empty by default. Specify which proxy URL to use when sending log data to the endpoint. Overrides the proxy setting.
-- **azure_cloud** - String, Empty by default. Used to specify the name of the Azure cloud that is being used, AzureCloud is set as default. Available values are: AzureCloud, AzureChinaCloud and AzureUSGovernment.
-#### Note: When setting an empty string as a value for a proxy setting, it will unset any system wide proxy setting.
-Security notice: We recommend not to implicitly state client_app_Id, client_app_secret, tenant_id, data_collection_endpoint, and dcr_immutable_id in your Logstash configuration for security reasons.
-                 It is best to store this sensitive information in a Logstash KeyStore as described here- ['Secrets Keystore'](<https://www.elastic.co/guide/en/logstash/current/keystore.html>)
-## 5. Basic logs transmission
-Here is an example configuration that parses Syslog incoming data into a custom stream named "Custom-MyTableRawData".
-### Example Configuration
-- Using filebeat input pipe
-```
-input {
-    beats {
-        port => "5044"
-    }
-}
- filter {
-}
-output {
-    microsoft-sentinel-log-analytics-logstash-output-plugin {
-      client_app_Id => "619c1731-15ca-4403-9c61-xxxxxxxxxxxx"
-      client_app_secret => "xxxxxxxxxxxxxxxx"
-      tenant_id => "72f988bf-86f1-41af-91ab-xxxxxxxxxxxx"
-      data_collection_endpoint => "https://my-customlogsv2-test-jz2a.eastus2-1.ingest.monitor.azure.com"
-      dcr_immutable_id => "dcr-xxxxxxxxxxxxxxxxac23b8978251433a"
-      dcr_stream_name => "Custom-MyTableRawData"
-      proxy_aad => "http://proxy.example.com"
-    }
-}
-```
-- Or using the tcp input pipe
-```
-input {
-    tcp {
-        port => "514"
-        type => syslog #optional, will effect log type in table
-    }
-}
- filter {
-}
-output {
-    microsoft-sentinel-log-analytics-logstash-output-plugin {
-      client_app_Id => "619c1731-15ca-4403-9c61-xxxxxxxxxxxx"
-      client_app_secret => "xxxxxxxxxxxxxxxx"
-      tenant_id => "72f988bf-86f1-41af-91ab-xxxxxxxxxxxx"
-      data_collection_endpoint => "https://my-customlogsv2-test-jz2a.eastus2-1.ingest.monitor.azure.com"
-      dcr_immutable_id => "dcr-xxxxxxxxxxxxxxxxac23b8978251433a"
-      dcr_stream_name => "Custom-MyTableRawData"
-    }
-}
-```
-<u>Advanced Configuration</u>
-```
-input {
-  syslog {
-    port => 514
-  }
-}
-output {
-    microsoft-sentinel-log-analytics-logstash-output-plugin {
-      client_app_Id => "${CLIENT_APP_ID}"
-      client_app_secret => "${CLIENT_APP_SECRET}"
-      tenant_id => "${TENANT_ID}"
-      data_collection_endpoint => "${DATA_COLLECTION_ENDPOINT}"
-      dcr_immutable_id => "${DCR_IMMUTABLE_ID}"
-      dcr_stream_name => "Custom-MyTableRawData"
-	  key_names => ['PRI','TIME_TAG','HOSTNAME','MSG']
-    }
-}
-```
-Now you are able to run logstash with the example configuration and send mock data using the 'logger' command.
-For example:
-```
-logger -p local4.warn --rfc3164 --tcp -t CEF "0|Microsoft|Device|cef-test|example|data|1|here is some more data for the example" -P 514 -d -n 127.0.0.1
-```
-Which will produce this content in the sample file:
-```
-[
-	{
-		"logsource": "logstashMachine",
-		"facility": 20,
-		"severity_label": "Warning",
-		"severity": 4,
-		"timestamp": "Apr  7 08:26:04",
-		"program": "CEF:",
-		"host": "127.0.0.1",
-		"facility_label": "local4",
-		"priority": 164,
-		"message": "0|Microsoft|Device|cef-test|example|data|1|here is some more data for the example",
-		"ls_timestamp": "2022-04-07T08:26:04.000Z",
-		"ls_version": "1"
-	}
-]
-```
+# Microsoft Sentinel output plugin for Logstash
+Microsoft Sentinel provides a new output plugin for Logstash. Use this output plugin to send any log via Logstash to the Microsoft Sentinel/Log Analytics workspace. This is done with the Log Analytics DCR-based API.
+You may send logs to custom or standard tables.
+Plugin version: v1.1.3
+Released on: 2024-10-10
+This plugin is currently in development and is free to use. We welcome contributions from the open source community on this project, and we request and appreciate feedback from users.
+## Installation Instructions
+1) Install the plugin
+2) Create a sample file
+3) Create the required DCR-related resources
+4) Configure Logstash configuration file
+5) Basic logs transmission
+## 1. Install the plugin
+Microsoft Sentinel provides Logstash output plugin to Log analytics workspace using DCR based logs API.
+The plugin is published on [RubyGems](https://rubygems.org/gems/microsoft-sentinel-log-analytics-logstash-output-plugin). To install to an existing logstash installation, run `logstash-plugin install microsoft-sentinel-log-analytics-logstash-output-plugin`.
+If you do not have a direct internet connection, you can install the plugin to another logstash installation, and then export and import a plugin bundle to the offline host. For more information, see [Logstash Offline Plugin Management instruction](<https://www.elastic.co/guide/en/logstash/current/offline-plugins.html>).
+Microsoft Sentinel's Logstash output plugin supports the following versions
+- 7.0 - 7.17.13
+- 8.0 - 8.9
+- 8.11 - 8.15
+Please note that when using Logstash 8, it is recommended to disable ECS in the pipeline. For more information refer to [Logstash documentation.](<https://www.elastic.co/guide/en/logstash/8.4/ecs-ls.html>)
+## 2. Create a sample file
+To create a sample file, follow the following steps:
+1)	Copy the output plugin configuration below to your Logstash configuration file:
+```
+output {
+    microsoft-sentinel-log-analytics-logstash-output-plugin {
+        create_sample_file => true
+        sample_file_path => "<enter the path to the file in which the sample data will be written>" #for example: "c:\\temp" (for windows) or "/var/log" for Linux.
+    }
+}
+```
+Note: make sure that the path exists before creating the sample file.
+2) Start Logstash. The plugin will collect up to 10 records to a sample.
+3) The file named "sampleFile<epoch seconds>.json" in the configured path will be created once there are 10 events to sample or when the Logstash process exited gracefully. (for example: "c:\temp\sampleFile1648453501.json").
+### Configurations:
+The following parameters are optional and should be used to create a sample file.
+- **create_sample_file** - Boolean, False by default. When enabled, up to 10 events will be written to a sample json file.
+- **sample_file_path** - Number, Empty by default. Required when create_sample_file is enabled. Should include a valid path in which to place the sample file generated.
+### Complete example
+1. set the pipeline.conf with the following configuration:
+```
+input {
+      generator {
+        lines => [ "This is a test log message"]
+        count => 10
+      }
+}
+output {
+    microsoft-sentinel-log-analytics-logstash-output-plugin {
+        create_sample_file => true
+        sample_file_path => "<enter the path to the file in which the sample data will be written>" #for example: "c:\\temp" (for windows) or "/var/log" for Linux.
+    }
+}
+```
+2. the following sample file will be generated:
+```
+[
+	{
+		"host": "logstashMachine",
+		"sequence": 0,
+		"message": "This is a test log message",
+		"ls_timestamp": "2022-10-29T13:19:28.116Z",
+		"ls_version": "1"
+	},
+	...
+]
+```
+## 3. Create the required DCR-related resources
+To configure Microsoft Sentinel Logstash plugin you first need to create the DCR-related resources. To create these resources, follow one of the following tutorials:
+1) To ingest the data to a custom table use [Tutorial - Send custom logs to Azure Monitor Logs (preview) - Azure Monitor | Microsoft Docs](<https://docs.microsoft.com/azure/azure-monitor/logs/tutorial-custom-logs>) tutorial. Note that as part of creating the table and the DCR you will need to provide the sample file that you've created in the previous section.
+2) To ingest the data to a standard table like Syslog or CommonSecurityLog use [Tutorial - Send custom logs to Azure Monitor Logs using resource manager templates - Azure Monitor | Microsoft Docs](<https://docs.microsoft.com/azure/azure-monitor/logs/tutorial-custom-logs-api>).
+## 4. Configure Logstash configuration file
+Use the tutorial from the previous section to retrieve the following attributes:
+- **client_app_Id** - String, The 'Application (client) ID' value created in step #3 of the "Configure Application" section of the tutorial you used in the previous step.
+- **client_app_secret** -String, The value of the client secret created in step #5 of the "Configure Application" section of the tutorial you used in the previous step.
+- **tenant_id** - String, Your subscription's tenant id. You can find in the following path: Home -> Azure Active Directory -> Overview Under 'Basic Information'.
+- **data_collection_endpoint** - String, - The value of the logsIngestion URI (see step #3 of the "Create data collection endpoint" section in Tutorial [Tutorial - Send custom logs to Azure Monitor Logs using resource manager templates - Azure Monitor | Microsoft Docs](<https://docs.microsoft.com/azure/azure-monitor/logs/tutorial-custom-logs-api#create-data-collection-endpoint>).
+- **dcr_immutable_id** - String, The value of the DCR immutableId (see the "Collect information from DCR" section in [Tutorial - Send custom logs to Azure Monitor Logs (preview) - Azure Monitor | Microsoft Docs](<https://docs.microsoft.com/azure/azure-monitor/logs/tutorial-custom-logs#collect-information-from-dcr>).
+- **dcr_stream_name** - String, The name of the data stream (Go to the json view of the DCR as explained in the "Collect information from DCR" section in [Tutorial - Send custom logs to Azure Monitor Logs (preview) - Azure Monitor | Microsoft Docs](<https://docs.microsoft.com/azure/azure-monitor/logs/tutorial-custom-logs#collect-information-from-dcr>) and copy the value of the "dataFlows -> streams" property (see circled in red in the below example).
+After retrieving the required values replace the output section of the Logstash configuration file created in the previous steps with the example below. Then, replace the strings in the brackets below with the corresponding values. Make sure you change the "create_sample_file" attribute to false.
+Here is an example for the output plugin configuration section:
+```
+output {
+    microsoft-sentinel-log-analytics-logstash-output-plugin {
+        client_app_Id => "<enter your client_app_id value here>"
+        client_app_secret => "<enter your client_app_secret value here>"
+        tenant_id => "<enter your tenant id here>"
+        data_collection_endpoint => "<enter your DCE logsIngestion URI here>"
+        dcr_immutable_id => "<enter your DCR immutableId here>"
+        dcr_stream_name => "<enter your stream name here>"
+        create_sample_file=> false
+        sample_file_path => "c:\\temp"
+    }
+}
+```
+### Optional configuration
+- **key_names** – Array of strings, if you wish to send a subset of the columns to Log Analytics.
+- **plugin_flush_interval** – Number, 5 by default. Defines the maximal time difference (in seconds) between sending two messages to Log Analytics.
+- **retransmission_time** - Number, 10 by default. This will set the amount of time in seconds given for retransmitting messages once sending has failed.
+- **compress_data** - Boolean, false by default. When this field is true, the event data is compressed before using the API. Recommended for high throughput pipelines
+- **proxy** - String, Empty by default. Specify which proxy URL to use for API calls for all of the communications with Azure.
+- **proxy_aad** - String, Empty by default. Specify which proxy URL to use for API calls for the Azure Active Directory service. Overrides the proxy setting.
+- **proxy_endpoint** - String, Empty by default. Specify which proxy URL to use when sending log data to the endpoint. Overrides the proxy setting.
+- **azure_cloud** - String, Empty by default. Used to specify the name of the Azure cloud that is being used, AzureCloud is set as default. Available values are: AzureCloud, AzureChinaCloud and AzureUSGovernment.
+#### Note: When setting an empty string as a value for a proxy setting, it will unset any system wide proxy setting.
+Security notice: We recommend not to implicitly state client_app_Id, client_app_secret, tenant_id, data_collection_endpoint, and dcr_immutable_id in your Logstash configuration for security reasons.
+                 It is best to store this sensitive information in a Logstash KeyStore as described here- ['Secrets Keystore'](<https://www.elastic.co/guide/en/logstash/current/keystore.html>)
+## 5. Basic logs transmission
+Here is an example configuration that parses Syslog incoming data into a custom stream named "Custom-MyTableRawData".
+### Example Configuration
+- Using filebeat input pipe
+```
+input {
+    beats {
+        port => "5044"
+    }
+}
+ filter {
+}
+output {
+    microsoft-sentinel-log-analytics-logstash-output-plugin {
+      client_app_Id => "619c1731-15ca-4403-9c61-xxxxxxxxxxxx"
+      client_app_secret => "xxxxxxxxxxxxxxxx"
+      tenant_id => "72f988bf-86f1-41af-91ab-xxxxxxxxxxxx"
+      data_collection_endpoint => "https://my-customlogsv2-test-jz2a.eastus2-1.ingest.monitor.azure.com"
+      dcr_immutable_id => "dcr-xxxxxxxxxxxxxxxxac23b8978251433a"
+      dcr_stream_name => "Custom-MyTableRawData"
+      proxy_aad => "http://proxy.example.com"
+    }
+}
+```
+- Or using the tcp input pipe
+```
+input {
+    tcp {
+        port => "514"
+        type => syslog #optional, will effect log type in table
+    }
+}
+ filter {
+}
+output {
+    microsoft-sentinel-log-analytics-logstash-output-plugin {
+      client_app_Id => "619c1731-15ca-4403-9c61-xxxxxxxxxxxx"
+      client_app_secret => "xxxxxxxxxxxxxxxx"
+      tenant_id => "72f988bf-86f1-41af-91ab-xxxxxxxxxxxx"
+      data_collection_endpoint => "https://my-customlogsv2-test-jz2a.eastus2-1.ingest.monitor.azure.com"
+      dcr_immutable_id => "dcr-xxxxxxxxxxxxxxxxac23b8978251433a"
+      dcr_stream_name => "Custom-MyTableRawData"
+    }
+}
+```
+<u>Advanced Configuration</u>
+```
+input {
+  syslog {
+    port => 514
+  }
+}
+output {
+    microsoft-sentinel-log-analytics-logstash-output-plugin {
+      client_app_Id => "${CLIENT_APP_ID}"
+      client_app_secret => "${CLIENT_APP_SECRET}"
+      tenant_id => "${TENANT_ID}"
+      data_collection_endpoint => "${DATA_COLLECTION_ENDPOINT}"
+      dcr_immutable_id => "${DCR_IMMUTABLE_ID}"
+      dcr_stream_name => "Custom-MyTableRawData"
+	  key_names => ['PRI','TIME_TAG','HOSTNAME','MSG']
+    }
+}
+```
+Now you are able to run logstash with the example configuration and send mock data using the 'logger' command.
+For example:
+```
+logger -p local4.warn --rfc3164 --tcp -t CEF "0|Microsoft|Device|cef-test|example|data|1|here is some more data for the example" -P 514 -d -n 127.0.0.1
+```
+Which will produce this content in the sample file:
+```
+[
+	{
+		"logsource": "logstashMachine",
+		"facility": 20,
+		"severity_label": "Warning",
+		"severity": 4,
+		"timestamp": "Apr  7 08:26:04",
+		"program": "CEF:",
+		"host": "127.0.0.1",
+		"facility_label": "local4",
+		"priority": 164,
+		"message": "0|Microsoft|Device|cef-test|example|data|1|here is some more data for the example",
+		"ls_timestamp": "2022-04-07T08:26:04.000Z",
+		"ls_version": "1"
+	}
+]
+```
+## Known issues
+When using Logstash installed on a Docker image of Lite Ubuntu, the following warning may appear:
+```
+java.lang.RuntimeException: getprotobyname_r failed
+```
+To resolve it, use the following commands to install the *netbase* package within your Dockerfile:
+```bash
+USER root
+RUN apt install netbase -y
+```
+For more information, see [JNR regression in Logstash 7.17.0 (Docker)](https://github.com/elastic/logstash/issues/13703).
+If your environment's event rate is low considering the number of allocated Logstash workers, we recommend increasing the value of *plugin_flush_interval* to 60 or more. This change will allow each worker to batch more events before uploading to the Data Collection Endpoint (DCE).  You can monitor the ingestion payload using [DCR metrics](https://learn.microsoft.com/azure/azure-monitor/essentials/data-collection-monitor#dcr-metrics).
+For more information on *plugin_flush_interval*, see the [Optional Configuration table](https://learn.microsoft.com/azure/sentinel/connect-logstash-data-connection-rules#optional-configuration) mentioned earlier.

data/lib/logstash/outputs/microsoft-sentinel-log-analytics-logstash-output-plugin.rb CHANGED Viewed

@@ -1,116 +1,116 @@
-# encoding: utf-8
-require "logstash/outputs/base"
-require "logstash/namespace"
-require "logstash/sentinel_la/logstashLoganalyticsConfiguration"
-require "logstash/sentinel_la/sampleFileCreator"
-require "logstash/sentinel_la/logsSender"
-class LogStash::Outputs::MicrosoftSentinelOutput < LogStash::Outputs::Base
-  config_name "microsoft-sentinel-log-analytics-logstash-output-plugin"
-  # Stating that the output plugin will run in concurrent mode
-  concurrency :shared
-  # Your registered app ID
-  config :client_app_Id, :validate => :string
-  # The registered app's secret, required by Azure Loganalytics REST API
-  config :client_app_secret, :validate => :string
-  # Your Operations Management Suite Tenant ID
-  config :tenant_id, :validate => :string
-  # Your data collection rule endpoint
-  config :data_collection_endpoint, :validate => :string
-  # Your data collection rule ID
-  config :dcr_immutable_id, :validate => :string
-  # Your dcr data stream name
-  config :dcr_stream_name, :validate => :string
-  # Subset of keys to send to the Azure Loganalytics workspace
-  config :key_names, :validate => :array, :default => []
-  # Max number of seconds to wait between flushes. Default 5
-  config :plugin_flush_interval, :validate => :number, :default => 5
-  # Factor for adding to the amount of messages sent
-  config :decrease_factor, :validate => :number, :default => 100
-  # This will trigger message amount resizing in a REST request to LA
-  config :amount_resizing, :validate => :boolean, :default => true
-  # Setting the default amount of messages sent
-  # it this is set with amount_resizing=false --> each message will have max_items
-  config :max_items, :validate => :number, :default => 2000
-  # Setting default proxy to be used for all communication with azure
-  config :proxy, :validate => :string
-  # Setting proxy_aad to be used for communicating with azure active directory service
-  config :proxy_aad, :validate => :string
-  # Setting proxy to be used for the LogAnalytics endpoint REST client
-  config :proxy_endpoint, :validate => :string
-  # This will set the amount of time given for retransmitting messages once sending is failed
-  config :retransmission_time, :validate => :number, :default => 10
-  # Compress the message body before sending to LA
-  config :compress_data, :validate => :boolean, :default => false
-  # Generate sample file from incoming events
-  config :create_sample_file, :validate => :boolean, :default => false
-  # Path where to place the sample file created
-  config :sample_file_path, :validate => :string
-  # Used to specify the name of the Azure cloud that is being used. By default, the value is set to "AzureCloud", which
-  # is the public Azure cloud. However, you can specify a different Azure cloud if you are
-  # using a different environment, such as Azure Government or Azure China.
-  config :azure_cloud, :validate => :string
-  public
-  def register
-    @logstash_configuration= build_logstash_configuration()
-    # Validate configuration correctness
-    @logstash_configuration.validate_configuration()
-    @events_handler = @logstash_configuration.create_sample_file ?
-                        LogStash::Outputs::MicrosoftSentinelOutputInternal::SampleFileCreator::new(@logstash_configuration) :
-                        LogStash::Outputs::MicrosoftSentinelOutputInternal::LogsSender::new(@logstash_configuration)
-  end # def register
-  def multi_receive(events)
-    @events_handler.handle_events(events)
-  end # def multi_receive
-  def close
-    @events_handler.close
-  end
-  #private
-  private
-  # Building the logstash object configuration from the output configuration provided by the user
-  # Return LogstashLoganalyticsOutputConfiguration populated with the configuration values
-  def build_logstash_configuration()
-    logstash_configuration= LogStash::Outputs::MicrosoftSentinelOutputInternal::LogstashLoganalyticsOutputConfiguration::new(@client_app_Id, @client_app_secret, @tenant_id, @data_collection_endpoint, @dcr_immutable_id, @dcr_stream_name, @compress_data, @create_sample_file, @sample_file_path, @logger)
-    logstash_configuration.key_names = @key_names
-    logstash_configuration.plugin_flush_interval = @plugin_flush_interval
-    logstash_configuration.decrease_factor = @decrease_factor
-    logstash_configuration.amount_resizing = @amount_resizing
-    logstash_configuration.max_items = @max_items
-    logstash_configuration.proxy_aad = @proxy_aad || @proxy || ENV['http_proxy']
-    logstash_configuration.proxy_endpoint = @proxy_endpoint || @proxy || ENV['http_proxy']
-    logstash_configuration.retransmission_time = @retransmission_time
-    logstash_configuration.azure_cloud = @azure_cloud || "AzureCloud"
-    return logstash_configuration
-  end # def build_logstash_configuration
-end # class LogStash::Outputs::MicrosoftSentinelOutput
+# encoding: utf-8
+require "logstash/outputs/base"
+require "logstash/namespace"
+require "logstash/sentinel_la/logstashLoganalyticsConfiguration"
+require "logstash/sentinel_la/sampleFileCreator"
+require "logstash/sentinel_la/logsSender"
+class LogStash::Outputs::MicrosoftSentinelOutput < LogStash::Outputs::Base
+  config_name "microsoft-sentinel-log-analytics-logstash-output-plugin"
+  # Stating that the output plugin will run in concurrent mode
+  concurrency :shared
+  # Your registered app ID
+  config :client_app_Id, :validate => :string
+  # The registered app's secret, required by Azure Loganalytics REST API
+  config :client_app_secret, :validate => :string
+  # Your Operations Management Suite Tenant ID
+  config :tenant_id, :validate => :string
+  # Your data collection rule endpoint
+  config :data_collection_endpoint, :validate => :string
+  # Your data collection rule ID
+  config :dcr_immutable_id, :validate => :string
+  # Your dcr data stream name
+  config :dcr_stream_name, :validate => :string
+  # Subset of keys to send to the Azure Loganalytics workspace
+  config :key_names, :validate => :array, :default => []
+  # Max number of seconds to wait between flushes. Default 5
+  config :plugin_flush_interval, :validate => :number, :default => 5
+  # Factor for adding to the amount of messages sent
+  config :decrease_factor, :validate => :number, :default => 100
+  # This will trigger message amount resizing in a REST request to LA
+  config :amount_resizing, :validate => :boolean, :default => true
+  # Setting the default amount of messages sent
+  # it this is set with amount_resizing=false --> each message will have max_items
+  config :max_items, :validate => :number, :default => 2000
+  # Setting default proxy to be used for all communication with azure
+  config :proxy, :validate => :string
+  # Setting proxy_aad to be used for communicating with azure active directory service
+  config :proxy_aad, :validate => :string
+  # Setting proxy to be used for the LogAnalytics endpoint REST client
+  config :proxy_endpoint, :validate => :string
+  # This will set the amount of time given for retransmitting messages once sending is failed
+  config :retransmission_time, :validate => :number, :default => 10
+  # Compress the message body before sending to LA
+  config :compress_data, :validate => :boolean, :default => false
+  # Generate sample file from incoming events
+  config :create_sample_file, :validate => :boolean, :default => false
+  # Path where to place the sample file created
+  config :sample_file_path, :validate => :string
+  # Used to specify the name of the Azure cloud that is being used. By default, the value is set to "AzureCloud", which
+  # is the public Azure cloud. However, you can specify a different Azure cloud if you are
+  # using a different environment, such as Azure Government or Azure China.
+  config :azure_cloud, :validate => :string
+  public
+  def register
+    @logstash_configuration= build_logstash_configuration()
+    # Validate configuration correctness
+    @logstash_configuration.validate_configuration()
+    @events_handler = @logstash_configuration.create_sample_file ?
+                        LogStash::Outputs::MicrosoftSentinelOutputInternal::SampleFileCreator::new(@logstash_configuration) :
+                        LogStash::Outputs::MicrosoftSentinelOutputInternal::LogsSender::new(@logstash_configuration)
+  end # def register
+  def multi_receive(events)
+    @events_handler.handle_events(events)
+  end # def multi_receive
+  def close
+    @events_handler.close
+  end
+  #private
+  private
+  # Building the logstash object configuration from the output configuration provided by the user
+  # Return LogstashLoganalyticsOutputConfiguration populated with the configuration values
+  def build_logstash_configuration()
+    logstash_configuration= LogStash::Outputs::MicrosoftSentinelOutputInternal::LogstashLoganalyticsOutputConfiguration::new(@client_app_Id, @client_app_secret, @tenant_id, @data_collection_endpoint, @dcr_immutable_id, @dcr_stream_name, @compress_data, @create_sample_file, @sample_file_path, @logger)
+    logstash_configuration.key_names = @key_names
+    logstash_configuration.plugin_flush_interval = @plugin_flush_interval
+    logstash_configuration.decrease_factor = @decrease_factor
+    logstash_configuration.amount_resizing = @amount_resizing
+    logstash_configuration.max_items = @max_items
+    logstash_configuration.proxy_aad = @proxy_aad || @proxy || ENV['http_proxy']
+    logstash_configuration.proxy_endpoint = @proxy_endpoint || @proxy || ENV['http_proxy']
+    logstash_configuration.retransmission_time = @retransmission_time
+    logstash_configuration.azure_cloud = @azure_cloud || "AzureCloud"
+    return logstash_configuration
+  end # def build_logstash_configuration
+end # class LogStash::Outputs::MicrosoftSentinelOutput

data/lib/logstash/sentinel_la/logAnalyticsAadTokenProvider.rb CHANGED Viewed

@@ -1,10 +1,10 @@
 # encoding: utf-8
 require "logstash/sentinel_la/logstashLoganalyticsConfiguration"
-require 'rest-client'
 require 'json'
 require 'openssl'
 require 'base64'
 require 'time'
+require 'excon'
 module LogStash; module Outputs; class MicrosoftSentinelOutputInternal
 class LogAnalyticsAadTokenProvider
@@ -64,14 +64,13 @@ class LogAnalyticsAadTokenProvider
     while true
       begin
         # Post REST request
-        response = RestClient::Request.execute(method: :post, url: @token_request_uri, payload: @token_request_body, headers: headers,
-                                              proxy: @logstashLoganalyticsConfiguration.proxy_aad)
-        if (response.code == 200 || response.code == 201)
+        response = Excon.post(@token_request_uri, :body => @token_request_body, :headers => headers, :proxy => @logstashLoganalyticsConfiguration.proxy_aad, expects: [200, 201])
+        if (response.status == 200 || response.status == 201)
           return JSON.parse(response.body)
         end
-      rescue RestClient::ExceptionWithResponse => ewr
-        @logger.error("Exception while authenticating with AAD API ['#{ewr.response}']")
+      rescue Excon::Error::HTTPStatus => ex
+        @logger.error("Error while authenticating with AAD [#{ex.class}: '#{ex.response.status}', Response: '#{ex.response.body}']")
       rescue Exception => ex
         @logger.trace("Exception while authenticating with AAD API ['#{ex}']")
       end

data/lib/logstash/sentinel_la/logAnalyticsClient.rb CHANGED Viewed

@@ -1,11 +1,11 @@
 # encoding: utf-8
 require "logstash/sentinel_la/version"
-require 'rest-client'
 require 'json'
 require 'openssl'
 require 'base64'
 require 'time'
 require 'rbconfig'
+require 'excon'
 module LogStash; module Outputs; class MicrosoftSentinelOutputInternal
 class LogAnalyticsClient
@@ -22,28 +22,78 @@ require "logstash/sentinel_la/logAnalyticsAadTokenProvider"
     @uri = sprintf("%s/dataCollectionRules/%s/streams/%s?api-version=%s",@logstashLoganalyticsConfiguration.data_collection_endpoint, @logstashLoganalyticsConfiguration.dcr_immutable_id, logstashLoganalyticsConfiguration.dcr_stream_name, la_api_version)
     @aadTokenProvider=LogAnalyticsAadTokenProvider::new(logstashLoganalyticsConfiguration)
     @userAgent = getUserAgent()
+    # Auto close connection after 60 seconds of inactivity
+    @connectionAutoClose = {
+      :last_use => Time.now,
+      :lock => Mutex.new,
+      :max_idel_time => 60,
+      :is_closed => true
+    }
+    @timer = Thread.new do
+      loop do
+        sleep @connectionAutoClose[:max_idel_time] / 2
+        if is_connection_stale?
+          @connectionAutoClose[:lock].synchronize do
+            if is_connection_stale?
+              reset_connection
+            end
+          end
+        end
+      end
+    end
   end # def initialize
   # Post the given json to Azure Loganalytics
   def post_data(body)
     raise ConfigError, 'no json_records' if body.empty?
+    response = nil
+    @connectionAutoClose[:lock].synchronize do
+      #close connection if its stale
+      if is_connection_stale?
+        reset_connection
+      end
+      if @connectionAutoClose[:is_closed]
+        open_connection
+      end
+      headers = get_header()
+      # Post REST request
+      response = @connection.request(method: :post, body: body, headers: headers)
+      @connectionAutoClose[:is_closed] = false
+      @connectionAutoClose[:last_use] = Time.now
+    end
+    return response
-    # Create REST request header
-    headers = get_header()
-    # Post REST request
-    return RestClient::Request.execute(method: :post, url: @uri, payload: body, headers: headers,
-                                        proxy: @logstashLoganalyticsConfiguration.proxy_endpoint, timeout: 240)
   end # def post_data
   # Static function to return if the response is OK or else
   def self.is_successfully_posted(response)
-    return (response.code >= 200 && response.code < 300 ) ? true : false
+    return (response.status >= 200 && response.status < 300 ) ? true : false
   end # def self.is_successfully_posted
   private
+  def open_connection
+    @connection = Excon.new(@uri, :persistent => true, :proxy => @logstashLoganalyticsConfiguration.proxy_endpoint,
+          expects: [200, 201, 202, 204, 206, 207, 208, 226, 300, 301, 302, 303, 304, 305, 306, 307, 308],
+          read_timeout: 240, write_timeout: 240, connect_timeout: 240)
+    @logger.trace("Connection to Azure LogAnalytics was opened.");
+  end
+  def reset_connection
+    @connection.reset
+    @connectionAutoClose[:is_closed] = true
+    @logger.trace("Connection to Azure LogAnalytics was closed due to inactivity.");
+  end
+  def is_connection_stale?
+    return Time.now - @connectionAutoClose[:last_use] > @connectionAutoClose[:max_idel_time] && !@connectionAutoClose[:is_closed]
+  end
   # Create a header for the given length
   def get_header()
     # Getting an authorization token bearer (if the token is expired, the method will post a request to get a new authorization token)

data/lib/logstash/sentinel_la/logStashEventsBatcher.rb CHANGED Viewed

@@ -2,7 +2,7 @@
 require "logstash/sentinel_la/logAnalyticsClient"
 require "logstash/sentinel_la/logstashLoganalyticsConfiguration"
+require "excon"
 # LogStashAutoResizeBuffer class setting a resizable buffer which is flushed periodically
 # The buffer resize itself according to Azure Loganalytics  and configuration limitations
 module LogStash; module Outputs; class MicrosoftSentinelOutputInternal
@@ -59,34 +59,35 @@ class LogStashEventsBatcher
                     return
                 else
                     @logger.trace("Rest client response ['#{response}']")
-                    @logger.error("#{api_name} request failed. Error code: #{response.code} #{try_get_info_from_error_response(response)}")
+                    @logger.error("#{api_name} request failed. Error code: #{response.pree} #{try_get_info_from_error_response(response)}")
                 end
-            rescue RestClient::Exceptions::Timeout => eto
-                @logger.trace("Timeout exception ['#{eto.display}'] when posting data to #{api_name}. Rest client response ['#{eto.response.display}']. [amount_of_documents=#{amount_of_documents}]")
-                @logger.error("Timeout exception while posting data to #{api_name}. [Exception: '#{eto}'] [amount of documents=#{amount_of_documents}]'")
-                force_retry = true
+                rescue Excon::Error::HTTPStatus => ewr
+                    response = ewr.response
+                    @logger.trace("Exception in posting data to #{api_name}. Rest client response ['#{response}']. [amount_of_documents=#{amount_of_documents} request payload=#{call_payload}]")
+                    @logger.error("Exception when posting data to #{api_name}. [Exception: '#{ewr.class}'] #{try_get_info_from_error_response(ewr.response)} [amount of documents=#{amount_of_documents}]'")
-            rescue RestClient::ExceptionWithResponse => ewr
-                response = ewr.response
-                @logger.trace("Exception in posting data to #{api_name}. Rest client response ['#{ewr.response}']. [amount_of_documents=#{amount_of_documents} request payload=#{call_payload}]")
-                @logger.error("Exception when posting data to #{api_name}. [Exception: '#{ewr}'] #{try_get_info_from_error_response(ewr.response)} [amount of documents=#{amount_of_documents}]'")
+                    if ewr.class ==  Excon::Error::BadRequest
+                        @logger.info("Not trying to resend since exception http code is 400")
+                        return
+                    elsif ewr.class ==  Excon::Error::RequestTimeout
+                        force_retry = true
+                    elsif ewr.class ==  Excon::Error::TooManyRequests
+                        # throttling detected, backoff before resending
+                        parsed_retry_after = response.data[:headers].include?('Retry-After') ? response.data[:headers]['Retry-After'].to_i : 0
+                        seconds_to_sleep = parsed_retry_after > 0 ? parsed_retry_after : 30
-                if ewr.http_code.to_f == 400
-                    @logger.info("Not trying to resend since exception http code is #{ewr.http_code}")
-                    return
-                elsif ewr.http_code.to_f == 408
+                        #force another retry even if the next iteration of the loop will be after the retransmission_timeout
+                        force_retry = true
+                    end
+                rescue Excon::Error::Socket => ex
+                    @logger.trace("Exception: '#{ex.class.name}]#{ex} in posting data to #{api_name}. [amount_of_documents=#{amount_of_documents}]'")
                     force_retry = true
-                elsif ewr.http_code.to_f == 429
-                    # thrutteling detected, backoff before resending
-                    parsed_retry_after = response.headers.include?(:retry_after) ? response.headers[:retry_after].to_i : 0
-                    seconds_to_sleep = parsed_retry_after > 0 ? parsed_retry_after : 30
-                    #force another retry even if the next iteration of the loop will be after the retransmission_timeout
+                rescue Excon::Error::Timeout => ex
+                    @logger.trace("Exception: '#{ex.class.name}]#{ex} in posting data to #{api_name}. [amount_of_documents=#{amount_of_documents}]'")
                     force_retry = true
-                end
-            rescue Exception => ex
-                @logger.trace("Exception in posting data to #{api_name}.[amount_of_documents=#{amount_of_documents} request payload=#{call_payload}]")
-                @logger.error("Exception in posting data to #{api_name}. [Exception: '#{ex}, amount of documents=#{amount_of_documents}]'")
+                rescue Exception => ex
+                    @logger.trace("Exception in posting data to #{api_name}.[amount_of_documents=#{amount_of_documents} request payload=#{call_payload}]")
+                    @logger.error("Exception in posting data to #{api_name}. [Exception: '[#{ex.class.name}]#{ex}, amount of documents=#{amount_of_documents}]'")
             end
             is_retry = true
             @logger.info("Retrying transmission to #{api_name} in #{seconds_to_sleep} seconds.")
@@ -110,8 +111,8 @@ class LogStashEventsBatcher
     def get_request_id_from_response(response)
         output =""
         begin
-            if !response.nil? && response.headers.include?(:x_ms_request_id)
-                output += response.headers[:x_ms_request_id]
+            if !response.nil? && response.data[:headers].include?("x-ms-request-id")
+                output += response.data[:headers]["x-ms-request-id"]
             end
         rescue Exception => ex
             @logger.debug("Error while getting reqeust id from success response headers: #{ex.display}")
@@ -124,12 +125,13 @@ class LogStashEventsBatcher
         begin
             output = ""
             if !response.nil?
-                if response.headers.include?(:x_ms_error_code)
-                    output += " [ms-error-code header: #{response.headers[:x_ms_error_code]}]"
+                if response.data[:headers].include?("x-ms-error-code")
+                    output += " [ms-error-code header: #{response.data[:headers]["x-ms-error-code"]}]"
             end
-            if response.headers.include?(:x_ms_request_id)
-                output += " [x-ms-request-id header: #{response.headers[:x_ms_request_id]}]"
+            if response.data[:headers].include?("x-ms-request-id")
+                output += " [x-ms-request-id header: #{response.data[:headers]["x-ms-request-id"]}]"
             end
+            output += " [response body: #{response.data[:body]}]"
         end
             return output
         rescue Exception => ex

data/lib/logstash/sentinel_la/version.rb CHANGED Viewed

@@ -1,10 +1,10 @@
 module LogStash; module Outputs;
 class MicrosoftSentinelOutputInternal
-  VERSION_INFO = [1, 1, 1].freeze
+  VERSION_INFO = [1, 1, 4].freeze
   VERSION = VERSION_INFO.map(&:to_s).join('.').freeze
   def self.version
     VERSION
   end
 end
-end;end
+end;end

data/microsoft-sentinel-log-analytics-logstash-output-plugin.gemspec CHANGED Viewed

@@ -1,27 +1,27 @@
-require File.expand_path('../lib/logstash/sentinel_la/version', __FILE__)
-Gem::Specification.new do |s|
-  s.name = 'microsoft-sentinel-log-analytics-logstash-output-plugin'
-  s.version = LogStash::Outputs::MicrosoftSentinelOutputInternal::VERSION
-  s.authors = ["Microsoft Sentinel"]
-  s.email   = 'AzureSentinel@microsoft.com'
-  s.summary = %q{Microsoft Sentinel provides a new output plugin for Logstash. Use this output plugin to send any log via Logstash to the Microsoft Sentinel/Log Analytics workspace. This is done with the Log Analytics DCR-based API.}
-  s.description = s.summary
-  s.homepage = "https://github.com/Azure/Azure-Sentinel"
-  s.licenses = ["MIT"]
-  s.require_paths = ["lib"]
-  # Files
-  s.files = Dir['lib/**/*','spec/**/*','vendor/**/*','*.gemspec','*.md','CONTRIBUTORS','Gemfile','LICENSE','NOTICE.TXT']
-   # Tests
-  s.test_files = s.files.grep(%r{^(test|spec|features)/})
-  # Special flag to let us know this is actually a logstash plugin
-  s.metadata = { "logstash_plugin" => "true", "logstash_group" => "output" }
-  # Gem dependencies
-  s.add_runtime_dependency "rest-client", ">= 2.1.0"
-  s.add_runtime_dependency "logstash-core-plugin-api", ">= 1.60", "<= 2.99"
-  s.add_runtime_dependency "logstash-codec-plain"
-  s.add_development_dependency "logstash-devutils"
-end
+require File.expand_path('../lib/logstash/sentinel_la/version', __FILE__)
+Gem::Specification.new do |s|
+  s.name = 'microsoft-sentinel-log-analytics-logstash-output-plugin'
+  s.version = LogStash::Outputs::MicrosoftSentinelOutputInternal::VERSION
+  s.authors = ["Microsoft Sentinel"]
+  s.email   = 'AzureSentinel@microsoft.com'
+  s.summary = %q{Microsoft Sentinel provides a new output plugin for Logstash. Use this output plugin to send any log via Logstash to the Microsoft Sentinel/Log Analytics workspace. This is done with the Log Analytics DCR-based API.}
+  s.description = s.summary
+  s.homepage = "https://github.com/Azure/Azure-Sentinel"
+  s.licenses = ["MIT"]
+  s.require_paths = ["lib"]
+  # Files
+  s.files = Dir['lib/**/*','spec/**/*','vendor/**/*','*.gemspec','*.md','CONTRIBUTORS','Gemfile','LICENSE','NOTICE.TXT']
+   # Tests
+  s.test_files = s.files.grep(%r{^(test|spec|features)/})
+  # Special flag to let us know this is actually a logstash plugin
+  s.metadata = { "logstash_plugin" => "true", "logstash_group" => "output" }
+  # Gem dependencies
+  s.add_runtime_dependency "logstash-core-plugin-api", ">= 1.60", "<= 2.99"
+  s.add_runtime_dependency "logstash-codec-plain"
+  s.add_runtime_dependency "excon", ">= 0.88.0", "< 1.0.0"
+  s.add_development_dependency "logstash-devutils"
+end

metadata CHANGED Viewed

@@ -1,70 +1,76 @@
 --- !ruby/object:Gem::Specification
 name: microsoft-sentinel-log-analytics-logstash-output-plugin
 version: !ruby/object:Gem::Version
-  version: 1.1.1
+  version: 1.1.4
 platform: ruby
 authors:
 - Microsoft Sentinel
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2024-01-17 00:00:00.000000000 Z
+date: 2024-12-10 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
-  name: rest-client
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: 2.1.0
+        version: '1.60'
+    - - "<="
+      - !ruby/object:Gem::Version
+        version: '2.99'
+  name: logstash-core-plugin-api
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: 2.1.0
+        version: '1.60'
+    - - "<="
+      - !ruby/object:Gem::Version
+        version: '2.99'
 - !ruby/object:Gem::Dependency
-  name: logstash-core-plugin-api
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: '1.60'
-    - - "<="
-      - !ruby/object:Gem::Version
-        version: '2.99'
+        version: '0'
+  name: logstash-codec-plain
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: '1.60'
-    - - "<="
-      - !ruby/object:Gem::Version
-        version: '2.99'
+        version: '0'
 - !ruby/object:Gem::Dependency
-  name: logstash-codec-plain
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: '0'
+        version: 0.88.0
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: 1.0.0
+  name: excon
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
-        version: '0'
+        version: 0.88.0
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: 1.0.0
 - !ruby/object:Gem::Dependency
-  name: logstash-devutils
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
+  name: logstash-devutils
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement