fluent-plugin-kusto 0.0.1.beta
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +7 -0
- data/Gemfile +8 -0
- data/LICENSE +21 -0
- data/README.md +201 -0
- data/lib/fluent/plugin/auth/aad_tokenprovider.rb +105 -0
- data/lib/fluent/plugin/auth/azcli_tokenprovider.rb +51 -0
- data/lib/fluent/plugin/auth/mi_tokenprovider.rb +92 -0
- data/lib/fluent/plugin/auth/tokenprovider_base.rb +57 -0
- data/lib/fluent/plugin/auth/wif_tokenprovider.rb +50 -0
- data/lib/fluent/plugin/client.rb +155 -0
- data/lib/fluent/plugin/conffile.rb +155 -0
- data/lib/fluent/plugin/ingester.rb +136 -0
- data/lib/fluent/plugin/kusto_error_handler.rb +126 -0
- data/lib/fluent/plugin/kusto_query.rb +67 -0
- data/lib/fluent/plugin/out_kusto.rb +423 -0
- data/test/helper.rb +9 -0
- data/test/plugin/test_azcli_tokenprovider.rb +37 -0
- data/test/plugin/test_e2e_kusto.rb +683 -0
- data/test/plugin/test_out_kusto_config.rb +86 -0
- data/test/plugin/test_out_kusto_format.rb +280 -0
- data/test/plugin/test_out_kusto_process.rb +150 -0
- data/test/plugin/test_out_kusto_start.rb +429 -0
- data/test/plugin/test_out_kusto_try_write.rb +382 -0
- data/test/plugin/test_out_kusto_write.rb +370 -0
- metadata +171 -0
checksums.yaml
ADDED
@@ -0,0 +1,7 @@
|
|
1
|
+
---
|
2
|
+
SHA256:
|
3
|
+
metadata.gz: 90834d37833dc31d3a1aaa3f3de064ee260f9c619753857be060c2e48ba4a79c
|
4
|
+
data.tar.gz: 854001eaf38c3065262d2fd4abdb20c14e95af65ba8f08418825ddf6ac56ad94
|
5
|
+
SHA512:
|
6
|
+
metadata.gz: 66e9089d652413b6a405dc49fafa2fb3fe5399461dd18fdd87025293790dc8d92c9fcf4b8122937f2c06ec766ea43a48716dce499a61c6d298ad03f23299d5c0
|
7
|
+
data.tar.gz: 93e5fb41d9e4a76a5a34bad266abf596161f24143b1763d27a3378cf7f22afb87601f0a23b80775bb5cb75dc52b9876f6077f35ba59ae6dd3f19182790e32e68
|
data/Gemfile
ADDED
data/LICENSE
ADDED
@@ -0,0 +1,21 @@
|
|
1
|
+
MIT License
|
2
|
+
|
3
|
+
Copyright (c) Microsoft Corporation.
|
4
|
+
|
5
|
+
Permission is hereby granted, free of charge, to any person obtaining a copy
|
6
|
+
of this software and associated documentation files (the "Software"), to deal
|
7
|
+
in the Software without restriction, including without limitation the rights
|
8
|
+
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
|
9
|
+
copies of the Software, and to permit persons to whom the Software is
|
10
|
+
furnished to do so, subject to the following conditions:
|
11
|
+
|
12
|
+
The above copyright notice and this permission notice shall be included in all
|
13
|
+
copies or substantial portions of the Software.
|
14
|
+
|
15
|
+
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
|
16
|
+
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
|
17
|
+
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
|
18
|
+
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
|
19
|
+
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
|
20
|
+
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
|
21
|
+
SOFTWARE
|
data/README.md
ADDED
@@ -0,0 +1,201 @@
|
|
1
|
+
# fluent-plugin-kusto
|
2
|
+
|
3
|
+
[Fluentd](https://fluentd.org/) output plugin for ingesting logs and data into [Azure Data Explorer (Kusto)](https://azure.microsoft.com/en-us/services/data-explorer/).
|
4
|
+
|
5
|
+
## Overview
|
6
|
+
|
7
|
+
This plugin allows you to send data from Fluentd to Azure Data Explorer (Kusto) using Azure Blob Storage and Queue for scalable, reliable ingestion. It supports both buffered and non-buffered modes, handles authentication via Azure AD or Managed Identity, and provides robust error handling and logging.
|
8
|
+
|
9
|
+
## Requirements
|
10
|
+
|
11
|
+
- Ruby 2.5 or later
|
12
|
+
Check version (Windows/Linux):
|
13
|
+
```bash
|
14
|
+
ruby --version
|
15
|
+
```
|
16
|
+
Install on Ubuntu/Linux:
|
17
|
+
```bash
|
18
|
+
sudo apt-get install ruby-full
|
19
|
+
```
|
20
|
+
Install on Windows (using RubyInstaller):
|
21
|
+
[Download RubyInstaller](https://rubyinstaller.org/)
|
22
|
+
[Official Ruby installation guide](https://www.ruby-lang.org/en/documentation/installation/)
|
23
|
+
|
24
|
+
- Fluentd v1.0 or later
|
25
|
+
Check version (Windows/Linux):
|
26
|
+
```bash
|
27
|
+
fluentd --version
|
28
|
+
```
|
29
|
+
Install on Ubuntu/Linux:
|
30
|
+
```bash
|
31
|
+
gem install fluentd
|
32
|
+
```
|
33
|
+
Install on Windows (in Command Prompt after Ruby is installed):
|
34
|
+
```cmd
|
35
|
+
gem install fluentd
|
36
|
+
```
|
37
|
+
[Official Fluentd installation guide](https://docs.fluentd.org/installation)
|
38
|
+
|
39
|
+
|
40
|
+
## Installation
|
41
|
+
|
42
|
+
### RubyGems
|
43
|
+
|
44
|
+
```sh
|
45
|
+
$ gem install fluent-plugin-kusto --pre
|
46
|
+
```
|
47
|
+
|
48
|
+
### Bundler
|
49
|
+
|
50
|
+
Add the following line to your Gemfile:
|
51
|
+
|
52
|
+
```ruby
|
53
|
+
gem "fluent-plugin-kusto", "~> 0.0.1.beta"
|
54
|
+
```
|
55
|
+
|
56
|
+
**Note:** This is a beta release. Use the `--pre` flag with gem install or specify the beta version in your Gemfile.
|
57
|
+
|
58
|
+
And then execute:
|
59
|
+
|
60
|
+
```sh
|
61
|
+
$ bundle
|
62
|
+
```
|
63
|
+
|
64
|
+
## Azure Data Explorer (Kusto) Prerequisites
|
65
|
+
The _Kusto_ output plugin lets you ingest your logs into an [Azure Data Explorer](https://azure.microsoft.com/en-us/services/data-explorer/) cluster, using the [Queued Ingestion](https://docs.microsoft.com/en-us/azure/data-explorer/kusto/api/netfx/about-kusto-ingest#queued-ingestion) mechanism.
|
66
|
+
|
67
|
+
## Ingest into Azure Data Explorer: create a Kusto cluster and database
|
68
|
+
|
69
|
+
Create an Azure Data Explorer cluster in one of the following ways:
|
70
|
+
|
71
|
+
* [Create a free-tier cluster](https://dataexplorer.azure.com/freecluster)
|
72
|
+
* [Create a fully featured cluster](https://docs.microsoft.com/en-us/azure/data-explorer/create-cluster-database-portal)
|
73
|
+
|
74
|
+
## Create an Azure registered application
|
75
|
+
|
76
|
+
FluentD uses the Azure application's credentials to ingest data into your cluster.
|
77
|
+
|
78
|
+
* [Register an application](https://docs.microsoft.com/en-us/azure/active-directory/develop/quickstart-register-app#register-an-application)
|
79
|
+
* [Add a client secret](https://docs.microsoft.com/en-us/azure/active-directory/develop/quickstart-register-app#add-a-client-secret)
|
80
|
+
* [Authorize the app in your database](https://docs.microsoft.com/en-us/azure/data-explorer/kusto/management/access-control/principals-and-identity-providers#azure-ad-tenants)
|
81
|
+
|
82
|
+
## Create a Managed Identity in Azure
|
83
|
+
|
84
|
+
- **System-assigned Managed Identity:**
|
85
|
+
- [Enable system-assigned managed identity](https://learn.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/how-to-enable-system-assigned-managed-identity)
|
86
|
+
|
87
|
+
- **User-assigned Managed Identity:**
|
88
|
+
- [Create and assign user-assigned managed identity](https://learn.microsoft.com/en-us/azure/active-directory/managed-identities-azure-resources/how-to-create-user-assigned-managed-identity)
|
89
|
+
|
90
|
+
- **Grant Permissions:**
|
91
|
+
- [Assign permissions to managed identity for Azure Data Explorer](https://learn.microsoft.com/en-us/azure/data-explorer/kusto/management/access-control/principals-and-identity-providers)
|
92
|
+
|
93
|
+
## Workload Identity Authentication
|
94
|
+
|
95
|
+
- [Follow workload_identity.md](workload_identity.md)
|
96
|
+
|
97
|
+
## Create a database
|
98
|
+
|
99
|
+
To create a new database in your Azure Data Explorer cluster, use the following KQL command:
|
100
|
+
|
101
|
+
```kql
|
102
|
+
.create database <database_name>
|
103
|
+
```
|
104
|
+
|
105
|
+
## Create a table
|
106
|
+
|
107
|
+
Fluent Bit ingests the event data into Kusto in a JSON format. By default, the table includes 3 properties:
|
108
|
+
|
109
|
+
* `record` - the actual event payload.
|
110
|
+
* `tag` - the event tag.
|
111
|
+
* `timestamp` - the event timestamp.
|
112
|
+
|
113
|
+
A table with the expected schema must exist in order for data to be ingested properly.
|
114
|
+
|
115
|
+
```kql
|
116
|
+
.create table <table_name> (tag:string, timestamp:datetime, record:dynamic)
|
117
|
+
```
|
118
|
+
|
119
|
+
## Configuration parameters
|
120
|
+
|
121
|
+
| Key | Description | Default |
|
122
|
+
| -------------------------------------- | ----------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- | ------------------------------ |
|
123
|
+
| `tenant_id` | The tenant/domain ID of the Azure Active Directory (AAD) registered application. Required if `managed_identity_client_id` isn't set. | _none_ |
|
124
|
+
| `client_id` | The client ID of the AAD registered application. Required if `managed_identity_client_id` isn't set. | _none_ |
|
125
|
+
| `client_secret` | The client secret of the AAD registered application ([App Secret](https://docs.microsoft.com/en-us/azure/active-directory/develop/howto-create-service-principal-portal#option-2-create-a-new-application-secret)). Required if `managed_identity_client_id` isn't set. | _none_ |
|
126
|
+
| `managed_identity_client_id` | The managed identity ID to authenticate with. Set to `SYSTEM` for system-assigned managed identity, or set to the MI client ID (`GUID`) for user-assigned managed identity. Required if `tenant_id`, `client_id`, and `client_secret` aren't set. | _none_ |
|
127
|
+
| `endpoint` | The cluster's endpoint, usually in the form `https://cluster_name.region.kusto.windows.net` | _none_ |
|
128
|
+
| `database_name` | The database name. | _none_ |
|
129
|
+
| `table_name` | The table name. | _none_ |
|
130
|
+
| `compression_enabled` | If enabled, sends compressed HTTP payload (gzip) to Kusto. | `true` |
|
131
|
+
| `workers` | The number of [workers](../../../administration/multithreading#outputs) to perform flush operations for this output. | `0` |
|
132
|
+
| `buffered` | Enable buffering into disk before ingesting into Azure Kusto. If `buffered` is `true`, buffered mode is activated. If `false`, non-buffered mode is used. | `true` |
|
133
|
+
| `delayed` | If `true`, enables delayed commit for buffer chunks. Only supported in buffered mode (`buffered` must be `true`). If `buffered` is `false`, delayed commit is not available. | `false` |
|
134
|
+
| `azure_cloud` | Azure cloud environment. E.g., `AzureCloud`, `AzureChinaCloud`, `AzureUSGovernmentCloud`, `AzureGermanCloud`. | `AzureCloud` |
|
135
|
+
| `chunk_keys` (buffer section) | Only in buffered mode. Keys to use for chunking the buffer. Possible values: `tag`, `time`, or a combination such as `["tag", "time"]`. Controls how data is grouped and flushed. | `["time"]` |
|
136
|
+
| `timekey` (buffer section) | Only in buffered mode. Time interval for buffer chunking. Possible values: integer seconds (e.g., `60`, `3600`, `86400`). | `86400` (1 day) |
|
137
|
+
| `timekey_wait` (buffer section) | Only in buffered mode. Wait time before flushing a timekey chunk after its time window closes. Possible values: duration string (e.g., `30s`, `5m`). | `30s` |
|
138
|
+
| `timekey_use_utc` (buffer section) | Only in buffered mode. Use UTC for timekey chunking. Possible values: `true`, `false`. | `true` |
|
139
|
+
| `flush_at_shutdown` (buffer section) | Only in buffered mode. Flush buffer at shutdown. Possible values: `true`, `false`. | `true` |
|
140
|
+
| `retry_max_times` (buffer section) | Only in buffered mode. Maximum number of retry attempts for buffer flush. Possible values: integer (e.g., `5`, `10`). | `5` |
|
141
|
+
| `retry_wait` (buffer section) | Only in buffered mode. Wait time between buffer flush retries. Possible values: duration string (e.g., `1s`, `10s`). | `1s` |
|
142
|
+
| `overflow_action` (buffer section) | Only in buffered mode. Action to take when buffer overflows. Possible values: `block`, `drop_oldest_chunk`, `throw_exception`. | `block` |
|
143
|
+
| `chunk_limit_size` (buffer section) | Only in buffered mode. Maximum size per buffer chunk. Possible values: size string (e.g., `256m`, `1g`). | `256m` |
|
144
|
+
| `total_limit_size` (buffer section) | Only in buffered mode. Maximum total buffer size. Possible values: size string (e.g., `2g`, `10g`). | `2g` |
|
145
|
+
| `flush_mode` (buffer section) | Only in buffered mode. Buffer flush mode. Possible values: `interval`, `immediate`, `lazy`. | `interval` |
|
146
|
+
| `flush_interval` (buffer section) | Only in buffered mode. Interval for buffer flush. Possible values: duration string (e.g., `10s`, `1m`). | `10s` |
|
147
|
+
| `logger_path` | Optional. File path for plugin log output. If not set, logs are written to stdout. | stdout(terminal) |
|
148
|
+
| `auth_type` | The authentication type to use. Possible values: `aad`, `user_managed_identity`, `system_managed_identity`,`workload_identity`. | `aad` |
|
149
|
+
| `workload_identity_client_id` | The client ID for Azure Workload Identity authentication. Required if using workload identity for authentication. | _none_ |
|
150
|
+
| `workload_identity_tenant_id` | The tenant ID for Azure Workload Identity authentication. Required if using workload identity for authentication. | _none_ |
|
151
|
+
| `workload_identity_token_file` | The file path to the token file for Azure Workload Identity authentication. Required if using workload identity for authentication. | `/var/run/secrets/azure/tokens/azure-identity-token` |
|
152
|
+
|
153
|
+
## Sample Configuration
|
154
|
+
|
155
|
+
```conf
|
156
|
+
<system>
|
157
|
+
workers 1
|
158
|
+
</system>
|
159
|
+
<match test.kusto>
|
160
|
+
@type kusto
|
161
|
+
@log_level debug
|
162
|
+
buffered true
|
163
|
+
delayed false
|
164
|
+
endpoint https://yourcluster.region.kusto.windows.net
|
165
|
+
database_name your-db
|
166
|
+
table_name your-table
|
167
|
+
tenant_id <your-tenant-id>
|
168
|
+
client_id <your-client-id>
|
169
|
+
managed_identity_client_id SYSTEM
|
170
|
+
compression_enabled true
|
171
|
+
azure_cloud AzureCloud
|
172
|
+
logger_path /var/log/azure-kusto-fluentd.log
|
173
|
+
<buffer>
|
174
|
+
@type memory
|
175
|
+
# To chunk by tag only:
|
176
|
+
# chunk_keys tag
|
177
|
+
# To chunk by tag and time:
|
178
|
+
# chunk_keys tag,time
|
179
|
+
timekey 1m
|
180
|
+
timekey_wait 30s
|
181
|
+
timekey_use_utc true
|
182
|
+
flush_at_shutdown true
|
183
|
+
retry_max_times 5
|
184
|
+
retry_wait 1s
|
185
|
+
overflow_action block
|
186
|
+
chunk_limit_size 256m
|
187
|
+
total_limit_size 2g
|
188
|
+
flush_mode interval
|
189
|
+
flush_interval 10s
|
190
|
+
</buffer>
|
191
|
+
</match>
|
192
|
+
```
|
193
|
+
|
194
|
+
# Fluentd Azure Data Explorer (Kusto) Output Plugin Architecture
|
195
|
+

|
196
|
+
|
197
|
+
This diagram shows the main components and data flow for the plugin, including configuration, error handling, token management, and Azure resource interactions.
|
198
|
+
|
199
|
+
## Copyright
|
200
|
+
|
201
|
+
* License: Apache License, Version 2.0
|
@@ -0,0 +1,105 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
# AadTokenProvider handles acquiring and refreshing Azure Active Directory tokens for Kusto ingestion.
|
4
|
+
#
|
5
|
+
# Responsibilities:
|
6
|
+
# - Build and send token requests to Azure AD endpoint
|
7
|
+
# - Cache and refresh tokens as needed
|
8
|
+
# - Support client credentials flow for authentication
|
9
|
+
require 'json'
|
10
|
+
require 'openssl'
|
11
|
+
require 'base64'
|
12
|
+
require 'time'
|
13
|
+
require 'net/http'
|
14
|
+
require 'uri'
|
15
|
+
require_relative '../kusto_error_handler'
|
16
|
+
require_relative 'tokenprovider_base'
|
17
|
+
|
18
|
+
class AadTokenProvider < AbstractTokenProvider
|
19
|
+
def initialize(outconfiguration)
|
20
|
+
super(outconfiguration)
|
21
|
+
token_request_params_set
|
22
|
+
end
|
23
|
+
|
24
|
+
# Use get_token from base class for token retrieval
|
25
|
+
|
26
|
+
private
|
27
|
+
|
28
|
+
def setup_config(outconfiguration)
|
29
|
+
@client_id = outconfiguration.client_app_id
|
30
|
+
@client_secret = outconfiguration.client_app_secret
|
31
|
+
@tenant_id = outconfiguration.tenant_id
|
32
|
+
@aad_uri = outconfiguration.aad_endpoint
|
33
|
+
@resource = outconfiguration.kusto_endpoint
|
34
|
+
@database_name = outconfiguration.database_name
|
35
|
+
@table_name = outconfiguration.table_name
|
36
|
+
@azure_cloud = outconfiguration.azure_cloud
|
37
|
+
@managed_identity_client_id = outconfiguration.managed_identity_client_id
|
38
|
+
end
|
39
|
+
|
40
|
+
def token_request_params_set
|
41
|
+
@token_request_uri = "#{@aad_uri}/#{@tenant_id}/oauth2/v2.0/token"
|
42
|
+
@scope = "#{@resource}/.default"
|
43
|
+
end
|
44
|
+
|
45
|
+
def fetch_token
|
46
|
+
response = post_token_request
|
47
|
+
{
|
48
|
+
access_token: response['access_token'],
|
49
|
+
expires_in: response['expires_in']
|
50
|
+
}
|
51
|
+
end
|
52
|
+
|
53
|
+
def post_token_request
|
54
|
+
headers = header
|
55
|
+
max_retries = 10
|
56
|
+
retries = 0
|
57
|
+
uri = URI.parse(@token_request_uri)
|
58
|
+
form_data = URI.encode_www_form(
|
59
|
+
'grant_type' => 'client_credentials',
|
60
|
+
'client_id' => @client_id,
|
61
|
+
'client_secret' => @client_secret,
|
62
|
+
'scope' => @scope
|
63
|
+
)
|
64
|
+
while retries < max_retries
|
65
|
+
begin
|
66
|
+
http = Net::HTTP.new(uri.host, uri.port)
|
67
|
+
http.use_ssl = (uri.scheme == 'https')
|
68
|
+
request = Net::HTTP::Post.new(uri.request_uri, headers)
|
69
|
+
request.body = form_data
|
70
|
+
|
71
|
+
response = http.request(request)
|
72
|
+
return JSON.parse(response.body) if [200, 201].include?(response.code.to_i)
|
73
|
+
|
74
|
+
begin
|
75
|
+
error_json = JSON.parse(response.body)
|
76
|
+
kusto_error_type = KustoErrorHandler.extract_kusto_error_type(error_json)
|
77
|
+
error = KustoErrorHandler.from_kusto_error_type(
|
78
|
+
kusto_error_type,
|
79
|
+
error_json['error_description'] || error_json['message'] || response.body
|
80
|
+
)
|
81
|
+
if error.permanent_error?
|
82
|
+
@logger.error("Permanent error encountered, not retrying. #{error.message}")
|
83
|
+
raise error
|
84
|
+
end
|
85
|
+
rescue JSON::ParserError
|
86
|
+
@logger.error("Failed to parse error response: #{response.body}")
|
87
|
+
raise "Permanent error while authenticating with AAD: #{response.body}"
|
88
|
+
end
|
89
|
+
end
|
90
|
+
retries += 1
|
91
|
+
@logger.error(
|
92
|
+
"Error while authenticating with AAD ('#{@aad_uri}'), retrying in 10 seconds. " \
|
93
|
+
"Attempt #{retries}/#{max_retries}"
|
94
|
+
)
|
95
|
+
sleep 10
|
96
|
+
end
|
97
|
+
raise "Failed to authenticate with AAD after #{max_retries} attempts."
|
98
|
+
end
|
99
|
+
|
100
|
+
def header
|
101
|
+
{
|
102
|
+
'Content-Type' => 'application/x-www-form-urlencoded'
|
103
|
+
}
|
104
|
+
end
|
105
|
+
end
|
@@ -0,0 +1,51 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
require 'json'
|
4
|
+
require 'time'
|
5
|
+
require 'open3'
|
6
|
+
require 'shellwords'
|
7
|
+
require_relative 'tokenprovider_base'
|
8
|
+
|
9
|
+
class AzCliTokenProvider < AbstractTokenProvider
|
10
|
+
def initialize(outconfiguration)
|
11
|
+
super(outconfiguration)
|
12
|
+
@resource = outconfiguration.kusto_endpoint
|
13
|
+
end
|
14
|
+
|
15
|
+
# Use get_token from base class for token retrieval
|
16
|
+
|
17
|
+
def fetch_token
|
18
|
+
token = acquire_token(@resource)
|
19
|
+
raise "No valid Azure CLI token found for resource: #{@resource}" unless token
|
20
|
+
|
21
|
+
{
|
22
|
+
access_token: token['accessToken'],
|
23
|
+
expires_in: (Time.parse(token['expiresOn']) - Time.now).to_i
|
24
|
+
}
|
25
|
+
end
|
26
|
+
|
27
|
+
def acquire_token(resource)
|
28
|
+
az_cli = locate_azure_cli
|
29
|
+
# Properly escape the Azure CLI executable path to prevent command injection
|
30
|
+
escaped_az_cli = Shellwords.escape(az_cli)
|
31
|
+
escaped_resource = Shellwords.escape(resource)
|
32
|
+
cmd = [escaped_az_cli, 'account', 'get-access-token', '--resource', escaped_resource, '--output', 'json']
|
33
|
+
stdout, stderr, status = Open3.capture3(*cmd)
|
34
|
+
raise "Failed to acquire Azure CLI token: #{stderr.strip}" if !status.success? && !status.success?
|
35
|
+
|
36
|
+
JSON.parse(stdout)
|
37
|
+
rescue Errno::ENOENT
|
38
|
+
raise "Azure CLI not found. Please install Azure CLI and run 'az login'."
|
39
|
+
end
|
40
|
+
|
41
|
+
def locate_azure_cli
|
42
|
+
exts = ENV['PATHEXT'] ? ENV['PATHEXT'].split(';') : ['']
|
43
|
+
ENV['PATH'].split(File::PATH_SEPARATOR).each do |path|
|
44
|
+
exts.each do |ext|
|
45
|
+
exe = File.join(path, "az#{ext}")
|
46
|
+
return exe if File.executable?(exe) && !File.directory?(exe)
|
47
|
+
end
|
48
|
+
end
|
49
|
+
raise "Azure CLI executable 'az' not found in PATH."
|
50
|
+
end
|
51
|
+
end
|
@@ -0,0 +1,92 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
# ManagedIdentityTokenProvider handles acquiring and refreshing Azure Managed Identity tokens for Kusto ingestion.
|
4
|
+
#
|
5
|
+
# Responsibilities:
|
6
|
+
# - Build and send token requests to Azure IMDS endpoint
|
7
|
+
# - Cache and refresh tokens as needed
|
8
|
+
# - Support both system-assigned and user-assigned managed identities
|
9
|
+
|
10
|
+
require 'net/http'
|
11
|
+
require 'uri'
|
12
|
+
require 'json'
|
13
|
+
require 'erb'
|
14
|
+
require_relative 'tokenprovider_base'
|
15
|
+
|
16
|
+
class ManagedIdentityTokenProvider < AbstractTokenProvider
|
17
|
+
IMDS_TOKEN_ACQUIRE_URL = 'http://169.254.169.254/metadata/identity/oauth2/token'
|
18
|
+
|
19
|
+
def initialize(outconfiguration)
|
20
|
+
super(outconfiguration)
|
21
|
+
token_request_params_set(outconfiguration)
|
22
|
+
end
|
23
|
+
|
24
|
+
# Use get_token from base class for token retrieval
|
25
|
+
|
26
|
+
private
|
27
|
+
|
28
|
+
def setup_config(outconfiguration)
|
29
|
+
@resource = outconfiguration.kusto_endpoint
|
30
|
+
@managed_identity_client_id = outconfiguration.managed_identity_client_id
|
31
|
+
val = @managed_identity_client_id.to_s.strip
|
32
|
+
@use_system_assigned = (val.upcase == 'SYSTEM')
|
33
|
+
@use_user_assigned = !val.empty? && val.upcase != 'SYSTEM'
|
34
|
+
end
|
35
|
+
|
36
|
+
def append_header(name, value)
|
37
|
+
"#{name}=#{value}"
|
38
|
+
end
|
39
|
+
|
40
|
+
def token_request_params_set(_outconfiguration)
|
41
|
+
token_acquire_url = IMDS_TOKEN_ACQUIRE_URL.dup + '?' + append_header('resource',
|
42
|
+
ERB::Util.url_encode(outconfiguration.kusto_endpoint)) + '&' + append_header(
|
43
|
+
'api-version', '2018-02-01'
|
44
|
+
)
|
45
|
+
unless @object_id.nil?
|
46
|
+
token_acquire_url = (token_acquire_url + '&' + append_header('object_id',
|
47
|
+
ERB::Util.url_encode(@object_id)))
|
48
|
+
end
|
49
|
+
unless @msi_res_id.nil?
|
50
|
+
token_acquire_url = (token_acquire_url + '&' + append_header('msi_res_id',
|
51
|
+
ERB::Util.url_encode(@msi_res_id)))
|
52
|
+
end
|
53
|
+
URI.parse(token_acquire_url)
|
54
|
+
return unless @use_user_assigned
|
55
|
+
|
56
|
+
(token_acquire_url + '&' + append_header('client_id',
|
57
|
+
ERB::Util.url_encode(@managed_identity_client_id)))
|
58
|
+
end
|
59
|
+
|
60
|
+
def fetch_token
|
61
|
+
response = post_token_request
|
62
|
+
{
|
63
|
+
access_token: response['access_token'],
|
64
|
+
expires_in: response['expires_in']
|
65
|
+
}
|
66
|
+
end
|
67
|
+
|
68
|
+
def post_token_request
|
69
|
+
headers = { 'Metadata' => 'true' }
|
70
|
+
max_retries = 2
|
71
|
+
retries = 0
|
72
|
+
uri = URI.parse(@token_acquire_url)
|
73
|
+
while retries < max_retries
|
74
|
+
begin
|
75
|
+
http = Net::HTTP.new(uri.host, uri.port)
|
76
|
+
request = Net::HTTP::Get.new(uri.request_uri, headers)
|
77
|
+
response = http.request(request)
|
78
|
+
return JSON.parse(response.body) if response.code.to_i == 200
|
79
|
+
|
80
|
+
@logger.error("Failed to get managed identity token: #{response.code} #{response.body}")
|
81
|
+
rescue StandardError => e
|
82
|
+
@logger.error("Error while requesting managed identity token: #{e.message}")
|
83
|
+
end
|
84
|
+
retries += 1
|
85
|
+
@logger.error(
|
86
|
+
"Retrying managed identity token request in 10 seconds. Attempt #{retries}/#{max_retries}"
|
87
|
+
)
|
88
|
+
sleep 10
|
89
|
+
end
|
90
|
+
raise "Failed to get managed identity token after #{max_retries} attempts."
|
91
|
+
end
|
92
|
+
end
|
@@ -0,0 +1,57 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
require 'logger'
|
4
|
+
|
5
|
+
# AbstractTokenProvider defines the interface and shared logic for all token providers.
|
6
|
+
class AbstractTokenProvider
|
7
|
+
def initialize(outconfiguration)
|
8
|
+
@logger = setup_logger(outconfiguration)
|
9
|
+
setup_config(outconfiguration)
|
10
|
+
@token_state = { access_token: nil, expiry_time: nil, token_details_mutex: Mutex.new }
|
11
|
+
end
|
12
|
+
|
13
|
+
# Abstract method: must be implemented by subclasses to fetch a new token.
|
14
|
+
def fetch_token
|
15
|
+
raise NotImplementedError, 'Subclasses must implement fetch_token'
|
16
|
+
end
|
17
|
+
|
18
|
+
# Public method to get a valid token, refreshing if needed.
|
19
|
+
def get_token
|
20
|
+
@token_state[:token_details_mutex].synchronize do
|
21
|
+
if saved_token_need_refresh?
|
22
|
+
@logger.info("Refreshing token. Previous expiry: #{@token_state[:expiry_time]}")
|
23
|
+
refresh_saved_token
|
24
|
+
@logger.info("New token expiry: #{@token_state[:expiry_time]}")
|
25
|
+
end
|
26
|
+
@token_state[:access_token]
|
27
|
+
end
|
28
|
+
end
|
29
|
+
|
30
|
+
private
|
31
|
+
|
32
|
+
def setup_logger(outconfiguration)
|
33
|
+
outconfiguration.logger || Logger.new($stdout)
|
34
|
+
end
|
35
|
+
|
36
|
+
def setup_config(_outconfiguration)
|
37
|
+
# To be optionally overridden by subclasses
|
38
|
+
end
|
39
|
+
|
40
|
+
def saved_token_need_refresh?
|
41
|
+
@token_state[:access_token].nil? || @token_state[:expiry_time].nil? || @token_state[:expiry_time] <= Time.now
|
42
|
+
end
|
43
|
+
|
44
|
+
def refresh_saved_token
|
45
|
+
token_response = fetch_token
|
46
|
+
@token_state[:access_token] = token_response[:access_token]
|
47
|
+
@token_state[:expiry_time] = get_token_expiry_time(token_response[:expires_in])
|
48
|
+
end
|
49
|
+
|
50
|
+
def get_token_expiry_time(expires_in_seconds)
|
51
|
+
if expires_in_seconds.nil? || expires_in_seconds.to_i <= 0
|
52
|
+
Time.now + 3540 # Default to 59 minutes if expires_in is not provided or invalid
|
53
|
+
else
|
54
|
+
Time.now + expires_in_seconds.to_i - 1
|
55
|
+
end
|
56
|
+
end
|
57
|
+
end
|
@@ -0,0 +1,50 @@
|
|
1
|
+
# frozen_string_literal: true
|
2
|
+
|
3
|
+
require 'net/http'
|
4
|
+
require 'uri'
|
5
|
+
require 'json'
|
6
|
+
require_relative 'tokenprovider_base'
|
7
|
+
|
8
|
+
class WorkloadIdentity < AbstractTokenProvider
|
9
|
+
DEFAULT_TOKEN_FILE = '/var/run/secrets/azure/tokens/azure-identity-token'
|
10
|
+
AZURE_OAUTH2_TOKEN_ENDPOINT = 'https://login.microsoftonline.com/%<tenant_id>s/oauth2/v2.0/token'
|
11
|
+
|
12
|
+
# Use get_token from base class for token retrieval
|
13
|
+
|
14
|
+
private
|
15
|
+
|
16
|
+
def setup_config(outconfiguration)
|
17
|
+
@client_id = outconfiguration.workload_identity_client_id
|
18
|
+
@tenant_id = outconfiguration.workload_identity_tenant_id
|
19
|
+
@token_file = outconfiguration.workload_identity_token_file_path || DEFAULT_TOKEN_FILE
|
20
|
+
@kusto_endpoint = outconfiguration.kusto_endpoint
|
21
|
+
@scope = "#{@kusto_endpoint}/.default"
|
22
|
+
end
|
23
|
+
|
24
|
+
def fetch_token
|
25
|
+
response = acquire_workload_identity_token
|
26
|
+
{
|
27
|
+
access_token: response['access_token'],
|
28
|
+
expires_in: response['expires_in']
|
29
|
+
}
|
30
|
+
end
|
31
|
+
|
32
|
+
def acquire_workload_identity_token
|
33
|
+
oidc_token = File.read(@token_file).strip
|
34
|
+
uri = URI.parse(format(AZURE_OAUTH2_TOKEN_ENDPOINT, tenant_id: @tenant_id))
|
35
|
+
req = Net::HTTP::Post.new(uri)
|
36
|
+
req.set_form_data(
|
37
|
+
'grant_type' => 'client_credentials',
|
38
|
+
'client_id' => @client_id,
|
39
|
+
'scope' => @scope,
|
40
|
+
'client_assertion_type' => 'urn:ietf:params:oauth:client-assertion-type:jwt-bearer',
|
41
|
+
'client_assertion' => oidc_token
|
42
|
+
)
|
43
|
+
http = Net::HTTP.new(uri.host, uri.port)
|
44
|
+
http.use_ssl = true
|
45
|
+
res = http.request(req)
|
46
|
+
raise "Failed to get access token: #{res.code} #{res.body}" unless res.is_a?(Net::HTTPSuccess)
|
47
|
+
|
48
|
+
JSON.parse(res.body)
|
49
|
+
end
|
50
|
+
end
|