fluent-plugin-azure-logs-ingestion 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: e5482480b26a287f04f3869ea12ddbe976dac865aa507f43538a99fcfc060c51
+  data.tar.gz: 171a565a9d11e5a020e254682540a85adafa8b90eb46e2732819b1a63f67dab6
+SHA512:
+  metadata.gz: 26314140847581088a5b2fa71e0fb4abeff7de0fae64ce31afcab42e40c5298d93df43def1809b1f0faf442075e57ebb4e796bd78a95989409550ead49da0037
+  data.tar.gz: 5006e5ddbeff5b09a2a2f6d7e79a68e78d1a7912ccf8e390e6aa997c71f97f151a6e0fefeec6b295daae0bbf014f3c361d6bc4c901c3886be0da33a049700da5

data/Gemfile

@@ -0,0 +1,8 @@
+# frozen_string_literal: true
+
+source 'https://rubygems.org'
+
+gemspec
+
+gem 'rake'
+gem 'test-unit'

data/LICENSE

@@ -0,0 +1,13 @@
+Copyright 2026- fukasawah
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

data/README.md

@@ -0,0 +1,209 @@
+# fluent-plugin-azure-logs-ingestion
+
+Fluentd output plugin that sends records to Log Analytics Workspace tables by using the Azure Monitor Logs Ingestion API.
+
+> [!WARNING]
+> This plugin is experimental and has not yet been sufficiently proven in serious production workloads.
+
+## Installation
+
+### RubyGems
+
+```bash
+fluent-gem install fluent-plugin-azure-logs-ingestion
+```
+
+If you use `td-agent`, use `td-agent-gem` instead of `fluent-gem`.
+
+### Bundler
+
+Add the following line to your Gemfile.
+
+```ruby
+gem 'fluent-plugin-azure-logs-ingestion'
+```
+
+Then run `bundle install`.
+
+### Install From GitHub With Bundler
+
+Bundler can point directly at the GitHub repository. Specify `ref` when you want to pin a specific revision.
+
+```ruby
+gem 'fluent-plugin-azure-logs-ingestion', git: 'https://github.com/fukasawah/fluent-plugin-azure-logs-ingestion.git', ref: 'abda3b5370ccd61282c8b234ca05042049e09d15'
+```
+
+Then run `bundle install`.
+
+
+## Configuration Example
+
+```conf
+
+<match azure.logs>
+	@type azure_logs_ingestion
+	endpoint https://example.japaneast-1.ingest.monitor.azure.com
+	dcr_immutable_id dcr-000a00a000a00000a000000aa000a0aa
+	stream_name Custom-MyTable
+
+	tenant_id YOUR_TENANT_ID
+	client_id YOUR_CLIENT_ID
+	client_secret YOUR_CLIENT_SECRET
+
+	<buffer>
+		@type file
+		path /var/log/fluent/azure-logs-ingestion-buffer.*.buf
+		chunk_limit_size 900KB
+	</buffer>
+</match>
+```
+
+## Configuration
+
+### Parameters
+
+| Parameter | Required | Default | Description |
+| --- | --- | --- | --- |
+| `endpoint` | yes | none | Logs Ingestion endpoint or DCE endpoint |
+| `dcr_immutable_id` | yes | none | Immutable DCR ID in `dcr-...` format |
+| `stream_name` | yes | none | DCR input stream name specified in the request URI |
+| `gzip` | no | `false` | Send the HTTP request body compressed with gzip |
+| `use_msi` | no | `false` | Use Managed Identity instead of a service principal |
+| `tenant_id` | no | `ENV['AZURE_TENANT_ID']` | Tenant ID used for service principal authentication |
+| `client_id` | no | `ENV['AZURE_CLIENT_ID']` | Service principal client ID, or user-assigned managed identity client ID |
+| `client_secret` | no | `ENV['AZURE_CLIENT_SECRET']` | Service principal client secret |
+| `authority_host` | no | `https://login.microsoftonline.com` | OAuth token endpoint base URL |
+| `logs_ingestion_scope` | no | `https://monitor.azure.com/.default` | OAuth scope for the Logs Ingestion API |
+| `token_refresh_skew` | no | `300s` | How many seconds before expiry to refresh the Azure access token |
+
+### Buffer Parameters
+
+This plugin changes only the buffer defaults needed for a production-friendly file buffer and a chunk size that is likely to fit within the Logs Ingestion API request size limit.
+
+| Buffer parameter | Default | Description |
+| --- | --- | --- |
+| `@type` | `file` | Use a file buffer by default |
+| `chunk_limit_size` | `900KB` | Chunk size with headroom against the Logs Ingestion API 1 MB request size limit |
+
+### Authentication
+
+Service principal credentials can be written directly in the Fluentd configuration or read from environment variables.
+
+Available environment variables:
+
+- `AZURE_TENANT_ID`
+- `AZURE_CLIENT_ID`
+- `AZURE_CLIENT_SECRET`
+
+When using Managed Identity, specify `use_msi true` and omit `tenant_id` and `client_secret`.
+When using User-assigned Managed Identity, specify the User-assigned Managed Identity client ID in `client_id`.
+
+### Managed Identity Example
+
+```conf
+<match azure.logs>
+	@type azure_logs_ingestion
+	endpoint https://example.japaneast-1.ingest.monitor.azure.com
+	dcr_immutable_id dcr-000a00a000a00000a000000aa000a0aa
+	stream_name Custom-MyTable
+	use_msi true
+	client_id YOUR_USER_ASSIGNED_MANAGED_IDENTITY_CLIENT_ID
+
+	<buffer>
+		@type file
+		path /var/log/fluent/azure-logs-ingestion-buffer.*.buf
+	</buffer>
+</match>
+```
+
+### Buffer Configuration Notes
+
+- `chunk_limit_size 900KB`: The Logs Ingestion API request size limit is 1 MB. One chunk should fit in one request, and starting around 900 KB is safer because JSON serialization can increase the API request size.
+- `flush_mode` and `flush_interval` use Fluentd defaults. If you need lower delivery latency, specify them explicitly as normal Fluentd buffer settings.
+
+### 30 Minute Limit On Auxiliary Tier
+
+When sending to the Log Analytics Workspace Auxiliary tier without converting `TimeGenerated` in a DCR transformation, the range of `TimeGenerated` values in one request must be less than 30 minutes. To satisfy this limit, treat the original log timestamp as the Fluentd event time, then split chunks by time with `<buffer time>` and `timekey`.
+
+For example, if the record field `created_at` is an ISO8601 string, convert it to event time with the input parser. Specify `keep_time_key true` when you also want to send `created_at` to Azure.
+
+```conf
+<source>
+	@type tail
+	path /var/log/myapp/app.log
+	tag azure.logs
+
+	<parse>
+		@type json
+		time_key created_at
+		time_format %iso8601
+		keep_time_key true
+	</parse>
+</source>
+
+<match azure.logs>
+	@type azure_logs_ingestion
+	# ...
+	<buffer time>
+		@type file
+		# ...
+		timekey 20m
+	</buffer>
+</match>
+```
+
+If you need to replace the event time after a record has already been ingested, you can use `renew_time_key` in a filter. The field specified in `renew_time_key` must be a Unix timestamp.
+
+```conf
+<filter azure.logs>
+	@type record_transformer
+	renew_time_key created_at
+</filter>
+
+
+<match azure.logs>
+	@type azure_logs_ingestion
+	# ...
+	<buffer time>
+		@type file
+		# ...
+		timekey 20m
+	</buffer>
+</match>
+```
+
+The `time` in `<buffer time>` is the Fluentd event time, not a `time` field inside the record. Merely leaving `created_at` or `TimeGenerated` in the payload does not make it available for time-based chunking.
+
+### Plugin Behavior
+
+- This plugin does not rewrite `TimeGenerated`. If the payload has an original timestamp field such as `time`, prefer creating it in the DCR transformation, for example `extend TimeGenerated = todatetime(['time'])`.
+- HTTP `400`, `401`, `403`, and `413` are treated as unrecoverable. `429` and `5xx` are retried by Fluentd.
+
+## Memo: Log Analytics Workspace / DCR / Logs Ingestion API Behavior
+
+- Currently, when the Log Analytics Workspace SKU is Auxiliary tier and the DCR transformation is not used, `TimeGenerated` in one request must stay within less than 30 minutes.
+  - > This limit only applies when ingesting to Auxiliary log tables. If the source entries for TimeGenerated are ingested without being transformed, the range of entries must be less than 30 minutes.
+    >
+	> https://learn.microsoft.com/en-us/azure/azure-monitor/fundamentals/service-limits#logs-ingestion-api
+- Logs Ingestion API request size must be kept to 1 MB or less.
+  - > Maximum size of API call | 1 MB
+    >
+	> https://learn.microsoft.com/en-us/azure/azure-monitor/fundamentals/service-limits#logs-ingestion-api
+- Log Analytics Workspace has no deduplication mechanism. If Azure accepts a request but Fluentd cannot confirm the response successfully, retrying can create duplicate records.
+
+## References
+
+- Azure Monitor Logs Ingestion API overview: https://learn.microsoft.com/azure/azure-monitor/logs/logs-ingestion-api-overview
+- Create data collection rules (DCRs) using JSON: https://learn.microsoft.com/azure/azure-monitor/data-collection/data-collection-rule-create-edit
+- Azure DCR structure: https://learn.microsoft.com/azure/azure-monitor/data-collection/data-collection-rule-structure
+- Azure custom tables and `_CL` suffix: https://learn.microsoft.com/azure/azure-monitor/logs/create-custom-table
+- Managed identity on Azure VM: https://learn.microsoft.com/entra/identity/managed-identities-azure-resources/how-to-use-vm-token
+- Managed identity on App Service / Functions: https://learn.microsoft.com/azure/app-service/overview-managed-identity
+- Fluentd output plugin API: https://docs.fluentd.org/plugin-development/api-plugin-output
+
+## Development
+
+```bash
+bundle install
+bundle exec rake test
+```

data/README_ja.md

@@ -0,0 +1,209 @@
+# fluent-plugin-azure-logs-ingestion
+
+Azure Monitor Logs Ingestion API を使い、Log Analytics Workspace のテーブルへ出力する Fluentd output plugin です。
+
+> [!WARNING]
+> この plugin は試験的な実装であり、本格的な production workload での動作実績はまだ十分ではありません。
+
+## インストール
+
+### RubyGems
+
+```bash
+fluent-gem install fluent-plugin-azure-logs-ingestion
+```
+
+`td-agent` を使う場合は `fluent-gem` の代わりに `td-agent-gem` を使ってください。
+
+### Bundler
+
+Gemfile に次の行を追加してください。
+
+```ruby
+gem 'fluent-plugin-azure-logs-ingestion'
+```
+
+その後、`bundle install` を実行してください。
+
+### GitHub から Bundler でインストール
+
+Bundler で GitHub repository を直接指定でき、特定の revision に固定したい場合は、`ref` を指定します。
+
+```ruby
+gem 'fluent-plugin-azure-logs-ingestion', git: 'https://github.com/fukasawah/fluent-plugin-azure-logs-ingestion.git', ref: 'abda3b5370ccd61282c8b234ca05042049e09d15'
+```
+
+その後、`bundle install` を実行してください。
+
+
+## 設定例
+
+```conf
+
+<match azure.logs>
+	@type azure_logs_ingestion
+	endpoint https://example.japaneast-1.ingest.monitor.azure.com
+	dcr_immutable_id dcr-000a00a000a00000a000000aa000a0aa
+	stream_name Custom-MyTable
+
+	tenant_id YOUR_TENANT_ID
+	client_id YOUR_CLIENT_ID
+	client_secret YOUR_CLIENT_SECRET
+
+	<buffer>
+		@type file
+		path /var/log/fluent/azure-logs-ingestion-buffer.*.buf
+		chunk_limit_size 900KB
+	</buffer>
+</match>
+```
+
+## 設定
+
+### パラメータ
+
+| Parameter | Required | Default | Description |
+| --- | --- | --- | --- |
+| `endpoint` | yes | none | Logs Ingestion endpoint または DCE endpoint |
+| `dcr_immutable_id` | yes | none | `dcr-...` 形式の immutable DCR ID |
+| `stream_name` | yes | none | request URI に指定する DCR の input stream 名 |
+| `gzip` | no | `false` | HTTP request body を gzip 圧縮して送信 |
+| `use_msi` | no | `false` | service principal ではなく Managed Identity を使う |
+| `tenant_id` | no | `ENV['AZURE_TENANT_ID']` | service principal 認証で使う tenant ID |
+| `client_id` | no | `ENV['AZURE_CLIENT_ID']` | service principal の client ID、または user-assigned managed identity の client ID |
+| `client_secret` | no | `ENV['AZURE_CLIENT_SECRET']` | service principal の client secret |
+| `authority_host` | no | `https://login.microsoftonline.com` | OAuth token endpoint の base URL |
+| `logs_ingestion_scope` | no | `https://monitor.azure.com/.default` | Logs Ingestion API 用の OAuth scope |
+| `token_refresh_skew` | no | `300s` | Azure のアクセストークンを期限の何秒前に再取得するか |
+
+### Buffer パラメータ
+
+この plugin は、production workload で使いやすい file buffer と、Logs Ingestion API の request size 上限に収まりやすい chunk size だけを buffer default として変更しています。
+
+| Buffer parameter | Default | Description |
+| --- | --- | --- |
+| `@type` | `file` | デフォルトで file buffer を使う |
+| `chunk_limit_size` | `900KB` | Logs Ingestion API の 1 MB request size 上限に対して余裕を持たせた chunk size |
+
+### 認証
+
+service principal の認証情報は Fluentd 設定に直接書くことも、環境変数から読むこともできます。
+
+利用できる環境変数:
+
+- `AZURE_TENANT_ID`
+- `AZURE_CLIENT_ID`
+- `AZURE_CLIENT_SECRET`
+
+Managed Identity を使う場合は `use_msi true` を指定し、`tenant_id` と `client_secret` は省略してください。
+User-assigned Managed Identity を使う場合は `client_id` に User-assigned Managed Identity の client ID を指定します。
+
+### Managed Identity の例
+
+```conf
+<match azure.logs>
+	@type azure_logs_ingestion
+	endpoint https://example.japaneast-1.ingest.monitor.azure.com
+	dcr_immutable_id dcr-000a00a000a00000a000000aa000a0aa
+	stream_name Custom-MyTable
+	use_msi true
+	client_id YOUR_USER_ASSIGNED_MANAGED_IDENTITY_CLIENT_ID
+
+	<buffer>
+		@type file
+		path /var/log/fluent/azure-logs-ingestion-buffer.*.buf
+	</buffer>
+</match>
+```
+
+### Buffer 設定の考え方
+
+- `chunk_limit_size 900KB`: Logs Ingestion API の request size 上限は 1 MB です。1 chunk を 1 request に収めるほうがよいですが、APIリクエストの際の JSON 化によるサイズ増加の余裕を見て 900KB 前後から始めるのが安全です。
+- `flush_mode` や `flush_interval` は Fluentd の default を使います。より短い遅延で送信したい場合は、通常の Fluentd buffer 設定として明示してください。
+
+### Auxiliary tier での 30 分制限
+
+Log Analytics Workspace の Auxiliary tier へ送信し、DCR transformation で `TimeGenerated` を変換しない場合、1 request 内の `TimeGenerated` の範囲は 30 分未満にする必要があります。この制限に対応するには、元ログの時刻を Fluentd の event time として扱い、`<buffer time>` と `timekey` で chunk を時間分割してください。
+
+たとえば record の `created_at` が ISO8601 文字列の場合、入力時の parser で event time に変換します。Azure 側にも `created_at` を送る場合は `keep_time_key true` を指定します。
+
+```conf
+<source>
+	@type tail
+	path /var/log/myapp/app.log
+	tag azure.logs
+
+	<parse>
+		@type json
+		time_key created_at
+		time_format %iso8601
+		keep_time_key true
+	</parse>
+</source>
+
+<match azure.logs>
+	@type azure_logs_ingestion
+	# ...
+	<buffer time>
+		@type file
+		# ...
+		timekey 20m
+	</buffer>
+</match>
+```
+
+すでに record として取り込まれた後に event time を差し替える場合は、filter で `renew_time_key` を使えます。ただし `renew_time_key` に指定するフィールドは Unix timestamp である必要があります。
+
+```conf
+<filter azure.logs>
+	@type record_transformer
+	renew_time_key created_at
+</filter>
+
+
+<match azure.logs>
+	@type azure_logs_ingestion
+	# ...
+	<buffer time>
+		@type file
+		# ...
+		timekey 20m
+	</buffer>
+</match>
+```
+
+`<buffer time>` の `time` は record 内の `time` フィールドではなく Fluentd の event time です。record の `created_at` や `TimeGenerated` を payload に残すだけでは chunk の時間分割には使われません。
+
+### Plugin 仕様
+
+- 本 plugin は `TimeGenerated` を書き換えません。payload に `time` のような元時刻フィールドがある場合は、DCR の transformation で `extend TimeGenerated = todatetime(['time'])` のように作成することをお勧めします。
+- HTTP `400`, `401`, `403`, `413` は unrecoverable として扱い、`429` と `5xx` は Fluentd の retry 対象です。
+
+## Memo: Log Analytics Workspace / DCR / Logs Ingestion API の仕様
+
+- 現状、Log Analytics Workspace の SKU が Auxiliary tier で DCR の transformation を使わない場合、1 request 内の `TimeGenerated` は 30 分未満に収める必要があります。
+  - > This limit only applies when ingesting to Auxiliary log tables. If the source entries for TimeGenerated are ingested without being transformed, the range of entries must be less than 30 minutes. 
+    >
+	> https://learn.microsoft.com/en-us/azure/azure-monitor/fundamentals/service-limits#logs-ingestion-api
+- Logs Ingestion API の request size は 1 MB 以下に抑える必要があります。
+  - > Maximum size of API call | 1 MB
+    >
+	> https://learn.microsoft.com/en-us/azure/azure-monitor/fundamentals/service-limits#logs-ingestion-api
+- Log Analytics Workspace には重複排除の仕組みがありません。Azure が受理した後で Fluentd が応答を正常に確認できなかった場合、再送により重複が発生します。
+
+## 参考資料
+
+- Azure Monitor Logs Ingestion API overview: https://learn.microsoft.com/azure/azure-monitor/logs/logs-ingestion-api-overview
+- Create data collection rules (DCRs) using JSON: https://learn.microsoft.com/azure/azure-monitor/data-collection/data-collection-rule-create-edit
+- Azure DCR structure: https://learn.microsoft.com/azure/azure-monitor/data-collection/data-collection-rule-structure
+- Azure custom tables and `_CL` suffix: https://learn.microsoft.com/azure/azure-monitor/logs/create-custom-table
+- Managed identity on Azure VM: https://learn.microsoft.com/entra/identity/managed-identities-azure-resources/how-to-use-vm-token
+- Managed identity on App Service / Functions: https://learn.microsoft.com/azure/app-service/overview-managed-identity
+- Fluentd output plugin API: https://docs.fluentd.org/plugin-development/api-plugin-output
+
+## 開発
+
+```bash
+bundle install
+bundle exec rake test
+```

data/Rakefile

@@ -0,0 +1,10 @@
+# frozen_string_literal: true
+
+require 'rake/testtask'
+
+Rake::TestTask.new do |test|
+  test.libs << 'test'
+  test.pattern = 'test/test_*.rb'
+end
+
+task default: :test

data/lib/fluent/plugin/azure_logs_ingestion/auth.rb

@@ -0,0 +1,184 @@
+# frozen_string_literal: true
+
+require 'json'
+require 'net/http'
+require 'uri'
+require 'time'
+require 'openssl'
+
+module Fluent
+  module Plugin
+    module AzureLogsIngestion
+      class Auth
+        Token = Struct.new(:value, :expires_at, keyword_init: true)
+
+        IMDS_API_VERSION = '2018-02-01'
+        APP_SERVICE_API_VERSION = '2019-08-01'
+        IMDS_ENDPOINT = 'http://169.254.169.254/metadata/identity/oauth2/token'
+
+        def initialize(use_msi:, tenant_id:, client_id:, client_secret:, authority_host:, logs_ingestion_scope:, token_refresh_skew:, logger:)
+          @use_msi = use_msi
+          @tenant_id = tenant_id
+          @client_id = client_id
+          @client_secret = client_secret
+          @authority_host = authority_host
+          @logs_ingestion_scope = logs_ingestion_scope
+          @token_refresh_skew = token_refresh_skew
+          @log = logger
+          @token = nil
+          @mutex = Mutex.new
+        end
+
+        def token
+          @mutex.synchronize do
+            if token_valid?(@token)
+              @log.debug('reusing cached access token', expires_at: @token.expires_at.utc.iso8601)
+              return @token.value
+            end
+
+            @token = @use_msi ? fetch_msi_token : fetch_service_principal_token
+            @log.debug('fetched new access token', mode: @use_msi ? 'managed_identity' : 'service_principal', expires_at: @token.expires_at.utc.iso8601)
+            @token.value
+          end
+        end
+
+        private
+
+        def token_valid?(token)
+          token && token.expires_at && (Time.now + @token_refresh_skew) < token.expires_at
+        end
+
+        def fetch_service_principal_token
+          uri = URI.join(normalized_authority_host, "#{@tenant_id}/oauth2/v2.0/token")
+          @log.debug('requesting service principal token', authority_host: normalized_authority_host, tenant_id: @tenant_id, scope: @logs_ingestion_scope)
+          request = Net::HTTP::Post.new(uri)
+          request.set_form_data(
+            'grant_type' => 'client_credentials',
+            'client_id' => @client_id,
+            'client_secret' => @client_secret,
+            'scope' => @logs_ingestion_scope
+          )
+
+          response = perform_request(uri, request)
+          body = parse_json_body(response)
+          build_token_from_token_response(body)
+        rescue JSON::ParserError => error
+          raise "failed to parse service principal token response: #{error.message}"
+        end
+
+        def fetch_msi_token
+          if ENV['IDENTITY_ENDPOINT'] && (ENV['IDENTITY_HEADER'] || ENV['MSI_SECRET'])
+            fetch_app_service_token
+          else
+            fetch_imds_token
+          end
+        end
+
+        def fetch_imds_token
+          uri = URI(ENV.fetch('AZURE_LOGS_INGESTION_IMDS_ENDPOINT', IMDS_ENDPOINT))
+          params = {
+            'api-version' => IMDS_API_VERSION,
+            'resource' => scope_resource(@logs_ingestion_scope)
+          }
+          params['client_id'] = @client_id if @client_id && !@client_id.empty?
+          uri.query = URI.encode_www_form(params)
+
+          request = Net::HTTP::Get.new(uri)
+          request['Metadata'] = 'true'
+          @log.debug('requesting managed identity token via IMDS', endpoint: uri.to_s, resource: params['resource'], client_id: params['client_id'])
+
+          response = perform_request(uri, request)
+          body = parse_json_body(response)
+          build_token_from_token_response(body)
+        rescue JSON::ParserError => error
+          raise "failed to parse IMDS token response: #{error.message}"
+        end
+
+        def fetch_app_service_token
+          uri = URI(ENV.fetch('IDENTITY_ENDPOINT'))
+          params = {
+            'api-version' => APP_SERVICE_API_VERSION,
+            'resource' => scope_resource(@logs_ingestion_scope)
+          }
+          params['client_id'] = @client_id if @client_id && !@client_id.empty?
+          existing = URI.decode_www_form(String(uri.query))
+          uri.query = URI.encode_www_form(existing + params.to_a)
+
+          request = Net::HTTP::Get.new(uri)
+          request['X-IDENTITY-HEADER'] = ENV['IDENTITY_HEADER'] if ENV['IDENTITY_HEADER']
+          request['Secret'] = ENV['MSI_SECRET'] if ENV['MSI_SECRET']
+          @log.debug('requesting managed identity token via app service endpoint', endpoint: uri.to_s, resource: params['resource'], client_id: params['client_id'])
+
+          response = perform_request(uri, request)
+          body = parse_json_body(response)
+          build_token_from_token_response(body)
+        rescue KeyError => error
+          raise Fluent::ConfigError, error.message
+        rescue JSON::ParserError => error
+          raise "failed to parse App Service managed identity response: #{error.message}"
+        end
+
+        def perform_request(uri, request)
+          http = Net::HTTP.new(uri.host, uri.port)
+          http.use_ssl = uri.scheme == 'https'
+          http.open_timeout = 10
+          http.read_timeout = 30
+          http.verify_mode = OpenSSL::SSL::VERIFY_PEER if http.use_ssl?
+
+          response = http.request(request)
+          handle_token_response_errors(response)
+          response
+        rescue Timeout::Error, Errno::ECONNREFUSED, Errno::EHOSTUNREACH, Errno::ECONNRESET, EOFError, SocketError, IOError, SystemCallError => error
+          raise "token request failed: #{error.message}"
+        end
+
+        def handle_token_response_errors(response)
+          return if response.is_a?(Net::HTTPSuccess)
+
+          message = "token endpoint returned #{response.code} #{response.message} #{String(response.body).strip}".strip
+          status = response.code.to_i
+
+          if @use_msi
+            raise message if retryable_msi_status?(status)
+            raise Fluent::UnrecoverableError, message if status >= 400 && status < 500
+            raise message
+          end
+
+          raise Fluent::UnrecoverableError, message if status >= 400 && status < 500
+          raise message
+        end
+
+        def retryable_msi_status?(status)
+          status == 404 || status == 410 || status == 429 || status >= 500
+        end
+
+        def parse_json_body(response)
+          JSON.parse(String(response.body))
+        end
+
+        def build_token_from_token_response(body)
+          token = body.fetch('access_token')
+          expires_at = parse_token_expiry(body)
+          Token.new(value: token, expires_at: expires_at)
+        end
+
+        def parse_token_expiry(body)
+          return Time.at(Integer(body['expires_on'])) if body['expires_on']
+          return Time.now + Integer(body['expires_in']) if body['expires_in']
+
+          raise Fluent::UnrecoverableError, 'token response did not include expires_on or expires_in'
+        rescue ArgumentError, TypeError
+          raise Fluent::UnrecoverableError, 'token response included an invalid expiration value'
+        end
+
+        def normalized_authority_host
+          @authority_host.end_with?('/') ? @authority_host : "#{@authority_host}/"
+        end
+
+        def scope_resource(scope)
+          scope.sub(%r{/\.default\z}, '/')
+        end
+      end
+    end
+  end
+end