fluent-plugin-dynatrace 0.1.0

data/README.md

@@ -0,0 +1,97 @@
+# fluent-plugin-dynatrace, a plugin for [fluentd](https://www.fluentd.org/)
+
+> This project is developed and maintained by Dynatrace R&D.
+Currently, this is a prototype and not intended for production use.
+It is not covered by Dynatrace support.
+
+A fluentd output plugin for sending logs to the Dynatrace [Generic log ingest API v2](https://www.dynatrace.com/support/help/how-to-use-dynatrace/log-monitoring/log-monitoring-v2/post-log-ingest/).
+
+## Requirements
+
+- An instance of fluentd from which logs should be exported
+- An ActiveGate with the Generic log ingest API v2 enabled as described in the [Dynatrace documentation](https://www.dynatrace.com/support/help/how-to-use-dynatrace/log-monitoring/log-monitoring-v2/log-data-ingestion/)
+- A [Dynatrace API token](https://www.dynatrace.com/support/help/dynatrace-api/basics/dynatrace-api-authentication/) with the `Log import` permission
+
+## Configuration options
+
+Below is an example configuration which sends all logs with tags starting with `dt.` to Dynatrace.
+
+```
+<match dt.*>
+  @type              dynatrace
+  active_gate_url    https://abc12345.live.dynatrace.com/api/v2/logs/ingest
+  api_token          api_token
+  ssl_verify_none    false
+</match>
+```
+
+### match directive
+
+- `required`
+
+The `match` directive is required to use an output plugin and tells fluentd which tags should be sent to the output plugin. In the above example, any tag that starts with `dt.` will be sent to Dynatrace. For more information see [how do match patterns work?](https://docs.fluentd.org/configuration/config-file#how-do-the-match-patterns-work). 
+
+### @type
+
+- `required`
+
+The `@type` directive tells fluentd which plugin should be used for the corresponding match block. This should always be `dynatrace` when you want to use the Dynatrace output plugin.
+
+### `active_gate_url`
+
+- `required`
+
+This is the full URL of the [Generic log ingest API v2](https://www.dynatrace.com/support/help/how-to-use-dynatrace/log-monitoring/log-monitoring-v2/post-log-ingest/) endpoint on your ActiveGate.
+
+### `api_token`
+
+- `required`
+
+This is the [Dynatrace API token](https://www.dynatrace.com/support/help/dynatrace-api/basics/dynatrace-api-authentication/) which will be used to authenticate log ingest requests. It should be assigned only the `Log import` permission.
+
+### `ssl_verify_none`
+
+- `optional`
+- `default: false`
+
+It is recommended to leave this optional configuration set to `false` unless absolutely required. Setting `ssl_verify_none` to `true` causes the output plugin to skip certificate verification when sending log ingest requests to SSL and TLS protected HTTPS endpoints. This option may be required if you are using a self-signed certificate, an expired certificate, or a certificate which was generated for a different domain than the one in use.
+
+## Development
+
+`fluent-plugin-dynatrace` supports Ruby versions `>= 2.4.0` but it is recommended that at least `2.7.2` is used for development. Ruby versions can be managed with tools like [chruby](https://github.com/postmodern/chruby) or [rbenv](https://github.com/rbenv/rbenv).
+
+### Install Dependencies
+
+```sh
+bundle install
+```
+
+### Run All Tests
+
+```sh
+rake test
+```
+
+### Run Specific Tests
+
+```sh
+# Run one test file
+rake test TEST=test/plugin/out_dynatrace_test.rb
+```
+
+### Code Style Checks
+
+```sh
+# Check for code style violations
+rake rubocop
+
+# Fix auto-fixable style violations
+rake rubocop:auto_correct
+```
+
+### Run all checks and build
+
+```sh
+# Runs rubocop, tests, and builds the gem
+rake check
+```

data/Rakefile

@@ -0,0 +1,37 @@
+#!/usr/bin/env rake
+# frozen_string_literal: true
+
+# Copyright 2021 Dynatrace LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+require 'bundler/gem_tasks'
+require 'rake/testtask'
+require 'rubocop/rake_task'
+
+RuboCop::RakeTask.new
+
+Rake::TestTask.new :test do |t|
+  t.libs << 'test'
+  t.libs << 'lib'
+  t.test_files = FileList['test/plugin/*_test.rb']
+end
+
+Rake::TestTask.new 'test:integration' do |t|
+  t.test_files = FileList['test/integration/*_test.rb']
+end
+
+desc 'check for style violations and test failures and build the gem'
+task check: %i[rubocop test build]
+
+task default: :build

data/fluent-plugin-dynatrace.gemspec

@@ -0,0 +1,30 @@
+# frozen_string_literal: true
+
+require './lib/fluent/plugin/dynatrace_constants'
+
+Gem::Specification.new do |gem|
+  gem.name          = 'fluent-plugin-dynatrace'
+  gem.version       = Fluent::Plugin::DynatraceOutputConstants.version
+  gem.authors       = ['Dynatrace Open Source Engineering']
+  gem.email         = ['opensource@dynatrace.com']
+  gem.summary       = 'A fluentd output plugin for sending logs to the Dynatrace Generic log ingest API v2'
+  gem.homepage      = 'https://www.dynatrace.com/'
+  gem.licenses      = ['Apache-2.0']
+
+  gem.metadata = {
+    'bug_tracker_uri' => 'https://github.com/dynatrace-oss/fluent-plugin-dynatrace/issues',
+    'documentation_uri' => 'https://github.com/dynatrace-oss/fluent-plugin-dynatrace',
+    'source_code_uri' => 'https://github.com/dynatrace-oss/fluent-plugin-dynatrace'
+  }
+
+  gem.files         = `git ls-files`.split($OUTPUT_RECORD_SEPARATOR)
+  gem.require_paths = ['lib']
+
+  gem.required_ruby_version = '>= 2.4.0'
+
+  gem.add_runtime_dependency 'fluentd', ['>= 0.14.22', '< 2']
+  gem.add_development_dependency 'bundler', ['>= 2', '<3']
+  gem.add_development_dependency 'rake', '13.0.3'
+  gem.add_development_dependency 'rubocop', '1.9.1'
+  gem.add_development_dependency 'rubocop-rake', '0.5.1'
+end

data/lib/fluent/plugin/dynatrace_constants.rb

@@ -0,0 +1,27 @@
+# frozen_string_literal: true
+
+# Copyright 2021 Dynatrace LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+module Fluent
+  module Plugin
+    # Constants for use in Dynatrace output plugin
+    class DynatraceOutputConstants
+      # The version of the Dynatrace output plugin
+      def self.version
+        '0.1.0'
+      end
+    end
+  end
+end

data/lib/fluent/plugin/out_dynatrace.rb

@@ -0,0 +1,139 @@
+# frozen_string_literal: true
+
+# Copyright 2021 Dynatrace LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+require 'fluent/plugin/output'
+require 'net/http'
+require_relative 'dynatrace_constants'
+
+module Fluent
+  module Plugin
+    # Fluentd output plugin for Dynatrace
+    class DynatraceOutput < Output
+      Fluent::Plugin.register_output('dynatrace', self)
+
+      helpers :compat_parameters # add :inject if need be
+
+      # Configurations
+      desc 'The full URL of the Dynatrace log ingestion endpoint, e.g. https://my-active-gate.example.com/api/logs/ingest'
+      config_param :active_gate_url, :string
+      desc 'The API token to use to authenticate requests to the log ingestion endpoint. Must have TODO scope'
+      config_param :api_token, :string, secret: true
+
+      desc 'Disable SSL validation by setting :verify_mode OpenSSL::SSL::VERIFY_NONE'
+      config_param :ssl_verify_none, :bool, default: false
+
+      #############################################
+
+      config_section :buffer do
+        config_set_default :flush_at_shutdown, true
+        config_set_default :chunk_limit_size, 10 * 1024
+      end
+
+      # Default injection parameters.
+      # Requires the :inject helper to be added to the helpers above and the
+      #   inject lines to be uncommented in the #write and #process methods
+      # config_section :inject do
+      #   config_set_default :time_type, :string
+      #   config_set_default :localtime, false
+      # end
+      #############################################
+
+      attr_accessor :uri, :agent
+
+      def configure(conf)
+        compat_parameters_convert(conf, :inject)
+        super
+
+        @uri = URI.parse(@active_gate_url)
+        @agent = Net::HTTP.new(@uri.host, @uri.port)
+
+        return unless uri.scheme == 'https'
+
+        @agent.use_ssl = true
+        @agent.verify_mode = OpenSSL::SSL::VERIFY_NONE if @ssl_verify_none
+      end
+
+      def shutdown
+        @agent.finish if @agent.started?
+        super
+      end
+
+      #############################################
+
+      def process(_tag, es)
+        # es = inject_values_to_event_stream(tag, es)
+        es.each do |_time, record|
+          send_to_dynatrace("#{record.to_json.chomp}\n")
+        end
+      end
+
+      def write(chunk)
+        body = []
+        chunk.each do |_time, record|
+          # body.push(inject_values_to_record(chunk.metadata.tag, time, record))
+          body.push(record)
+        end
+
+        send_to_dynatrace("#{body.to_json.chomp}\n")
+      end
+
+      #############################################
+
+      def prefer_buffered_processing
+        true
+      end
+
+      def multi_workers_ready?
+        false
+      end
+
+      #############################################
+
+      def user_agent
+        "fluent-plugin-dynatrace v#{DynatraceOutputConstants.version}"
+      end
+
+      def prepare_request(uri)
+        req = Net::HTTP::Post.new(uri, { 'User-Agent' => user_agent })
+        req['Content-Type'] = 'application/json; charset=utf-8'
+        req['Authorization'] = "Api-Token #{@api_token}"
+
+        req
+      end
+
+      def send_to_dynatrace(body)
+        agent.start unless agent.started?
+
+        req = prepare_request(@uri)
+        res = @agent.request(req, body)
+
+        return if res.is_a?(Net::HTTPSuccess)
+
+        raise failure_message res
+      end
+
+      def failure_message(res)
+        res_summary = if res
+                        "#{res.code} #{res.message}"
+                      else
+                        'res=nil'
+                      end
+
+        "failed to #{req.method} #{uri} (#{res_summary})"
+      end
+    end
+  end
+end

data/test/.rubocop.yml

@@ -0,0 +1,5 @@
+inherit_from: ../.rubocop.yml
+
+Metrics:
+  Enabled: false
+

data/test/integration/fixtures/docker-compose.yaml

@@ -0,0 +1,27 @@
+# Copyright 2021 Dynatrace LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+services:
+  fluent:
+    build:
+      context: ../../../
+      dockerfile: test/integration/fixtures/fluent/Dockerfile
+    ports:
+      - 8080:8080
+    links:
+      - logsink
+  logsink:
+    build: logsink
+    expose:
+      - 8080

data/test/integration/fixtures/fluent/Dockerfile

@@ -0,0 +1,30 @@
+# Copyright 2021 Dynatrace LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+FROM fluent/fluentd:edge
+
+LABEL maintainer="Daniel Dyla <Daniel.Dyla@dynatrace.com>"
+USER root
+
+# the build context is the root of the repo to allow access to the plugin rb
+COPY test/integration/fixtures/fluent/entrypoint.sh /fluentd/entrypoint.sh
+RUN chmod +x /fluentd/entrypoint.sh
+
+COPY test/integration/fixtures/fluent/fluent.conf /fluentd/etc/fluent.conf
+COPY lib/fluent/plugin/out_dynatrace.rb /fluentd/plugins/
+COPY lib/fluent/plugin/dynatrace_constants.rb /fluentd/plugins/
+
+ENTRYPOINT ["tini", "--", "/fluentd/entrypoint.sh"]
+
+USER fluent

data/test/integration/fixtures/fluent/entrypoint.sh

@@ -0,0 +1,17 @@
+#!/usr/bin/env sh
+
+# Copyright 2021 Dynatrace LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+exec fluentd -c /fluentd/etc/fluent.conf -p /fluentd/plugins

data/test/integration/fixtures/fluent/fluent.conf

@@ -0,0 +1,30 @@
+# Copyright 2021 Dynatrace LLC
+#
+# Licensed under the Apache License, Version 2.0 (the "License");
+# you may not use this file except in compliance with the License.
+# You may obtain a copy of the License at
+#
+#    http://www.apache.org/licenses/LICENSE-2.0
+#
+# Unless required by applicable law or agreed to in writing, software
+# distributed under the License is distributed on an "AS IS" BASIS,
+# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+# See the License for the specific language governing permissions and
+# limitations under the License.
+
+<source>
+  @type http
+  port 8080
+</source>
+
+<match dt.*>
+  @type dynatrace
+
+  active_gate_url    http://logsink:8080/api/v2/logs/ingest
+  api_token          my_token
+
+  <buffer>
+    chunk_limit_records 5
+    flush_interval 5s
+  </buffer>
+</match>