logstash-output-harbor_beacon 0.3.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 12a689753ad344a88d1c675b3d3bbd4424b6e6aa054b9e9a4baf9db4d6cbb7ee
+  data.tar.gz: d5cc214c27a3b2b01f07339a2ebd824c3c0d4a7b3f3b1ef2a2e4097869e7c270
+SHA512:
+  metadata.gz: 9452adf6b1d1c049f24974a1d62f9b9e662ecf479309f27db9beb0ba2c6b11ac9e14bf8445254120606288aa04fb773975510ed07d922ab417a14459a7a315b8
+  data.tar.gz: 1e980c6f23eef6be5bd1eb1799d58cea358f7a7927a1ff8d5367f53323c32a8cc684bfbeacbba6ea04e0b23434d02c527ff4a01fa58c27847c03dd591e8b29a0

data/CHANGELOG.md

@@ -0,0 +1,2 @@
+## 0.1.0
+  - Plugin created with the logstash plugin generator

data/CONTRIBUTORS

@@ -0,0 +1,10 @@
+The following is a list of people who have contributed ideas, code, bug
+reports, or in general have helped logstash along its way.
+
+Contributors:
+* Scott Matheson - scott@hrbr.io
+
+Note: If you've sent us patches, bug reports, or otherwise contributed to
+Logstash, and you aren't on the list above and want to be, please let us know
+and we'll make sure you're here. Contributions from folks like you are what make
+open source awesome.

data/Gemfile

@@ -0,0 +1,3 @@
+source 'https://rubygems.org'
+gemspec
+

data/LICENSE

@@ -0,0 +1,11 @@
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

data/README.md

@@ -0,0 +1,86 @@
+# Logstash Plugin
+
+This is a plugin for [Logstash](https://github.com/elastic/logstash).
+
+It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
+
+## Documentation
+
+Logstash provides infrastructure to automatically generate documentation for this plugin. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html. All plugin documentation are placed under one [central location](http://www.elastic.co/guide/en/logstash/current/).
+
+- For formatting code or config example, you can use the asciidoc `[source,ruby]` directive
+- For more asciidoc formatting tips, see the excellent reference here https://github.com/elastic/docs#asciidoc-guide
+
+## Need Help?
+
+Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/logstash discussion forum.
+
+## Developing
+
+### 1. Plugin Developement and Testing
+
+#### Code
+- To get started, you'll need JRuby with the Bundler gem installed.
+
+- Create a new plugin or clone and existing from the GitHub [logstash-plugins](https://github.com/logstash-plugins) organization. We also provide [example plugins](https://github.com/logstash-plugins?query=example).
+
+- Install dependencies
+```sh
+bundle install
+```
+
+#### Test
+
+- Update your dependencies
+
+```sh
+bundle install
+```
+
+- Run tests
+
+```sh
+bundle exec rspec
+```
+
+### 2. Running your unpublished Plugin in Logstash
+
+#### 2.1 Run in a local Logstash clone
+
+- Edit Logstash `Gemfile` and add the local plugin path, for example:
+```ruby
+gem "logstash-filter-awesome", :path => "/your/local/logstash-filter-awesome"
+```
+- Install plugin
+```sh
+bin/logstash-plugin install --no-verify
+```
+- Run Logstash with your plugin
+```sh
+bin/logstash -e 'filter {awesome {}}'
+```
+At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.
+
+#### 2.2 Run in an installed Logstash
+
+You can use the same **2.1** method to run your plugin in an installed Logstash by editing its `Gemfile` and pointing the `:path` to your local plugin development directory or you can build the gem and install it using:
+
+- Build your plugin gem
+```sh
+gem build logstash-filter-awesome.gemspec
+```
+- Install the plugin from the Logstash home
+```sh
+bin/logstash-plugin install /your/local/plugin/logstash-filter-awesome.gem
+```
+- Start Logstash and proceed to test the plugin
+
+## Contributing
+
+All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.
+
+Programming is not a required skill. Whatever you've seen about open source and maintainers or community members  saying "send patches or die" - you will not see that here.
+
+It is more important to the community that you are able to contribute.
+
+For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file.

data/lib/logstash/outputs/harbor_beacon.rb

@@ -0,0 +1,422 @@
+# encoding: utf-8
+require "logstash/outputs/base"
+require "logstash/namespace"
+require "logstash/json"
+require "uri"
+require "logstash/plugin_mixins/http_client"
+require "zlib"
+
+# An harbor_beacon output that does nothing.
+class LogStash::Outputs::HarborBeacon < LogStash::Outputs::Base
+  include LogStash::PluginMixins::HttpClient
+
+  concurrency :shared
+
+  attr_accessor :is_batch
+
+  RETRYABLE_MANTICORE_EXCEPTIONS = [
+    ::Manticore::Timeout,
+    ::Manticore::SocketException,
+    ::Manticore::ClientProtocolException,
+    ::Manticore::ResolutionFailure,
+    ::Manticore::SocketTimeout
+  ]
+
+  # This output lets you send events to a
+  # generic HTTP(S) endpoint
+  #
+  # This output will execute up to 'pool_max' requests in parallel for performance.
+  # Consider this when tuning this plugin for performance.
+  #
+  # Additionally, note that when parallel execution is used strict ordering of events is not
+  # guaranteed!
+  #
+  # Beware, this gem does not yet support codecs. Please use the 'format' option for now.
+
+  config_name "harbor_beacon"
+
+  # URL to use
+  @@url = "https://harbor-stream.hrbr.io/beacon"
+
+  # The HTTP Verb. Must be "post"
+  @@http_method = "post"
+
+  # Content type
+  @@content_type = "application/json"
+
+
+  # Custom headers to use
+  # format is `headers => ["X-My-Header", "%{host}"]`
+  config :headers, :validate => :hash, :default => {}
+
+  # API Key
+  config :api_key, :validate => :string, :required => :true
+
+  # App Version Id
+  config :app_version_id, :validate => :string, :required => :true
+
+  # Beacon Version Id
+  config :beacon_version_id, :validate => :string, :required => :true
+
+  # Beacon Message Type
+  config :beacon_message_type, :validate => :string, :required => :true
+
+  # Beacon Instance ID
+  config :beacon_instance_id, :validate => :string
+
+  # Data Timestamp
+  config :data_timestamp, :validate => :string
+
+  # Set this to false if you don't want this output to retry failed requests
+  config :retry_failed, :validate => :boolean, :default => true
+
+  # If encountered as response codes this plugin will retry these requests
+  config :retryable_codes, :validate => :number, :list => true, :default => [429, 500, 502, 503, 504]
+
+  # If you would like to consider some non-2xx codes to be successes
+  # enumerate them here. Responses returning these codes will be considered successes
+  config :ignorable_codes, :validate => :number, :list => true
+
+  # This lets you choose the structure and parts of the event that are sent.
+  #
+  #
+  # For example:
+  # [source,ruby]
+  #    mapping => {"foo" => "%{host}"
+  #               "bar" => "%{type}"}
+  config :mapping, :validate => :hash
+
+  # Set the format of the http body.
+  #
+  # If message, then the body will be the result of formatting the event according to message
+  #
+  # Otherwise, the event is sent as json.
+  config :format, :validate => ["json", "message"], :default => "json"
+
+  # Set this to true if you want to enable gzip compression for your http requests
+  #
+  # Compression is not supported at this time
+  #
+  # config :http_compression, :validate => :boolean, :default => false
+
+  config :message, :validate => :string
+
+  public
+  def register
+
+    # We count outstanding requests with this queue
+    # This queue tracks the requests to create backpressure
+    # When this queue is empty no new requests may be sent,
+    # tokens must be added back by the client on success
+    @request_tokens = SizedQueue.new(@pool_max)
+    @pool_max.times {|t| @request_tokens << true }
+
+    @requests = Array.new
+
+    # Compression is not supported at this time
+    #
+    @http_compression = false
+
+    @is_batch = false
+
+    @headers["Content-Type"] = @@content_type
+
+    validate_api_required!
+
+    @headers["apiKey"] = @api_key
+    @headers["appVersionId"] = @app_version_id
+    @headers["beaconVersionId"] = @beacon_version_id
+    @headers["beaconMessageType"] = @beacon_message_type
+
+    if !@beacon_instance_id.nil?
+      @headers["beaconInstanceId"] = @beacon_instance_id.to_s
+    end
+
+    if !@data_timestamp.nil?
+      @headers["dataTimestamp"] = @data_timestamp.to_s
+    end
+
+    validate_format!
+
+    # Run named Timer as daemon thread
+    @timer = java.util.Timer.new("HTTP Output #{self.params['id']}", true)
+  end # def register
+
+  def multi_receive(events)
+    return if events.empty?
+    send_events(events)
+  end
+
+  class RetryTimerTask < java.util.TimerTask
+    def initialize(pending, event, attempt)
+      @pending = pending
+      @event = event
+      @attempt = attempt
+      super()
+    end
+
+    def run
+      @pending << [@event, @attempt]
+    end
+  end
+
+  def log_retryable_response(response)
+    if (response.code == 429)
+      @logger.debug? && @logger.debug("Encountered a 429 response, will retry. This is not serious, just flow control via HTTP")
+    else
+      @logger.warn("Encountered a retryable HTTP request in HTTP output, will retry", :code => response.code, :body => response.body)
+    end
+  end
+
+  def log_error_response(response, url, event)
+    log_failure(
+              "Encountered non-2xx HTTP code #{response.code}",
+              :response_code => response.code,
+              :url => url,
+              :event => event
+            )
+  end
+
+  def send_events(events)
+    successes = java.util.concurrent.atomic.AtomicInteger.new(0)
+    failures  = java.util.concurrent.atomic.AtomicInteger.new(0)
+    retries = java.util.concurrent.atomic.AtomicInteger.new(0)
+    event_count = @is_batch ? 1 : events.size
+
+    pending = Queue.new
+    if @is_batch
+      pending << [events, 0]
+    else
+      events.each {|e| pending << [e, 0]}
+    end
+
+    while popped = pending.pop
+      break if popped == :done
+
+      event, attempt = popped
+
+      action, event, attempt = send_event(event, attempt)
+      begin
+        action = :failure if action == :retry && !@retry_failed
+
+        case action
+        when :success
+          successes.incrementAndGet
+        when :retry
+          retries.incrementAndGet
+
+          next_attempt = attempt+1
+          sleep_for = sleep_for_attempt(next_attempt)
+          @logger.info("Retrying http request, will sleep for #{sleep_for} seconds")
+          timer_task = RetryTimerTask.new(pending, event, next_attempt)
+          @timer.schedule(timer_task, sleep_for*1000)
+        when :failure
+          failures.incrementAndGet
+        else
+          raise "Unknown action #{action}"
+        end
+
+        if action == :success || action == :failure
+          if successes.get+failures.get == event_count
+            pending << :done
+          end
+        end
+      rescue => e
+        # This should never happen unless there's a flat out bug in the code
+        @logger.error("Error sending HTTP Request",
+          :class => e.class.name,
+          :message => e.message,
+          :backtrace => e.backtrace)
+        failures.incrementAndGet
+        raise e
+      end
+    end
+  rescue => e
+    @logger.error("Error in http output loop",
+            :class => e.class.name,
+            :message => e.message,
+            :backtrace => e.backtrace)
+    raise e
+  end
+
+  def sleep_for_attempt(attempt)
+    sleep_for = attempt**2
+    sleep_for = sleep_for <= 60 ? sleep_for : 60
+    (sleep_for/2) + (rand(0..sleep_for)/2)
+  end
+
+  def send_event(event, attempt)
+    body = event_body(event)
+
+    # Send the request
+    url = @is_batch ? @@url : event.sprintf(@@url)
+    headers = @is_batch ? @headers : event_headers(event)
+
+    # Compress the body and add appropriate header
+    if @http_compression == true
+      headers["Content-Encoding"] = "gzip"
+      body = gzip(body)
+    end
+
+    # Create an async request
+    response = client.send(@@http_method, url, :body => body, :headers => headers).call
+
+    if !response_success?(response)
+      if retryable_response?(response)
+        log_retryable_response(response)
+        return :retry, event, attempt
+      else
+        log_error_response(response, url, event)
+        return :failure, event, attempt
+      end
+    else
+      return :success, event, attempt
+    end
+
+  rescue => exception
+    will_retry = retryable_exception?(exception)
+    log_failure("Could not fetch URL",
+                :url => url,
+                :method => @@http_method,
+                :body => body,
+                :headers => headers,
+                :message => exception.message,
+                :class => exception.class.name,
+                :backtrace => exception.backtrace,
+                :will_retry => will_retry
+    )
+
+    if will_retry
+      return :retry, event, attempt
+    else
+      return :failure, event, attempt
+    end
+  end
+
+  def close
+    @timer.cancel
+    client.close
+  end
+
+  private
+
+  def response_success?(response)
+    code = response.code
+    return true if @ignorable_codes && @ignorable_codes.include?(code)
+    return code >= 200 && code <= 299
+  end
+
+  def retryable_response?(response)
+    @retryable_codes && @retryable_codes.include?(response.code)
+  end
+
+  def retryable_exception?(exception)
+    RETRYABLE_MANTICORE_EXCEPTIONS.any? {|me| exception.is_a?(me) }
+  end
+
+  # This is split into a separate method mostly to help testing
+  def log_failure(message, opts)
+    @logger.error("[HTTP Output Failure] #{message}", opts)
+  end
+
+  # Format the HTTP body
+  def event_body(event)
+    # TODO: Create an HTTP post data codec, use that here
+    if @format == "json"
+      LogStash::Json.dump(map_event(event))
+    elsif @format == "message"
+      event.sprintf(@message)
+    else
+      encode(map_event(event))
+    end
+  end
+
+  # gzip data
+  def gzip(data)
+    gz = StringIO.new
+    gz.set_encoding("BINARY")
+    z = Zlib::GzipWriter.new(gz)
+    z.write(data)
+    z.close
+    gz.string
+  end
+
+  def convert_mapping(mapping, event)
+    if mapping.is_a?(Hash)
+      mapping.reduce({}) do |acc, kv|
+        k, v = kv
+        acc[k] = convert_mapping(v, event)
+        acc
+      end
+    elsif mapping.is_a?(Array)
+      mapping.map { |elem| convert_mapping(elem, event) }
+    else
+      event.sprintf(mapping)
+    end
+  end
+
+  def map_event(event)
+    if @mapping
+      convert_mapping(@mapping, event)
+    else
+      event.to_hash
+    end
+  end
+
+  def event_headers(event)
+    custom_headers(event) || {}
+  end
+
+  def custom_headers(event)
+    return nil unless @headers
+
+    @headers.reduce({}) do |acc,kv|
+      k,v = kv
+      acc[k] = event.sprintf(v)
+      acc
+    end
+  end
+
+  #TODO Extract this to a codec
+  def encode(hash)
+    return hash.collect do |key, value|
+      CGI.escape(key) + "=" + CGI.escape(value.to_s)
+    end.join("&")
+  end
+
+
+  def validate_format!
+    if @format == "message"
+      if @message.nil?
+        raise "message must be set if message format is used"
+      end
+
+      unless @mapping.nil?
+        @logger.warn "mapping is not supported and will be ignored if message format is used"
+      end
+    end
+  end
+
+  def validate_api_required!
+    if @api_key.nil?
+      raise "api_key must be set"
+    end
+
+    if @app_version_id.nil?
+      raise "app_version_id must be set"
+    end
+
+    if @beacon_version_id.nil?
+      raise "beacon_version_id must be set"
+    end
+
+    if @beacon_message_type.nil?
+      raise "beacon_message_type must be set"
+    end
+
+  #  # unless @mapping.nil?
+  #  #   @logger.warn "mapping is not supported and will be ignored if message format is used"
+  #  # end
+  end
+
+end # class LogStash::Outputs::HarborBeacon

data/logstash-output-harbor_beacon.gemspec

@@ -0,0 +1,30 @@
+Gem::Specification.new do |s|
+  s.name          = 'logstash-output-harbor_beacon'
+  s.version       = '0.3.0'
+  s.licenses      = ['Apache-2.0']
+  s.summary       = 'Emit output messages (beacons) to Harbor Beacon service.'
+  s.description   = 'This gem is a Logstash output plugin to be installed on top of the Logstash'\
+                    'core  pipeline using $LS_HOME/bin/logstash-plugin install gemname.'\
+                    'This gem is not a stand-alone program.'
+  s.homepage      = 'https://docs.hrbr.io/'
+  s.authors       = ['Harbor']
+  s.email         = 'info@hrbr.io'
+  s.require_paths = ['lib']
+
+  # Files
+  s.files = Dir['lib/**/*','spec/**/*','vendor/**/*','*.gemspec','*.md','CONTRIBUTORS','Gemfile','LICENSE','NOTICE.TXT']
+   # Tests
+  s.test_files = s.files.grep(%r{^(test|spec|features)/})
+
+  # Special flag to let us know this is actually a logstash plugin
+  s.metadata = { "logstash_plugin" => "true", "logstash_group" => "output" }
+
+  # Gem dependencies
+  s.add_runtime_dependency "logstash-core-plugin-api", "~> 2.0"
+  s.add_runtime_dependency "logstash-mixin-http_client", ">= 6.0.0", "< 7.0.0"
+  s.add_runtime_dependency "logstash-codec-plain", "~> 3.0"
+
+  s.add_development_dependency "logstash-devutils", "~> 1.3"
+  # s.add_development_dependency 'sinatra', '~> 1.4'
+  # s.add_development_dependency 'webrick', '~> 1.4'
+end

data/spec/outputs/harbor_beacon_spec.rb

@@ -0,0 +1,35 @@
+require "logstash/devutils/rspec/spec_helper"
+require "logstash/outputs/harbor_beacon"
+require "logstash/codecs/plain"
+
+
+class LogStash::Outputs::HarborBeacon
+  attr_writer :agent
+  attr_reader :request_tokens
+end
+
+describe LogStash::Outputs::HarborBeacon do
+
+  let(:sample_event) { LogStash::Event.new({"message" => "hi"}) }
+
+  let(:api_key) { "ABC" }
+  let(:app_version_id) { "DEF" }
+  let(:beacon_version_id) { "GHI" }
+  let(:beacon_message_type) { "JKL" }
+  let(:sample_config) { {"api_key" => api_key, "app_version_id" => app_version_id, "beacon_version_id" => beacon_version_id, "beacon_message_type" => beacon_message_type } }
+
+  let(:output) { LogStash::Outputs::HarborBeacon.new(sample_config) }
+
+  before do
+      output.register
+  end
+
+  describe "receive message" do
+    subject { output.receive(sample_event) }
+
+    it "returns a string" do
+      #expect(subject).to eq("Event received")
+    end
+  end
+
+end

metadata

@@ -0,0 +1,118 @@
+--- !ruby/object:Gem::Specification
+name: logstash-output-harbor_beacon
+version: !ruby/object:Gem::Version
+  version: 0.3.0
+platform: ruby
+authors:
+- Harbor
+autorequire:
+bindir: bin
+cert_chain: []
+date: 2018-12-21 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
+  name: logstash-core-plugin-api
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 6.0.0
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: 7.0.0
+  name: logstash-mixin-http_client
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 6.0.0
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: 7.0.0
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.0'
+  name: logstash-codec-plain
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3.0'
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.3'
+  name: logstash-devutils
+  prerelease: false
+  type: :development
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.3'
+description: This gem is a Logstash output plugin to be installed on top of the Logstashcore  pipeline
+  using $LS_HOME/bin/logstash-plugin install gemname.This gem is not a stand-alone
+  program.
+email: info@hrbr.io
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- CONTRIBUTORS
+- Gemfile
+- LICENSE
+- README.md
+- lib/logstash/outputs/harbor_beacon.rb
+- logstash-output-harbor_beacon.gemspec
+- spec/outputs/harbor_beacon_spec.rb
+homepage: https://docs.hrbr.io/
+licenses:
+- Apache-2.0
+metadata:
+  logstash_plugin: 'true'
+  logstash_group: output
+post_install_message:
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project:
+rubygems_version: 2.7.6
+signing_key:
+specification_version: 4
+summary: Emit output messages (beacons) to Harbor Beacon service.
+test_files:
+- spec/outputs/harbor_beacon_spec.rb