logstash-output-sumologic 1.2.2 → 1.4.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: b6f148933e50f3fad54125d1e006def5f8515add3a030d4e163aca083e6d0c6d
-  data.tar.gz: f1bcdaa22d833c71ac6b6779993427afd500d798417f7f33a498f150367b2f47
+  metadata.gz: ab9e1ae438a4d6030629b00a67c0536aba6995f6a2bae1f952b9737cf2c28189
+  data.tar.gz: fdfec75b2d7c4bcd5c242c1d142d041648d450e3dfcb7a576de9531986f5438d
 SHA512:
-  metadata.gz: 580d1c927da3234976e7c71beea83dcffe5e8dabb2e495f50197f91b8ac156e57740d33433c13a985d332542b2ae38613a3e3167adbe23c3ee3ad4ca2a91452f
-  data.tar.gz: be8464e519361f94ee835c93440ebfc9cb1b169b81cc62a477118753ec048a3742896a112fd82cc9123a8b864be756e7ec1114fcc725b5fa03bee370ba13248f
+  metadata.gz: 66620931f50b2c241835824e26e16e0c05ef8c4ec9ac5ad3770d579ae345ef4486c68de23de8c7c284e3005598231738f75bef59704977a65e3911b992907951
+  data.tar.gz: 294a5910f5ec91ace98405439411d26ddf1e3c858e6d17cefb67d8b2b3c0b534c101cdd93faeab08c34dd42f77f5fb078e03b853e6937d63b1ed9f56ba69b840

data/CHANGELOG.md

@@ -1,5 +1,27 @@
 # Change Log
 
+## 1.4.0 (2021-09-27)
+
+- [#68](https://github.com/SumoLogic/logstash-output-sumologic/pull/68) feat: retry on 502 error code
+
+## 1.3.2 (2021-02-03)
+
+- [#60](https://github.com/SumoLogic/logstash-output-sumologic/pull/60) Fix plugin metrics not being sent
+
+## 1.3.1 (2020-12-18)
+
+- [#53](https://github.com/SumoLogic/logstash-output-sumologic/pull/53) Fix "undefined method `blank?'"
+- [#52](https://github.com/SumoLogic/logstash-output-sumologic/pull/52) Fix logstash-plugin-http_client version conflict in Logstash 7
+
+## 1.3.0
+
+- [#41](https://github.com/SumoLogic/logstash-output-sumologic/pull/41) Provide Docker image with Logstash 6.6 + output plugin on docker hub
+- [#41](https://github.com/SumoLogic/logstash-output-sumologic/pull/41) Kubernetes support with Logstash beats to SumoLogic
+- [#41](https://github.com/SumoLogic/logstash-output-sumologic/pull/41) CI improving 
+- [#36](https://github.com/SumoLogic/logstash-output-sumologic/pull/36) Update sender to send in batch.
+- [#36](https://github.com/SumoLogic/logstash-output-sumologic/pull/36) Support %{} field evaluation in `source_category`, `source_name`, `source_host` parameters
+- [#39](https://github.com/SumoLogic/logstash-output-sumologic/pull/39) Disable cookies by default
+
 ## 1.2.2
 
 - Bug fix: memory leak when using `%{@json}` in format
@@ -21,4 +43,4 @@
 
 ## 1.0.0
 
-- Initial release
+- Initial release

data/CONTRIBUTING.md

@@ -0,0 +1,75 @@
+# Contributing
+
+## Set up development environment
+
+You can use the preconfigured Vagrant virtual machine for development
+It includes everything that is needed to start coding.
+See [./vagrant/README.md](./vagrant/README.md).
+
+If you don't want to or cannot use Vagrant, you need to install the following dependencies:
+
+- [Java SE](http://www.oracle.com/technetwork/java/javase/downloads/index.html) as a prerequisite to JRuby,
+- [JRuby](https://www.jruby.org/),
+- [Bundler](https://bundler.io/),
+- optionally [Docker](https://docs.docker.com/get-docker/), if you want to build and run container images.
+
+When your machine is ready (either in Vagrant or not), run this in the root directory of this repository:
+
+```sh
+bundle install
+```
+
+## Running tests
+
+Some of the tests try to send actual data to an actual Sumo Logic account.
+To run those tests, you need to make `sumo_url` environment variable available.
+If the `sumo_url` environment variable is not present, the tests reaching Sumo Logic will be skipped.
+
+```sh
+export sumo_url=https://events.sumologic.net/receiver/v1/http/XXXXXXXXXX
+```
+
+To run the tests, execute:
+
+```sh
+bundle exec rspec
+```
+
+To run tests in Docker, execute:
+
+```sh
+docker build -t logstash-output-plugin .
+docker run --rm -it -e 'sumo_url=https://events.sumologic.net/receiver/v1/http/XXXXXXXXXX' logstash-output-plugin
+```
+
+## How to build .gem file from repository
+
+Open logstash-output-sumologic.gemspec and make any necessary configuration changes.
+In your local Git clone, run:
+
+```bash
+gem build logstash-output-sumologic.gemspec
+```
+
+You will get a .gem file in the same directory as `logstash-output-sumologic-x.y.z.gem`
+Remove old version of plugin (optional):
+
+```bash
+bin/logstash-plugin remove logstash-output-sumologic
+```
+
+And then install the plugin locally:
+
+```bash
+bin/logstash-plugin install <full path of .gem>
+```
+
+## Continuous Integration
+
+The project uses GitHub Actions for:
+
+- Testing pull requests and main branch commits: [.github/workflows/ci.yml](.github/workflows/ci.yml)
+- Publishing new version of gem to RubyGems.org after tagging: [.github/workflows/publish.yml](.github/workflows/publish.yml)
+
+Before publishing a new version, make sure the RubyGems account has MFA disabled for API access.
+Go to [Settings](https://rubygems.org/settings/edit) and set `MFA Level` to `UI only`.

data/README.md

@@ -1,6 +1,6 @@
 # Logstash Sumo Logic Output Plugin
 
-[![Build Status](https://travis-ci.org/SumoLogic/logstash-output-sumologic.svg?branch=master)](https://travis-ci.org/SumoLogic/logstash-output-sumologic)  [![Gem Version](https://badge.fury.io/rb/logstash-output-sumologic.svg)](https://badge.fury.io/rb/logstash-output-sumologic)
+[![Build Status](https://travis-ci.org/SumoLogic/logstash-output-sumologic.svg?branch=master)](https://travis-ci.org/SumoLogic/logstash-output-sumologic)  [![Gem Version](https://badge.fury.io/rb/logstash-output-sumologic.svg)](https://badge.fury.io/rb/logstash-output-sumologic)  [![Docker Pulls](https://img.shields.io/docker/pulls/sumologic/logstash-output-sumologic.svg)](https://hub.docker.com/r/sumologic/logstash-output-sumologic)
 
 This is an output plugin for [Logstash](https://github.com/elastic/logstash).
 It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
@@ -112,6 +112,7 @@ Logon to Sumo Logic [web app](https://service.sumologic.com/) and run
 | `fields_include`       | array   | No        | all fields    | Working with `fields_as_metrics` parameter, only the fields which full name matching these RegEx pattern(s) will be included in metrics.
 | `fields_exclude`       | array   | No        | none          | Working with `fields_as_metrics` parameter, the fields which full name matching these RegEx pattern(s) will be ignored in metrics.
 | `sleep_before_requeue` | number  | No        | `30`          | The message failed to send to server will be retried after (x) seconds. Not retried if negative number set
+| `stats_category`       | string  | No        | `Logstash.stats` | The source category this plugin's stats will be sent with when `stats_enabled` is `true`
 | `stats_enabled`        | boolean | No        | `false`       | If `true`, stats of this plugin will be sent as metrics
 | `stats_interval`       | number  | No        | `60`          | The stats will be sent every (x) seconds
 
@@ -136,7 +137,7 @@ On the other side, this version is marked as thread safe so if necessary, multip
 
 ### Monitor throughput in metrics
 
-If your Sumo Logic account supports metrics feature, you can enable the stats monitor of this plugin with configuring `stats_enabled` to `true`. For every `stats_interval` seconds, a batch of metrics data points will be sent to Sumo Logic with source category `XXX.stats` (`XXX` is the source category of main output):
+If your Sumo Logic account supports metrics feature, you can enable the stats monitor of this plugin with configuring `stats_enabled` to `true`. For every `stats_interval` seconds, a batch of metrics data points will be sent to Sumo Logic with source category specified by `stats_category` parameter:
 
 | Metric                          | Description                                                 |
 | ------------------------------- | ----------------------------------------------------------- |

data/lib/logstash/outputs/sumologic/batch.rb

@@ -0,0 +1,13 @@
+# encoding: utf-8
+
+module LogStash; module Outputs; class SumoLogic;
+  class Batch
+    
+    attr_accessor :headers, :payload
+  
+    def initialize(headers, payload)
+        @headers, @payload = headers, payload
+    end
+  
+  end
+end; end; end;

data/lib/logstash/outputs/sumologic/common.rb

@@ -1,9 +1,9 @@
 # encoding: utf-8
-require "date"
-
 module LogStash; module Outputs; class SumoLogic;
   module Common
 
+    require "date"
+
     # global constants
     DEFAULT_LOG_FORMAT = "%{@timestamp} %{host} %{message}"
     METRICS_NAME_PLACEHOLDER = "*"
@@ -12,6 +12,23 @@ module LogStash; module Outputs; class SumoLogic;
     DEFLATE = "deflate"
     GZIP = "gzip"
     STATS_TAG = "STATS_TAG"
+    STOP_TAG = "PLUGIN STOPPED"
+
+    CONTENT_TYPE = "Content-Type"
+    CONTENT_TYPE_LOG = "text/plain"
+    CONTENT_TYPE_GRAPHITE = "application/vnd.sumologic.graphite"
+    CONTENT_TYPE_CARBON2 = "application/vnd.sumologic.carbon2"
+    CONTENT_ENCODING = "Content-Encoding"
+
+    CATEGORY_HEADER = "X-Sumo-Category"
+    CATEGORY_HEADER_DEFAULT = "Logstash"
+    CATEGORY_HEADER_DEFAULT_STATS = "Logstash.stats"
+    HOST_HEADER = "X-Sumo-Host"
+    NAME_HEADER = "X-Sumo-Name"
+    NAME_HEADER_DEFAULT = "logstash-output-sumologic"
+
+    CLIENT_HEADER = "X-Sumo-Client"
+    CLIENT_HEADER_VALUE = "logstash-output-sumologic"
 
     # for debugging test
     LOG_TO_CONSOLE = false
@@ -53,5 +70,15 @@ module LogStash; module Outputs; class SumoLogic;
       end
     end # def log_dbg
 
+    def blank?(value)
+      if value.kind_of?(NilClass)
+        true
+      elsif value.kind_of?(String)
+        value !~ /\S/
+      else
+        value.respond_to?(:empty?) ? value.empty? : !value
+      end
+    end
+
   end
-end; end; end
+end; end; end

data/lib/logstash/outputs/sumologic/compressor.rb

@@ -1,11 +1,11 @@
 # encoding: utf-8
-require "stringio"
-require "zlib"
-require "logstash/outputs/sumologic/common"
 
 module LogStash; module Outputs; class SumoLogic;
   class Compressor
 
+    require "stringio"
+    require "zlib"
+    require "logstash/outputs/sumologic/common"
     include LogStash::Outputs::SumoLogic::Common
 
     def initialize(config)

data/lib/logstash/outputs/sumologic/header_builder.rb

@@ -1,31 +1,17 @@
 # encoding: utf-8
-require "socket"
-require "logstash/outputs/sumologic/common"
 
 module LogStash; module Outputs; class SumoLogic;
   class HeaderBuilder
 
+    require "socket"
+    require "logstash/outputs/sumologic/common"
     include LogStash::Outputs::SumoLogic::Common
 
-    CONTENT_TYPE = "Content-Type"
-    CONTENT_TYPE_LOG = "text/plain"
-    CONTENT_TYPE_GRAPHITE = "application/vnd.sumologic.graphite"
-    CONTENT_TYPE_CARBON2 = "application/vnd.sumologic.carbon2"
-    CONTENT_ENCODING = "Content-Encoding"
-
-    CATEGORY_HEADER = "X-Sumo-Category"
-    CATEGORY_HEADER_DEFAULT = "Logstash"
-    HOST_HEADER = "X-Sumo-Host"
-    NAME_HEADER = "X-Sumo-Name"
-    NAME_HEADER_DEFAULT = "logstash-output-sumologic"
-
-    CLIENT_HEADER = "X-Sumo-Client"
-    CLIENT_HEADER_VALUE = "logstash-output-sumologic"
-
     def initialize(config)
       
       @extra_headers = config["extra_headers"] ||= {}
       @source_category = config["source_category"] ||= CATEGORY_HEADER_DEFAULT
+      @stats_category = config["stats_category"] ||= CATEGORY_HEADER_DEFAULT_STATS
       @source_host = config["source_host"] ||= Socket.gethostname
       @source_name = config["source_name"] ||= NAME_HEADER_DEFAULT
       @metrics = config["metrics"]
@@ -36,31 +22,31 @@ module LogStash; module Outputs; class SumoLogic;
 
     end # def initialize
     
-    def build()
-      headers = build_common()
-      headers[CATEGORY_HEADER] = @source_category unless @source_category.blank?
+    def build(event)
+      headers = Hash.new
+      headers.merge!(@extra_headers)
+      headers[CLIENT_HEADER] = CLIENT_HEADER_VALUE
+      headers[CATEGORY_HEADER] = event.sprintf(@source_category) unless blank?(@source_category)
+      headers[HOST_HEADER] = event.sprintf(@source_host) unless blank?(@source_host)
+      headers[NAME_HEADER] = event.sprintf(@source_name) unless blank?(@source_name)
       append_content_header(headers)
+      append_compress_header(headers)
       headers
     end # def build
 
     def build_stats()
-      headers = build_common()
-      headers[CATEGORY_HEADER] = "#{@source_category}.stats"
-      headers[CONTENT_TYPE] = CONTENT_TYPE_CARBON2
-      headers
-    end # def build_stats
-
-    private
-    def build_common()
-      headers = Hash.new()
+      headers = Hash.new
       headers.merge!(@extra_headers)
       headers[CLIENT_HEADER] = CLIENT_HEADER_VALUE
-      headers[HOST_HEADER] = @source_host unless @source_host.blank?
-      headers[NAME_HEADER] = @source_name unless @source_name.blank?
+      headers[CATEGORY_HEADER] = @stats_category
+      headers[HOST_HEADER] = Socket.gethostname
+      headers[NAME_HEADER] = NAME_HEADER_DEFAULT
+      headers[CONTENT_TYPE] = CONTENT_TYPE_CARBON2 
       append_compress_header(headers)
       headers
-    end # build_common
+    end # def build_stats
 
+    private
     def append_content_header(headers)
       contentType = CONTENT_TYPE_LOG
       if @metrics || @fields_as_metrics
@@ -76,4 +62,4 @@ module LogStash; module Outputs; class SumoLogic;
     end # append_compress_header
 
   end
-end; end; end
+end; end; end

data/lib/logstash/outputs/sumologic/message_queue.rb

@@ -1,38 +1,42 @@
 # encoding: utf-8
-require "logstash/outputs/sumologic/common"
-require "logstash/outputs/sumologic/statistics"
-
 module LogStash; module Outputs; class SumoLogic;
   class MessageQueue
 
+    require "logstash/outputs/sumologic/common"
+    require "logstash/outputs/sumologic/statistics"
     include LogStash::Outputs::SumoLogic::Common
 
     def initialize(stats, config)
       @queue_max = (config["queue_max"] ||= 1) < 1 ? 1 : config["queue_max"]
       @queue = SizedQueue::new(@queue_max)
       log_info("initialize memory queue", :max => @queue_max)
+      @queue_bytesize = Concurrent::AtomicFixnum.new
       @stats = stats
     end # def initialize
 
-    def enq(obj)
-      if (obj.bytesize > 0)
-        @queue.enq(obj)
-        @stats.record_enque(obj)
+    def enq(batch)
+      batch_size = batch.payload.bytesize
+      if (batch_size > 0)
+        @queue.enq(batch)
+        @stats.record_enque(batch_size)
+        @queue_bytesize.update { |v| v + batch_size }
         log_dbg("enqueue",
           :objects_in_queue => size,
-          :bytes_in_queue => @stats.current_queue_bytes,
-          :size => obj.bytesize)
-      end
+          :bytes_in_queue => @queue_bytesize,
+          :size => batch_size)
+        end
     end # def enq
 
     def deq()
-      obj = @queue.deq()
-      @stats.record_deque(obj)
+      batch = @queue.deq()
+      batch_size = batch.payload.bytesize
+      @stats.record_deque(batch_size)
+      @queue_bytesize.update { |v| v - batch_size }
       log_dbg("dequeue",
         :objects_in_queue => size,
-        :bytes_in_queue => @stats.current_queue_bytes,
-        :size => obj.bytesize)
-      obj
+        :bytes_in_queue => @queue_bytesize,
+        :size => batch_size)
+      batch
     end # def deq
 
     def drain()
@@ -44,6 +48,10 @@ module LogStash; module Outputs; class SumoLogic;
     def size()
       @queue.size()
     end # size
+
+    def bytesize()
+      @queue_bytesize.value
+    end # bytesize
     
   end
 end; end; end

data/lib/logstash/outputs/sumologic/monitor.rb

@@ -1,11 +1,11 @@
 # encoding: utf-8
-require "logstash/outputs/sumologic/common"
-require "logstash/outputs/sumologic/statistics"
-require "logstash/outputs/sumologic/message_queue"
 
 module LogStash; module Outputs; class SumoLogic;
   class Monitor
 
+    require "logstash/outputs/sumologic/common"
+    require "logstash/outputs/sumologic/statistics"
+    require "logstash/outputs/sumologic/message_queue"
     include LogStash::Outputs::SumoLogic::Common
 
     attr_reader :is_pile
@@ -14,6 +14,7 @@ module LogStash; module Outputs; class SumoLogic;
       @queue = queue
       @stats = stats
       @stopping = Concurrent::AtomicBoolean.new(false)
+      @header_builder = HeaderBuilder.new(config)
 
       @enabled = config["stats_enabled"] ||= false
       @interval = config["stats_interval"] ||= 60
@@ -27,8 +28,8 @@ module LogStash; module Outputs; class SumoLogic;
         @monitor_t = Thread.new { 
           while @stopping.false?
             Stud.stoppable_sleep(@interval) { @stopping.true? }
-            if @stats.total_input_events.value > 0
-              @queue.enq(build_stats_payload())
+            if @stats.total_log_lines.value > 0 || @stats.total_metrics_datapoints.value > 0
+              @queue.enq(Batch.new(headers(), build_stats_payload()))
             end
           end # while
         }
@@ -72,5 +73,9 @@ module LogStash; module Outputs; class SumoLogic;
       "metric=#{key} interval=#{@interval}  category=monitor #{value} #{timestamp}"
     end # def build_metric_line
 
+    def headers()
+      @headers ||= @header_builder.build_stats()
+    end
+
   end
-end; end; end        
+end; end; end

data/lib/logstash/outputs/sumologic/payload_builder.rb

@@ -1,12 +1,11 @@
 # encoding: utf-8
-require "logstash/json"
-require "logstash/event"
-
-require "logstash/outputs/sumologic/common"
 
 module LogStash; module Outputs; class SumoLogic;
   class PayloadBuilder
     
+    require "logstash/json"
+    require "logstash/event"
+    require "logstash/outputs/sumologic/common"
     include LogStash::Outputs::SumoLogic::Common
 
     TIMESTAMP_FIELD = "@timestamp"

data/lib/logstash/outputs/sumologic/piler.rb

@@ -1,11 +1,11 @@
 # encoding: utf-8
-require "logstash/outputs/sumologic/common"
-require "logstash/outputs/sumologic/statistics"
-require "logstash/outputs/sumologic/message_queue"
 
 module LogStash; module Outputs; class SumoLogic;
   class Piler
 
+    require "logstash/outputs/sumologic/common"
+    require "logstash/outputs/sumologic/statistics"
+    require "logstash/outputs/sumologic/message_queue"
     include LogStash::Outputs::SumoLogic::Common
 
     attr_reader :is_pile
@@ -17,14 +17,14 @@ module LogStash; module Outputs; class SumoLogic;
       @queue = queue
       @stats = stats
       @stopping = Concurrent::AtomicBoolean.new(false)
+      @payload_builder = PayloadBuilder.new(@stats, config)
+      @header_builder = HeaderBuilder.new(config)
       @is_pile = (@interval > 0 && @pile_max > 0)
-
       if (@is_pile)
-        @pile = Array.new
-        @pile_size = 0
+        @pile = Hash.new("")
         @semaphore = Mutex.new
       end
-
+  
     end # def initialize
 
     def start()
@@ -46,46 +46,44 @@ module LogStash; module Outputs; class SumoLogic;
     def stop()
       @stopping.make_true()
       if (@is_pile)
-        log_info("shutting down piler...")
-        @piler_t.join
+        log_info("shutting down piler in #{@interval * 2} secs ...")
+        @piler_t.join(@interval * 2)
         log_info("piler is fully shutted down")
       end
     end # def stop
 
-    def input(entry)
+    def input(event)
       if (@stopping.true?)
-        log_warn("piler is shutting down, message dropped", 
-          "message" => entry)
-      elsif (@is_pile)
-        @semaphore.synchronize {
-          if @pile_size + entry.bytesize > @pile_max
-            @queue.enq(@pile.join($/))
-            @pile.clear
-            @pile_size = 0
-            @stats.record_clear_pile()
-          end
-          @pile << entry
-          @pile_size += entry.bytesize
-          @stats.record_input(entry)
-        }
+        log_warn("piler is shutting down, event is dropped", 
+          "event" => event)
       else
-        @queue.enq(entry)
-      end # if
+        headers = @header_builder.build(event)
+        payload = @payload_builder.build(event)
+        if (@is_pile)
+          @semaphore.synchronize {
+            content = @pile[headers]
+            size = content.bytesize
+            if size + payload.bytesize > @pile_max
+              @queue.enq(Batch.new(headers, content))
+              @pile[headers] = ""
+            end
+            @pile[headers] = blank?(@pile[headers]) ? payload : "#{@pile[headers]}\n#{payload}"
+          }
+        else
+          @queue.enq(Batch.new(headers, payload))
+        end # if
+      end
     end # def input
 
     private
     def enq_and_clear()
-      if (@pile.size > 0)
-        @semaphore.synchronize {
-          if (@pile.size > 0)
-            @queue.enq(@pile.join($/))
-            @pile.clear
-            @pile_size = 0
-            @stats.record_clear_pile()
-          end
-        }
-      end
+      @semaphore.synchronize {
+        @pile.each do |headers, content|
+          @queue.enq(Batch.new(headers, content))
+        end
+        @pile.clear()
+      }
     end # def enq_and_clear
 
   end
-end; end; end
+end; end; end