logstash-output-sumologic 1.3.1 → 1.4.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 596877e694163207291385444d3e10ced75cba12422d57b21503126395002f03
-  data.tar.gz: 157f6fc6262ef8990d44d597066bbbcde226e27dc3d48d332a29970d38bd962c
+  metadata.gz: a93e4aba1af87a7e0c3ea3570e5ff177e4307736dacf2fde1da78dbe4150a2da
+  data.tar.gz: 9ceb227f390a49467f7195daa96152335922275b714dda87215f11c16273d060
 SHA512:
-  metadata.gz: 2da6958efa3878652965d1620206a3af61172e1f5ff3f369ed738a0346c0c14949cad1c5e36c4d889e8e3dd75373cf8a6f6f0d6be0c8800dfda14b6be855f74b
-  data.tar.gz: b6021badd961cbe617108676906a55e16beeaa169e9f1a3e8a0b1244508c8271e661f7df8c951b38b7132d61588631a00f9595573abf50de43d69ce4dbcb6693
+  metadata.gz: 88b55069b26026545a9cbb755555fa53e0570b120ea37ded4ae487da989f0054a6fb1259998a8e9eb038c0314a4450afa70fe658d372030f4eae0f25e7247b5b
+  data.tar.gz: 6ca3a09f3f297c87c06707dd6a3b74d8210cb46ac60dde3f2663b2509cd87274d92d67d49203194a4def5e70dd314190a4887ac9401199889ecbaf79b22bb77a

data/CHANGELOG.md

@@ -1,5 +1,17 @@
 # Change Log
 
+## 1.4.1 (2022-03-09)
+
+- [#73](https://github.com/SumoLogic/logstash-output-sumologic/pull/73) fix: remove deadlock possibility by adding resend_queue queue
+
+## 1.4.0 (2021-09-27)
+
+- [#68](https://github.com/SumoLogic/logstash-output-sumologic/pull/68) feat: retry on 502 error code
+
+## 1.3.2 (2021-02-03)
+
+- [#60](https://github.com/SumoLogic/logstash-output-sumologic/pull/60) Fix plugin metrics not being sent
+
 ## 1.3.1 (2020-12-18)
 
 - [#53](https://github.com/SumoLogic/logstash-output-sumologic/pull/53) Fix "undefined method `blank?'"

data/CONTRIBUTING.md

@@ -0,0 +1,75 @@
+# Contributing
+
+## Set up development environment
+
+You can use the preconfigured Vagrant virtual machine for development
+It includes everything that is needed to start coding.
+See [./vagrant/README.md](./vagrant/README.md).
+
+If you don't want to or cannot use Vagrant, you need to install the following dependencies:
+
+- [Java SE](http://www.oracle.com/technetwork/java/javase/downloads/index.html) as a prerequisite to JRuby,
+- [JRuby](https://www.jruby.org/),
+- [Bundler](https://bundler.io/),
+- optionally [Docker](https://docs.docker.com/get-docker/), if you want to build and run container images.
+
+When your machine is ready (either in Vagrant or not), run this in the root directory of this repository:
+
+```sh
+bundle install
+```
+
+## Running tests
+
+Some of the tests try to send actual data to an actual Sumo Logic account.
+To run those tests, you need to make `sumo_url` environment variable available.
+If the `sumo_url` environment variable is not present, the tests reaching Sumo Logic will be skipped.
+
+```sh
+export sumo_url=https://events.sumologic.net/receiver/v1/http/XXXXXXXXXX
+```
+
+To run the tests, execute:
+
+```sh
+bundle exec rspec
+```
+
+To run tests in Docker, execute:
+
+```sh
+docker build -t logstash-output-plugin .
+docker run --rm -it -e 'sumo_url=https://events.sumologic.net/receiver/v1/http/XXXXXXXXXX' logstash-output-plugin
+```
+
+## How to build .gem file from repository
+
+Open logstash-output-sumologic.gemspec and make any necessary configuration changes.
+In your local Git clone, run:
+
+```bash
+gem build logstash-output-sumologic.gemspec
+```
+
+You will get a .gem file in the same directory as `logstash-output-sumologic-x.y.z.gem`
+Remove old version of plugin (optional):
+
+```bash
+bin/logstash-plugin remove logstash-output-sumologic
+```
+
+And then install the plugin locally:
+
+```bash
+bin/logstash-plugin install <full path of .gem>
+```
+
+## Continuous Integration
+
+The project uses GitHub Actions for:
+
+- Testing pull requests and main branch commits: [.github/workflows/ci.yml](.github/workflows/ci.yml)
+- Publishing new version of gem to RubyGems.org after tagging: [.github/workflows/publish.yml](.github/workflows/publish.yml)
+
+Before publishing a new version, make sure the RubyGems account has MFA disabled for API access.
+Go to [Settings](https://rubygems.org/settings/edit) and set `MFA Level` to `UI only`.

data/README.md

@@ -112,6 +112,7 @@ Logon to Sumo Logic [web app](https://service.sumologic.com/) and run
 | `fields_include`       | array   | No        | all fields    | Working with `fields_as_metrics` parameter, only the fields which full name matching these RegEx pattern(s) will be included in metrics.
 | `fields_exclude`       | array   | No        | none          | Working with `fields_as_metrics` parameter, the fields which full name matching these RegEx pattern(s) will be ignored in metrics.
 | `sleep_before_requeue` | number  | No        | `30`          | The message failed to send to server will be retried after (x) seconds. Not retried if negative number set
+| `stats_category`       | string  | No        | `Logstash.stats` | The source category this plugin's stats will be sent with when `stats_enabled` is `true`
 | `stats_enabled`        | boolean | No        | `false`       | If `true`, stats of this plugin will be sent as metrics
 | `stats_interval`       | number  | No        | `60`          | The stats will be sent every (x) seconds
 
@@ -136,7 +137,7 @@ On the other side, this version is marked as thread safe so if necessary, multip
 
 ### Monitor throughput in metrics
 
-If your Sumo Logic account supports metrics feature, you can enable the stats monitor of this plugin with configuring `stats_enabled` to `true`. For every `stats_interval` seconds, a batch of metrics data points will be sent to Sumo Logic with source category `XXX.stats` (`XXX` is the source category of main output):
+If your Sumo Logic account supports metrics feature, you can enable the stats monitor of this plugin with configuring `stats_enabled` to `true`. For every `stats_interval` seconds, a batch of metrics data points will be sent to Sumo Logic with source category specified by `stats_category` parameter:
 
 | Metric                          | Description                                                 |
 | ------------------------------- | ----------------------------------------------------------- |

data/lib/logstash/outputs/sumologic/common.rb

@@ -22,6 +22,7 @@ module LogStash; module Outputs; class SumoLogic;
 
     CATEGORY_HEADER = "X-Sumo-Category"
     CATEGORY_HEADER_DEFAULT = "Logstash"
+    CATEGORY_HEADER_DEFAULT_STATS = "Logstash.stats"
     HOST_HEADER = "X-Sumo-Host"
     NAME_HEADER = "X-Sumo-Name"
     NAME_HEADER_DEFAULT = "logstash-output-sumologic"

data/lib/logstash/outputs/sumologic/header_builder.rb

@@ -11,6 +11,7 @@ module LogStash; module Outputs; class SumoLogic;
       
       @extra_headers = config["extra_headers"] ||= {}
       @source_category = config["source_category"] ||= CATEGORY_HEADER_DEFAULT
+      @stats_category = config["stats_category"] ||= CATEGORY_HEADER_DEFAULT_STATS
       @source_host = config["source_host"] ||= Socket.gethostname
       @source_name = config["source_name"] ||= NAME_HEADER_DEFAULT
       @metrics = config["metrics"]
@@ -33,6 +34,18 @@ module LogStash; module Outputs; class SumoLogic;
       headers
     end # def build
 
+    def build_stats()
+      headers = Hash.new
+      headers.merge!(@extra_headers)
+      headers[CLIENT_HEADER] = CLIENT_HEADER_VALUE
+      headers[CATEGORY_HEADER] = @stats_category
+      headers[HOST_HEADER] = Socket.gethostname
+      headers[NAME_HEADER] = NAME_HEADER_DEFAULT
+      headers[CONTENT_TYPE] = CONTENT_TYPE_CARBON2 
+      append_compress_header(headers)
+      headers
+    end # def build_stats
+
     private
     def append_content_header(headers)
       contentType = CONTENT_TYPE_LOG

data/lib/logstash/outputs/sumologic/message_queue.rb

@@ -27,8 +27,8 @@ module LogStash; module Outputs; class SumoLogic;
         end
     end # def enq
 
-    def deq()
-      batch = @queue.deq()
+    def deq(non_block: false)
+      batch = @queue.deq(non_block: non_block)
       batch_size = batch.payload.bytesize
       @stats.record_deque(batch_size)
       @queue_bytesize.update { |v| v - batch_size }

data/lib/logstash/outputs/sumologic/monitor.rb

@@ -14,6 +14,7 @@ module LogStash; module Outputs; class SumoLogic;
       @queue = queue
       @stats = stats
       @stopping = Concurrent::AtomicBoolean.new(false)
+      @header_builder = HeaderBuilder.new(config)
 
       @enabled = config["stats_enabled"] ||= false
       @interval = config["stats_interval"] ||= 60
@@ -27,8 +28,8 @@ module LogStash; module Outputs; class SumoLogic;
         @monitor_t = Thread.new { 
           while @stopping.false?
             Stud.stoppable_sleep(@interval) { @stopping.true? }
-            if @stats.total_input_events.value > 0
-              @queue.enq(build_stats_payload())
+            if @stats.total_log_lines.value > 0 || @stats.total_metrics_datapoints.value > 0
+              @queue.enq(Batch.new(headers(), build_stats_payload()))
             end
           end # while
         }
@@ -72,5 +73,9 @@ module LogStash; module Outputs; class SumoLogic;
       "metric=#{key} interval=#{@interval}  category=monitor #{value} #{timestamp}"
     end # def build_metric_line
 
+    def headers()
+      @headers ||= @header_builder.build_stats()
+    end
+
   end
-end; end; end        
+end; end; end

data/lib/logstash/outputs/sumologic/sender.rb

@@ -24,10 +24,14 @@ module LogStash; module Outputs; class SumoLogic;
       @sender_max = (config["sender_max"] ||= 1) < 1 ? 1 : config["sender_max"]
       @sleep_before_requeue = config["sleep_before_requeue"] ||= 30
       @stats_enabled = config["stats_enabled"] ||= false
+      @iteration_sleep = 0.3
 
       @tokens = SizedQueue.new(@sender_max)
       @sender_max.times { |t| @tokens << t }
 
+      # Make resend_queue twice as big as sender_max,
+      # because if one batch is processed, the next one is already waiting in the thread
+      @resend_queue = SizedQueue.new(2*@sender_max)
       @compressor = LogStash::Outputs::SumoLogic::Compressor.new(config)
 
     end # def initialize
@@ -39,9 +43,24 @@ module LogStash; module Outputs; class SumoLogic;
       @stopping.make_false()
       @sender_t = Thread.new {
         while @stopping.false?
-          batch = @queue.deq()
+          begin
+            # Resend batch if any in the queue
+            batch = @resend_queue.deq(non_block: true)
+          rescue ThreadError
+            # send new batch otherwise
+            begin
+              batch = @queue.deq(non_block: true)
+            rescue ThreadError
+              Stud.stoppable_sleep(@iteration_sleep) { @stopping.true? }
+              next
+            end
+          end
           send_request(batch)
         end # while
+        @resend_queue.size.times.map { |queue|
+          batch = queue.deq()
+          send_request(batch)
+        }
         @queue.drain().map { |batch| 
           send_request(batch)
         }
@@ -98,12 +117,11 @@ module LogStash; module Outputs; class SumoLogic;
         return
       end
       
+      # wait for token so we do not exceed number of request in background
       token = @tokens.pop()
 
       if @stats_enabled && content.start_with?(STATS_TAG)
         body = @compressor.compress(content[STATS_TAG.length..-1])
-        headers[CATEGORY_HEADER] = "#{headers[CATEGORY_HEADER]}.stats"
-        headers[CONTENT_TYPE] = CONTENT_TYPE_CARBON2
       else
         body = @compressor.compress(content)
       end
@@ -113,11 +131,9 @@ module LogStash; module Outputs; class SumoLogic;
         :content_size => content.size,
         :content => content[0..20],
         :payload_size => body.size)
+
+      # send request in background
       request = @client.send(:background).send(:post, @url, :body => body, :headers => headers)
-      
-      request.on_complete do
-        @tokens << token
-      end
   
       request.on_success do |response|
         @stats.record_response_success(response.code)
@@ -127,13 +143,17 @@ module LogStash; module Outputs; class SumoLogic;
             :code => response.code,
             :headers => headers,
             :contet => content[0..20])
-          if response.code == 429 || response.code == 503 || response.code == 504
+          if response.code == 429 || response.code == 502 || response.code == 503 || response.code == 504
+            # requeue and release token
             requeue_message(batch)
+            @tokens << token
           end
         else
           log_dbg("request accepted",
             :token => token,
             :code => response.code)
+          # release token
+          @tokens << token
         end
       end
   
@@ -145,6 +165,8 @@ module LogStash; module Outputs; class SumoLogic;
           :class => exception.class.name,
           :backtrace => exception.backtrace)
         requeue_message(batch)
+        # requeue and release token
+        @tokens << token
       end      
 
       @stats.record_request(content.bytesize, body.bytesize)
@@ -164,9 +186,9 @@ module LogStash; module Outputs; class SumoLogic;
           :content => content[0..20],
           :headers => batch.headers)
         Stud.stoppable_sleep(@sleep_before_requeue) { @stopping.true? }
-        @queue.enq(batch)
+        @resend_queue.enq(batch)
       end
     end # def reque_message
 
   end
-end; end; end
+end; end; end

data/lib/logstash/outputs/sumologic.rb

@@ -102,9 +102,11 @@ class LogStash::Outputs::SumoLogic < LogStash::Outputs::Base
   # For carbon2 metrics format only, define the meta tags (which will NOT be used to identify the metrics)
   config :meta_tags, :validate => :hash, :default => {}
   
-  # For messages fail to send or get 429/503/504 response, try to resend after (x) seconds; don't resend if (x) < 0
+  # For messages fail to send or get 429/502/503/504 response, try to resend after (x) seconds; don't resend if (x) < 0
   config :sleep_before_requeue, :validate => :number, :default => 30
 
+  config :stats_category, :validate => :string, :default => CATEGORY_HEADER_DEFAULT_STATS
+
   # Sending throughput data as metrics
   config :stats_enabled, :validate => :boolean, :default => false
 

data/logstash-output-sumologic.gemspec

@@ -1,6 +1,6 @@
 Gem::Specification.new do |s|
   s.name          = 'logstash-output-sumologic'
-  s.version       = '1.3.1'
+  s.version       = '1.4.1'
   s.licenses      = ['Apache-2.0']
   s.summary       = 'Deliever the log to Sumo Logic cloud service.'
   s.description   = 'This gem is a Logstash output plugin to deliver the log or metrics to Sumo Logic cloud service. Go to https://github.com/SumoLogic/logstash-output-sumologic for getting help, reporting issues, etc.'

data/spec/outputs/sumologic/header_builder_spec.rb

@@ -241,4 +241,90 @@ describe LogStash::Outputs::SumoLogic::HeaderBuilder do
 
   end # context
 
+  context "should build headers for stats" do
+    let(:builder) {
+      LogStash::Outputs::SumoLogic::HeaderBuilder.new("url" => "http://localhost/1234")
+    }
+
+    specify {
+      stats_result = builder.build_stats()
+      expected = {
+        "X-Sumo-Client" => "logstash-output-sumologic",
+        "X-Sumo-Name" => "logstash-output-sumologic",
+        "X-Sumo-Host" => Socket.gethostname,
+        "X-Sumo-Category" => "Logstash.stats",
+        "Content-Type" => "application/vnd.sumologic.carbon2"
+      }
+      expect(stats_result).to eq(expected)
+    }
+  end
+
+  context "should build headers for stats with overridden source category" do
+    let(:builder) {
+      LogStash::Outputs::SumoLogic::HeaderBuilder.new("url" => "http://localhost/1234", "stats_category" => "custom")
+    }
+
+    specify {
+      stats_result = builder.build_stats()
+      expect(stats_result["X-Sumo-Category"]).to eq("custom")
+    }
+  end
+
+  context "should build headers for stats with compression" do
+    let(:builder) {
+      LogStash::Outputs::SumoLogic::HeaderBuilder.new("url" => "http://localhost/1234", "compress" => true, "compress_encoding" => "gzip")
+    }
+
+    specify {
+      stats_result = builder.build_stats()
+      expect(stats_result["Content-Encoding"]).to eq("gzip")
+    }
+  end
+
+  context "should build headers for stats with extra_headers" do
+    let(:builder) {
+      LogStash::Outputs::SumoLogic::HeaderBuilder.new(
+        "url" => "http://localhost/1234",
+        "extra_headers" => {
+          "foo" => "bar"
+        })
+    }
+
+    specify {
+      stats_result = builder.build_stats()
+      expect(stats_result.count).to eq(6)
+      expect(stats_result["foo"]).to eq("bar")
+    }
+  end
+
+  context "should build headers for stats with extra_headers but never overwrite pre-defined headers" do
+    
+    let(:builder) {
+      LogStash::Outputs::SumoLogic::HeaderBuilder.new(
+        "url" => "http://localhost/1234",
+        "extra_headers" => {
+          "foo" => "bar",
+          "X-Sumo-Client" => "a",
+          "X-Sumo-Name" => "b",
+          "X-Sumo-Host" => "c",
+          "X-Sumo-Category" => "d",
+          "Content-Type" => "e"
+      })
+    }
+
+    specify {
+      stats_result = builder.build_stats()
+      expected = {
+        "foo" => "bar",
+        "X-Sumo-Client" => "logstash-output-sumologic",
+        "X-Sumo-Name" => "logstash-output-sumologic",
+        "X-Sumo-Host" => Socket.gethostname,
+        "X-Sumo-Category" => "Logstash.stats",
+        "Content-Type" => "application/vnd.sumologic.carbon2"
+      }
+      expect(stats_result).to eq(expected)
+    }
+
+  end # context
+
 end # describe

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: logstash-output-sumologic
 version: !ruby/object:Gem::Version
-  version: 1.3.1
+  version: 1.4.1
 platform: ruby
 authors:
 - Sumo Logic
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2020-12-18 00:00:00.000000000 Z
+date: 2022-03-09 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: manticore
@@ -107,7 +107,7 @@ extensions: []
 extra_rdoc_files: []
 files:
 - CHANGELOG.md
-- DEVELOPER.md
+- CONTRIBUTING.md
 - Gemfile
 - LICENSE
 - README.md

data/DEVELOPER.md

@@ -1,49 +0,0 @@
-# Development Guide
-
-Logstash output plugin for delivering log to Sumo Logic cloud service through HTTP source.
-
-## How to build .gem file from repository
-
-Open logstash-output-sumologic.gemspec and make any necessary configuration changes.
-In your local Git clone, run:
-
-```bash
-gem build logstash-output-sumologic.gemspec
-```
-
-You will get a .gem file in the same directory as `logstash-output-sumologic-x.y.z.gem`
-Remove old version of plugin (optional):
-
-```bash
-bin/logstash-plugin remove logstash-output-sumologic
-```
-
-And then install the plugin locally:
-
-```bash
-bin/logstash-plugin install <full path of .gem>
-```
-
-## How to run test with rspec
-
-### Running in Docker
-
-```bash
-docker build -t logstash-output-plugin .
-docker run --rm -it -e 'sumo_url=https://events.sumologic.net/receiver/v1/http/XXXXXXXXXX' logstash-output-plugin
-```
-
-### Running on bare metal
-
-The test requires JRuby to run. So you need to install [JRuby](http://jruby.org/), [bundle](https://bundler.io/bundle_install.html) and [RVM](https://rvm.io/) (for switching between JRuby and Ruby) first.
-And then run:
-
-```bash
-rvm use jruby
-bundle install
-export sumo_url=https://events.sumologic.net/receiver/v1/http/XXXXXXXXXX
-rspec spec/
-```
-
-The project is integrated to Travis CI now.
-Make sure [all test passed](https://travis-ci.com/SumoLogic/logstash-output-sumologic) before creating PR