logstash-output-scalyr 0.1.3 → 0.1.8

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: f07d1addbee89789c0819a0566262a22754990d87a70f479b33534f1d793410d
-  data.tar.gz: 72a1ebd3076e811232cfca2851b923e5fa0a2ef444cf3710d3268de41935532b
+  metadata.gz: 67c7ee2ee65ab144dc7eebd1fa6a553ef1fb4d6c89558c023f334d8a728e5f2e
+  data.tar.gz: 48027aa03b3d0d5ccf51db08e0b2f2bed125c2d571a07bd64dac2e853485cd48
 SHA512:
-  metadata.gz: b1e01a579378b28d69c184e6513d9c33f86963a4ce4bc6078cbbd78c9a380367605e369013417792041eb7b40efa3380c4e2b39b6b3b2bb2ec6761421329b92b
-  data.tar.gz: b5554805e01d55a70b20bfc51c70b1e1bf96164d24c43fe5e56dd1444c3891434a5202aa181341d318787605aacc92a6f21c2ff0601872636ff621de929dc292
+  metadata.gz: 0cb175245132cb45a1e243383b544b9ee1483084b10eac4a4e44ce7fc1e149fb7bba083a3e6df399e090a25693d4637117abf0f5f765a63fad3cb8c293337b4d
+  data.tar.gz: bb8fc93d41d6a05f517c87bbc1e2e86ec4ceb939f1d08b017434a43057e54aa2919f6e052efe435c7aacb1f9ccc6e61a9ee05409fcebccc5ac44a07d72d556b6

data/CHANGELOG.md

@@ -1,4 +1,20 @@
 # Beta
+## 0.1.8
+
+- Add additional metrics.
+- Correctly handle flatten_nested_values_duration metric
+- Add support for setting sampling rate for per event level metrics. It defaults to %5 (``0.05``.)
+- Status log line format has been updated so it doesn't include comma between key=value pair to
+  make parser definition a bit simpler.
+
+## 0.1.7
+ - Tracking of new statistics such as `multi_receive` method duration and batch sizes.
+ - Addition of percentiles for both existing and new stats.
+ - Add ability to define a parser name for status messages using the `status_parser` configuration option.
+
+## 0.1.6
+ - Allow for a customer delimiter when flattening values using the `flatten_nested_values_delimiter` configuration option
+
 ## 0.1.2
  - Remove special treatment of `origin` field in favor of `serverHost`
  - Change concurrency type to `single` to help guarantee one-time ordered delivery in Scalyr

data/Gemfile

@@ -12,3 +12,4 @@ end
 
 gem 'pry'
 gem 'pry-nav'
+gem 'quantile'

data/README.md

@@ -1,7 +1,7 @@
 
 [![CircleCI](https://circleci.com/gh/scalyr/logstash-output-scalyr.svg?style=svg)](https://circleci.com/gh/scalyr/logstash-output-scalyr)
 
-# [Scalyr output plugin for Logstash (Alpha release)]
+# [Scalyr output plugin for Logstash (Beta release)]
 
 This plugin implements a Logstash output plugin that uploads data to [Scalyr](http://www.scalyr.com).
 
@@ -10,7 +10,7 @@ You can view documentation for this plugin [on the Scalyr website](https://app.s
 # Quick start
 
 1. Build the gem, run `gem build logstash-output-scalyr.gemspec` 
-2. Install the gem into a Logstash installation, run `/usr/share/logstash/bin/logstash-plugin install logstash-output-scalyr-0.1.3.gem` or follow the latest official instructions on working with plugins from Logstash.
+2. Install the gem into a Logstash installation, run `/usr/share/logstash/bin/logstash-plugin install logstash-output-scalyr-0.1.8.gem` or follow the latest official instructions on working with plugins from Logstash.
 3. Configure the output plugin (e.g. add it to a pipeline .conf)
 4. Restart Logstash 
 
@@ -48,7 +48,7 @@ In the above example, the Logstash pipeline defines a file input that reads from
 
 ---
 
-- If your Scalyr backend is located in other geographies (such as Europe which would use `https://agent.eu.scalyr.com/`), you may need to modify this
+- If you have an EU-based Scalyr account, please use https://eu.scalyr.com/
 
 `config :scalyr_server, :validate => :string, :default => "https://agent.scalyr.com/"`
 
@@ -56,7 +56,7 @@ In the above example, the Logstash pipeline defines a file input that reads from
 
 - Path to SSL bundle file.
 
-`config :ssl_ca_bundle_path, :validate => :string, :default =>  "/etc/ssl/certs/ca-bundle.crt"`
+`config :ssl_ca_bundle_path, :validate => :string, :default => nil`
 
 ---
 
@@ -115,6 +115,12 @@ In the above example, the Logstash pipeline defines a file input that reads from
 
 ---
 
+- If set, this will change the delimiter used when concatenating nested keys
+
+`config :flatten_nested_values_delimiter, :validate => :string, :default => "_"`
+
+---
+
 - If true, the 'tags' field will be flattened into key-values where each key is a tag and each value is set to
  :flat_tag_value
 
@@ -152,7 +158,6 @@ In the above example, the Logstash pipeline defines a file input that reads from
 
 # Conceptual Overview
 
-
 ## Persistence
 
 Logstash itself supports [Persistent Queues](https://www.elastic.co/guide/en/logstash/current/persistent-queues.html) with at-least-once delivery semantics.  It expects output plugins to retry uploads until success or else to write failures into a Dead-Letter Queue (DLQ). Since Logstash offers Persistent Queues, the Scalyr plugin does not perform its own buffering or persistence.  More specifically, invocation of `multi_receive` is synchronously retried until success or written to the DLQ upon failure.  Note: the `multi_receive` interface does not provide a feedback mechanism (outcome codes etc).
@@ -301,11 +306,64 @@ This repo has been configured to run a full-cycle smoketest on CircleCI as follo
 This repo has unit tests that can be run by running:
 
 ```
-bundle exec rspec
+sudo bundle exec rspec
 ```
 
 in the root of the repo.
 
+By default this will run all the tests (including integration ones which require sudo access and
+may not pass everywhere).
+
+If you want to run just the unit tests, you can run the command displayed below.
+
+
+```bash
+bundle exec rspec spec/logstash/outputs/scalyr_spec.rb spec/scalyr/common/util_spec.rb
+```
+
+## Instrumentation and metrics
+
+By default, plugin logs a special line with metrics to Scalyr every 5 minutes. This line contains
+various batch, request and event level metrics.
+
+Example line is shown below:
+
+```bash
+plugin_status: total_requests_sent=6 total_requests_failed=0 total_request_bytes_sent=240586 total_compressed_request_bytes_sent=6396 total_response_bytes_received=222 total_request_latency_secs=1.018 total_serialization_duration_secs=0.024 total_compression_duration_secs=0.046 compression_type=deflate compression_level=6 request_latency_p50=0.048 request_latency_p90=0.104 request_latency_p99=0.104 serialization_duration_secs_p50=0.003 serialization_duration_secs_p90=0.005 serialization_duration_secs_p99=0.005 compression_duration_secs_p50=0.006 compression_duration_secs_p90=0.013 compression_duration_secs_p99=0.013 bytes_sent_p50=40116 bytes_sent_p90=40116 bytes_sent_p99=40116 total_multi_receive_secs=1.404 multi_receive_duration_p50=0.083 multi_receive_duration_p90=0.192 multi_receive_duration_p99=0.192 multi_receive_event_count_p50=100 multi_receive_event_count_p90=100 multi_receive_event_count_p99=100 event_attributes_count_p50=8 event_attributes_count_p90=8 event_attributes_count_p99=8 flatten_values_duration_secs_p50=0 flatten_values_duration_secs_p90=0 flatten_values_duration_secs_p99=0
+```
+
+Those lines also have ``logstash_plugin_metrics`` ``parser`` attribute defined which means you
+can easily parse them using a parser definition similar to the one below.
+
+```javascript
+{
+    patterns: {
+      nonQuoted: "[^\"\n ]+"
+  },
+
+  formats: [
+    {
+      format: ".* $_=identifier$=$_=nonQuoted$", repeat: true
+    }
+  ]
+}
+```
+
+Do keep in mind that this line contains quite a lot of metrics so parsing it may make it more likely
+to hit server side defined per 5 minute period unique number of attributes limit in case you are already
+very close to tke limit.
+
+For various request level metrics we track totals (either counts or duration). Those metrics names start
+with ``total_`` (e.g. ``total_requests_failed``). To be able to derive average per request values you
+can do that by dividing the total value with the value of ``total_requests_sent`` metrics.
+
+Because averages are not all that useful we also track percentiles for various request, batch and
+event level metrics. Those metrics names end with ``_p{percentile}``. For example ``_p50`` represents
+50th percentile and ``_p99`` represents 99th percentile (e.g. ``request_latency_p99``).
+
+If you want to change status reporting interval you can do that by changing the
+``status_report_interval`` config option (in seconds).
+
 # Releasing
 
 ## Updating version

data/lib/logstash/outputs/scalyr.rb

@@ -14,6 +14,7 @@ require 'net/https'
 require 'rbzip2'
 require 'zlib'
 require 'stringio'
+require 'quantile'
 
 require 'scalyr/common/client'
 require "scalyr/common/util"
@@ -33,11 +34,14 @@ class LogStash::Outputs::Scalyr < LogStash::Outputs::Base
   # The Scalyr API write token, these are available at https://www.scalyr.com/keys.  This is the only compulsory configuration field required for proper upload
   config :api_write_token, :validate => :string, :required => true
 
-  # If your Scalyr backend is located in other geographies (such as Europe which would use `https://agent.eu.scalyr.com/`), you may need to modify this
+  # If you have an EU-based Scalyr account, please use https://eu.scalyr.com/
   config :scalyr_server, :validate => :string, :default => "https://agent.scalyr.com/"
 
   # Path to SSL bundle file.
-  config :ssl_ca_bundle_path, :validate => :string, :default =>  "/etc/ssl/certs/ca-bundle.crt"
+  config :ssl_ca_bundle_path, :validate => :string, :default => "/etc/ssl/certs/ca-bundle.crt"
+
+  # If we should append our built-in Scalyr cert to the one we find at `ssl_ca_bundle_path`.
+  config :append_builtin_cert, :validate => :boolean, :default => true
 
   # server_attributes is a dictionary of key value pairs that represents/identifies the logstash aggregator server
   # (where this plugin is running).  Keys are arbitrary except for the 'serverHost' key which holds special meaning to
@@ -73,6 +77,7 @@ class LogStash::Outputs::Scalyr < LogStash::Outputs::Base
   # If true, nested values will be flattened (which changes keys to delimiter-separated concatenation of all
   # nested keys).
   config :flatten_nested_values, :validate => :boolean, :default => false
+  config :flatten_nested_values_delimiter, :validate => :string, :default => "_" 
 
   # If true, the 'tags' field will be flattened into key-values where each key is a tag and each value is set to
   # :flat_tag_value
@@ -100,6 +105,30 @@ class LogStash::Outputs::Scalyr < LogStash::Outputs::Base
   # An int containing the compression level of compression to use, from 1-9. Defaults to 6
   config :compression_level, :validate => :number, :default => 6
 
+  # How often to log and report status metrics to Scalyr. Defaults to every 5
+  # minutes.
+  config :status_report_interval, :validate => :number, :default => 300
+
+  # Whether or not to count status event uploads in the statistics such as request latency etc.
+  config :record_stats_for_status, :validate => :boolean, :default => false
+
+  # Sample rate for event level metrics (flattening time, number of attributes per event, etc,).
+  # It's important to set this in case there are many events coming in per seconds, because
+  # instrumentation does add some overhead. By default, we sample 5% of the events. Keep in
+  # mind that we use simple random based sampling. Maximum possible value is 1 (aka no sampling
+  # - record metrics for every single event).
+  # We use sampling since Quantile.observe() operation is more expensive than simple counter
+  # based metric so we need to ensure recording a metric doesn't add too much overhead.
+  # Based on micro benchmark, random based sampler is about 5x faster than quantile.observe()
+  config :event_metrics_sample_rate, :validate => :number, :default => 0.05
+
+  # Parser to attach to status events
+  config :status_parser, :validate => :string, :default => "logstash_plugin_metrics"
+
+  # Whether or not to create fresh quantile estimators after a status send. Depending on what you want to gather from
+  # these stas this might be wanted or not.
+  config :flush_quantile_estimates_on_status_send, :validate => :boolean, :default => false
+
   def close
     @running = false
     @client_session.close if @client_session
@@ -107,6 +136,13 @@ class LogStash::Outputs::Scalyr < LogStash::Outputs::Base
 
   public
   def register
+    @prng = Random.new
+
+    if @event_metrics_sample_rate < 0 or @event_metrics_sample_rate > 1
+      raise LogStash::ConfigurationError, "Minimum possible value for 'event_metrics_sample_rate' is 0 (dont sample any events) and maximum is 1 (sample every event)"
+    end
+
+    @node_hostname = Socket.gethostname
 
     if @log_constants and not @log_constants.all? { |x| x.is_a? String }
       raise LogStash::ConfigurationError, "All elements of 'log_constants' must be strings."
@@ -154,7 +190,7 @@ class LogStash::Outputs::Scalyr < LogStash::Outputs::Base
       # Note: Use strings rather than symbols for the key, because keys coming
       # from the config file will be strings
       unless @server_attributes.key? 'serverHost'
-        @server_attributes['serverHost'] = Socket.gethostname
+        @server_attributes['serverHost'] = @node_hostname
       end
     end
 
@@ -172,12 +208,20 @@ class LogStash::Outputs::Scalyr < LogStash::Outputs::Base
     @last_status_transmit_time = nil
     @last_status_ = false
 
+    # Plugin level (either per batch or event level metrics). Other request
+    # level metrics are handled by the HTTP Client class.
+    @multi_receive_statistics = {
+      :total_multi_receive_secs => 0
+    }
+    @plugin_metrics = get_new_metrics
+
     # create a client session for uploading to Scalyr
     @running = true
     @client_session = Scalyr::Common::Client::ClientSession.new(
         @logger, @add_events_uri,
         @compression_type, @compression_level,
-        @ssl_verify_peer, @ssl_ca_bundle_path, @ssl_verify_depth
+        @ssl_verify_peer, @ssl_ca_bundle_path, @ssl_verify_depth,
+        @append_builtin_cert, @record_stats_for_status, @flush_quantile_estimates_on_status_send
     )
 
     @logger.info("Started Scalyr output plugin", :class => self.class.name)
@@ -187,6 +231,15 @@ class LogStash::Outputs::Scalyr < LogStash::Outputs::Base
 
   end # def register
 
+  # Convenience method to create a fresh quantile estimator
+  def get_new_metrics
+    return {
+      :multi_receive_duration_secs => Quantile::Estimator.new,
+      :multi_receive_event_count => Quantile::Estimator.new,
+      :event_attributes_count => Quantile::Estimator.new,
+      :flatten_values_duration_secs => Quantile::Estimator.new
+    }
+  end
 
   # Receive an array of events and immediately upload them (without buffering).
   # The Logstash framework will call this plugin method whenever there is a list of events to upload to Scalyr.
@@ -200,6 +253,7 @@ class LogStash::Outputs::Scalyr < LogStash::Outputs::Base
   #
   public
   def multi_receive(events)
+    start_time = Time.now.to_f
 
     multi_event_request_array = build_multi_event_request_array(events)
     # Loop over all array of multi-event requests, sending each multi-event to Scalyr
@@ -209,18 +263,24 @@ class LogStash::Outputs::Scalyr < LogStash::Outputs::Base
     total_batches = multi_event_request_array.length unless multi_event_request_array.nil?
 
     result = []
+    records_count = events.to_a.length
+
     while !multi_event_request_array.to_a.empty?
       begin
         multi_event_request = multi_event_request_array.pop
-        @client_session.post_add_events(multi_event_request[:body])
-        sleep_interval = 0
-        result.push(multi_event_request)
+        # For some reason a retry on the multi_receive may result in the request array containing `nil` elements, we
+        # ignore these.
+        if !multi_event_request.nil?
+          @client_session.post_add_events(multi_event_request[:body], false, multi_event_request[:serialization_duration])
+          sleep_interval = 0
+          result.push(multi_event_request)
+        end
 
       rescue OpenSSL::SSL::SSLError => e
         # cannot rely on exception message, so we always log the following warning
-        @logger.error "SSL certificate verification failed.  "
-        "Please make sure your certificate bundle is configured correctly and points to a valid file.  "
-        "You can configure this with the ssl_ca_bundle_path configuration option.  "
+        @logger.error "SSL certificate verification failed.  " +
+        "Please make sure your certificate bundle is configured correctly and points to a valid file.  " +
+        "You can configure this with the ssl_ca_bundle_path configuration option.  " +
         "The current value of ssl_ca_bundle_path is '#{@ssl_ca_bundle_path}'"
         @logger.error e.message
         @logger.error "Discarding buffer chunk without retrying."
@@ -229,7 +289,7 @@ class LogStash::Outputs::Scalyr < LogStash::Outputs::Base
         sleep_interval = sleep_for(sleep_interval)
         message = "Error uploading to Scalyr (will backoff-retry)"
         exc_data = {
-            :url => e.url.sanitized.to_s,
+            :url => e.url.to_s,
             :message => e.message,
             :batch_num => batch_num,
             :total_batches => total_batches,
@@ -247,7 +307,7 @@ class LogStash::Outputs::Scalyr < LogStash::Outputs::Base
           # all other failed uploads should be errors
           @logger.error(message, exc_data)
         end
-        retry if @running.true?
+        retry if @running
 
       rescue => e
         # Any unexpected errors should be fully logged
@@ -259,10 +319,16 @@ class LogStash::Outputs::Scalyr < LogStash::Outputs::Base
         )
         @logger.debug("Failed multi_event_request", :multi_event_request => multi_event_request)
         sleep_interval = sleep_for(sleep_interval)
-        retry if @running.true?
+        retry if @running
       end
     end
 
+    if records_count > 0
+      @multi_receive_statistics[:total_multi_receive_secs] += (Time.now.to_f - start_time)
+      @plugin_metrics[:multi_receive_duration_secs].observe(Time.now.to_f - start_time)
+      @plugin_metrics[:multi_receive_event_count].observe(records_count)
+    end
+
     send_status
     return result
   end  # def multi_receive
@@ -286,6 +352,9 @@ class LogStash::Outputs::Scalyr < LogStash::Outputs::Base
   # Tags are either propagated as a comma-separated string, or optionally transposed into key-values where the keys
   # are tag names and the values are 1 (may be configured.)
   def build_multi_event_request_array(logstash_events)
+    if logstash_events.nil? or logstash_events.empty?
+     return []
+    end
 
     multi_event_request_array = Array.new
     total_bytes = 0
@@ -328,8 +397,8 @@ class LogStash::Outputs::Scalyr < LogStash::Outputs::Base
         if standard_field != renamed_field
           if record.key? renamed_field
             if record.key? standard_field
-              @logger.warn "Overwriting log record field '#{standard_field}'.  You are seeing this warning because in "
-              "your LogStash config file you have configured the '#{renamed_field}' field to be converted to the "
+              @logger.warn "Overwriting log record field '#{standard_field}'.  You are seeing this warning because in " +
+              "your LogStash config file you have configured the '#{renamed_field}' field to be converted to the " +
               "'#{standard_field}' field, but the event already contains a field called '#{standard_field}' and "
               "this is now being overwritten."
             end
@@ -353,16 +422,25 @@ class LogStash::Outputs::Scalyr < LogStash::Outputs::Base
 
       # Rename user-specified serverHost field -> 'serverHost'
       rename.call(@serverhost_field, 'serverHost')
-      record.delete(@serverhost_field)
 
       # Rename user-specified logfile field -> 'logfile'
       rename.call(@logfile_field, 'logfile')
-      record.delete(@logfile_field)
+
+      # Set a default parser is none is present in the event
+      if record['parser'].to_s.empty?
+        record['parser'] = "logstashParser"
+      end
+
       # Set logfile field if empty and serverHost is supplied
       if record['logfile'].to_s.empty? and serverHost
         record['logfile'] = "/logstash/#{serverHost}"
       end
 
+      # Set a default if no serverHost value is present.
+      if serverHost.nil?
+        record['serverHost'] = "Logstash"
+      end
+
       log_identifier = nil
       add_log = false
       if serverHost
@@ -406,8 +484,25 @@ class LogStash::Outputs::Scalyr < LogStash::Outputs::Base
         record.delete('tags')
       end
 
+      # Record per-event level metrics (flatten duration, event attributes count). Doing this for every single
+      # event would be somewhat expensive so we use sampling.
+      should_sample_event_metrics = should_sample?
+
       # flatten record
-      record = Scalyr::Common::Util.flatten(record) if @flatten_nested_values
+      if @flatten_nested_values
+        start_time = Time.now.to_f
+        record = Scalyr::Common::Util.flatten(record, delimiter=@flatten_nested_values_delimiter)
+        end_time = Time.now.to_f
+        flatten_nested_values_duration = end_time - start_time
+      end
+
+      if should_sample_event_metrics
+        @plugin_metrics[:event_attributes_count].observe(record.count)
+
+        if @flatten_nested_values
+          @plugin_metrics[:flatten_values_duration_secs].observe(flatten_nested_values_duration)
+        end
+      end
 
       # Use LogStash event.timestamp as the 'ts' Scalyr timestamp.  Note that this may be overwritten by input
       # filters so may not necessarily reflect the actual originating timestamp.
@@ -484,7 +579,6 @@ class LogStash::Outputs::Scalyr < LogStash::Outputs::Base
   end
 
 
-
   # Helper method that adds a client_timestamp to a batch addEvents request body
   def add_client_timestamp_to_body(body)
     current_time_millis = DateTime.now.strftime('%Q').to_i
@@ -494,7 +588,6 @@ class LogStash::Outputs::Scalyr < LogStash::Outputs::Base
   end
 
 
-
   # A request comprises multiple Scalyr Events.  This function creates a request hash for
   # final upload to Scalyr (from an array of events, and an optional hash of current threads)
   # Note: The request body field will be json-encoded.
@@ -529,11 +622,46 @@ class LogStash::Outputs::Scalyr < LogStash::Outputs::Base
     # add serverAttributes
     body[:sessionInfo] = @server_attributes if @server_attributes
 
-    { :body => body.to_json, :record_count => scalyr_events.size }
+    # We time serialization to get some insight on how long it takes to serialize the request body
+    start_time = Time.now.to_f
+    serialized_body = body.to_json
+    end_time = Time.now.to_f
+    serialization_duration = end_time - start_time
+    { :body => serialized_body, :record_count => scalyr_events.size, :serialization_duration => serialization_duration }
 
   end  # def create_multi_event_request
 
 
+  # Retrieve batch and other event level metric values
+  def get_stats
+    current_stats = @multi_receive_statistics.clone
+
+    current_stats[:multi_receive_duration_p50] = @plugin_metrics[:multi_receive_duration_secs].query(0.5)
+    current_stats[:multi_receive_duration_p90] = @plugin_metrics[:multi_receive_duration_secs].query(0.9)
+    current_stats[:multi_receive_duration_p99] = @plugin_metrics[:multi_receive_duration_secs].query(0.99)
+
+    current_stats[:multi_receive_event_count_p50] = @plugin_metrics[:multi_receive_event_count].query(0.5)
+    current_stats[:multi_receive_event_count_p90] = @plugin_metrics[:multi_receive_event_count].query(0.9)
+    current_stats[:multi_receive_event_count_p99] = @plugin_metrics[:multi_receive_event_count].query(0.99)
+
+    current_stats[:event_attributes_count_p50] = @plugin_metrics[:event_attributes_count].query(0.5)
+    current_stats[:event_attributes_count_p90] = @plugin_metrics[:event_attributes_count].query(0.9)
+    current_stats[:event_attributes_count_p99] = @plugin_metrics[:event_attributes_count].query(0.99)
+
+    if @flatten_nested_values
+      # We only return those metrics in case flattening is enabled
+      current_stats[:flatten_values_duration_secs_p50] = @plugin_metrics[:flatten_values_duration_secs].query(0.5)
+      current_stats[:flatten_values_duration_secs_p90] = @plugin_metrics[:flatten_values_duration_secs].query(0.9)
+      current_stats[:flatten_values_duration_secs_p99] = @plugin_metrics[:flatten_values_duration_secs].query(0.99)
+    end
+
+    if @flush_quantile_estimates_on_status_send
+      @logger.debug "Recreating / reseting quantile estimator classes for plugin metrics"
+      @plugin_metrics = get_new_metrics
+    end
+
+    current_stats
+  end
 
 
   # Sends a status update to Scalyr by posting a log entry under the special logfile of 'logstash_plugin.log'
@@ -554,44 +682,59 @@ class LogStash::Outputs::Scalyr < LogStash::Outputs::Base
 
     if !@last_status_transmit_time
       status_event[:attrs]['message'] = "Started Scalyr LogStash output plugin."
+      status_event[:attrs]['serverHost'] = @node_hostname
     else
       cur_time = Time.now()
-      return if (cur_time.to_i - @last_status_transmit_time.to_i) < 300
+      return if (cur_time.to_i - @last_status_transmit_time.to_i) < @status_report_interval
       # echee TODO: get instance stats from session and create a status log line
       msg = 'plugin_status: '
       cnt = 0
       @client_session.get_stats.each do |k, v|
-        val = v.instance_of?(Float) ? sprintf("%.1f", v) : v
-        msg << ', ' if cnt > 0
+        val = v.instance_of?(Float) ? sprintf("%.3f", v) : v
+        val = val.nil? ? 0 : val
+        msg << ' ' if cnt > 0
+        msg << "#{k.to_s}=#{val}"
+        cnt += 1
+      end
+      get_stats.each do |k, v|
+        val = v.instance_of?(Float) ? sprintf("%.3f", v) : v
+        val = val.nil? ? 0 : val
+        msg << ' ' if cnt > 0
         msg << "#{k.to_s}=#{val}"
         cnt += 1
       end
       status_event[:attrs]['message'] = msg
+      status_event[:attrs]['serverHost'] = @node_hostname
+      status_event[:attrs]['parser'] = @status_parser
     end
     multi_event_request = create_multi_event_request([status_event], nil, nil)
-    @client_session.post_add_events(multi_event_request[:body])
+    @client_session.post_add_events(multi_event_request[:body], true, 0)
     @last_status_transmit_time = Time.now()
+    status_event
   end
 
+  # Returns true if we should sample and record metrics for a specific event based on the sampling
+  # rate and random value
+  def should_sample?
+    return @prng.rand(0.0..1.0) < @event_metrics_sample_rate
+  end
 
 
   # Returns true if it is time to transmit status
   def should_transmit_status?
     @last_status_transmit_time_lock.synchronize do
       saved_last_time = @last_status_transmit_time
-      if Time.now.to_i - saved_last_time.to_i > 300
+      if Time.now.to_i - saved_last_time.to_i > @status_report_interval
         @last_status_transmit_time = Float::INFINITY
         return saved_last_time
       end
-    ensure
-
     end
   end
 
 
   # Helper method that performs synchronous sleep for a certain time interval
   def sleep_for(sleep_interval)
-    Stud.stoppable_sleep(sleep_interval) { @running.false? }
+    Stud.stoppable_sleep(sleep_interval) { !@running }
     get_sleep_sec(sleep_interval)
   end
 
@@ -603,12 +746,10 @@ class LogStash::Outputs::Scalyr < LogStash::Outputs::Base
   end
 
 
-
   # Helper method to check if the dead-letter queue is enabled
   def dlq_enabled?
     # echee TODO submit to DLQ
     respond_to?(:execution_context) && execution_context.respond_to?(:dlq_writer) &&
         !execution_context.dlq_writer.inner_writer.is_a?(::LogStash::Util::DummyDeadLetterQueueWriter)
   end
-
 end