logstash-output-scalyr 0.1.7 → 0.1.12

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 9a0be59b3c2ecaeb444f9fb76e6c97129ab9e1eeaa4a907976e4f6e62a44a582
-  data.tar.gz: 4ddecf13bd1b7800d6d5bad0a271014257159aedfb7eb332ebc8f1c1dd0dd3c5
+  metadata.gz: 8e13b3be77498d8ff0613a34733b477a5c648d48af1070c4ed083d3cd178b69c
+  data.tar.gz: 2873fd90e70e8ac42bfdda09041fc2863eda1a0760efca7203bdc8e63bcbaa5d
 SHA512:
-  metadata.gz: 8ed321b57270cf4bc6fbbc3375f516d08df0592d0476fa71d15af40a7afc1ddb4a97634678b65735101f3c4a17422c0d73e3f40524db06444f8c99cfd3f3a375
-  data.tar.gz: 818943ede0fcad23642366f465c63134500cd70fea625d52313dcc628b3bde3f19aecc6739766690f7d5985c9188af78b5c32e63bcfc4e9a1ca60ee0ceac8743
+  metadata.gz: 192c043d86b0b2b7e12532e16e0c94654789d8a6e959994b317b699b02f318b36a220f1935c5a214174d67f46a5d3dea77b01a3299d290246c59d9ed652de150
+  data.tar.gz: 7b9b18c86a9abe7141c6b0a74ec99a37ee2cc301dcaabf4e2c25bff02673a2388912a3f155af2534a22658165c5998278ba194851a8b96317b02ed3ff127b94f

data/CHANGELOG.md

@@ -1,4 +1,36 @@
 # Beta
+
+## 0.1.12
+- Add logging of successful request retries after an error for additional clarity.
+- Add debug level logging of request body on error.
+
+## 0.1.11.beta
+- Fixes to retry mechanisms.
+- More thorough catching of events, preferring to retry requests rather than crashing the plugin.
+
+## 0.1.10.beta
+
+- Switch to shared concurrency to allow the use of multiple worker threads for increased
+  throughput.
+- Switch HTTP client library to `manticore` to work better with new shared concurrency.
+
+## 0.1.9
+
+- Add support for logging status messages with metrics to stdout in addition to sending this
+  data to Scalyr by setting ``log_status_messages_to_stdout`` config option. By default those
+  lines are logged under INFO log level and you may need to enable / configure pluggin logging
+  as per https://www.elastic.co/guide/en/logstash/current/logging.html.
+- Update metric reporting code to round float values to 4 decimal points so we also record sub
+  millisecond values for per event metrics.
+
+## 0.1.8
+
+- Add additional metrics.
+- Correctly handle flatten_nested_values_duration metric
+- Add support for setting sampling rate for per event level metrics. It defaults to %5 (``0.05``.)
+- Status log line format has been updated so it doesn't include comma between key=value pair to
+  make parser definition a bit simpler.
+
 ## 0.1.7
  - Tracking of new statistics such as `multi_receive` method duration and batch sizes.
  - Addition of percentiles for both existing and new stats.

data/Gemfile

@@ -10,6 +10,11 @@ if Dir.exist?(logstash_path) && use_logstash_source
   gem 'logstash-core-plugin-api', :path => "#{logstash_path}/logstash-core-plugin-api"
 end
 
+group :test do
+  gem "webmock"
+end
+
 gem 'pry'
 gem 'pry-nav'
 gem 'quantile'
+gem 'manticore', platform: :jruby

data/README.md

@@ -10,7 +10,7 @@ You can view documentation for this plugin [on the Scalyr website](https://app.s
 # Quick start
 
 1. Build the gem, run `gem build logstash-output-scalyr.gemspec` 
-2. Install the gem into a Logstash installation, run `/usr/share/logstash/bin/logstash-plugin install logstash-output-scalyr-0.1.7.gem` or follow the latest official instructions on working with plugins from Logstash.
+2. Install the gem into a Logstash installation, run `/usr/share/logstash/bin/logstash-plugin install logstash-output-scalyr-0.1.12.gem` or follow the latest official instructions on working with plugins from Logstash.
 3. Configure the output plugin (e.g. add it to a pipeline .conf)
 4. Restart Logstash 
 
@@ -321,12 +321,55 @@ If you want to run just the unit tests, you can run the command displayed below.
 bundle exec rspec spec/logstash/outputs/scalyr_spec.rb spec/scalyr/common/util_spec.rb
 ```
 
+## Instrumentation and metrics
+
+By default, plugin logs a special line with metrics to Scalyr every 5 minutes. This line contains
+various batch, request and event level metrics.
+
+Example line is shown below:
+
+```bash
+plugin_status: total_requests_sent=6 total_requests_failed=0 total_request_bytes_sent=240586 total_compressed_request_bytes_sent=6396 total_response_bytes_received=222 total_request_latency_secs=1.018 total_serialization_duration_secs=0.024 total_compression_duration_secs=0.046 compression_type=deflate compression_level=6 request_latency_p50=0.048 request_latency_p90=0.104 request_latency_p99=0.104 serialization_duration_secs_p50=0.003 serialization_duration_secs_p90=0.005 serialization_duration_secs_p99=0.005 compression_duration_secs_p50=0.006 compression_duration_secs_p90=0.013 compression_duration_secs_p99=0.013 bytes_sent_p50=40116 bytes_sent_p90=40116 bytes_sent_p99=40116 total_multi_receive_secs=1.404 multi_receive_duration_p50=0.083 multi_receive_duration_p90=0.192 multi_receive_duration_p99=0.192 multi_receive_event_count_p50=100 multi_receive_event_count_p90=100 multi_receive_event_count_p99=100 event_attributes_count_p50=8 event_attributes_count_p90=8 event_attributes_count_p99=8 flatten_values_duration_secs_p50=0 flatten_values_duration_secs_p90=0 flatten_values_duration_secs_p99=0
+```
+
+Those lines also have ``logstash_plugin_metrics`` ``parser`` attribute defined which means you
+can easily parse them using a parser definition similar to the one below.
+
+```javascript
+{
+    patterns: {
+      nonQuoted: "[^\"\n ]+"
+  },
+
+  formats: [
+    {
+      format: ".* $_=identifier$=$_=nonQuoted$", repeat: true
+    }
+  ]
+}
+```
+
+Do keep in mind that this line contains quite a lot of metrics so parsing it may make it more likely
+to hit server side defined per 5 minute period unique number of attributes limit in case you are already
+very close to tke limit.
+
+For various request level metrics we track totals (either counts or duration). Those metrics names start
+with ``total_`` (e.g. ``total_requests_failed``). To be able to derive average per request values you
+can do that by dividing the total value with the value of ``total_requests_sent`` metrics.
+
+Because averages are not all that useful we also track percentiles for various request, batch and
+event level metrics. Those metrics names end with ``_p{percentile}``. For example ``_p50`` represents
+50th percentile and ``_p99`` represents 99th percentile (e.g. ``request_latency_p99``).
+
+If you want to change status reporting interval you can do that by changing the
+``status_report_interval`` config option (in seconds).
+
 # Releasing
 
 ## Updating version
 
 Currently references to the version need to be manually updated, files to look in for this are `logstash-putput-scalyr.gemspec`,
- `client.rb`, twice in the CircleCI `Dockerfile`, and under "Quick Start" in this `README.md`.
+ `lib/scalyr/constants.rb`, and under "Quick Start" in this `README.md`.
 
 The changelog should also be updated with the latest version and changes of note.
 
@@ -335,6 +378,7 @@ The changelog should also be updated with the latest version and changes of note
 To deploy the current code on your machine run these commands:
 
 ```
+rm -rf vendor/
 bundle check --path vendor/bundle || bundle install --deployment
 curl -u RUBY_USER:RUBY_PASSWORD https://rubygems.org/api/v1/api_key.yaml > ~/.gem/credentials
 chmod 0600 ~/.gem/credentials

data/lib/logstash/outputs/scalyr.rb

@@ -8,9 +8,7 @@ require "thread" # for safe queueing
 require "uri" # for escaping user input
 require 'json' # for converting event object to JSON for upload
 
-require 'net/http'
-require 'net/http/persistent'
-require 'net/https'
+require 'manticore'
 require 'rbzip2'
 require 'zlib'
 require 'stringio'
@@ -18,6 +16,7 @@ require 'quantile'
 
 require 'scalyr/common/client'
 require "scalyr/common/util"
+require "scalyr/constants"
 
 
 #---------------------------------------------------------------------------------------------------------------------
@@ -27,9 +26,7 @@ class LogStash::Outputs::Scalyr < LogStash::Outputs::Base
 
   config_name "scalyr"
 
-  # For correctness reasons we need to limit this plugin to a single worker, a single worker will be single concurrency
-  # anyway but we should be explicit.
-  concurrency :single
+  concurrency :shared
 
   # The Scalyr API write token, these are available at https://www.scalyr.com/keys.  This is the only compulsory configuration field required for proper upload
   config :api_write_token, :validate => :string, :required => true
@@ -37,12 +34,6 @@ class LogStash::Outputs::Scalyr < LogStash::Outputs::Base
   # If you have an EU-based Scalyr account, please use https://eu.scalyr.com/
   config :scalyr_server, :validate => :string, :default => "https://agent.scalyr.com/"
 
-  # Path to SSL bundle file.
-  config :ssl_ca_bundle_path, :validate => :string, :default => "/etc/ssl/certs/ca-bundle.crt"
-
-  # If we should append our built-in Scalyr cert to the one we find at `ssl_ca_bundle_path`.
-  config :append_builtin_cert, :validate => :boolean, :default => true
-
   # server_attributes is a dictionary of key value pairs that represents/identifies the logstash aggregator server
   # (where this plugin is running).  Keys are arbitrary except for the 'serverHost' key which holds special meaning to
   # Scalyr and is given special treatment in the Scalyr UI.  All of these attributes are optional (not required for logs
@@ -91,9 +82,14 @@ class LogStash::Outputs::Scalyr < LogStash::Outputs::Base
   # Set max interval in seconds between bulk retries.
   config :retry_max_interval, :validate => :number, :default => 64
 
-  # The following two settings pertain to preventing Man-in-the-middle (MITM) attacks  # echee TODO: eliminate?
+  # Whether or not to verify the connection to Scalyr, only set to false for debugging.
   config :ssl_verify_peer, :validate => :boolean, :default => true
-  config :ssl_verify_depth, :validate => :number, :default => 5
+
+  # Path to SSL bundle file.
+  config :ssl_ca_bundle_path, :validate => :string, :default => "/etc/ssl/certs/ca-bundle.crt"
+
+  # If we should append our built-in Scalyr cert to the one we find at `ssl_ca_bundle_path`.
+  config :append_builtin_cert, :validate => :boolean, :default => true
 
   config :max_request_buffer, :validate => :number, :default => 5500000  # echee TODO: eliminate?
   config :force_message_encoding, :validate => :string, :default => nil
@@ -109,15 +105,47 @@ class LogStash::Outputs::Scalyr < LogStash::Outputs::Base
   # minutes.
   config :status_report_interval, :validate => :number, :default => 300
 
+  # Set to true to also log status messages with various metrics to stdout in addition to sending
+  # this data to Scalyr
+  config :log_status_messages_to_stdout, :validate => :boolean, :default => false
+
   # Whether or not to count status event uploads in the statistics such as request latency etc.
   config :record_stats_for_status, :validate => :boolean, :default => false
 
+  # Sample rate for event level metrics (flattening time, number of attributes per event, etc,).
+  # It's important to set this in case there are many events coming in per seconds, because
+  # instrumentation does add some overhead. By default, we sample 5% of the events. Keep in
+  # mind that we use simple random based sampling. Maximum possible value is 1 (aka no sampling
+  # - record metrics for every single event).
+  # We use sampling since Quantile.observe() operation is more expensive than simple counter
+  # based metric so we need to ensure recording a metric doesn't add too much overhead.
+  # Based on micro benchmark, random based sampler is about 5x faster than quantile.observe()
+  config :event_metrics_sample_rate, :validate => :number, :default => 0.05
+
   # Parser to attach to status events
   config :status_parser, :validate => :string, :default => "logstash_plugin_metrics"
 
   # Whether or not to create fresh quantile estimators after a status send. Depending on what you want to gather from
   # these stas this might be wanted or not.
   config :flush_quantile_estimates_on_status_send, :validate => :boolean, :default => false
+  
+  # Causes this plugin to act as if it successfully uploaded the logs, while actually returning as quickly as possible
+  # after no work being done.
+  config :noop_mode, :validate => :boolean, :default => false
+
+  # Manticore related options
+  config :http_connect_timeout, :validate => :number, :default => 10
+  config :http_socket_timeout, :validate => :number, :default => 10
+  config :http_request_timeout, :validate => :number, :default => 60
+  config :http_pool_max, :validate => :number, :default => 50
+  config :http_pool_max_per_route, :validate => :number, :default => 25
+
+  def initialize(*params)
+    super
+    # Request statistics are accumulated across multiple threads and must be accessed through a mutex
+    @stats_lock = Mutex.new
+    @send_stats = Mutex.new
+  end
 
   def close
     @running = false
@@ -126,6 +154,14 @@ class LogStash::Outputs::Scalyr < LogStash::Outputs::Base
 
   public
   def register
+    # This prng is used exclusively to determine when to sample statistics and no security related purpose, for this
+    # reason we do not ensure thread safety for it.
+    @prng = Random.new
+
+    if @event_metrics_sample_rate < 0 or @event_metrics_sample_rate > 1
+      raise LogStash::ConfigurationError, "Minimum possible value for 'event_metrics_sample_rate' is 0 (dont sample any events) and maximum is 1 (sample every event)"
+    end
+
     @node_hostname = Socket.gethostname
 
     if @log_constants and not @log_constants.all? { |x| x.is_a? String }
@@ -191,23 +227,24 @@ class LogStash::Outputs::Scalyr < LogStash::Outputs::Base
     @last_status_transmit_time_lock = Mutex.new
     @last_status_transmit_time = nil
     @last_status_ = false
-    
-    # Per-multi-receive statistics
+
+    # Plugin level (either per batch or event level metrics). Other request
+    # level metrics are handled by the HTTP Client class.
     @multi_receive_statistics = {
       :total_multi_receive_secs => 0
     }
-    @multi_receive_metrics = get_new_multi_receive_metrics
+    @plugin_metrics = get_new_metrics
 
     # create a client session for uploading to Scalyr
     @running = true
     @client_session = Scalyr::Common::Client::ClientSession.new(
         @logger, @add_events_uri,
-        @compression_type, @compression_level,
-        @ssl_verify_peer, @ssl_ca_bundle_path, @ssl_verify_depth,
-        @append_builtin_cert, @record_stats_for_status, @flush_quantile_estimates_on_status_send
+        @compression_type, @compression_level, @ssl_verify_peer, @ssl_ca_bundle_path, @append_builtin_cert,
+        @record_stats_for_status, @flush_quantile_estimates_on_status_send,
+        @http_connect_timeout, @http_socket_timeout, @http_request_timeout, @http_pool_max, @http_pool_max_per_route
     )
 
-    @logger.info("Started Scalyr output plugin", :class => self.class.name)
+    @logger.info(sprintf("Started Scalyr output plugin (%s)." % [PLUGIN_VERSION]), :class => self.class.name)
 
     # Finally, send a status line to Scalyr
     send_status
@@ -215,10 +252,13 @@ class LogStash::Outputs::Scalyr < LogStash::Outputs::Base
   end # def register
 
   # Convenience method to create a fresh quantile estimator
-  def get_new_multi_receive_metrics
+  def get_new_metrics
     return {
       :multi_receive_duration_secs => Quantile::Estimator.new,
-      :multi_receive_event_count => Quantile::Estimator.new
+      :multi_receive_event_count => Quantile::Estimator.new,
+      :event_attributes_count => Quantile::Estimator.new,
+      :flatten_values_duration_secs => Quantile::Estimator.new,
+      :batches_per_multi_receive => Quantile::Estimator.new
     }
   end
 
@@ -234,84 +274,129 @@ class LogStash::Outputs::Scalyr < LogStash::Outputs::Base
   #
   public
   def multi_receive(events)
-    start_time = Time.now.to_f
+    # Just return and pretend we did something if running in noop mode
+    return events if @noop_mode
 
-    multi_event_request_array = build_multi_event_request_array(events)
-    # Loop over all array of multi-event requests, sending each multi-event to Scalyr
+    begin
+      start_time = Time.now.to_f
 
-    sleep_interval = @retry_initial_interval
-    batch_num = 1
-    total_batches = multi_event_request_array.length unless multi_event_request_array.nil?
+      multi_event_request_array = build_multi_event_request_array(events)
+      # Loop over all array of multi-event requests, sending each multi-event to Scalyr
 
-    result = []
-    records_count = events.to_a.length
+      sleep_interval = @retry_initial_interval
+      batch_num = 1
+      total_batches = multi_event_request_array.length unless multi_event_request_array.nil?
 
-    while !multi_event_request_array.to_a.empty?
-      begin
+      result = []
+      records_count = events.to_a.length
+
+      while !multi_event_request_array.to_a.empty?
         multi_event_request = multi_event_request_array.pop
-        # For some reason a retry on the multi_receive may result in the request array containing `nil` elements, we
-        # ignore these.
-        if !multi_event_request.nil?
-          @client_session.post_add_events(multi_event_request[:body], false, multi_event_request[:serialization_duration], multi_event_request[:flatten_nested_values_duration])
-          sleep_interval = 0
-          result.push(multi_event_request)
+        # Variables to hold information about exceptions we run into, and our handling of retries for this request. We
+        # track this to log it when the retries succeed so we can be sure logs are going through.
+        # General exception info we log in the error
+        exc_data = nil
+        # Whether the exception is commonly retried or not, for determining log level
+        exc_commonly_retried = false
+        # Count of retries attempted for this request
+        exc_retries = 0
+        # Total time spent sleeping while retrying this request due to backoff
+        exc_sleep = 0
+        begin
+          # For some reason a retry on the multi_receive may result in the request array containing `nil` elements, we
+          # ignore these.
+          if !multi_event_request.nil?
+            @client_session.post_add_events(multi_event_request[:body], false, multi_event_request[:serialization_duration])
+
+            sleep_interval = @retry_initial_interval
+            batch_num += 1
+            result.push(multi_event_request)
+          end
+
+        rescue Scalyr::Common::Client::ServerError, Scalyr::Common::Client::ClientError => e
+          sleep_interval = sleep_for(sleep_interval)
+          exc_sleep += sleep_interval
+          exc_retries += 1
+          message = "Error uploading to Scalyr (will backoff-retry)"
+          exc_data = {
+              :url => e.url.to_s,
+              :message => e.message,
+              :batch_num => batch_num,
+              :total_batches => total_batches,
+              :record_count => multi_event_request[:record_count],
+              :payload_size => multi_event_request[:body].bytesize,
+              :will_retry_in_seconds => sleep_interval,
+          }
+          exc_data[:code] = e.code if e.code
+          if @logger.debug? and e.body
+            exc_data[:body] = e.body
+          elsif e.message == "Invalid JSON response from server" and e.body
+            exc_data[:body] = Scalyr::Common::Util.truncate(e.body, 512)
+          end
+          exc_data[:payload] = "\tSample payload: #{request[:body][0,1024]}..." if @logger.debug?
+          if e.is_commonly_retried?
+            # well-known retriable errors should be debug
+            @logger.debug(message, exc_data)
+            exc_commonly_retried = true
+          else
+            # all other failed uploads should be errors
+            @logger.error(message, exc_data)
+            exc_commonly_retried = false
+          end
+          retry if @running
+
+        rescue => e
+          # Any unexpected errors should be fully logged
+          @logger.error(
+              "Unexpected error occurred while uploading to Scalyr (will backoff-retry)",
+              :error_message => e.message,
+              :error_class => e.class.name,
+              :backtrace => e.backtrace
+          )
+          @logger.debug("Failed multi_event_request", :multi_event_request => multi_event_request)
+          sleep_interval = sleep_for(sleep_interval)
+          exc_data = {
+            :error_message => e.message,
+            :error_class => e.class.name,
+            :backtrace => e.backtrace,
+            :multi_event_request => multi_event_request
+          }
+          exc_sleep += sleep_interval
+          exc_retries += 1
+          retry if @running
         end
 
-      rescue OpenSSL::SSL::SSLError => e
-        # cannot rely on exception message, so we always log the following warning
-        @logger.error "SSL certificate verification failed.  " +
-        "Please make sure your certificate bundle is configured correctly and points to a valid file.  " +
-        "You can configure this with the ssl_ca_bundle_path configuration option.  " +
-        "The current value of ssl_ca_bundle_path is '#{@ssl_ca_bundle_path}'"
-        @logger.error e.message
-        @logger.error "Discarding buffer chunk without retrying."
-
-      rescue Scalyr::Common::Client::ServerError, Scalyr::Common::Client::ClientError => e
-        sleep_interval = sleep_for(sleep_interval)
-        message = "Error uploading to Scalyr (will backoff-retry)"
-        exc_data = {
-            :url => e.url.to_s,
-            :message => e.message,
-            :batch_num => batch_num,
-            :total_batches => total_batches,
-            :record_count => multi_event_request[:record_count],
-            :payload_size => multi_event_request[:body].bytesize,
-            :will_retry_in_seconds => sleep_interval,
-        }
-        exc_data[:code] = e.response_code if e.code
-        exc_data[:body] = e.response_body if @logger.debug? and e.body
-        exc_data[:payload] = "\tSample payload: #{request[:body][0,1024]}..." if @logger.debug?
-        if e.is_commonly_retried?
-          # well-known retriable errors should be debug
-          @logger.debug(message, exc_data)
-        else
-          # all other failed uploads should be errors
-          @logger.error(message, exc_data)
+        if !exc_data.nil?
+          message = "Retry successful after error."
+          if exc_commonly_retried
+            @logger.debug(message, :error_data => exc_data, :retries => exc_retries, :sleep_time => exc_sleep)
+          else
+            @logger.info(message, :error_data => exc_data, :retries => exc_retries, :sleep_time => exc_sleep)
+          end
         end
-        retry if @running
+      end
 
-      rescue => e
-        # Any unexpected errors should be fully logged
-        @logger.error(
-            "Unexpected error occurred while uploading to Scalyr (will backoff-retry)",
-            :error_message => e.message,
-            :error_class => e.class.name,
-            :backtrace => e.backtrace
-        )
-        @logger.debug("Failed multi_event_request", :multi_event_request => multi_event_request)
-        sleep_interval = sleep_for(sleep_interval)
-        retry if @running
+      if records_count > 0
+        @stats_lock.synchronize do
+          @multi_receive_statistics[:total_multi_receive_secs] += (Time.now.to_f - start_time)
+          @plugin_metrics[:multi_receive_duration_secs].observe(Time.now.to_f - start_time)
+          @plugin_metrics[:multi_receive_event_count].observe(records_count)
+          @plugin_metrics[:batches_per_multi_receive].observe(total_batches)
+        end
       end
-    end
 
-    if records_count > 0
-      @multi_receive_statistics[:total_multi_receive_secs] += (Time.now.to_f - start_time)
-      @multi_receive_metrics[:multi_receive_duration_secs].observe(Time.now.to_f - start_time)
-      @multi_receive_metrics[:multi_receive_event_count].observe(records_count)
+      send_status
+      return result
+
+    rescue => e
+      # Any unexpected errors should be fully logged
+      @logger.error(
+          "Unexpected error occurred while executing multi_receive.",
+          :error_message => e.message,
+          :error_class => e.class.name,
+          :backtrace => e.backtrace
+      )
     end
-
-    send_status
-    return result
   end  # def multi_receive
 
 
@@ -352,8 +437,6 @@ class LogStash::Outputs::Scalyr < LogStash::Outputs::Base
     logs_ids = Hash.new
     next_log_id = 1
 
-    flatten_nested_values_duration = 0
-
     logstash_events.each {|l_event|
 
       record = l_event.to_hash
@@ -467,16 +550,28 @@ class LogStash::Outputs::Scalyr < LogStash::Outputs::Base
         record.delete('tags')
       end
 
-      # flatten record
-      flatten_nested_values_duration = 0
+      # Record per-event level metrics (flatten duration, event attributes count). Doing this for every single
+      # event would be somewhat expensive so we use sampling.
+      should_sample_event_metrics = should_sample?
 
+      # flatten record
       if @flatten_nested_values
         start_time = Time.now.to_f
-        record = Scalyr::Common::Util.flatten(record, delimiter=@flatten_nested_values_delimiter) if @flatten_nested_values
+        record = Scalyr::Common::Util.flatten(record, delimiter=@flatten_nested_values_delimiter)
         end_time = Time.now.to_f
         flatten_nested_values_duration = end_time - start_time
       end
 
+      if should_sample_event_metrics
+        @stats_lock.synchronize do
+          @plugin_metrics[:event_attributes_count].observe(record.count)
+
+          if @flatten_nested_values
+            @plugin_metrics[:flatten_values_duration_secs].observe(flatten_nested_values_duration)
+          end
+        end
+      end
+
       # Use LogStash event.timestamp as the 'ts' Scalyr timestamp.  Note that this may be overwritten by input
       # filters so may not necessarily reflect the actual originating timestamp.
       scalyr_event = {
@@ -526,7 +621,7 @@ class LogStash::Outputs::Scalyr < LogStash::Outputs::Base
           scalyr_events << scalyr_event
           append_event = false
         end
-        multi_event_request = self.create_multi_event_request(scalyr_events, current_threads, logs, flatten_nested_values_duration)
+        multi_event_request = self.create_multi_event_request(scalyr_events, current_threads, logs)
         multi_event_request_array << multi_event_request
 
         total_bytes = 0
@@ -546,13 +641,12 @@ class LogStash::Outputs::Scalyr < LogStash::Outputs::Base
     }
 
     # create a final request with any left over events
-    multi_event_request = self.create_multi_event_request(scalyr_events, current_threads, logs, flatten_nested_values_duration)
+    multi_event_request = self.create_multi_event_request(scalyr_events, current_threads, logs)
     multi_event_request_array << multi_event_request
     multi_event_request_array
   end
 
 
-
   # Helper method that adds a client_timestamp to a batch addEvents request body
   def add_client_timestamp_to_body(body)
     current_time_millis = DateTime.now.strftime('%Q').to_i
@@ -562,14 +656,13 @@ class LogStash::Outputs::Scalyr < LogStash::Outputs::Base
   end
 
 
-
   # A request comprises multiple Scalyr Events.  This function creates a request hash for
   # final upload to Scalyr (from an array of events, and an optional hash of current threads)
   # Note: The request body field will be json-encoded.
-  def create_multi_event_request(scalyr_events, current_threads, current_logs, flatten_nested_values_duration = 0)
+  def create_multi_event_request(scalyr_events, current_threads, current_logs)
 
     body = {
-      :session => @session_id,
+      :session => @session_id + Thread.current.object_id.to_s,
       :token => @api_write_token,
       :events => scalyr_events,
     }
@@ -602,31 +695,49 @@ class LogStash::Outputs::Scalyr < LogStash::Outputs::Base
     serialized_body = body.to_json
     end_time = Time.now.to_f
     serialization_duration = end_time - start_time
-    { :body => serialized_body, :record_count => scalyr_events.size, :serialization_duration => serialization_duration,
-      :flatten_nested_values_duration => flatten_nested_values_duration }
+    { :body => serialized_body, :record_count => scalyr_events.size, :serialization_duration => serialization_duration }
 
   end  # def create_multi_event_request
 
 
+  # Retrieve batch and other event level metric values
+  def get_stats
+    @stats_lock.synchronize do
+      current_stats = @multi_receive_statistics.clone
+
+      current_stats[:multi_receive_duration_p50] = @plugin_metrics[:multi_receive_duration_secs].query(0.5)
+      current_stats[:multi_receive_duration_p90] = @plugin_metrics[:multi_receive_duration_secs].query(0.9)
+      current_stats[:multi_receive_duration_p99] = @plugin_metrics[:multi_receive_duration_secs].query(0.99)
 
-  def get_batch_stats
-    current_stats = @multi_receive_statistics.clone
+      current_stats[:multi_receive_event_count_p50] = @plugin_metrics[:multi_receive_event_count].query(0.5)
+      current_stats[:multi_receive_event_count_p90] = @plugin_metrics[:multi_receive_event_count].query(0.9)
+      current_stats[:multi_receive_event_count_p99] = @plugin_metrics[:multi_receive_event_count].query(0.99)
 
-    current_stats[:multi_receive_duration_p50] = @multi_receive_metrics[:multi_receive_duration_secs].query(0.5)
-    current_stats[:multi_receive_duration_p90] = @multi_receive_metrics[:multi_receive_duration_secs].query(0.9)
-    current_stats[:multi_receive_duration_p99] = @multi_receive_metrics[:multi_receive_duration_secs].query(0.99)
-    current_stats[:multi_receive_event_count_p50] = @multi_receive_metrics[:multi_receive_event_count].query(0.5)
-    current_stats[:multi_receive_event_count_p90] = @multi_receive_metrics[:multi_receive_event_count].query(0.9)
-    current_stats[:multi_receive_event_count_p99] = @multi_receive_metrics[:multi_receive_event_count].query(0.99)
+      current_stats[:event_attributes_count_p50] = @plugin_metrics[:event_attributes_count].query(0.5)
+      current_stats[:event_attributes_count_p90] = @plugin_metrics[:event_attributes_count].query(0.9)
+      current_stats[:event_attributes_count_p99] = @plugin_metrics[:event_attributes_count].query(0.99)
 
-    if @flush_quantile_estimates_on_status_send
-      @multi_receive_metrics = get_new_multi_receive_metrics
+      current_stats[:batches_per_multi_receive_p50] = @plugin_metrics[:batches_per_multi_receive].query(0.5)
+      current_stats[:batches_per_multi_receive_p90] = @plugin_metrics[:batches_per_multi_receive].query(0.9)
+      current_stats[:batches_per_multi_receive_p99] = @plugin_metrics[:batches_per_multi_receive].query(0.99)
+
+      if @flatten_nested_values
+        # We only return those metrics in case flattening is enabled
+        current_stats[:flatten_values_duration_secs_p50] = @plugin_metrics[:flatten_values_duration_secs].query(0.5)
+        current_stats[:flatten_values_duration_secs_p90] = @plugin_metrics[:flatten_values_duration_secs].query(0.9)
+        current_stats[:flatten_values_duration_secs_p99] = @plugin_metrics[:flatten_values_duration_secs].query(0.99)
+      end
+
+      if @flush_quantile_estimates_on_status_send
+        @logger.debug "Recreating / reseting quantile estimator classes for plugin metrics"
+        @plugin_metrics = get_new_metrics
+      end
+
+      current_stats
     end
-    current_stats
   end
 
 
-
   # Sends a status update to Scalyr by posting a log entry under the special logfile of 'logstash_plugin.log'
   # Instead of creating a separate thread, let this method be invoked once at startup and then every 5 minutes
   # at most.  (If no events are received, no status update will be sent even if 5 minutes has elapsed).
@@ -642,40 +753,59 @@ class LogStash::Outputs::Scalyr < LogStash::Outputs::Base
         'plugin_id' => self.id,
       }
     }
-
-    if !@last_status_transmit_time
-      status_event[:attrs]['message'] = "Started Scalyr LogStash output plugin."
-      status_event[:attrs]['serverHost'] = @node_hostname
-    else
-      cur_time = Time.now()
-      return if (cur_time.to_i - @last_status_transmit_time.to_i) < @status_report_interval
-      # echee TODO: get instance stats from session and create a status log line
-      msg = 'plugin_status: '
-      cnt = 0
-      @client_session.get_stats.each do |k, v|
-        val = v.instance_of?(Float) ? sprintf("%.3f", v) : v
-        val = val.nil? ? 0 : val
-        msg << ', ' if cnt > 0
-        msg << "#{k.to_s}=#{val}"
-        cnt += 1
-      end
-      get_batch_stats.each do |k, v|
-        val = v.instance_of?(Float) ? sprintf("%.3f", v) : v
-        val = val.nil? ? 0 : val
-        msg << ', ' if cnt > 0
-        msg << "#{k.to_s}=#{val}"
-        cnt += 1
+    @send_stats.synchronize do
+      if !@last_status_transmit_time
+        status_event[:attrs]['message'] = sprintf("Started Scalyr LogStash output plugin (%s)." % [PLUGIN_VERSION])
+        status_event[:attrs]['serverHost'] = @node_hostname
+      else
+        cur_time = Time.now()
+        return if (cur_time.to_i - @last_status_transmit_time.to_i) < @status_report_interval
+        # echee TODO: get instance stats from session and create a status log line
+        msg = 'plugin_status: '
+        cnt = 0
+        @client_session.get_stats.each do |k, v|
+          val = v.instance_of?(Float) ? sprintf("%.4f", v) : v
+          val = val.nil? ? 0 : val
+          msg << ' ' if cnt > 0
+          msg << "#{k.to_s}=#{val}"
+          cnt += 1
+        end
+        get_stats.each do |k, v|
+          val = v.instance_of?(Float) ? sprintf("%.4f", v) : v
+          val = val.nil? ? 0 : val
+          msg << ' ' if cnt > 0
+          msg << "#{k.to_s}=#{val}"
+          cnt += 1
+        end
+        status_event[:attrs]['message'] = msg
+        status_event[:attrs]['serverHost'] = @node_hostname
+        status_event[:attrs]['parser'] = @status_parser
       end
-      status_event[:attrs]['message'] = msg
-      status_event[:attrs]['serverHost'] = @node_hostname
-      status_event[:attrs]['parser'] = @status_parser
     end
-    multi_event_request = create_multi_event_request([status_event], nil, nil, nil)
-    @client_session.post_add_events(multi_event_request[:body], true, 0, 0)
+    multi_event_request = create_multi_event_request([status_event], nil, nil)
+    begin
+      @client_session.post_add_events(multi_event_request[:body], true, 0)
+    rescue => e
+      @logger.warn(
+        "Unexpected error occurred while uploading status to Scalyr",
+        :error_message => e.message,
+        :error_class => e.class.name
+      )
+      return
+    end
     @last_status_transmit_time = Time.now()
+
+    if @log_status_messages_to_stdout
+      @logger.info msg
+    end
     status_event
   end
 
+  # Returns true if we should sample and record metrics for a specific event based on the sampling
+  # rate and random value
+  def should_sample?
+    return @prng.rand(0.0..1.0) < @event_metrics_sample_rate
+  end
 
 
   # Returns true if it is time to transmit status
@@ -704,12 +834,10 @@ class LogStash::Outputs::Scalyr < LogStash::Outputs::Base
   end
 
 
-
   # Helper method to check if the dead-letter queue is enabled
   def dlq_enabled?
     # echee TODO submit to DLQ
     respond_to?(:execution_context) && execution_context.respond_to?(:dlq_writer) &&
         !execution_context.dlq_writer.inner_writer.is_a?(::LogStash::Util::DummyDeadLetterQueueWriter)
   end
-
 end