logstash-output-scalyr 0.1.2

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: ef843ec2b97b5a4f7f1f3005500cdc7e89bcf32a3a7766eb1528f866d4a9c7ee
+  data.tar.gz: ebdd84bc5c603ec0eaaf54467a4dff9cfe94badcaa2f32c172067eeadab8b759
+SHA512:
+  metadata.gz: 2803a5befd9d127b23c0fa2a17aa0333d7e4616d6c2976584018b1244bab7d9e55ba205ebb65bfc701dd2fe500c4d0ad549f7d4fd2783b410b99c647827b9c53
+  data.tar.gz: cc745e6249def709126df87c5333469fe44ebbf6c1f6919c804ed1e424f72087c4bd8bc3ca64dfad69f0e2595c14a799dd659ccb4a2dd0b66ce5fc6a81fa7117

data/CHANGELOG.md

@@ -0,0 +1,21 @@
+# Beta
+## 0.1.2
+ - Remove special treatment of `origin` field in favor of `serverHost`
+ - Change concurrency type to `single` to help guarantee one-time ordered delivery in Scalyr
+ - Change default compression to `deflate`
+ - Don't use aggregator hostname as `serverHost` by default
+ - Update upload request format to match latest Scalyr API
+ - Add `User-Agent` header to Scalyr requests
+
+# Alpha
+## 0.0.4
+  - Docs: Set the default_codec doc attribute.
+## 0.0.3
+ - Docs: Add documentation template
+## 0.0.2
+ - Add encoding: utf-8 to spec files. This can help prevent issues during testing.
+## 0.0.1
+ - Plugins were updated to follow the new shutdown semantic, this mainly allows Logstash to instruct input plugins to terminate gracefully, 
+   instead of using Thread.raise on the plugins' threads. Ref: https://github.com/elastic/logstash/pull/3895
+ - Dependency on logstash-core update to 2.0
+

data/CONTRIBUTORS

@@ -0,0 +1,4 @@
+The following is a list of people who have contributed to the Logsstash output plugin for Scalyr.
+
+Contributors:
+* Edward Chee

data/DEVELOPER.md

@@ -0,0 +1,2 @@
+# logstash-output-example
+Example output plugin. This should help bootstrap your effort to write your own output plugin!

data/Gemfile

@@ -0,0 +1,14 @@
+source 'https://rubygems.org'
+
+gemspec
+
+logstash_path = ENV["LOGSTASH_PATH"] || "../../logstash"
+use_logstash_source = ENV["LOGSTASH_SOURCE"] && ENV["LOGSTASH_SOURCE"].to_s == "1"
+
+if Dir.exist?(logstash_path) && use_logstash_source
+  gem 'logstash-core', :path => "#{logstash_path}/logstash-core"
+  gem 'logstash-core-plugin-api', :path => "#{logstash_path}/logstash-core-plugin-api"
+end
+
+gem 'pry'
+gem 'pry-nav'

data/LICENSE

@@ -0,0 +1,13 @@
+Copyright (c) 2019 Scalyr <http://www.scalyr.com>
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

data/NOTICE.TXT

@@ -0,0 +1,5 @@
+Scalyr
+Copyright 2019 Scalyr
+
+This product includes software developed by The Apache Software
+Foundation (http://www.apache.org/).

data/README.md

@@ -0,0 +1,307 @@
+
+[![CircleCI](https://circleci.com/gh/scalyr/logstash-output-scalyr.svg?style=svg)](https://circleci.com/gh/scalyr/logstash-output-scalyr)
+
+# [Scalyr output plugin for Logstash (Alpha release)]
+
+This plugin implements a Logstash output plugin that uploads data to [Scalyr](http://www.scalyr.com).
+
+You can view documentation for this plugin [on the Scalyr website](https://www.scalyr.com/solutions/logstash).
+
+# Quick start
+
+1. Build the gem, run `gem build logstash-output-scalyr.gemspec` 
+2. Install the gem into a Logstash installation, run `/usr/share/logstash/bin/logstash-plugin install logstash-output-scalyr-0.1.2.gem` or follow the latest official instructions on working with plugins from Logstash.
+3. Configure the output plugin (e.g. add it to a pipeline .conf)
+4. Restart Logstash 
+
+# Configuration
+
+The Scalyr output plugin has a number of sensible defaults so the minimum configuration only requires your `api_write_token` for upload access.
+
+Plugin configuration is achieved by adding an output section to the appropriate config file for your Logstash event pipeline: 
+
+```
+my_pipeline.conf
+
+input {   
+  file {  
+    path => "/var/log/messages"  
+  }  
+}
+
+output {
+ scalyr {
+   api_write_token => 'SCALYR_API_KEY'
+   serverhost_field => 'host'
+   logfile_field => 'path'
+ }
+}
+```
+
+In the above example, the Logstash pipeline defines a file input that reads from `/var/log/messages`.  Log events from this source have the `host` and `path` fields.  The pipeline then outputs to the scalyr plugin, which in this example is configured to remap `host`->`serverHost` and `path`->`logfile`, thus facilitating filtering in the Scalyr UI.
+
+## Options
+
+- The Scalyr API write token, these are available at https://www.scalyr.com/keys.  This is the only compulsory configuration field required for proper upload
+
+`config :api_write_token, :validate => :string, :required => true`
+
+---
+
+- If your Scalyr backend is located in other geographies (such as Europe which would use `https://agent.eu.scalyr.com/`), you may need to modify this
+
+`config :scalyr_server, :validate => :string, :default => "https://agent.scalyr.com/"`
+
+---
+
+- Path to SSL bundle file.
+
+`config :ssl_ca_bundle_path, :validate => :string, :default =>  "/etc/ssl/certs/ca-bundle.crt"`
+
+---
+
+- server_attributes is a dictionary of key value pairs that represents/identifies the logstash aggregator server
+ (where this plugin is running).  Keys are arbitrary except for the 'serverHost' key which holds special meaning to
+ Scalyr and is given special treatment in the Scalyr UI.  All of these attributes are optional (not required for logs
+ to be correctly uploaded)
+
+`config :server_attributes, :validate => :hash, :default => nil`
+
+---
+
+- Related to the server_attributes dictionary above, if you do not define the 'serverHost' key in server_attributes,
+ the plugin will automatically set it, using the aggregator hostname as value, if this value is true.
+ 
+`config :use_hostname_for_serverhost, :validate => :boolean, :default => false`
+
+---
+
+- Field that represents the origin of the log event. (Warning: events with an existing 'serverHost' field, it will be overwritten)
+
+`config :serverhost_field, :validate => :string, :default => 'serverHost'`
+
+---
+
+- The 'logfile' fieldname has special meaning for the Scalyr UI.  Traditionally, it represents the origin logfile
+ which users can search for in a dedicated widget in the Scalyr UI. If your Events capture this in a different field
+ you can specify that fieldname here and the Scalyr Output Plugin will rename it to 'logfile' before upload.
+ (Warning: events with an existing 'logfile' field, it will be overwritten)
+
+`config :logfile_field, :validate => :string, :default => 'logfile'`
+
+---
+
+- The Scalyr Output Plugin expects the main log message to be contained in the Event['message'].  If your main log
+ content is contained in a different field, specify it here.  It will be renamed to 'message' before upload.
+ (Warning: events with an existing 'message' field, it will be overwritten)
+
+`config :message_field, :validate => :string, :default => "message"`
+
+---
+
+- A list of fieldnames that are constant for any logfile. Any fields listed here will be sent to Scalyr as part of
+ the `logs` array instead of inside every event to save on transmitted bytes. What constitutes a single "logfile"
+ for correctness is a combination of logfile_field value and serverhost_field value. Only events with a serverHost
+ value with have fields moved.
+
+`config :log_constants, :validate => :array, :default => nil`
+
+---
+
+- If true, nested values will be flattened (which changes keys to underscore-separated concatenation of all
+ nested keys).
+
+`config :flatten_nested_values, :validate => :boolean, :default => false`
+
+---
+
+- If true, the 'tags' field will be flattened into key-values where each key is a tag and each value is set to
+ :flat_tag_value
+
+`config :flatten_tags, :validate => :boolean, :default => false`
+
+`config :flat_tag_prefix, :validate => :string, :default => 'tag_'`
+
+`config :flat_tag_value, :default => 1`
+
+---
+
+- Initial interval in seconds between bulk retries. Doubled on each retry up to `retry_max_interval`
+
+`config :retry_initial_interval, :validate => :number, :default => 1`
+
+---
+
+- Set max interval in seconds between bulk retries.
+
+`config :retry_max_interval, :validate => :number, :default => 64`
+
+---
+
+- Valid options are bz2, deflate, or none.
+
+`config :compression_type, :validate => :string, :default => 'deflate'`
+
+---
+
+- An int containing the compression level of compression to use, from 1-9. Defaults to 6
+
+`config :compression_level, :validate => :number, :default => 6`
+
+---
+
+# Conceptual Overview
+
+
+## Persistence
+
+Logstash itself supports [Persistent Queues](https://www.elastic.co/guide/en/logstash/current/persistent-queues.html) with at-least-once delivery semantics.  It expects output plugins to retry uploads until success or else to write failures into a Dead-Letter Queue (DLQ). Since Logstash offers Persistent Queues, the Scalyr plugin does not perform its own buffering or persistence.  More specifically, invocation of `multi_receive` is synchronously retried until success or written to the DLQ upon failure.  Note: the `multi_receive` interface does not provide a feedback mechanism (outcome codes etc).
+
+## Concurrency
+
+The plugin does not manage its own internal concurrency - no threads are started to increase parallelism. To ensure correct ordering of events in Scalyr configure your pipeline with `pipeline.workers: 1`.
+
+## Data model
+
+Logstash Events are arbitrary nested JSON.  Scalyr, however, supports a flat key-value model.  Users are encouraged to pay attention to the mapping of Logstash Events to Scalyr key-values.
+
+### Special fields
+
+Scalyr assigns semantics to certain fields. These semantics allow Scalyr to know which field contains the main message, and also facilitates searching of data. For example, a user may restrict searches to specific combination of `serverHost` and `logfile` in the [Scalyr UI](https://www.scalyr.com/help/log-overview), whereby these 2 fields have dedicated input widgets in the UI.
+
+Mapping/renaming of Logstash event fields to these special fields an important configuration step. For example, if the main message is contained in a field named `text_msg`, then you should configure the plugin's `message_field`  parameter to `text_msg`. This instructs the plugin to rename event `text_msg` to `message`, thus enabling the Scalyr backend to correctly receive the main log message.
+
+Here is the Scalyr API data shape and a description of the special fields:
+```
+{
+  "ts": <Time as epoch nanoseconds>
+  "attrs": 
+  {
+    "message": <The main log message>
+    "logfile": <Log file name (at the originating server) for the message>
+    "serverHost": <The originating source/server for the message>
+    "parser": <What Scalyr parser will be used for server side parsing>
+    <Any other keys / values>
+    ...
+  }
+}
+```
+
+You can use the `mutate` filter to add these fields or rename existing fields to them. Here is an example of a filter configuration you can use to add these fields:
+
+```
+filter {
+    mutate {
+        add_field => { "parser" => "logstash_parser" }
+        add_field => { "serverHost" => "my hostname" }
+        rename => { "path" => "logfile" }
+        rename => { "data" => "message" }
+    }
+}
+```
+
+Note: the only required fields above are `ts` and `attrs/message`.  Omitting optional fields such as `serverHost` or `logfile` merely precludes ability to filter on these fields, but you are still able to search for the log event by any of the event's key/values including the main message field.
+
+
+### Flattening nested values
+
+By default, event attribute values that are nested JSON are converted into strings when uploaded to Scalyr.  However, flattening of nested values is supported. For example, the Logstash Event shape might be as follows:
+
+```
+{
+  "message": "Some log line",
+  "k1": "Key 1 value",
+  "k2": {
+    "A": 100,
+    "B": "Some text",
+  }
+}
+```
+
+Without flattening, the event uploads to Scalyr as a log entry with 3 string values as follows:
+```
+{
+  "message": "Some log line",
+  "k1": "Key 1 value",
+  "k2": "{ \"A\": 100, \"B\": \"Some text\" }"  
+}
+```
+
+Whereas flattening will result in the following data shape:
+```
+{
+  "message": "Some log line",
+  "k1": "Key 1 value",
+  "k2_A": 100,
+  "k2_B": "Some text",
+}
+```
+
+(Notice that multi-level keys are transformed to a flat key by concatenation with a separator.  The default separator is `-`, but is configurable to any character.)
+
+
+## Flattening of nested arrays
+
+Consider the following event where "key2" has a JSON array as its value:
+
+```
+{
+  "message": "Some log line",
+  "k1": "Key 1 value",
+  "k2": [
+    {
+      "A": 100,
+    }, 
+    {
+      "A": 200,
+      "B": 300,
+    }
+  ] 
+}
+```
+
+Without flattening, the event uploads to Scalyr as a log entry with 3 string values as follows:
+```
+{
+  `message`: `Some log line`,
+  "k1": "Key 1 value",
+  "k2": "[{ 'A': 100 }, { 'A': 200, 'B': 300 }]"
+}
+```
+
+Whereas flattening will result in the following data shape:
+```
+{
+  "message": "Some log line",
+  "k1": "Key 1 value",
+  "k2_0_A: 100,
+  "k2_1_A": 200,
+  "k2_1_B": 300,
+}
+```
+
+## Log attributes
+
+If every message that comes from the same `logfile` and `serverHost` has fields that stay constant for that logfile and serverHost you can define this as a log constant. 
+Any fields marked as such will be sent only once per request to scalyr for a serverhost and logfile, which can result in better throughput due to fewer bytes sent.
+
+# Testing
+
+## Smoke test
+
+This repo has been configured to run a full-cycle smoketest on CircleCI as follows:
+
+1. Build the gem
+2. Configure a logstash docker image with pipeline that has file input & Scalyr output
+3. Launch a lightweight "Uploader" docker container that verifies the plugin is active, then writes to a bind-mounted file.  (The bind-mounted file is configured as the input source to Logstash.)
+4. Launch a lightweight "Verifier" docker container that verifies the plugin is active, then executes queries against Scalyr to verify that the Logstash/Scalyr output plugin had uploaded events from the bind-mounted input file to Scalyr. 
+
+## Unit tests
+
+This repo has unit tests that can be run by running:
+
+```
+bundle exec rspec
+```
+
+in the root of the repo.

data/lib/logstash/outputs/scalyr.rb

@@ -0,0 +1,614 @@
+# encoding: utf-8
+require "logstash/outputs/base"
+require "logstash/namespace"
+require "concurrent"
+require "stud/buffer"
+require "socket" # for Socket.gethostname
+require "thread" # for safe queueing
+require "uri" # for escaping user input
+require 'json' # for converting event object to JSON for upload
+
+require 'net/http'
+require 'net/http/persistent'
+require 'net/https'
+require 'rbzip2'
+require 'zlib'
+require 'stringio'
+
+require 'scalyr/common/client'
+require "scalyr/common/util"
+
+
+#---------------------------------------------------------------------------------------------------------------------
+# Implements the Scalyr output plugin
+#---------------------------------------------------------------------------------------------------------------------
+class LogStash::Outputs::Scalyr < LogStash::Outputs::Base
+
+  config_name "scalyr"
+
+  # For correctness reasons we need to limit this plugin to a single worker, a single worker will be single concurrency
+  # anyway but we should be explicit.
+  concurrency :single
+
+  # The Scalyr API write token, these are available at https://www.scalyr.com/keys.  This is the only compulsory configuration field required for proper upload
+  config :api_write_token, :validate => :string, :required => true
+
+  # If your Scalyr backend is located in other geographies (such as Europe which would use `https://agent.eu.scalyr.com/`), you may need to modify this
+  config :scalyr_server, :validate => :string, :default => "https://agent.scalyr.com/"
+
+  # Path to SSL bundle file.
+  config :ssl_ca_bundle_path, :validate => :string, :default =>  "/etc/ssl/certs/ca-bundle.crt"
+
+  # server_attributes is a dictionary of key value pairs that represents/identifies the logstash aggregator server
+  # (where this plugin is running).  Keys are arbitrary except for the 'serverHost' key which holds special meaning to
+  # Scalyr and is given special treatment in the Scalyr UI.  All of these attributes are optional (not required for logs
+  # to be correctly uploaded)
+  config :server_attributes, :validate => :hash, :default => nil
+
+  # Related to the server_attributes dictionary above, if you do not define the 'serverHost' key in server_attributes,
+  # the plugin will automatically set it, using the aggregator hostname as value, if this value is true.
+  config :use_hostname_for_serverhost, :validate => :boolean, :default => false
+
+  # Field that represents the origin of the log event.
+  # (Warning: events with an existing 'serverHost' field, it will be overwritten)
+  config :serverhost_field, :validate => :string, :default => 'serverHost'
+
+  # The 'logfile' fieldname has special meaning for the Scalyr UI.  Traditionally, it represents the origin logfile
+  # which users can search for in a dedicated widget in the Scalyr UI. If your Events capture this in a different field
+  # you can specify that fieldname here and the Scalyr Output Plugin will rename it to 'logfile' before upload.
+  # (Warning: events with an existing 'logfile' field, it will be overwritten)
+  config :logfile_field, :validate => :string, :default => 'logfile'
+
+  # The Scalyr Output Plugin expects the main log message to be contained in the Event['message'].  If your main log
+  # content is contained in a different field, specify it here.  It will be renamed to 'message' before upload.
+  # (Warning: events with an existing 'message' field, it will be overwritten)
+  config :message_field, :validate => :string, :default => "message"
+
+  # A list of fieldnames that are constant for any logfile. Any fields listed here will be sent to Scalyr as part of
+  # the `logs` array instead of inside every event to save on transmitted bytes. What constitutes a single "logfile"
+  # for correctness is a combination of logfile_field value and serverhost_field value. Only events with a serverHost
+  # value with have fields moved.
+  config :log_constants, :validate => :array, :default => nil
+
+  # If true, nested values will be flattened (which changes keys to delimiter-separated concatenation of all
+  # nested keys).
+  config :flatten_nested_values, :validate => :boolean, :default => false
+
+  # If true, the 'tags' field will be flattened into key-values where each key is a tag and each value is set to
+  # :flat_tag_value
+  config :flatten_tags, :validate => :boolean, :default => false
+  config :flat_tag_prefix, :validate => :string, :default => 'tag_'
+  config :flat_tag_value, :default => 1
+
+  # Initial interval in seconds between bulk retries. Doubled on each retry up to `retry_max_interval`
+  config :retry_initial_interval, :validate => :number, :default => 1
+
+  # Set max interval in seconds between bulk retries.
+  config :retry_max_interval, :validate => :number, :default => 64
+
+  # The following two settings pertain to preventing Man-in-the-middle (MITM) attacks  # echee TODO: eliminate?
+  config :ssl_verify_peer, :validate => :boolean, :default => true
+  config :ssl_verify_depth, :validate => :number, :default => 5
+
+  config :max_request_buffer, :validate => :number, :default => 5500000  # echee TODO: eliminate?
+  config :force_message_encoding, :validate => :string, :default => nil
+  config :replace_invalid_utf8, :validate => :boolean, :default => false
+
+  # Valid options are bz2, deflate, or none.
+  config :compression_type, :validate => :string, :default => 'deflate'
+
+  # An int containing the compression level of compression to use, from 1-9. Defaults to 6
+  config :compression_level, :validate => :number, :default => 6
+
+  def close
+    @running = false
+    @client_session.close if @client_session
+  end
+
+  public
+  def register
+
+    if @log_constants and not @log_constants.all? { |x| x.is_a? String }
+      raise LogStash::ConfigurationError, "All elements of 'log_constants' must be strings."
+    end
+
+    if @max_request_buffer > 6000000
+      @logger.warn "Maximum request buffer > 6 MB.  This may result in requests being rejected by Scalyr."
+    end
+
+    @dlq_writer = dlq_enabled? ? execution_context.dlq_writer : nil
+
+    @message_encoding = nil
+    if @force_message_encoding.to_s != ''
+      begin
+        @message_encoding = Encoding.find(@force_message_encoding)
+        @logger.debug "Forcing message encoding to '#{@force_message_encoding}'"
+      rescue ArgumentError
+        @logger.warn "Encoding '#{@force_message_encoding}' not found. Ignoring."
+      end
+    end
+
+    #evaluate any statements in string value of the server_attributes object
+    if @server_attributes
+      new_attributes = {}
+      @server_attributes.each do |key, value|
+        if value.is_a?( String )
+          m = /^\#{(.*)}$/.match( value )
+          if m
+            new_attributes[key] = eval( m[1] )
+          else
+            new_attributes[key] = value
+          end
+        end
+      end
+      @server_attributes = new_attributes
+    end
+
+    # See if we should use the hostname as the server_attributes.serverHost
+    if @use_hostname_for_serverhost
+      if @server_attributes.nil?
+        @server_attributes = {}
+      end
+
+      # only set serverHost if it doesn't currently exist in server_attributes
+      # Note: Use strings rather than symbols for the key, because keys coming
+      # from the config file will be strings
+      unless @server_attributes.key? 'serverHost'
+        @server_attributes['serverHost'] = Socket.gethostname
+      end
+    end
+
+    # Add monitor server attribute to identify this as coming from a plugin
+    @server_attributes['monitor'] = 'pluginLogstash'
+
+    @scalyr_server << '/' unless @scalyr_server.end_with?('/')
+
+    @add_events_uri = URI(@scalyr_server) + "addEvents"
+
+    @logger.info "Scalyr LogStash Plugin ID - #{self.id}"
+
+    @session_id = SecureRandom.uuid
+    @last_status_transmit_time_lock = Mutex.new
+    @last_status_transmit_time = nil
+    @last_status_ = false
+
+    # create a client session for uploading to Scalyr
+    @running = true
+    @client_session = Scalyr::Common::Client::ClientSession.new(
+        @logger, @add_events_uri,
+        @compression_type, @compression_level,
+        @ssl_verify_peer, @ssl_ca_bundle_path, @ssl_verify_depth
+    )
+
+    @logger.info("Started Scalyr output plugin", :class => self.class.name)
+
+    # Finally, send a status line to Scalyr
+    send_status
+
+  end # def register
+
+
+  # Receive an array of events and immediately upload them (without buffering).
+  # The Logstash framework will call this plugin method whenever there is a list of events to upload to Scalyr.
+  # The plugin is expected to retry until success, or else to write failures to the Dead-letter Queue.
+  # No buffering/queuing is done -- ie a synchronous upload to Scalyr is attempted and retried upon failure.
+  #
+  # If there are any network errors, exponential backoff occurs.
+  #
+  # Also note that event uploads are broken up into batches such that each batch is less than max_request_buffer.
+  # Increasing max_request_buffer beyond 3MB will lead to failed requests.
+  #
+  public
+  def multi_receive(events)
+
+    multi_event_request_array = build_multi_event_request_array(events)
+    # Loop over all array of multi-event requests, sending each multi-event to Scalyr
+
+    sleep_interval = @retry_initial_interval
+    batch_num = 1
+    total_batches = multi_event_request_array.length unless multi_event_request_array.nil?
+
+    result = []
+    while !multi_event_request_array.to_a.empty?
+      begin
+        multi_event_request = multi_event_request_array.pop
+        @client_session.post_add_events(multi_event_request[:body])
+        sleep_interval = 0
+        result.push(multi_event_request)
+
+      rescue OpenSSL::SSL::SSLError => e
+        # cannot rely on exception message, so we always log the following warning
+        @logger.error "SSL certificate verification failed.  "
+        "Please make sure your certificate bundle is configured correctly and points to a valid file.  "
+        "You can configure this with the ssl_ca_bundle_path configuration option.  "
+        "The current value of ssl_ca_bundle_path is '#{@ssl_ca_bundle_path}'"
+        @logger.error e.message
+        @logger.error "Discarding buffer chunk without retrying."
+
+      rescue Scalyr::Common::Client::ServerError, Scalyr::Common::Client::ClientError => e
+        sleep_interval = sleep_for(sleep_interval)
+        message = "Error uploading to Scalyr (will backoff-retry)"
+        exc_data = {
+            :url => e.url.sanitized.to_s,
+            :message => e.message,
+            :batch_num => batch_num,
+            :total_batches => total_batches,
+            :record_count => multi_event_request[:record_count],
+            :payload_size => multi_event_request[:body].bytesize,
+            :will_retry_in_seconds => sleep_interval,
+        }
+        exc_data[:code] = e.response_code if e.code
+        exc_data[:body] = e.response_body if @logger.debug? and e.body
+        exc_data[:payload] = "\tSample payload: #{request[:body][0,1024]}..." if @logger.debug?
+        if e.is_commonly_retried?
+          # well-known retriable errors should be debug
+          @logger.debug(message, exc_data)
+        else
+          # all other failed uploads should be errors
+          @logger.error(message, exc_data)
+        end
+        retry if @running.true?
+
+      rescue => e
+        # Any unexpected errors should be fully logged
+        @logger.error(
+            "Unexpected error occurred while uploading to Scalyr (will backoff-retry)",
+            :error_message => e.message,
+            :error_class => e.class.name,
+            :backtrace => e.backtrace
+        )
+        @logger.debug("Failed multi_event_request", :multi_event_request => multi_event_request)
+        sleep_interval = sleep_for(sleep_interval)
+        retry if @running.true?
+      end
+    end
+
+    send_status
+    return result
+  end  # def multi_receive
+
+
+  # Builds an array of multi-event requests from LogStash events
+  # Each array element is a request that groups multiple events (to be posted to Scalyr's addEvents endpoint)
+  #
+  # This function also performs data transformations to support special fields and, optionally, flatten JSON values.
+  #
+  # Special fields are those that have special semantics to Scalyr, i.e. 'message' contains the main log message,
+  # 'serverHost' and 'logfile' have a dedicated search boxes to facilitate filtering.  All Logstash event key/values will
+  # be marshalled into a Scalyr addEvents `attr` key/value unless they are identified as alternate names for special
+  # fields. The special fields ('message', 'serverHost', 'logfile') may be remapped from other fields (configured by setting
+  # 'message_field', 'serverhost_field', 'logfile_field')
+  #
+  # Values that are nested JSON may be optionally flattened (See README.md for some examples).
+  #
+  # Certain fields are removed (e.g. @timestamp and @version)
+  #
+  # Tags are either propagated as a comma-separated string, or optionally transposed into key-values where the keys
+  # are tag names and the values are 1 (may be configured.)
+  def build_multi_event_request_array(logstash_events)
+
+    multi_event_request_array = Array.new
+    total_bytes = 0
+    # Set of unique scalyr threads for this chunk
+    current_threads = Hash.new
+    # Create a Scalyr event object for each record in the chunk
+    scalyr_events = Array.new
+
+    thread_ids = Hash.new
+    next_id = 1 #incrementing thread id for the session
+
+    # per-logfile attributes
+    logs = Hash.new
+    logs_ids = Hash.new
+    next_log_id = 1
+
+    logstash_events.each {|l_event|
+
+      record = l_event.to_hash
+
+      # Create optional threads hash if serverHost is non-nil
+      # echee: TODO I don't think threads are necessary.  Too much info?
+      # they seem to be a second level of granularity within a logfile
+      serverHost = record.fetch(@serverhost_field, nil)
+
+      if serverHost
+        # get thread id or add a new one if we haven't seen this serverHost before
+        if thread_ids.key? serverHost
+          thread_id = thread_ids[serverHost]
+        else
+          thread_id = next_id
+          thread_ids[serverHost] = thread_id
+          next_id += 1
+        end
+        # then update the map of threads for this chunk
+        current_threads[serverHost] = thread_id
+      end
+
+      rename = lambda do |renamed_field, standard_field|
+        if standard_field != renamed_field
+          if record.key? renamed_field
+            if record.key? standard_field
+              @logger.warn "Overwriting log record field '#{standard_field}'.  You are seeing this warning because in "
+              "your LogStash config file you have configured the '#{renamed_field}' field to be converted to the "
+              "'#{standard_field}' field, but the event already contains a field called '#{standard_field}' and "
+              "this is now being overwritten."
+            end
+            record[standard_field] = record[renamed_field]
+            record.delete(renamed_field)
+          end
+        end
+      end
+
+      # Rename user-specified message field -> 'message'
+      rename.call(@message_field, 'message')
+      # Fix message encoding
+      if @message_encoding and !record['message'].to_s.empty?
+        if @replace_invalid_utf8 and @message_encoding == Encoding::UTF_8
+          record["message"] = record["message"].encode("UTF-8", :invalid => :replace,
+                                                       :undef => :replace, :replace => "<?>").force_encoding('UTF-8')
+        else
+          record["message"].force_encoding(@message_encoding)
+        end
+      end
+
+      # Rename user-specified serverHost field -> 'serverHost'
+      rename.call(@serverhost_field, 'serverHost')
+      record.delete(@serverhost_field)
+
+      # Rename user-specified logfile field -> 'logfile'
+      rename.call(@logfile_field, 'logfile')
+      record.delete(@logfile_field)
+      # Set logfile field if empty and serverHost is supplied
+      if record['logfile'].to_s.empty? and serverHost
+        record['logfile'] = "/logstash/#{serverHost}"
+      end
+
+      log_identifier = nil
+      add_log = false
+      if serverHost
+       log_identifier = serverHost + record['logfile']
+      end
+      if log_identifier and not logs.key? log_identifier
+        add_log = true
+        logs[log_identifier] = {
+          'id' => next_log_id,
+          'attrs' => Hash.new
+        }
+        if not record['serverHost'].to_s.empty?
+          logs[log_identifier]['attrs']['serverHost'] = record['serverHost']
+          record.delete('serverHost')
+        end
+        if not record['logfile'].to_s.empty?
+          logs[log_identifier]['attrs']['logfile'] = record['logfile']
+          record.delete('logfile')
+        end
+        if @log_constants
+          @log_constants.each {|log_constant|
+            if record.key? log_constant
+              logs[log_identifier]['attrs'][log_constant] = record[log_constant]
+              record.delete(log_constant)
+            end
+          }
+        end
+        logs_ids[log_identifier] = next_log_id
+        next_log_id += 1
+      end
+
+      # Delete unwanted fields from record
+      record.delete('@version')
+      record.delete('@timestamp')
+
+      # flatten tags
+      if @flatten_tags and record.key? 'tags'
+        record['tags'].each do |tag|
+          record["#{@flat_tag_prefix}#{tag}"] = @flat_tag_value
+        end
+        record.delete('tags')
+      end
+
+      # flatten record
+      record = Scalyr::Common::Util.flatten(record) if @flatten_nested_values
+
+      # Use LogStash event.timestamp as the 'ts' Scalyr timestamp.  Note that this may be overwritten by input
+      # filters so may not necessarily reflect the actual originating timestamp.
+      scalyr_event = {
+        :ts => (l_event.timestamp.time.to_f * (10**9)).round,
+        :attrs => record
+      }
+
+      # optionally set thread
+      if serverHost
+        scalyr_event[:thread] = thread_id.to_s
+        scalyr_event[:log] = logs_ids[log_identifier]
+      end
+
+      # get json string of event to keep track of how many bytes we are sending
+      begin
+        event_json = scalyr_event.to_json
+        log_json = nil
+        if add_log
+          log_json = logs[log_identifier].to_json
+        end
+      rescue JSON::GeneratorError, Encoding::UndefinedConversionError => e
+        @logger.warn "#{e.class}: #{e.message}"
+
+        # Send the faulty event to a label @ERROR block and allow to handle it there (output to exceptions file for ex)
+        # TODO
+        # atime = Fluent::EventTime.new( sec, nsec )
+        # router.emit_error_event(serverHost, time, record, e)
+
+        scalyr_event[:attrs].each do |key, value|
+          @logger.debug "\t#{key} (#{value.encoding.name}): '#{value}'"
+          scalyr_event[:attrs][key] = value.encode(
+              "UTF-8", :invalid => :replace, :undef => :replace, :replace => "<?>"
+          ).force_encoding('UTF-8')
+        end
+        event_json = scalyr_event.to_json
+      end
+
+      # generate new request if json size of events in the array exceed maximum request buffer size
+      append_event = true
+      add_bytes =  event_json.bytesize
+      if log_json
+        add_bytes = add_bytes + log_json.bytesize
+      end
+      if total_bytes + add_bytes > @max_request_buffer
+        # make sure we always have at least one event
+        if scalyr_events.size == 0
+          scalyr_events << scalyr_event
+          append_event = false
+        end
+        multi_event_request = self.create_multi_event_request(scalyr_events, current_threads, logs)
+        multi_event_request_array << multi_event_request
+
+        total_bytes = 0
+        current_threads = Hash.new
+        logs = Hash.new
+        logs_ids = Hash.new
+        scalyr_events = Array.new
+      end
+
+      # if we haven't consumed the current event already
+      # add it to the end of our array and keep track of the json bytesize
+      if append_event
+        scalyr_events << scalyr_event
+        total_bytes += add_bytes
+      end
+
+    }
+
+    # create a final request with any left over events
+    multi_event_request = self.create_multi_event_request(scalyr_events, current_threads, logs)
+    multi_event_request_array << multi_event_request
+    multi_event_request_array
+  end
+
+
+
+  # Helper method that adds a client_timestamp to a batch addEvents request body
+  def add_client_timestamp_to_body(body)
+    current_time_millis = DateTime.now.strftime('%Q').to_i
+    # echee TODO scalyr_agent code suggests this should be "client_time", not "client_timestamp"
+    # however, I cannot find any documentation anywhere. Is it even used?
+    body[:client_timestamp] = current_time_millis.to_s
+  end
+
+
+
+  # A request comprises multiple Scalyr Events.  This function creates a request hash for
+  # final upload to Scalyr (from an array of events, and an optional hash of current threads)
+  # Note: The request body field will be json-encoded.
+  def create_multi_event_request(scalyr_events, current_threads, current_logs)
+
+    body = {
+      :session => @session_id,
+      :token => @api_write_token,
+      :events => scalyr_events,
+    }
+
+    add_client_timestamp_to_body body
+
+    # build the scalyr thread JSON object
+    if current_threads
+      threads = Array.new
+      current_threads.each do |thread_name, id|
+        threads << { :id => id.to_s, :name => "LogStash: #{thread_name}" }
+      end
+      body[:threads] = threads
+    end
+
+    # build the scalyr thread logs object
+    if current_logs
+      logs = Array.new
+      current_logs.each do |identifier, log|
+        logs << log
+      end
+      body[:logs] = logs
+    end
+
+    # add serverAttributes
+    body[:sessionInfo] = @server_attributes if @server_attributes
+
+    { :body => body.to_json, :record_count => scalyr_events.size }
+
+  end  # def create_multi_event_request
+
+
+
+
+  # Sends a status update to Scalyr by posting a log entry under the special logfile of 'logstash_plugin.log'
+  # Instead of creating a separate thread, let this method be invoked once at startup and then every 5 minutes
+  # at most.  (If no events are received, no status update will be sent even if 5 minutes has elapsed).
+  # Finally, note that there could be multiple instances of this plugin (one per worker), in which case each worker
+  # thread sends their own status updates.  This is intentional so that we know how much data each worker thread is
+  # uploading to Scalyr over time.
+  def send_status
+
+    status_event = {
+      :ts => (Time.now.to_f * (10**9)).round,
+      :attrs => {
+        'logfile' => "scalyr_logstash.log",
+        'plugin_id' => self.id,
+      }
+    }
+
+    if !@last_status_transmit_time
+      status_event[:attrs]['message'] = "Started Scalyr LogStash output plugin."
+    else
+      cur_time = Time.now()
+      return if (cur_time.to_i - @last_status_transmit_time.to_i) < 300
+      # echee TODO: get instance stats from session and create a status log line
+      msg = 'plugin_status: '
+      cnt = 0
+      @client_session.get_stats.each do |k, v|
+        val = v.instance_of?(Float) ? sprintf("%.1f", v) : v
+        msg << ', ' if cnt > 0
+        msg << "#{k.to_s}=#{val}"
+        cnt += 1
+      end
+      status_event[:attrs]['message'] = msg
+    end
+    multi_event_request = create_multi_event_request([status_event], nil, nil)
+    @client_session.post_add_events(multi_event_request[:body])
+    @last_status_transmit_time = Time.now()
+  end
+
+
+
+  # Returns true if it is time to transmit status
+  def should_transmit_status?
+    @last_status_transmit_time_lock.synchronize do
+      saved_last_time = @last_status_transmit_time
+      if Time.now.to_i - saved_last_time.to_i > 300
+        @last_status_transmit_time = Float::INFINITY
+        return saved_last_time
+      end
+    ensure
+
+    end
+  end
+
+
+  # Helper method that performs synchronous sleep for a certain time interval
+  def sleep_for(sleep_interval)
+    Stud.stoppable_sleep(sleep_interval) { @running.false? }
+    get_sleep_sec(sleep_interval)
+  end
+
+
+  # Helper method that gets the next sleep time for exponential backoff, capped at a defined maximum
+  def get_sleep_sec(current_interval)
+    doubled = current_interval * 2
+    doubled > @retry_max_interval ? @retry_max_interval : doubled
+  end
+
+
+
+  # Helper method to check if the dead-letter queue is enabled
+  def dlq_enabled?
+    # echee TODO submit to DLQ
+    respond_to?(:execution_context) && execution_context.respond_to?(:dlq_writer) &&
+        !execution_context.dlq_writer.inner_writer.is_a?(::LogStash::Util::DummyDeadLetterQueueWriter)
+  end
+
+end