fluent-plugin-scalyr 0.8.5 → 0.8.10

checksums.yaml

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: 891497d6c594d6e115191cae7b8d9d48c428770d
-  data.tar.gz: e1f90aac40a8ed67aaa103ec552924eddf09d54c
+SHA256:
+  metadata.gz: 607dd0cf45280387ab9c19b460eab081f2111b1059f8036032e13c1c5c4fd901
+  data.tar.gz: 3d0fa711e084ce2f204dbf69c249363a88d73293a6874c8eb6a175ee920e5cc5
 SHA512:
-  metadata.gz: c1be90d4ae35f8e434269493d19321e27108cf3a608d3874ad4a08d151f69706479f304688630669c8ad79b9785b628a0bf7f4764e955e7e0e7a4bdcc486684b
-  data.tar.gz: 29ca4a9a59d7e1047fd675a6baaaa9a37c45379e48587e586c929db78ea6b1c9323055db7f9c33792765a36cf58a02329286e3cf7ed35ea1a515a0a437d15a01
+  metadata.gz: f6075b808c1126d4b4d8ba7ea30fb5d899facc78c38f4d0d95afc8c97f5eca8f28aefdf414b267dd45695ebe70feb2dd976be76babb8f8458d1240382077a3f0
+  data.tar.gz: deeafb8a4886f51ce635541756e33ba5c3a98b31e6c4ac6f0a5a37a9f8f8862e2a32d0be2026d90f6f7828157e43ac77261529364f1b2bf697b9a2948f7fd3e4

data/README.md

@@ -1,5 +1,5 @@
 Scalyr output plugin for Fluentd
-=========================
+================================
 
 **Note:** Fluentd introduced breaking changes to their plugin API between
 version 0.12 and 0.14.
@@ -24,7 +24,7 @@ Fluentd may format log messages into json or some other format.  If you want to
   format none
 ```
 
-The Scalyr output plugin assigns a unique Scalyr session id for each Fluentd <match> block.  It is recommended that a single machine doesn't create too many simultaneous Scalyr sessions, so if possible you should try to have a single match for all logs you wish to send to Scalyr.
+The Scalyr output plugin assigns a unique Scalyr session id for each Fluentd <match> block, or for each worker.  It is recommended that a single machine doesn't create too many simultaneous Scalyr sessions, so if possible you should try to have a single match for all logs you wish to send to Scalyr.
 
 This can be done by specifying tags such as scalyr.apache, scalyr.maillog etc and matching on scalyr.\*
 
@@ -33,7 +33,7 @@ Fluentd tag names will be used for the logfile name in Scalyr.
 Scalyr Parsers and Custom Fields
 --------------------------------
 
-You may also need to specify a Scalyr parser for your log message or add custom fields to each log event. This can be done using Fluentd's filter mechanism, in particular the [record_transformer filter](http://docs.fluentd.org/articles/filter_record_transformer).
+You may also need to specify a Scalyr parser for your log message or add custom fields to each log event. This can be done using Fluentd's filter mechanism, in particular the [record_transformer filter](https://docs.fluentd.org/filter/record_transformer).
 
 For example, if you want to use Scalyr's ```accessLog``` parser for all events with the ```scalyr.access``` tag you would add the following to your fluent.conf file:
 
@@ -66,7 +66,9 @@ The following configuration options are also supported:
 
   #scalyr specific options
   api_write_token YOUR_SCALYR_WRITE_TOKEN
-  compression_type bz2
+  compression_type deflate
+  compression_level 6
+  use_hostname_for_serverhost true
   server_attributes {
     "serverHost": "front-1",
     "serverType": "frontend",
@@ -79,7 +81,7 @@ The following configuration options are also supported:
   ssl_verify_depth 5
   message_field message
 
-  max_request_buffer 1048576
+  max_request_buffer 5500000
 
   force_message_encoding nil
   replace_invalid_utf8 false
@@ -91,14 +93,18 @@ The following configuration options are also supported:
     retry_max_interval 30s
     flush_interval 5s
     flush_thread_count 1
-    chunk_limit_size 100k
+    chunk_limit_size 2.5m
     queue_limit_length 1024
   </buffer>
 
 </match>
 ```
 
-####Scalyr specific options
+For some additional examples of configuration for different setups, please refer to the
+[examples/configs/](https://github.com/scalyr/scalyr-fluentd/tree/master/examples/configs/)
+directory.
+
+### Scalyr specific options
 
 ***compression_type*** - compress Scalyr traffic to reduce network traffic. Options are `bz2` and `deflate`. See [here](https://www.scalyr.com/help/scalyr-agent#compressing) for more details.  This feature is optional.
 
@@ -106,6 +112,8 @@ The following configuration options are also supported:
 
 ***server_attributes*** - a JSON hash containing custom server attributes you want to include with each log request.  This value is optional and defaults to *nil*.
 
+***use_hostname_for_serverhost*** - if `true` then if `server_attributes` is nil or it does *not* include a field called `serverHost` then the plugin will add the `serverHost` field with the value set to the hostname that fluentd is running on.  Defaults to `true`.
+
 ***scalyr_server*** - the Scalyr server to send API requests to. This value is optional and defaults to https://agent.scalyr.com/
 
 ***ssl_ca_bundle_path*** - a path on your server pointing to a valid certificate bundle.  This value is optional and defaults to */etc/ssl/certs/ca-bundle.crt*.
@@ -126,13 +134,13 @@ The cURL project maintains CA certificate bundles automatically converted from m
 
 ***message_field*** - Scalyr expects all log events to have a 'message' field containing the contents of a log message.  If your event has the log message stored in another field, you can specify the field name here, and the plugin will rename that field to 'message' before sending the data to Scalyr.  **Note:** this will override any existing 'message' field if the log record contains both a 'message' field and the field specified by this config option.
 
-***max_request_buffer*** - The maximum size in bytes of each request to send to Scalyr.  Defaults to 1,048,576 (1MB).  Fluentd chunks that generate JSON requests larger than the max_request_buffer will be split in to multiple separate requests.  **Note:** If you set this value too large Scalyr may reject your requests.
+***max_request_buffer*** - The maximum size in bytes of each request to send to Scalyr.  Defaults to 5,500,000 (5.5MB).  Fluentd chunks that generate JSON requests larger than the max_request_buffer will be split in to multiple separate requests.  **Note:** The maximum size the Scalyr servers accept for this value is 6MB and requests containing data larger than this will be rejected.
 
 ***force_message_encoding*** - Set a specific encoding for all your log messages (defaults to nil).  If your log messages are not in UTF-8, this can cause problems when converting the message to JSON in order to send to the Scalyr server.  You can avoid these problems by setting an encoding for your log messages so they can be correctly converted.
 
 ***replace_invalid_utf8*** - If this value is true and ***force_message_encoding*** is set to 'UTF-8' then all invalid UTF-8 sequences in log messages will be replaced with <?>.  Defaults to false.  This flag has no effect if ***force_message_encoding*** is not set to 'UTF-8'.
 
-####Buffer options
+### Buffer options
 
 ***retry_max_times*** - the maximum number of times to retry a failed post request before giving up.  Defaults to *40*.
 
@@ -144,7 +152,7 @@ The cURL project maintains CA certificate bundles automatically converted from m
 
 ***flush_thread_count*** - the number of threads to use to upload logs.  This is currently fixed to 1 will cause fluentd to fail with a ConfigError if set to anything greater.
 
-***chunk_limit_size*** - the maximum amount of log data to send to Scalyr in a single request.  Defaults to *100KB*.  **Note:** if you set this value too large, then Scalyr may reject your requests.  Requests smaller than 1MB will typically be accepted by Scalyr, but note that the 1MB limit also includes the entire request body and all associated JSON keys and punctuation, which may be considerably larger than the raw log data.
+***chunk_limit_size*** - the maximum amount of log data to send to Scalyr in a single request.  Defaults to *2.5MB*.  **Note:** if you set this value too large, then Scalyr may reject your requests.  Requests smaller than 6 MB will typically be accepted by Scalyr, but note that the 6 MB limit also includes the entire request body and all associated JSON keys and punctuation, which may be considerably larger than the raw log data.  This value should be set lower than the `max_request_buffer` option.
 
 ***queue_limit_length*** - the maximum number of chunks to buffer before dropping new log requests.  Defaults to *1024*.  Combines with ***chunk_limit_size*** to give you the total amount of buffer to use in the event of request failures before dropping requests.
 

data/VERSION

@@ -1 +1 @@
-0.8.5
+0.8.10

data/fluent-plugin-scalyr.gemspec

@@ -9,7 +9,6 @@ Gem::Specification.new do |gem|
   gem.authors = ["Imron Alston"]
   gem.licenses = ["Apache-2.0"]
   gem.email = "imron@scalyr.com"
-  gem.has_rdoc = false
   gem.platform = Gem::Platform::RUBY
   gem.files = Dir['AUTHORS', 'Gemfile', 'LICENSE', 'README.md', 'Rakefile', 'VERSION', 'fluent-plugin-scalyr.gemspec', 'fluent.conf.sample', 'lib/**/*', 'test/**/*']
   gem.test_files = Dir.glob("{test,spec,features}/**/*")

data/fluent.conf.sample

@@ -1,7 +1,7 @@
 <match scalyr.*>
   @type scalyr
   api_write_token YOUR_WRITE_LOGS_API_TOKEN
-  compression_type bz2
+  compression_type deflate
 
   ##Scalyr specific options
   # server_attributes {

data/lib/fluent/plugin/out_scalyr.rb

@@ -26,6 +26,7 @@ require 'rbzip2'
 require 'stringio'
 require 'zlib'
 require 'securerandom'
+require 'socket'
 require 'thread'
 
 module Scalyr
@@ -36,23 +37,24 @@ module Scalyr
 
     config_param :api_write_token, :string
     config_param :server_attributes, :hash, :default => nil
+    config_param :use_hostname_for_serverhost, :bool, :default => true
     config_param :scalyr_server, :string, :default => "https://agent.scalyr.com/"
     config_param :ssl_ca_bundle_path, :string, :default => "/etc/ssl/certs/ca-bundle.crt"
     config_param :ssl_verify_peer, :bool, :default => true
     config_param :ssl_verify_depth, :integer, :default => 5
     config_param :message_field, :string, :default => "message"
-    config_param :max_request_buffer, :integer, :default => 1024*1024
+    config_param :max_request_buffer, :integer, :default => 5500000
     config_param :force_message_encoding, :string, :default => nil
     config_param :replace_invalid_utf8, :bool, :default => false
     config_param :compression_type, :string, :default => nil #Valid options are bz2, deflate or None. Defaults to None.
-    config_param :compression_level, :integer, :default => 9 #An int containing the compression level of compression to use, from 1-9. Defaults to 9 (max)
+    config_param :compression_level, :integer, :default => 6 #An int containing the compression level of compression to use, from 1-9. Defaults to 6
 
     config_section :buffer do
       config_set_default :retry_max_times, 40 #try a maximum of 40 times before discarding
       config_set_default :retry_max_interval,  30 #wait a maximum of 30 seconds per retry
       config_set_default :retry_wait, 5 #wait a minimum of 5 seconds per retry
       config_set_default :flush_interval, 5 #default flush interval of 5 seconds
-      config_set_default :chunk_limit_size, 1024*100 #default chunk size of 100k
+      config_set_default :chunk_limit_size, 2500000 #default chunk size of 2.5mb
       config_set_default :queue_limit_length, 1024 #default queue size of 1024
     end
 
@@ -65,6 +67,10 @@ module Scalyr
       true
     end
 
+    def multi_workers_ready?
+      true
+    end
+
     def configure( conf )
 
       if conf.elements('buffer').empty?
@@ -75,12 +81,12 @@ module Scalyr
 
       super
 
-      if @buffer.chunk_limit_size > 1024*1024
-        $log.warn "Buffer chunk size is greater than 1Mb.  This may result in requests being rejected by Scalyr"
+      if @buffer.chunk_limit_size > 6000000
+        $log.warn "Buffer chunk size is greater than 6Mb.  This may result in requests being rejected by Scalyr"
       end
 
-      if @max_request_buffer > (1024*1024*3)
-        $log.warn "Maximum request buffer > 3Mb.  This may result in requests being rejected by Scalyr"
+      if @max_request_buffer > 6000000
+        $log.warn "Maximum request buffer > 6Mb.  This may result in requests being rejected by Scalyr"
       end
 
       @message_encoding = nil
@@ -109,6 +115,22 @@ module Scalyr
         @server_attributes = new_attributes
       end
 
+      # See if we should use the hostname as the server_attributes.serverHost
+      if @use_hostname_for_serverhost
+
+        # ensure server_attributes is not nil
+        if @server_attributes.nil?
+          @server_attributes = {}
+        end
+
+        # only set serverHost if it doesn't currently exist in server_attributes
+        # Note: Use strings rather than symbols for the key, because keys coming
+        # from the config file will be strings
+        if !@server_attributes.key? 'serverHost'
+          @server_attributes['serverHost'] = Socket.gethostname
+        end
+      end
+
       @scalyr_server << '/' unless @scalyr_server.end_with?('/')
 
       @add_events_uri = URI @scalyr_server + "addEvents"
@@ -121,15 +143,10 @@ module Scalyr
 
     def start
       super
-      $log.info "Scalyr Fluentd Plugin ID - #{self.plugin_id()}"
       #Generate a session id.  This will be called once for each <match> in fluent.conf that uses scalyr
       @session = SecureRandom.uuid
 
-      @sync = Mutex.new
-      #the following variables are all under the control of the above mutex
-        @thread_ids = Hash.new #hash of tags -> id
-        @next_id = 1 #incrementing thread id for the session
-        @last_timestamp = 0 #timestamp of most recent event in nanoseconds since epoch
+      $log.info "Scalyr Fluentd Plugin ID id=#{self.plugin_id()} worker=#{fluentd_worker_id} session=#{@session}"
 
     end
 
@@ -140,8 +157,14 @@ module Scalyr
           time = Fluent::Engine.now
         end
 
+        # handle timestamps that are not EventTime types
         if time.is_a?( Integer )
           time = Fluent::EventTime.new( time )
+        elsif time.is_a?( Float )
+          components = time.divmod 1 #get integer and decimal components
+          sec = components[0].to_i
+          nsec = (components[1] * 10**9).to_i
+          time = Fluent::EventTime.new( sec, nsec )
         end
 
         if @message_field != "message"
@@ -154,7 +177,7 @@ module Scalyr
           end
         end
 
-        if @message_encoding
+        if @message_encoding and record.key? "message" and record["message"]
           if @replace_invalid_utf8 and @message_encoding == Encoding::UTF_8
             record["message"] = record["message"].encode("UTF-8", :invalid => :replace, :undef => :replace, :replace => "<?>").force_encoding('UTF-8')
           else
@@ -310,22 +333,7 @@ module Scalyr
 
         timestamp = self.to_nanos( sec, nsec )
 
-        thread_id = 0
-
-        @sync.synchronize {
-          #ensure timestamp is at least 1 nanosecond greater than the last one
-          timestamp = [timestamp, @last_timestamp + 1].max
-          @last_timestamp = timestamp
-
-          #get thread id or add a new one if we haven't seen this tag before
-          if @thread_ids.key? tag
-            thread_id = @thread_ids[tag]
-          else
-            thread_id = @next_id
-            @thread_ids[tag] = thread_id
-            @next_id += 1
-          end
-        }
+        thread_id = tag
 
         #then update the map of threads for this chunk
         current_threads[tag] = thread_id

data/test/test_config.rb

@@ -17,12 +17,15 @@
 
 
 require 'helper'
+require 'socket'
 
 class ConfigTest  < Scalyr::ScalyrOutTest
 
   def test_default_params
     d = create_driver
-    assert_nil( d.instance.server_attributes, "Default server_attributes not nil" )
+    hostname = Socket.gethostname
+    assert_not_nil( d.instance.server_attributes, "Default server_attributes should not be nil" )
+    assert_equal( hostname, d.instance.server_attributes['serverHost'], "Default serverHost is not hostname" )
     assert( d.instance.ssl_verify_peer, "Default ssl_verify_peer should be true" )
 
     #check default buffer limits because they are set outside of the config_set_default
@@ -30,6 +33,17 @@ class ConfigTest  < Scalyr::ScalyrOutTest
     assert_equal( 1024, d.instance.buffer.buffer_queue_limit, "Buffer queue limit should be 1024" )
   end
 
+  def test_custom_serverhost_not_overwritten
+    hostname = "customHost"
+    d = create_driver CONFIG + "server_attributes { \"serverHost\":\"#{hostname}\" }\nuse_hostname_for_serverhost true"
+    assert_equal( hostname, d.instance.server_attributes['serverHost'], "Custom serverHost should not be overwritten" )
+  end
+
+  def test_configure_use_hostname_for_serverhost
+    d = create_driver CONFIG + 'use_hostname_for_serverhost false'
+    assert_nil( d.instance.server_attributes, "Default server_attributes should be nil" )
+  end
+
   def test_configure_ssl_verify_peer
     d = create_driver CONFIG + 'ssl_verify_peer false'
     assert( !d.instance.ssl_verify_peer, "Config failed to set ssl_verify_peer" )

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: fluent-plugin-scalyr
 version: !ruby/object:Gem::Version
-  version: 0.8.5
+  version: 0.8.10
 platform: ruby
 authors:
 - Imron Alston
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2018-11-22 00:00:00.000000000 Z
+date: 2020-07-24 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: fluentd
@@ -168,14 +168,13 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubyforge_project: 
-rubygems_version: 2.5.2
+rubygems_version: 3.0.6
 signing_key: 
 specification_version: 4
 summary: Scalyr plugin for fluentd
 test_files:
-- test/test_events.rb
-- test/test_config.rb
-- test/helper.rb
 - test/test_handle_response.rb
 - test/test_ssl_verify.rb
+- test/test_config.rb
+- test/test_events.rb
+- test/helper.rb