logstash-output-scalyr 0.1.25.beta → 0.2.1.beta

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 5d84626fc9f17fb9cc549552722612e2620c10ead3700689cb5625715a5441bc
-  data.tar.gz: a49c2c03b3f97678df4a20f66ceec0ec2fdcf4fedcac732a130abe642a469057
+  metadata.gz: a42ad86b12842e9e684f6e155a5a18f72e976a116af35b3bd7a15b38f4ab4fe8
+  data.tar.gz: 7b89fcda801a9d8b2a0df34c95288d4d1599c4c08795f600aba78206fe45407c
 SHA512:
-  metadata.gz: 015e7216df91f4bcc696038ea305936275194d87eb68f5fe5d067e704932663ab5a0cdbb8440810682964c6bfa7a9f052774065ed9fb498fc175900b77eaf161
-  data.tar.gz: e08e06532da18aaf597937482fdee0561ea816b09f3b77c3eee259aeaaaaececcdded5e4dbd2101aacf02018167764c8e98faa7db1c9abe76e4f1094d7fc68ca
+  metadata.gz: 2da035c07ee249d16c5f1041f6dcc72fbe3ae39edbc161ddc602ea2af3cac3823584563d5ccb03bc0e099dc256bc0396898d26562558dcf03777a5eccc690920
+  data.tar.gz: d9491cd2bb02bc2f5d5e612b69ca9ba3923b773d21c37e8b2bc22d270c467552fe1569a85e0755961857f360d8a59841134156e390b4a5c536ede5299f196b89

data/CHANGELOG.md

@@ -1,5 +1,36 @@
 # Beta
 
+## 0.2.1.beta
+
+- Update plugin to fail fast on register and throw more user-friendly error on invalid URL for
+  ``scalyr_server`` configuration option value and other fatal server errors (e.g. invalid
+  hostname).
+- On plugin init / register we now perform connectivity check and verify that we can talk to
+  Scalyr API and validate that the API key is valid. This ensures that the plugin doesn't start and
+  start consuming events until we can successfully perform a connectivity check. This means
+  we can't end up in situation when we could potentially drop some events to the ground in case of
+  an invalid API key or similar when reaching retry limit and DLQ disabled. If you want to disable
+  this check on register, you can set ``perform_connectivity_check`` config option to ``false``.
+
+## 0.2.0.beta, 0.2.0
+
+- Fix a bug and correctly handle ``serverHost`` event level attribute. Now if an event contains
+``serverHost`` attribute, this attribute will be correctly set on the event level and available for
+ "Sources" filtering in the UI.
+- Plugin doesn't set ``serverHost`` attribute with a fixed value of ``Logstash`` on each event
+level anymore. If you still want this behavior, you can achieve that with logstash mutate filter.
+ - Session level ``serverHost`` value now defaults to logstash aggregator node hostname
+ (``use_hostname_for_serverhost`` config option now defaults to true).
+- ``host`` attribute is not removed by default from all the events. By default, logstash adds
+  ``host`` attribute which contains logstash aggregator host to each event. This is now redundant
+  and unncessary with the fixed and improved serverHost behavior (host and serverHost would contain
+  the same value by default). If you want to change this behavior and and still include ``host``
+  attribute on each event you can do that by setting ``remove_host_attribute_from_events`` config
+  option to false.
+
+## 0.1.26.beta
+- Add support for new ``json_library`` config option. Valid values are ``stdlib`` (default) are ``jrjackson``. The later may offer 2-4x faster JSON serialization.
+
 ## 0.1.23.beta
 - Add testing support for disabling estimation of serialized event size for each event in the batch.
 

data/Gemfile

@@ -12,9 +12,13 @@ end
 
 group :test do
   gem "webmock"
+
+  # Require the specific version of `json` used in logstash while testing
+  gem 'json', '1.8.6'
 end
 
 gem 'pry'
 gem 'pry-nav'
 gem 'quantile', '~> 0.2.1'
 gem 'manticore', '~> 0.7.1', platform: :jruby
+gem 'jrjackson', '~> 0.4.14', platform: :jruby

data/README.md

@@ -1,7 +1,6 @@
-
 [![CircleCI](https://circleci.com/gh/scalyr/logstash-output-scalyr.svg?style=svg)](https://circleci.com/gh/scalyr/logstash-output-scalyr)
 
-# [Scalyr output plugin for Logstash (Beta release)]
+# [Scalyr output plugin for Logstash]
 
 This plugin implements a Logstash output plugin that uploads data to [Scalyr](http://www.scalyr.com).
 
@@ -10,7 +9,9 @@ You can view documentation for this plugin [on the Scalyr website](https://app.s
 # Quick start
 
 1. Build the gem, run `gem build logstash-output-scalyr.gemspec` 
-2. Install the gem into a Logstash installation, run `/usr/share/logstash/bin/logstash-plugin install logstash-output-scalyr-0.1.22.beta.gem` or follow the latest official instructions on working with plugins from Logstash.
+2. Install the gem into a Logstash installation, run `/usr/share/logstash/bin/logstash-plugin install logstash-output-scalyr-0.2.0.gem` 
+   or follow the latest official instructions on working with plugins from Logstash. As an alternative, you can directly install latest
+   stable version from RubyGems - ``/usr/share/logstash/bin/logstash-plugin --version 0.2.0 logstash-output-scalyr``
 3. Configure the output plugin (e.g. add it to a pipeline .conf)
 4. Restart Logstash 
 
@@ -40,6 +41,51 @@ output {
 
 In the above example, the Logstash pipeline defines a file input that reads from `/var/log/messages`.  Log events from this source have the `host` and `path` fields.  The pipeline then outputs to the scalyr plugin, which in this example is configured to remap `host`->`serverHost` and `path`->`logfile`, thus facilitating filtering in the Scalyr UI.
 
+## Notes on serverHost attribute handling
+
+> Some of this functionality has been fixed and changed in the v0.2.0 release. In previous
+  versions, plugin added ``serverHost`` attribute with a value of ``Logstash`` to each event and
+  this attribute was not handled correctly - it was treated as a regular event level attribute
+  and not a special attribute which can be used for Source functionality and filtering.
+
+By default this plugin will set ``serverHost`` for all the events in a batch to match hostname of
+the logstash node where the output plugin is running.
+
+You can change that either by setting ``serverHost`` attribute in the ``server_attributes`` config
+option hash or by setting ``serverHost`` attribute on the event level via logstash record attribute.
+
+In both scenarios, you will be able to utilize this value for "Sources" functionality and filterin
+in the Scalyr UI.
+
+For example:
+
+1. Define static value for all the events handled by specific plugin instance
+
+```
+output {
+ scalyr {
+   api_write_token => 'SCALYR_API_KEY'
+   server_attributes => {'serverHost' => 'my-host-1'}
+ }
+}
+```
+
+2. Define static value on the event level which is set via logstash filter
+
+```
+  mutate {
+    add_field => { "serverHost" => "my hostname" }
+  }
+```
+
+3. Define dynamic value on the event level which is set via logstash filter
+
+```
+  mutate {
+    add_field => { "serverHost" => "%{[host][name]}" }
+  }
+```
+
 ## Options
 
 - The Scalyr API write token, these are available at https://www.scalyr.com/keys.  This is the only compulsory configuration field required for proper upload
@@ -72,7 +118,7 @@ In the above example, the Logstash pipeline defines a file input that reads from
 - Related to the server_attributes dictionary above, if you do not define the 'serverHost' key in server_attributes,
  the plugin will automatically set it, using the aggregator hostname as value, if this value is true.
  
-`config :use_hostname_for_serverhost, :validate => :boolean, :default => false`
+`config :use_hostname_for_serverhost, :validate => :boolean, :default => true`
 
 ---
 
@@ -393,5 +439,82 @@ bundle exec rspec
 bundle exec rake publish_gem
 ```
 
+Or as an alternative if ``rake publish_gem`` task doesn't appear to work for whatever reason
+(``publish_gem`` logstash Rake task silently swallows all the errors):
+
+```
+rm -rf vendor/
+bundle check --path vendor/bundle || bundle install --deployment
+curl -u RUBY_USER:RUBY_PASSWORD https://rubygems.org/api/v1/api_key.yaml > ~/.gem/credentials
+chmod 0600 ~/.gem/credentials
+bundle exec rake vendor
+bundle exec rspec
+rvm use jruby
+bundle exec gem build logstash-output-scalyr.gemspec
+bundle exec gem push logstash-output-scalyr-<version>.gem
+```
+
 `RUBY_USER` and `RUBY_PASSWORD` should be replaced with the username and password to the RubyGems.org account you wish to release to,
  these credentials should be found in Keeper.
+
+# Testing Plugin Changes
+
+This section describes how to test the plugin and changes using docker compose setup from
+lostash-config-tester repo available at https://github.com/Kami/logstash-config-tester.
+
+This repo has been forked and already contains some changes which make testing the plugin
+easier.
+
+The logstash configuration in that repo is set up to receive records encoded as JSON via
+standard input, decode the JSON into the event object and print it to stdout + send it
+to the Scalyr output.
+
+```bash
+# 0. Clone the tester repo
+git clone https://github.com/Kami/logstash-config-tester ~/
+
+# 1. Build the plugin
+gem build logstash-output-scalyr.gemspec
+
+# 2. Copy it to the config test repo
+cp logstash-output-scalyr-0.2.0.gem ~/logstash-config-test/logstash-output-scalyr.gem
+
+# 3. Build docker image with the latest dev version of the plugin (may take a while)
+docker-compose build
+
+# 4. Configure API key in docker-compose.yml and make any changes to plugin config in
+# pipeline/scalyr_output.conf.j2, if necessary
+vim docker-compose.yml
+
+vim pipeline/scalyr_output.conf.j2
+
+# 4. Run logstash with the stdin input and stdout + scalyr output
+docker-compose run logstash
+```
+
+A couple of things to keep in mind:
+
+1. Logstash log level is set to debug which is quite verbose, but it makes troubleshooting and
+  testing easier.
+2. Plugin accepts records (events) as JSON via standard input. This means to inject the mock
+   event you can simply copy the JSON event representation string to stdin and press enter. If you
+   want to submit multiples events to be handled as a single batch, paste each event one at a time
+   and press enter at the end. Logstash pipeline has been configured to wait up to 5 seconds before
+   handling the batch which should give you enough time to test batches with multiple events (this
+   setting can be adjusted in ``config/logstash.yml`` - ``pipeline.batch.delay``, if needed)
+
+Example values you can enter into stdin:
+
+1. Single event batch
+
+```javascript
+{"foo": "bar", "message": "test logstash"}
+```
+
+2. Batch with 3 events
+
+```javascript
+{"serverHost": "host-1", "bar": "baz", "message": "test logstash 1"}
+{"serverHost": "host-2", "bar": "baz", "message": "test logstash 2"}
+{"bar": "baz", "message": "test logstash 3"}
+```