logstash-output-sumologic 1.0.4 → 1.1.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 4bf404a07cdd75110c532fb0279686b95d3627df
-  data.tar.gz: f6a8e2b65b662c8f20575ed56ffba768bad20773
+  metadata.gz: e52d2f2af928de4e3f0318396239ad13552c8381
+  data.tar.gz: adbeece1b46c5a0409eff70626e81ac717e4dfb1
 SHA512:
-  metadata.gz: 0b01a0de20c6daa4ce28d0ece3e3d34b6933a7b2ded07807e9fc0fdd8f48898f04fb5c5b3f97a4d8eac0cdd14764a2b458f742c5045650fdd806f465fa151126
-  data.tar.gz: c0a8f598ad9a36610bc1c3bbfd986e8c1fbef2988dece36d46288402f0ff6a1e06292f2a14253290345e8e0a83cc6dd2d78f2d2f27aab2314fd9250dcc27b193
+  metadata.gz: b34303f871a5efbbd9a9a47f0f0eee6edd59672eb321daa7c25883fb0d4972d249eea597e64e4cf50a26e2f9ed9271ce86b010214e60f3357a16b093e6cd1cfb
+  data.tar.gz: 6b0e002735a97349cf933f35d16bb7a5cf69c77833c295535bf25c7ecafe4e7c28efbb73d6548ba9573f6d2edb368051bf169e652ada4c792e93ad258ca3ca21

data/CHANGELOG.md

@@ -1,23 +1,5 @@
-## 1.0.0
- - First public release
-
-### 1.0.1
- - Update gem description 
+## 1.1.0
+ - Support metrics sending
 
-### 1.0.2
- - Support using `%{@json}` in format to send event in json format
- - Support pararmeter `json_mapping` to filter json fields. For example:
-```
-json_mapping => {
-    "foo" => "%{@timestamp}"
-    "bar" => "%{message}"
-}
-```
-will create message as:
-```
-{"foo":"2016-07-27T18:37:59.460Z","bar":"hello world"}
-{"foo":"2016-07-27T18:38:01.222Z","bar":"bye!"}
-```
-
-### 1.0.3
- - Remove version limitation so it works with Log Stash 5.0.0 core
+## 1.0.0
+ - Initial release

data/DEVELOPER.md

@@ -1,2 +1,27 @@
 # logstash-output-sumologic
 Logstash output plugin for delivering log to Sumo Logic cloud service through HTTP source.
+
+# How to build .gem file from repository
+Open logstash-output-sumologic.gemspec and make any necessary configuration changes.
+In your local Git clone, run:
+```sh
+gem build logstash-output-sumologic.gemspec
+```
+You will get a .gem file in the same directory as `logstash-output-sumologic-x.y.z.gem`
+Remove old version of plugin (optional):
+```sh
+bin/logstash-plugin remove logstash-output-sumologic
+```
+And then install the plugin locally:
+```sh
+bin/logstash-plugin install <full path of .gem>
+```
+
+# How to run test with rspec
+The test requires JRuby to run. So you need to install [JRuby](http://jruby.org/) and [RVM](https://rvm.io/) (for switching between JRuby and Ruby) first.
+And then run:
+```bash
+rvm use jruby
+rspec spec/
+```
+

data/README.md

@@ -1,9 +1,10 @@
 # Logstash Sumo Logic Output Plugin
 
-This is a plugin for [Logstash](https://github.com/elastic/logstash).
+This is an output plugin for [Logstash](https://github.com/elastic/logstash).
 It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
 
 ## Getting Started
+This guide is for the users just want download the binary and make the plugin work. For the developer, please refer to the [Developer Guide](DEVELOPER.md)
 
 ### 1. Create a Sumo Logic HTTP source
 Create a [Sumo Logic](https://www.sumologic.com/) free account if you currently don't have one.
@@ -16,57 +17,67 @@ https://events.sumologic.net/receiver/v1/http/XXXXXXXXXX
 ### 2. Install LogStash on your machine
 Following this [instruction](https://www.elastic.co/guide/en/logstash/current/getting-started-with-logstash.html) to download and install LogStash. This plugin requires Logstash 2.3 or higher to work.
 
-### 3. Build your plugin gem
-In your local Git clone, running:
+### 3. Install latest Logstash Sumo Logic Output plugin from [RubyGems](https://rubygems.org/gems/logstash-output-sumologic)
 ```sh
-gem build logstash-output-sumologic.gemspec
+bin/logstash-plugin install logstash-output-sumologic
 ```
-You will get a .gem file as `logstash-output-sumologic-1.0.0.gem`
-
-### 4. Install plugin into LogStash
-In the Logstash home, running:
-```sh
-bin/logstash-plugin install <path of .gem>
-```
-
-### 5. Start Logstash and send log
+### 4. Start Logstash and send log
 In the Logstash home, running:
 ```sh
-bin/logstash -e 'input{stdin{}}output{sumologic{url=>"<url from step 1>"}}'
+bin/logstash -e 'input{stdin{}}output{sumologic{url=>"<URL from step 1>"}}'
 ```
 This will send any input from console to Sumo Logic cloud service.
 
-### 6. Get result from Sumo Logic web app
-Logon to Sumo Logic [web app](https://prod-www.sumologic.net/ui/) and run [Search](http://help.sumologic.com/Search) or [Live Tail](http://help.sumologic.com/Search/Live_Tail)
-
-### Furthermore
-- Try it with different input/filter/codec plugins
-- Start LogStash as a service/daemon in your production environment 
-- Report any issue or idea through [Git Hub](https://github.com/SumoLogic/logstash-output-sumologic)
-
-## Parameters
-This plugin is based on [logstash-mixin-http_client](https://github.com/logstash-plugins/logstash-mixin-http_client) thus it supports all parameters like proxy, authentication, retry, etc.
+### 5. Try out samples
+Open samples/sample-logs.conf, replace #URL# placeholder as real URL got from step 1
 
-And it supports following additional prarmeters:
+Launch sample with:
+```sh
+bin/logstash -f samples/log.conf
 ```
-  # The URL to send logs to. This should be given when creating a HTTP Source
-  # on Sumo Logic web app. See http://help.sumologic.com/Send_Data/Sources/HTTP_Source
-  config :url, :validate => :string, :required => true
-
-  # Include extra HTTP headers on request if needed 
-  config :extra_headers, :validate => :hash, :default => []
-
-  # The formatter of message, by default is message with timestamp and host as prefix
-  config :format, :validate => :string, :default => "%{@timestamp} %{host} %{message}"
-
-  # Hold messages for at least (x) seconds as a pile; 0 means sending every events immediately  
-  config :interval, :validate => :number, :default => 0
+The input from console will be sent to Sumo Logic cloud service as log lines.
 
-  # Compress the payload 
-  config :compress, :validate => :boolean, :default => false
+Open samples/sample-metrics.conf, replace #URL# placeholder as real URL got from step 1
+(This sample may require installing the [plugins-filters-metrics](https://www.elastic.co/guide/en/logstash/current/plugins-filters-metrics.html) plugin first)
 
+Launch sample with:
+```sh
+bin/logstash -f samples/metrics.conf
 ```
+A mocked event will be sent to Sumo Logic cloud service as 1 minute and 15 minutes rate metrics.
 
+### 6. Get result from Sumo Logic web app
+Logon to Sumo Logic [web app](https://prod-www.sumologic.net/ui/) and run 
+ - [Log Search](http://help.sumologic.com/Search)
+ - [Live Tail](http://help.sumologic.com/Search/Live_Tail)
+ - [Metrics Search](https://help.sumologic.com/Metrics)
 
+## What's Next
+- Try it with different input/filter/codec plugins
+- Start LogStash as a service/daemon in your production environment 
+- Report any issue or idea through [Git Hub](https://github.com/SumoLogic/logstash-output-sumologic)
 
+## Parameters of Plugin
+| Parameter           | Type    | Required? | Default value | Decription            |
+| ------------------- | ------- | --------- | ------------- | --------------------- |
+| `url`               | string  | Yes       |               | HTTP Source URL
+| `source_category`   | string  | No        |               | Source category to appear when searching in Sumo Logic by `_sourceCategory`. If not specified, the source category of the HTTP source will be used.
+| `source_name`       | string  | No        |               | Source name to appear when searching in Sumo Logic by `_sourceName`.
+| `source_host`       | string  | No        |               | Source host to appear when searching in Sumo Logic by `_sourceHost`. If not specified, it will be the machine host name.
+| `extra_headers`     | hash    | No        |               | Extra fields need to be send in HTTP header.
+| `compress`          | boolean | No        | `false`       | Enable or disable compression.
+| `compress_encoding` | string  | No        | `'deflate'`   | Encoding method of comressing, can only be `'deflate'` or `'gzip'`.
+| `interval`          | number  | No        | `0`           | The maximum time for waiting before send in batch, in ms. 
+| `format`            | string  | No        | `"%{@timestamp} %{host} %{message}"` | For log only, the formatter of log lines. Use `%{@json}` as the placeholder for whole event json.
+| `json_mapping`      | hash    | No        |               | Override the structure of `{@json}` tag with the given key value pairs.
+| `metrics`           | hash    | No        |               | If defined, the event will be sent as metrics. Keys will be the metrics name and values will be the metrics value.
+| `metrics_format`    | string  | No        | `'cabon2'`    | Metrics format, can only be `'graphite'` or `'carbon2'`.
+| `metrics_name`      | string  | No        | `*`           | Define the metric name looking, the placeholder '*' will be replaced with the actual metric name.
+| `intrinsic_tags`    | hash    | No        |               | For carbon2 format only, send extra intrinsic key-value pairs other than `metric` (which is the metric name).
+| `meta_tags`         | hash    | No        |               | For carbon2 format only, send metadata key-value pairs.
+| `fields_as_metrics` | boolean | No        | `false`       | If `true`, all fields in logstash event with number value will be sent as a metrics (with filtering by `fields_include` and `fields_exclude` ; the `metics` parameter is ignored.
+| `fields_include`    | array   | No        | all fields    | Working with `fields_as_metrics` parameter, only the fields which full name matching these RegEx pattern(s) will be included in metrics.
+| `fields_exclude`    | array   | No        | none          | Working with `fields_as_metrics` parameter, the fields which full name matching these RegEx pattern(s) will be ignored in metrics.
+
+This plugin is based on [logstash-mixin-http_client](https://github.com/logstash-plugins/logstash-mixin-http_client) thus we also support all HTTP layer parameters like proxy, authentication, retry, etc.
 

data/lib/logstash/outputs/sumologic.rb

@@ -6,6 +6,7 @@ require "logstash/plugin_mixins/http_client"
 require 'thread'
 require "uri"
 require "zlib"
+require "stringio"
 
 # Now you can use logstash to deliver logs to Sumo Logic
 #
@@ -17,35 +18,98 @@ class LogStash::Outputs::SumoLogic < LogStash::Outputs::Base
   include LogStash::PluginMixins::HttpClient
   
   config_name "sumologic"
-  
+
+  CONTENT_TYPE = "Content-Type"
+  CONTENT_TYPE_LOG = "text/plain"
+  CONTENT_TYPE_GRAPHITE = "application/vnd.sumologic.graphite"
+  CONTENT_TYPE_CARBON2 = "application/vnd.sumologic.carbon2"
+  CATEGORY_HEADER = "X-Sumo-Category"
+  HOST_HEADER = "X-Sumo-Host"
+  NAME_HEADER = "X-Sumo-Name"
+  CLIENT_HEADER = "X-Sumo-Client"
+  TIMESTAMP_FIELD = "@timestamp"
+  METRICS_NAME_PLACEHOLDER = "*"
+  GRAPHITE = "graphite"
+  CARBON2 = "carbon2"
+  CONTENT_ENCODING = "Content-Encoding"
+  DEFLATE = "deflate"
+  GZIP = "gzip"
+  ALWAYS_EXCLUDED = [ "@timestamp", "@version" ]
+
   # The URL to send logs to. This should be given when creating a HTTP Source
   # on Sumo Logic web app. See http://help.sumologic.com/Send_Data/Sources/HTTP_Source
   config :url, :validate => :string, :required => true
 
+  # Define the source category metadata
+  config :source_category, :validate => :string
+
+  # Define the source host metadata
+  config :source_host, :validate => :string
+
+  # Define the source name metadata
+  config :source_name, :validate => :string
+
   # Include extra HTTP headers on request if needed 
-  config :extra_headers, :validate => :hash, :default => []
+  config :extra_headers, :validate => :hash
 
-  # The formatter of message, by default is message with timestamp and host as prefix
-  # use %{@json} tag to send whole event
-  config :format, :validate => :string, :default => "%{@timestamp} %{host} %{message}"
+  # Compress the payload 
+  config :compress, :validate => :boolean, :default => false
+
+  # The encoding method of compress
+  config :compress_encoding, :validate =>:string, :default => DEFLATE
 
   # Hold messages for at least (x) seconds as a pile; 0 means sending every events immediately  
   config :interval, :validate => :number, :default => 0
 
-  # Compress the payload 
-  config :compress, :validate => :boolean, :default => false
+  # The formatter of log message, by default is message with timestamp and host as prefix
+  # Use %{@json} tag to send whole event
+  config :format, :validate => :string, :default => "%{@timestamp} %{host} %{message}"
 
-  # This lets you choose the structure and parts of the event that are sent in @json tag.
+  # Override the structure of @json tag with the given key value pairs
   config :json_mapping, :validate => :hash
+  
+  # Send metric(s) if configured. This is a hash with k as metric name and v as metric value
+  # Both metric names and values support dynamic strings like %{host}
+  # For example: 
+  #     metrics => { "%{host}/uptime" => "%{uptime_1m}" }
+  config :metrics, :validate => :hash
+
+  # Create metric(s) automatically from @json fields if configured. 
+  config :fields_as_metrics, :validate => :boolean, :default => false
+  
+  config :fields_include, :validate => :array, :default => [ ]
+  
+  config :fields_exclude, :validate => :array, :default => [ ]
+
+  # Defines the format of the metric, support "graphite" or "carbon2"
+  config :metrics_format, :validate => :string, :default => CARBON2
 
+  # Define the metric name looking, the placeholder '*' will be replaced with the actual metric name
+  # For example:
+  #     metrics => { "uptime.1m" => "%{uptime_1m}" }
+  #     metrics_name => "mynamespace.*"
+  # will produce metrics as:
+  #     "mynamespace.uptime.1m xxx 1234567"
+  config :metrics_name, :validate => :string, :default => METRICS_NAME_PLACEHOLDER
+
+  # For carbon2 metrics format only, define the intrinsic tags (which will be used to identify the metrics)
+  # There is always an intrinsic tag as "metric" which value is from metrics_name
+  config :intrinsic_tags, :validate => :hash, :default => {}
+
+  # For carbon2 metrics format only, define the meta tags (which will NOT be used to identify the metrics)
+  config :meta_tags, :validate => :hash, :default => {}
+  
   public
   def register
+    @source_host = `hostname`.strip unless @source_host
+
     # initialize request pool
     @request_tokens = SizedQueue.new(@pool_max)
     @pool_max.times { |t| @request_tokens << true }
     @timer = Time.now
     @pile = Array.new
     @semaphore = Mutex.new
+    connect
   end # def register
 
   public
@@ -61,23 +125,9 @@ class LogStash::Outputs::SumoLogic < LogStash::Outputs::Base
       return
     end
 
-    content = format_event(event)
+    content = event2content(event)
+    queue_and_send(content)
     
-    if @interval <= 0 # means send immediately
-      send_request(content)
-      return
-    end
-
-    @semaphore.synchronize {
-      now = Time.now
-      @pile << content
-
-      if now - @timer > @interval # ready to send
-        send_request(@pile.join($/))
-        @timer = now
-        @pile.clear
-      end
-    }
   end # def receive
 
   public
@@ -88,15 +138,35 @@ class LogStash::Outputs::SumoLogic < LogStash::Outputs::Base
     }
     client.close
   end # def close
+
+
+  private
+  def connect
+    # TODO: ping endpoint make sure config correct
+  end # def connect
   
   private
-  def send_request(content)
-    token = @request_tokens.pop
-    body = if @compress
-      Zlib::Deflate.deflate(content)
+  def queue_and_send(content)
+    if @interval <= 0 # means send immediately
+      send_request(content)
     else
-      content
+      @semaphore.synchronize {
+        now = Time.now
+        @pile << event
+
+        if now - @timer > @interval # ready to send
+          send_request(@pile.join($/))
+          @timer = now
+          @pile.clear
+        end
+      }
     end
+  end
+
+  private
+  def send_request(content)
+    token = @request_tokens.pop
+    body = compress(content)
     headers = get_headers()
 
     request = client.send(:parallel).send(:post, @url, :body => body, :headers => headers)
@@ -128,29 +198,146 @@ class LogStash::Outputs::SumoLogic < LogStash::Outputs::Base
     request.call
   end # def send_request
   
+  private
+  def compress(content)
+    if @compress
+      if @compress_encoding == GZIP
+        result = gzip(content)
+        result.bytes.to_a.pack('c*')
+      else
+        Zlib::Deflate.deflate(content)
+      end
+    else
+      content
+    end
+  end # def compress
+  
+  private
+  def gzip(content)
+    stream = StringIO.new("w")
+    stream.set_encoding("ASCII")
+    gz = Zlib::GzipWriter.new(stream)
+    gz.write(content)
+    gz.close
+    stream.string.bytes.to_a.pack('c*')
+  end # def gzip
+
   private
   def get_headers()
-    base = { "Content-Type" => "text/plain" }
-    base["Content-Encoding"] = "deflate" if @compress
-    base.merge(@extra_headers)
+
+    base = {}
+    base = @extra_headers if @extra_headers.is_a?(Hash)
+
+    base[CATEGORY_HEADER] = @source_category if @source_category
+    base[HOST_HEADER] = @source_host if @source_host
+    base[NAME_HEADER] = @source_name if @source_name
+    base[CLIENT_HEADER] = 'logstash-output-sumologic'
+    
+    if @compress
+      if @compress_encoding == GZIP
+        base[CONTENT_ENCODING] = GZIP
+      elsif 
+        base[CONTENT_ENCODING] = DEFLATE
+      else
+        log_failure(
+          "Unrecogonized compress encoding",
+          :encoding => @compress_encoding
+        )
+      end
+    end
+
+    if @metrics || @fields_as_metrics
+      if @metrics_format == CARBON2
+        base[CONTENT_TYPE] = CONTENT_TYPE_CARBON2
+      elsif @metrics_format == GRAPHITE
+        base[CONTENT_TYPE] = CONTENT_TYPE_GRAPHITE
+      else
+        log_failure(
+          "Unrecogonized metrics format",
+          :format => @metrics_format
+        )
+      end
+    else
+      base[CONTENT_TYPE] = CONTENT_TYPE_LOG
+    end
+    
+    base
+
   end # def get_headers
 
   private 
-  def format_event(event)
-    if @format.to_s.strip.length == 0
-      LogStash::Json.dump(map_event(event))
+  def event2content(event)
+    if @metrics || @fields_as_metrics
+      event2metrics(event)
     else
-      f = if @format.include? "%{@json}"
-        @format.gsub("%{@json}", LogStash::Json.dump(map_event(event)))
+      event2log(event)
+    end
+  end # def event2content
+
+  private
+  def event2log(event)
+    @format = "%{@json}" if @format.nil? || @format.empty?
+    expand(@format, event)
+  end # def event2log
+
+  private
+  def event2metrics(event)
+    timestamp = get_timestamp(event)
+    source = expand_hash(@metrics, event) unless @fields_as_metrics
+    source = event_as_metrics(event) if @fields_as_metrics
+    source.flat_map { |key, value|
+      get_single_line(event, key, value, timestamp)
+    }.reject(&:nil?).join("\n")
+  end # def event2metrics
+
+  def event_as_metrics(event)
+    hash = event2hash(event)
+    acc = {}
+    hash.keys.each do |field|
+      value = hash[field]
+      dotify(acc, field, value, nil)
+    end
+    acc
+  end # def event_as_metrics
+
+  def get_single_line(event, key, value, timestamp)
+    full = get_metrics_name(event, key)
+    if !ALWAYS_EXCLUDED.include?(full) &&  \
+      (fields_include.empty? || fields_include.any? { |regexp| full.match(regexp) }) && \
+      !(fields_exclude.any? {|regexp| full.match(regexp)}) && \
+      is_number?(value)
+      if @metrics_format == CARBON2
+        @intrinsic_tags["metric"] = full
+        "#{hash2line(@intrinsic_tags, event)} #{hash2line(@meta_tags, event)}#{value} #{timestamp}"
       else
-        @format
+        "#{full} #{value} #{timestamp}" 
       end
-      event.sprintf(f)
     end
-  end # def format_event
+end # def get_single_line
+
+def dotify(acc, key, value, prefix)
+  pk = prefix ? "#{prefix}.#{key}" : key.to_s
+  if value.is_a?(Hash)
+    value.each do |k, v|
+      dotify(acc, k, v, pk)
+    end
+  elsif value.is_a?(Array)
+    value.each_with_index.map { |v, i|
+      dotify(acc, i.to_s, v, pk)
+    }
+  else
+    acc[pk] = value
+  end
+end # def dotify
+
+  private
+  def expand(template, event)
+    template = template.gsub("%{@json}", LogStash::Json.dump(event2hash(event))) if template.include? "%{@json}"
+    event.sprintf(template)
+  end # def expand
 
   private 
-  def map_event(event)
+  def event2hash(event)
     if @json_mapping
       @json_mapping.reduce({}) do |acc, kv|
         k, v = kv
@@ -161,7 +348,45 @@ class LogStash::Outputs::SumoLogic < LogStash::Outputs::Base
       event.to_hash
     end
   end # def map_event
+
+  private
+  def is_number?(me)
+    me.to_f.to_s == me.to_s || me.to_i.to_s == me.to_s
+  end
+
+ private
+  def expand_hash(hash, event)
+    hash.reduce({}) do |acc, kv|
+      k, v = kv
+      exp_k = expand(k, event)
+      exp_v = expand(v, event)
+      acc[exp_k] = exp_v
+      acc
+    end # def expand_hash
+  end
   
+  private
+  def get_timestamp(event)
+    event.get(TIMESTAMP_FIELD).to_i
+  end # def get_timestamp
+
+  private
+  def get_metrics_name(event, name)
+    name = @metrics_name.gsub(METRICS_NAME_PLACEHOLDER, name) if @metrics_name
+    event.sprintf(name)
+  end # def get_metrics_name
+
+  private
+  def hash2line(hash, event)
+    if (hash.is_a?(Hash) && !hash.empty?)
+      expand_hash(hash, event).flat_map { |k, v|
+        "#{k}=#{v} "
+      }.join()
+    else
+      ""
+    end
+  end # hash2line
+
   private
   def log_failure(message, opts)
     @logger.error(message, opts)