logstash-filter-rest2 0.5.3

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA256:
+  metadata.gz: 5093c9df4843c58afdcf73d5340d27b56b2df0e13df0af8f34e9d83f4346653a
+  data.tar.gz: ea0db1681f4c9e948c9bf02e8b628575e9319c3e1ac23855aa904cbb3f7dba0d
+SHA512:
+  metadata.gz: 705f79dae08026d09fd0355f9caa229573cc794cfec2380d2773e34dbdd8d7c3d1b804d0ac054e95306ed772ddf5af937d9bb1f148b214d22acb1b4601e1a986
+  data.tar.gz: 77eed5e5c69da9a35b928f9db1f9909f36d4f42c682f49ccf58121d2e258553eaf94e3dff51999e563933d528819ce162f6a8c6b403c4ea345e0639285705fc5

data/CHANGELOG.md

@@ -0,0 +1,24 @@
+## 0.5.3
+  - freeze all instance variables
+  - fix parallel processing by creating a `deep_clone` for each event
+  - use `LogStash::Util.deep_clone` for object cloning
+  - only dump body as json, if json is enabled in config (default)
+  - delete empty target testcase, as catched by upper logstash `LogStash::ConfigurationError`
+  - fix `sprintf` find and merge for more complex structures
+
+## 0.5.2
+  - Fix behavior, where a referenced field (`%{...}`) has `ruby` chars
+  (i.e., consisting of `:`)
+  - Field interpolation is done by assigning explicit values instead
+  of converting the `sprintf` string back into a `hash`
+
+## 0.5.0
+  - Relax constraint on logstash-core-plugin-api to >= 1.60 <= 2.99
+  - Require devutils >= 0 to make `bundler` update the package
+  - Use Event API for LS-5
+  - Implicit `sprintf`, deprecating the setting
+  - `target` is now required, dropping support to write into top-level in favor of only using new Event API
+    - this follows other logstash-plugins like `logstash-filter-json`
+  - if the response is empty, add the restfailure tags
+  - Some logging moved before code
+  - Testcases adapted to new behavior with error check

data/CONTRIBUTORS

@@ -0,0 +1,11 @@
+The following is a list of people who have contributed ideas, code, bug
+reports, or in general have helped logstash along its way.
+
+Contributors:
+* Aaron Mildenstein (untergeek)
+* Pier-Hugues Pellerin (ph)
+
+Note: If you've sent us patches, bug reports, or otherwise contributed to
+Logstash, and you aren't on the list above and want to be, please let us know
+and we'll make sure you're here. Contributions from folks like you are what make
+open source awesome.

data/Gemfile

@@ -0,0 +1,2 @@
+source 'https://rubygems.org'
+gemspec

data/LICENSE

@@ -0,0 +1,13 @@
+Copyright (c) 2012–2015 Elasticsearch <http://www.elastic.co>
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

data/README.md

@@ -0,0 +1,92 @@
+# Logstash REST Filter [![Build Status](https://travis-ci.org/gandalfb/logstash-filter-rest.svg?branch=master)](https://travis-ci.org/gandalfb/logstash-filter-rest)
+
+This is a filter plugin for [Logstash](https://github.com/elasticsearch/logstash).
+
+It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
+
+## Documentation
+
+This logstash filter provides an easy way to access RESTful Resources within logstash. It can be used to post data to a REST API or to gather data and save it in your log file.
+
+## Usage
+### 1. Installation
+You can use the built-in plugin tool of Logstash to install the filter:
+```
+$LS_HOME/bin/logstash-plugin install logstash-filter-rest
+```
+
+Or you can build it yourself:
+```
+git clone https://github.com/lucashenning/logstash-filter-rest.git
+bundle install
+gem build logstash-filter-rest.gemspec
+$LS_HOME/bin/logstash-plugin install logstash-filter-rest-0.1.0.gem
+```
+
+### 2. Filter Configuration
+Add the following inside the filter section of your logstash configuration:
+
+```sh
+filter {
+  rest {
+    request => {
+      url => "http://example.com"        # string (required, with field reference: "http://example.com?id=%{id}" or params, if defined)
+      method => "post"                   # string (optional, default = "get")
+      headers => {                       # hash (optional)
+        "key1" => "value1"
+        "key2" => "value2"
+      }
+      auth => {
+        user => "AzureDiamond"
+        password => "hunter2"
+      }
+      params => {                        # hash (optional, available for method => "get" and "post"; if post it will be transformed into body hash and posted as json)
+        "key1" => "value1"
+        "key2" => "value2"
+        "key3" => "%{somefield}"         # sprintf is used implicitly
+      }
+    }
+    json => true                         # boolean (optional, default = true)
+    target => "my_key"                   # string (mandatory, no default)
+    fallback => {                        # hash describing a default in case of error
+      "key1" => "value1"
+      "key2" => "value2"
+    }
+  }
+}
+```
+
+Print plugin version:
+
+``` bash
+bin/logstash-plugin list --verbose | grep rest
+```
+
+Examples for running logstash from `cli`:
+
+``` bash
+bin/logstash --debug -e 'input { stdin{} } filter { rest { request => { url => "https://jsonplaceholder.typicode.com/posts" method => "post" params => { "userId" => "%{message}" } headers => { "Content-Type" => "application/json" } } target => 'rest' } } output {stdout { codec => rubydebug }}'
+```
+
+``` bash
+bin/logstash --debug -e 'input { stdin{} } filter { rest { request => { url => "https://jsonplaceholder.typicode.com/posts" method => "post" body => { "userId" => "%{message}" } headers => { "Content-Type" => "application/json" } } target => 'rest' } } output {stdout { codec => rubydebug }}'
+```
+
+``` bash
+bin/logstash --debug -e 'input { stdin{} } filter { rest { request => { url => "http://jsonplaceholder.typicode.com/users/%{message}" } target => 'rest' } } output {stdout { codec => rubydebug }}'
+```
+
+``` bash
+bin/logstash --debug -e 'input { stdin{} } filter { rest { request => { url => "https://jsonplaceholder.typicode.com/posts" method => "get" params => { "userId" => "%{message}" } headers => { "Content-Type" => "application/json" } } target => 'rest' } } output {stdout { codec => rubydebug }}'
+```
+
+
+## Contributing
+
+All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.
+
+Programming is not a required skill. Whatever you've seen about open source and maintainers or community members  saying "send patches or die" - you will not see that here.
+
+It is more important to the community that you are able to contribute.
+
+For more information about contributing, see the [CONTRIBUTING](https://github.com/elasticsearch/logstash/blob/master/CONTRIBUTING.md) file.

data/lib/logstash/filters/rest.rb

@@ -0,0 +1,320 @@
+# encoding: utf-8
+require 'logstash/filters/base'
+require 'logstash/namespace'
+require 'logstash/plugin_mixins/http_client'
+require 'logstash/json'
+
+# Monkey Patch hsh with a recursive compact and deep freeze
+class Hash
+  def compact
+    delete_if { |_k, v| v.respond_to?(:each) ? v.compact.empty? : v.nil? }
+  end
+
+  def deep_freeze
+    each { |_k, v| v.deep_freeze if v.respond_to? :deep_freeze }
+    freeze
+  end
+end
+
+#  Monkey Patch Array with deep freeze
+class Array
+  def compact
+    delete_if { |v| v.respond_to?(:each) ? v.compact.empty? : v.nil? }
+  end
+
+  def deep_freeze
+    each { |j| j.deep_freeze if j.respond_to? :deep_freeze }
+    freeze
+  end
+end
+
+# Logstash REST Filter
+# This filter calls a defined URL and saves the answer into a specified field.
+#
+class LogStash::Filters::Rest < LogStash::Filters::Base
+  include LogStash::PluginMixins::HttpClient
+
+  config_name 'rest'
+
+  # Configure the rest request send via HttpClient Plugin
+  # with hash objects used by the mixin plugin
+  #
+  # For example, if you want the data to be put in the `doc` field:
+  # [source,ruby]
+  #    filter {
+  #      rest {
+  #        request => {
+  #          url => "http://example.com"       # string (required, with field reference: "http://example.com?id=%{id}" or params, if defined)
+  #          method => "post"                  # string (optional, default = "get")
+  #          headers => {                      # hash (optional)
+  #            "key1" => "value1"
+  #            "key2" => "value2"
+  #          }
+  #          auth => {
+  #            user => "AzureDiamond"
+  #            password => "hunter2"
+  #          }
+  #          params => {                       # hash (optional, available for method => "get" and "post"; if post it will be transformed into body hash and posted as json)
+  #            "key1" => "value1"
+  #            "key2" => "value2"
+  #            "key3" => "%{somefield}"        # Field references are found implicitly on startup
+  #          }
+  #        }
+  #        target => "doc"
+  #      }
+  #    }
+  #
+  # NOTE: for further details, please reference https://github.com/logstash-plugins/logstash-mixin-http_client[logstash-mixin-http_client]
+  config :request, :validate => :hash, :required => true
+
+  # The plugin is written json centric, which defaults to true
+  # the response body will be parsed to json if true
+  #
+  # [source,ruby]
+  #     filter {
+  #       rest {
+  #         json => true
+  #       }
+  #     }
+  config :json, :validate => :boolean, :default => true
+
+  # If true, references to event fields can be made in
+  # url, params or body by using '%{somefield}'
+  #
+  # [source,ruby]
+  #     filter {
+  #       rest {
+  #         request => { .. }
+  #         sprintf => true
+  #       }
+  #     }
+  config :sprintf, :validate => :boolean, :default => false, :deprecated => true
+
+  # Define the target field for placing the response data. This setting is
+  # required and may not be omitted. It is not possible to place the response
+  # into the event top-level.
+  #
+  # For example, if you want the data to be put in the `doc` field:
+  # [source,ruby]
+  #     filter {
+  #       rest {
+  #         target => "doc"
+  #       }
+  #     }
+  #
+  # Rest response will be expanded into a data structure in the `target` field.
+  #
+  # NOTE: if the `target` field already exists, it will be overwritten!
+  config :target, :validate => :string, :required => true
+
+  # If set, any error like json parsing or invalid http response
+  # will result in this hash to be added to target instead of error tags
+  #
+  # For example, if you want the fallback data to be put in the `target` field:
+  # [source,ruby]
+  #     filter {
+  #       rest {
+  #         fallback => {
+  #           'key1' => 'value1'
+  #           'key2' => 'value2'
+  #           ...
+  #         }
+  #       }
+  #     }
+  config :fallback, :validate => :hash, :default => {}
+
+  # Append values to the `tags` field when there has been no
+  # successful match or json parsing error
+  config :tag_on_rest_failure, :validate => :array, :default => ['_restfailure']
+  config :tag_on_json_failure, :validate => :array, :default => ['_jsonparsefailure']
+
+  public
+
+  def register
+    @request = normalize_request(@request).deep_freeze
+    @sprintf_fields = find_sprintf(
+      LogStash::Util.deep_clone(@request)
+    ).deep_freeze
+    @sprintf_needed = !@sprintf_fields.empty?
+    @target = normalize_target(@target).freeze
+  end # def register
+
+  private
+
+  def normalize_target(target)
+    # make sure @target is in the format [field name] if defined,
+    # i.e. not empty and surrounded by brakets
+    raise LogStash::ConfigurationError, 'target config string is empty, please set a valid field name' if target.empty?
+    target = "[#{target}]" if target && target !~ /^\[[^\[\]]+\]$/
+    target
+  end
+
+  private
+
+  def normalize_request(url_or_spec)
+    if url_or_spec.is_a?(String)
+      res = [:get, url_or_spec]
+    elsif url_or_spec.is_a?(Hash)
+      # The client will expect keys / values
+      spec = Hash[url_or_spec.clone.map { |k, v| [k.to_sym, v] }]
+
+      # method and url aren't really part of the options, so we pull them out
+      method = (spec.delete(:method) || :get).to_sym.downcase
+      url = spec.delete(:url)
+
+      # if it is a post and json, it is used as body string, not params
+      spec[:body] = spec.delete(:params) if method == :post && spec[:params]
+
+      # We need these strings to be keywords!
+      spec[:auth] = { user: spec[:auth]['user'], pass: spec[:auth]['password'] } if spec[:auth]
+
+      res = [method.freeze, url, spec]
+    else
+      raise LogStash::ConfigurationError, "Invalid URL or request spec: '#{url_or_spec}', expected a String or Hash!"
+    end
+
+    validate_request!(url_or_spec, res)
+    res
+  end
+
+  private
+
+  def validate_request!(url_or_spec, request)
+    method, url, spec = request
+
+    raise LogStash::ConfigurationError, "No URL provided for request! #{url_or_spec}" unless url
+    raise LogStash::ConfigurationError, "Not supported request method #{method}" unless [ :get, :post ].include?( method )
+
+    if spec && spec[:auth]
+      raise LogStash::ConfigurationError, "Auth was specified, but 'user' was not!" unless spec[:auth][:user]
+      raise LogStash::ConfigurationError, "Auth was specified, but 'password' was not!" unless spec[:auth][:pass]
+    end
+
+    request
+  end
+
+  private
+
+  def find_sprintf(config)
+    field_matcher = /%\{[^}]+\}/
+    if config.is_a?(Hash)
+      config.keep_if do |_k, v|
+        find_sprintf(v)
+      end.compact
+    elsif config.is_a?(Array)
+      config.keep_if do |v|
+        find_sprintf(v)
+      end.compact
+    elsif config.is_a?(String) && config =~ field_matcher
+      config
+    end
+  end
+
+  private
+
+  def request_http(request)
+    if request[2].key?(:body) && @json
+      request[2][:body] = LogStash::Json.dump(request[2][:body])
+    end
+    @logger.debug? && @logger.debug('fetching request',
+                                    :request => request)
+
+    method, url, *request_opts = request
+    response = client.http(method, url, *request_opts)
+    [response.code, response.body]
+  end
+
+  private
+
+  def process_response(response, event)
+    if @json
+      begin
+        parsed = LogStash::Json.load(response)
+        if parsed.empty?
+          @logger.warn('rest response empty',
+                       :response => response, :event => event)
+          @tag_on_rest_failure.each { |tag| event.tag(tag) }
+        else
+          event.set(@target, parsed)
+        end
+      rescue
+        if @fallback.empty?
+          @logger.warn('JSON parsing error',
+                       :response => response, :event => event)
+          @tag_on_json_failure.each { |tag| event.tag(tag) }
+        else
+          event.set(@target, @fallback)
+        end
+      end
+    else
+      event.set(@target, response.strip)
+    end
+  end
+
+  private
+
+  def field_intrpl(intrpl_fields, event)
+    case intrpl_fields
+    when String
+      result = event.sprintf(intrpl_fields)
+    when Array
+      result = []
+      intrpl_fields.each do |v|
+        result << field_intrpl(v, event)
+      end
+    when Hash
+      result = {}
+      intrpl_fields.each do |k, v|
+        result[k] = field_intrpl(v, event)
+      end
+    else
+      result = intrpl_fields
+    end
+    result
+  end
+
+  public
+
+  def filter(event)
+    return unless filter?(event)
+    request = LogStash::Util.deep_clone(@request)
+    @logger.debug? && @logger.debug('processing request',
+                                    :request => request,
+                                    :sprintf_needed => @sprintf_needed)
+
+    if @sprintf_needed
+      request = field_intrpl(request, event)
+      @logger.debug? && @logger.debug('interpolated request',
+                                      :request => request)
+    end
+
+    client_error = nil
+    begin
+      code, body = request_http(request)
+    rescue StandardError => e
+      client_error = e
+    end
+
+    if !client_error && code.between?(200, 299)
+      @logger.debug? && @logger.debug('success received',
+                                      :code => code, :body => body)
+      process_response(body, event)
+    else
+      @logger.debug? && @logger.debug('http error received',
+                                      :code => code, :body => body,
+                                      :client_error => client_error)
+      if @fallback.empty?
+        @logger.error('error in rest filter',
+                      :request => request, :json => @json,
+                      :code => code, :body => body,
+                      :client_error => client_error)
+        @tag_on_rest_failure.each { |tag| event.tag(tag) }
+      else
+        @logger.debug? && @logger.debug('setting fallback',
+                                        :fallback => @fallback)
+        event.set(@target, @fallback)
+      end
+    end
+    filter_matched(event)
+  end # def filter
+end # class LogStash::Filters::Rest

data/logstash-filter-rest.gemspec

@@ -0,0 +1,33 @@
+Gem::Specification.new do |s|
+  s.name = 'logstash-filter-rest2'
+  s.version = '0.5.3'
+  s.licenses = ['Apache-2.0']
+  s.summary = 'This filter requests data from a RESTful Web Service.'
+  s.description = 'This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install logstash-filter-rest2. This gem is not a stand-alone program'
+  s.authors = ['Lucas Henning', 'Gandalf Buscher', 'Boris Gorbylev']
+  s.email = 'ekho@ekho.name'
+  s.homepage = 'https://github.com/ekho/logstash-filter-rest/'
+  s.require_paths = ['lib']
+
+  # Files
+  s.files = Dir['lib/**/*',
+                'spec/**/*',
+                'vendor/**/*',
+                '*.gemspec',
+                '*.md',
+                'CONTRIBUTORS',
+                'Gemfile',
+                'LICENSE',
+                'NOTICE.TXT']
+  # Tests
+  s.test_files = s.files.grep(%r{^(test|spec|features)/})
+
+  # Special flag to let us know this is actually a logstash plugin
+  s.metadata = { 'logstash_plugin' => 'true', 'logstash_group' => 'filter' }
+
+  # Gem dependencies
+  s.add_runtime_dependency 'logstash-core-plugin-api', '>= 1.60', '<= 2.99'
+  s.add_runtime_dependency 'logstash-mixin-http_client', '>= 2.2.4', '< 5.0.0'
+
+  s.add_development_dependency 'logstash-devutils', '>= 0', '< 2.0.0'
+end

data/spec/filters/rest_spec.rb

@@ -0,0 +1,406 @@
+require 'logstash/devutils/rspec/spec_helper'
+require 'logstash/filters/rest'
+
+describe LogStash::Filters::Rest do
+  describe 'Set to Rest Filter Get without params' do
+    let(:config) do <<-CONFIG
+      filter {
+        rest {
+          request => {
+            url => 'http://jsonplaceholder.typicode.com/users/10'
+          }
+          json => true
+          target => 'rest'
+        }
+      }
+    CONFIG
+    end
+
+    sample('message' => 'some text') do
+      expect(subject).to include('rest')
+      expect(subject.get('rest')).to include('id')
+      expect(subject.get('[rest][id]')).to eq(10)
+      expect(subject.get('rest')).to_not include('fallback')
+    end
+  end
+  describe 'Set to Rest Filter Get without params custom target' do
+    let(:config) do <<-CONFIG
+      filter {
+        rest {
+          request => {
+            url => 'http://jsonplaceholder.typicode.com/users/10'
+          }
+          json => true
+          target => 'testing'
+        }
+      }
+    CONFIG
+    end
+
+    sample('message' => 'some text') do
+      expect(subject).to include('testing')
+      expect(subject.get('testing')).to include('id')
+      expect(subject.get('[testing][id]')).to eq(10)
+      expect(subject.get('testing')).to_not include('fallback')
+    end
+  end
+  describe 'Set to Rest Filter Get without params and sprintf' do
+    let(:config) do <<-CONFIG
+      filter {
+        rest {
+          request => {
+            url => "http://jsonplaceholder.typicode.com/users/%{message}"
+          }
+          json => true
+          sprintf => true
+          target => 'rest'
+        }
+      }
+    CONFIG
+    end
+
+    sample('message' => '10') do
+      expect(subject).to include('rest')
+      expect(subject.get('rest')).to include('id')
+      expect(subject.get('[rest][id]')).to eq(10)
+      expect(subject.get('rest')).to_not include('fallback')
+    end
+    sample('message' => '9') do
+      expect(subject).to include('rest')
+      expect(subject.get('rest')).to include('id')
+      expect(subject.get('[rest][id]')).to eq(9)
+      expect(subject.get('rest')).to_not include('fallback')
+    end
+  end
+  describe 'Set to Rest Filter Get without params http error' do
+    let(:config) do <<-CONFIG
+      filter {
+        rest {
+          request => {
+            url => 'http://httpstat.us/404'
+          }
+          json => true
+          target => 'rest'
+        }
+      }
+    CONFIG
+    end
+
+    sample('message' => 'some text') do
+      expect(subject).to_not include('rest')
+      expect(subject.get('tags')).to include('_restfailure')
+    end
+  end
+  describe 'Set to Rest Filter Get with params' do
+    let(:config) do <<-CONFIG
+      filter {
+        rest {
+          request => {
+            url => 'https://jsonplaceholder.typicode.com/posts'
+            params => {
+              userId => 10
+            }
+            headers => {
+              'Content-Type' => 'application/json'
+            }
+          }
+          json => true
+          target => 'rest'
+        }
+      }
+    CONFIG
+    end
+
+    sample('message' => 'some text') do
+      expect(subject).to include('rest')
+      expect(subject.get('[rest][0]')).to include('userId')
+      expect(subject.get('[rest][0][userId]')).to eq(10)
+      expect(subject.get('rest')).to_not include('fallback')
+    end
+  end
+  describe 'empty response' do
+    let(:config) do <<-CONFIG
+      filter {
+        rest {
+          request => {
+            url => 'https://jsonplaceholder.typicode.com/posts'
+            params => {
+              userId => 0
+            }
+            headers => {
+              'Content-Type' => 'application/json'
+            }
+          }
+          target => 'rest'
+        }
+      }
+    CONFIG
+    end
+
+    sample('message' => 'some text') do
+      expect(subject).to_not include('rest')
+      expect(subject.get('tags')).to include('_restfailure')
+    end
+  end
+  describe 'Set to Rest Filter Get with params sprintf' do
+    let(:config) do <<-CONFIG
+      filter {
+        rest {
+          request => {
+            url => 'https://jsonplaceholder.typicode.com/posts'
+            params => {
+              userId => "%{message}"
+              id => "%{message}"
+            }
+            headers => {
+              'Content-Type' => 'application/json'
+            }
+          }
+          json => true
+          target => 'rest'
+        }
+      }
+    CONFIG
+    end
+
+    sample('message' => '1') do
+      expect(subject).to include('rest')
+      expect(subject.get('[rest][0]')).to include('userId')
+      expect(subject.get('[rest][0][userId]')).to eq(1)
+      expect(subject.get('[rest][0][id]')).to eq(1)
+      expect(subject.get('rest').length).to eq(1)
+      expect(subject.get('rest')).to_not include('fallback')
+    end
+  end
+  describe 'Set to Rest Filter Post with params' do
+    let(:config) do <<-CONFIG
+      filter {
+        rest {
+          request => {
+            url => 'https://jsonplaceholder.typicode.com/posts'
+            method => 'post'
+            params => {
+              title => 'foo'
+              body => 'bar'
+              userId => 42
+            }
+            headers => {
+              'Content-Type' => 'application/json'
+            }
+          }
+          json => true
+          target => 'rest'
+        }
+      }
+    CONFIG
+    end
+
+    sample('message' => 'some text') do
+      expect(subject).to include('rest')
+      expect(subject.get('rest')).to include('id')
+      expect(subject.get('[rest][userId]')).to eq(42)
+      expect(subject.get('rest')).to_not include('fallback')
+    end
+  end
+  describe 'Set to Rest Filter Post with params sprintf' do
+    let(:config) do <<-CONFIG
+      filter {
+        rest {
+          request => {
+            url => 'https://jsonplaceholder.typicode.com/posts'
+            method => 'post'
+            params => {
+              title => '%{message}'
+              body => 'bar'
+              userId => "%{message}"
+            }
+            headers => {
+              'Content-Type' => 'application/json'
+            }
+          }
+          json => true
+          target => 'rest'
+        }
+      }
+    CONFIG
+    end
+
+    sample('message' => '42') do
+      expect(subject).to include('rest')
+      expect(subject.get('rest')).to include('id')
+      expect(subject.get('[rest][title]')).to eq(42)
+      expect(subject.get('[rest][userId]')).to eq(42)
+      expect(subject.get('rest')).to_not include('fallback')
+    end
+    sample('message' => ':5e?#!-_') do
+      expect(subject).to include('rest')
+      expect(subject.get('rest')).to include('id')
+      expect(subject.get('[rest][title]')).to eq(':5e?#!-_')
+      expect(subject.get('[rest][userId]')).to eq(':5e?#!-_')
+      expect(subject.get('rest')).to_not include('fallback')
+    end
+    sample('message' => ':4c43=>') do
+      expect(subject).to include('rest')
+      expect(subject.get('rest')).to include('id')
+      expect(subject.get('[rest][title]')).to eq(':4c43=>')
+      expect(subject.get('[rest][userId]')).to eq(':4c43=>')
+      expect(subject.get('rest')).to_not include('fallback')
+    end
+  end
+  describe 'Set to Rest Filter Post with body sprintf' do
+    let(:config) do <<-CONFIG
+      filter {
+        rest {
+          request => {
+            url => 'https://jsonplaceholder.typicode.com/posts'
+            method => 'post'
+            body => {
+              title => 'foo'
+              body => 'bar'
+              userId => "%{message}"
+            }
+            headers => {
+              'Content-Type' => 'application/json'
+            }
+          }
+          json => true
+          target => 'rest'
+        }
+      }
+    CONFIG
+    end
+
+    sample('message' => '42') do
+      expect(subject).to include('rest')
+      expect(subject.get('rest')).to include('id')
+      expect(subject.get('[rest][userId]')).to eq(42)
+      expect(subject.get('rest')).to_not include('fallback')
+    end
+  end
+  describe 'Set to Rest Filter Post with body sprintf nested params' do
+    let(:config) do <<-CONFIG
+      filter {
+        rest {
+          request => {
+            url => 'https://jsonplaceholder.typicode.com/posts'
+            method => 'post'
+            body => {
+              key1 => [
+                {
+                  "filterType" => "text"
+                  "text" => "salmon"
+                  "boolean" => false
+                },
+                {
+                  "filterType" => "unique"
+                }
+              ]
+              key2 => [
+                {
+                  "message" => "123%{message}"
+                  "boolean" => true
+                }
+              ]
+              key3 => [
+                {
+                  "text" => "%{message}123"
+                  "filterType" => "text"
+                  "number" => 44
+                },
+                {
+                  "filterType" => "unique"
+                  "null" => nil
+                }
+              ]
+              userId => "%{message}"
+            }
+            headers => {
+              'Content-Type' => 'application/json'
+            }
+          }
+          target => 'rest'
+        }
+      }
+    CONFIG
+    end
+
+    sample('message' => '42') do
+      expect(subject).to include('rest')
+      expect(subject.get('rest')).to include('key1')
+      expect(subject.get('[rest][key1][0][boolean]')).to eq('false')
+      expect(subject.get('[rest][key1][1][filterType]')).to eq('unique')
+      expect(subject.get('[rest][key2][0][message]')).to eq('12342')
+      expect(subject.get('[rest][key2][0][boolean]')).to eq('true')
+      expect(subject.get('[rest][key3][0][text]')).to eq('42123')
+      expect(subject.get('[rest][key3][0][filterType]')).to eq('text')
+      expect(subject.get('[rest][key3][0][number]')).to eq(44)
+      expect(subject.get('[rest][key3][1][filterType]')).to eq('unique')
+      expect(subject.get('[rest][key3][1][null]')).to eq('nil')
+      expect(subject.get('[rest][userId]')).to eq(42)
+      expect(subject.get('rest')).to_not include('fallback')
+    end
+  end
+  describe 'fallback' do
+    let(:config) do <<-CONFIG
+      filter {
+        rest {
+          request => {
+            url => 'http://jsonplaceholder.typicode.com/users/0'
+          }
+          json => true
+          fallback => {
+            'fallback1' => true
+            'fallback2' => true
+          }
+          target => 'rest'
+        }
+      }
+    CONFIG
+    end
+
+    sample('message' => 'some text') do
+      expect(subject).to include('rest')
+      expect(subject.get('rest')).to include('fallback1')
+      expect(subject.get('rest')).to include('fallback2')
+      expect(subject.get('rest')).to_not include('id')
+    end
+  end
+  describe 'empty target exception' do
+    let(:config) do <<-CONFIG
+      filter {
+        rest {
+          request => {
+            url => 'http://jsonplaceholder.typicode.com/users/0'
+          }
+          json => true
+          fallback => {
+            'fallback1' => true
+            'fallback2' => true
+          }
+          target => ''
+        }
+      }
+    CONFIG
+    end
+    sample('message' => 'some text') do
+      expect { subject }.to raise_error(LogStash::ConfigurationError)
+    end
+  end
+  describe 'http client throws exception' do
+    let(:config) do <<-CONFIG
+      filter {
+        rest {
+          request => {
+            url => 'invalid_url'
+          }
+          target => 'rest'
+        }
+      }
+    CONFIG
+    end
+    sample('message' => 'some text') do
+      expect(subject).to_not include('rest')
+      expect(subject.get('tags')).to include('_restfailure')
+    end
+  end
+end

metadata

@@ -0,0 +1,118 @@
+--- !ruby/object:Gem::Specification
+name: logstash-filter-rest2
+version: !ruby/object:Gem::Version
+  version: 0.5.3
+platform: ruby
+authors:
+- Lucas Henning
+- Gandalf Buscher
+- Boris Gorbylev
+autorequire:
+bindir: bin
+cert_chain: []
+date: 2017-07-04 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '1.60'
+    - - "<="
+      - !ruby/object:Gem::Version
+        version: '2.99'
+  name: logstash-core-plugin-api
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '1.60'
+    - - "<="
+      - !ruby/object:Gem::Version
+        version: '2.99'
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 2.2.4
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: 5.0.0
+  name: logstash-mixin-http_client
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 2.2.4
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: 5.0.0
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: 2.0.0
+  name: logstash-devutils
+  prerelease: false
+  type: :development
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: 2.0.0
+description: This gem is a logstash plugin required to be installed on top of the
+  Logstash core pipeline using $LS_HOME/bin/logstash-plugin install logstash-filter-rest2.
+  This gem is not a stand-alone program
+email: ekho@ekho.name
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- CONTRIBUTORS
+- Gemfile
+- LICENSE
+- README.md
+- lib/logstash/filters/rest.rb
+- logstash-filter-rest.gemspec
+- spec/filters/rest_spec.rb
+homepage: https://github.com/ekho/logstash-filter-rest/
+licenses:
+- Apache-2.0
+metadata:
+  logstash_plugin: 'true'
+  logstash_group: filter
+post_install_message:
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project:
+rubygems_version: 2.6.11
+signing_key:
+specification_version: 4
+summary: This filter requests data from a RESTful Web Service.
+test_files:
+- spec/filters/rest_spec.rb