logstash-codec-json_stream 0.0.1

Sign up to get free protection for your applications and to get access to all the features.
checksums.yaml ADDED
@@ -0,0 +1,7 @@
1
+ ---
2
+ SHA1:
3
+ metadata.gz: 0f139c6c6288fd48811d138546e72b7c172e4c76
4
+ data.tar.gz: a27785bee26ce029a6ce6a28d45a04d85750199c
5
+ SHA512:
6
+ metadata.gz: bea1ff89d17b91b7f75be9b326568967b826c196f7608643e843db4e160aca6049c5561de9eb81ce433f08bbbacae7a821170c1db5b56e06234161bb7bff685d
7
+ data.tar.gz: 95fbaa6505fb63bcce99e80365f2d208e63809c2554ae517ce295d5c496a745eb11f5d47ffb703b38f6ad9ee6bd9966ea8ca65cf3821999b787e237a6e49bafa
data/CHANGELOG.md ADDED
@@ -0,0 +1,3 @@
1
+ ## 0.0.1
2
+ - Forked from Json_lines codec by elastic @elastic: Thank you
3
+ - Added bracket counter to extract jsons.
data/CONTRIBUTORS ADDED
@@ -0,0 +1,19 @@
1
+ The following is a list of people who have contributed ideas, code, bug
2
+ reports, or in general have helped logstash along its way.
3
+
4
+ Contributors:
5
+ * Colin Surprenant (colinsurprenant)
6
+ * Greg Mefford (GregMefford)
7
+ * Jordan Sissel (jordansissel)
8
+ * João Duarte (jsvd)
9
+ * Nick Ethier (nickethier)
10
+ * Pier-Hugues Pellerin (ph)
11
+ * Richard Pijnenburg (electrical)
12
+ * Suyog Rao (suyograo)
13
+ * Tal Levy (talevy)
14
+ * Guy Boertje (guyboertje)
15
+
16
+ Note: If you've sent us patches, bug reports, or otherwise contributed to
17
+ Logstash, and you aren't on the list above and want to be, please let us know
18
+ and we'll make sure you're here. Contributions from folks like you are what make
19
+ open source awesome.
data/Gemfile ADDED
@@ -0,0 +1,11 @@
1
+ source 'https://rubygems.org'
2
+
3
+ gemspec
4
+
5
+ logstash_path = ENV["LOGSTASH_PATH"] || "../../logstash"
6
+ use_logstash_source = ENV["LOGSTASH_SOURCE"] && ENV["LOGSTASH_SOURCE"].to_s == "1"
7
+
8
+ if Dir.exist?(logstash_path) && use_logstash_source
9
+ gem 'logstash-core', :path => "#{logstash_path}/logstash-core"
10
+ gem 'logstash-core-plugin-api', :path => "#{logstash_path}/logstash-core-plugin-api"
11
+ end
data/LICENSE ADDED
@@ -0,0 +1,13 @@
1
+ Copyright (c) 2012-2018 Elasticsearch <http://www.elastic.co>
2
+
3
+ Licensed under the Apache License, Version 2.0 (the "License");
4
+ you may not use this file except in compliance with the License.
5
+ You may obtain a copy of the License at
6
+
7
+ http://www.apache.org/licenses/LICENSE-2.0
8
+
9
+ Unless required by applicable law or agreed to in writing, software
10
+ distributed under the License is distributed on an "AS IS" BASIS,
11
+ WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
12
+ See the License for the specific language governing permissions and
13
+ limitations under the License.
data/NOTICE.TXT ADDED
@@ -0,0 +1,5 @@
1
+ Elasticsearch
2
+ Copyright 2012-2015 Elasticsearch
3
+
4
+ This product includes software developed by The Apache Software
5
+ Foundation (http://www.apache.org/).
data/README.md ADDED
@@ -0,0 +1,98 @@
1
+ # Logstash Plugin
2
+
3
+ [![Travis Build Status](https://travis-ci.org/logstash-plugins/logstash-codec-json_lines.svg)](https://travis-ci.org/logstash-plugins/logstash-codec-json_lines)
4
+
5
+ This is a plugin for [Logstash](https://github.com/elastic/logstash).
6
+
7
+ It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
8
+
9
+ ## Documentation
10
+
11
+ Logstash provides infrastructure to automatically generate documentation for this plugin. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html. All plugin documentation are placed under one [central location](http://www.elastic.co/guide/en/logstash/current/).
12
+
13
+ - For formatting code or config example, you can use the asciidoc `[source,ruby]` directive
14
+ - For more asciidoc formatting tips, see the excellent reference here https://github.com/elastic/docs#asciidoc-guide
15
+
16
+ ## Need Help?
17
+
18
+ Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/logstash discussion forum.
19
+
20
+ ## Developing
21
+
22
+ ### 1. Plugin Developement and Testing
23
+
24
+ #### Code
25
+ - To get started, you'll need JRuby with the Bundler gem installed.
26
+
27
+ - Create a new plugin or clone and existing from the GitHub [logstash-plugins](https://github.com/logstash-plugins) organization. We also provide [example plugins](https://github.com/logstash-plugins?query=example).
28
+
29
+ - Install dependencies
30
+ ```sh
31
+ bundle install
32
+ ```
33
+
34
+ #### Test
35
+
36
+ - Update your dependencies
37
+
38
+ ```sh
39
+ bundle install
40
+ ```
41
+
42
+ - Run tests
43
+
44
+ ```sh
45
+ bundle exec rspec
46
+ ```
47
+
48
+ ### 2. Running your unpublished Plugin in Logstash
49
+
50
+ #### 2.1 Run in a local Logstash clone
51
+
52
+ - Edit Logstash `Gemfile` and add the local plugin path, for example:
53
+ ```ruby
54
+ gem "logstash-filter-awesome", :path => "/your/local/logstash-filter-awesome"
55
+ ```
56
+ - Install plugin
57
+ ```sh
58
+ # Logstash 2.3 and higher
59
+ bin/logstash-plugin install --no-verify
60
+
61
+ # Prior to Logstash 2.3
62
+ bin/plugin install --no-verify
63
+
64
+ ```
65
+ - Run Logstash with your plugin
66
+ ```sh
67
+ bin/logstash -e 'filter {awesome {}}'
68
+ ```
69
+ At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.
70
+
71
+ #### 2.2 Run in an installed Logstash
72
+
73
+ You can use the same **2.1** method to run your plugin in an installed Logstash by editing its `Gemfile` and pointing the `:path` to your local plugin development directory or you can build the gem and install it using:
74
+
75
+ - Build your plugin gem
76
+ ```sh
77
+ gem build logstash-filter-awesome.gemspec
78
+ ```
79
+ - Install the plugin from the Logstash home
80
+ ```sh
81
+ # Logstash 2.3 and higher
82
+ bin/logstash-plugin install --no-verify
83
+
84
+ # Prior to Logstash 2.3
85
+ bin/plugin install --no-verify
86
+
87
+ ```
88
+ - Start Logstash and proceed to test the plugin
89
+
90
+ ## Contributing
91
+
92
+ All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.
93
+
94
+ Programming is not a required skill. Whatever you've seen about open source and maintainers or community members saying "send patches or die" - you will not see that here.
95
+
96
+ It is more important to the community that you are able to contribute.
97
+
98
+ For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file.
@@ -0,0 +1,67 @@
1
+ :plugin: json_lines
2
+ :type: codec
3
+
4
+ ///////////////////////////////////////////
5
+ START - GENERATED VARIABLES, DO NOT EDIT!
6
+ ///////////////////////////////////////////
7
+ :version: %VERSION%
8
+ :release_date: %RELEASE_DATE%
9
+ :changelog_url: %CHANGELOG_URL%
10
+ :include_path: ../../../../logstash/docs/include
11
+ ///////////////////////////////////////////
12
+ END - GENERATED VARIABLES, DO NOT EDIT!
13
+ ///////////////////////////////////////////
14
+
15
+ [id="plugins-{type}s-{plugin}"]
16
+
17
+ === Json_lines codec plugin
18
+
19
+ include::{include_path}/plugin_header.asciidoc[]
20
+
21
+ ==== Description
22
+
23
+ This codec will decode streamed JSON that is newline delimited.
24
+ Encoding will emit a single JSON string ending in a `@delimiter`
25
+ NOTE: Do not use this codec if your source input is line-oriented JSON, for
26
+ example, redis or file inputs. Rather, use the json codec.
27
+ More info: This codec is expecting to receive a stream (string) of newline
28
+ terminated lines. The file input will produce a line string without a newline.
29
+ Therefore this codec cannot work with line oriented inputs.
30
+
31
+ [id="plugins-{type}s-{plugin}-options"]
32
+ ==== Json_lines Codec Configuration Options
33
+
34
+ [cols="<,<,<",options="header",]
35
+ |=======================================================================
36
+ |Setting |Input type|Required
37
+ | <<plugins-{type}s-{plugin}-charset>> |<<string,string>>, one of `["ASCII-8BIT", "UTF-8", "US-ASCII", "Big5", "Big5-HKSCS", "Big5-UAO", "CP949", "Emacs-Mule", "EUC-JP", "EUC-KR", "EUC-TW", "GB2312", "GB18030", "GBK", "ISO-8859-1", "ISO-8859-2", "ISO-8859-3", "ISO-8859-4", "ISO-8859-5", "ISO-8859-6", "ISO-8859-7", "ISO-8859-8", "ISO-8859-9", "ISO-8859-10", "ISO-8859-11", "ISO-8859-13", "ISO-8859-14", "ISO-8859-15", "ISO-8859-16", "KOI8-R", "KOI8-U", "Shift_JIS", "UTF-16BE", "UTF-16LE", "UTF-32BE", "UTF-32LE", "Windows-31J", "Windows-1250", "Windows-1251", "Windows-1252", "IBM437", "IBM737", "IBM775", "CP850", "IBM852", "CP852", "IBM855", "CP855", "IBM857", "IBM860", "IBM861", "IBM862", "IBM863", "IBM864", "IBM865", "IBM866", "IBM869", "Windows-1258", "GB1988", "macCentEuro", "macCroatian", "macCyrillic", "macGreek", "macIceland", "macRoman", "macRomania", "macThai", "macTurkish", "macUkraine", "CP950", "CP951", "IBM037", "stateless-ISO-2022-JP", "eucJP-ms", "CP51932", "EUC-JIS-2004", "GB12345", "ISO-2022-JP", "ISO-2022-JP-2", "CP50220", "CP50221", "Windows-1256", "Windows-1253", "Windows-1255", "Windows-1254", "TIS-620", "Windows-874", "Windows-1257", "MacJapanese", "UTF-7", "UTF8-MAC", "UTF-16", "UTF-32", "UTF8-DoCoMo", "SJIS-DoCoMo", "UTF8-KDDI", "SJIS-KDDI", "ISO-2022-JP-KDDI", "stateless-ISO-2022-JP-KDDI", "UTF8-SoftBank", "SJIS-SoftBank", "BINARY", "CP437", "CP737", "CP775", "IBM850", "CP857", "CP860", "CP861", "CP862", "CP863", "CP864", "CP865", "CP866", "CP869", "CP1258", "Big5-HKSCS:2008", "ebcdic-cp-us", "eucJP", "euc-jp-ms", "EUC-JISX0213", "eucKR", "eucTW", "EUC-CN", "eucCN", "CP936", "ISO2022-JP", "ISO2022-JP2", "ISO8859-1", "ISO8859-2", "ISO8859-3", "ISO8859-4", "ISO8859-5", "ISO8859-6", "CP1256", "ISO8859-7", "CP1253", "ISO8859-8", "CP1255", "ISO8859-9", "CP1254", "ISO8859-10", "ISO8859-11", "CP874", "ISO8859-13", "CP1257", "ISO8859-14", "ISO8859-15", "ISO8859-16", "CP878", "MacJapan", "ASCII", "ANSI_X3.4-1968", "646", "CP65000", "CP65001", "UTF-8-MAC", "UTF-8-HFS", "UCS-2BE", "UCS-4BE", "UCS-4LE", "CP932", "csWindows31J", "SJIS", "PCK", "CP1250", "CP1251", "CP1252", "external", "locale"]`|No
38
+ | <<plugins-{type}s-{plugin}-delimiter>> |<<string,string>>|No
39
+ |=======================================================================
40
+
41
+ &nbsp;
42
+
43
+ [id="plugins-{type}s-{plugin}-charset"]
44
+ ===== `charset`
45
+
46
+ * Value can be any of: `ASCII-8BIT`, `UTF-8`, `US-ASCII`, `Big5`, `Big5-HKSCS`, `Big5-UAO`, `CP949`, `Emacs-Mule`, `EUC-JP`, `EUC-KR`, `EUC-TW`, `GB2312`, `GB18030`, `GBK`, `ISO-8859-1`, `ISO-8859-2`, `ISO-8859-3`, `ISO-8859-4`, `ISO-8859-5`, `ISO-8859-6`, `ISO-8859-7`, `ISO-8859-8`, `ISO-8859-9`, `ISO-8859-10`, `ISO-8859-11`, `ISO-8859-13`, `ISO-8859-14`, `ISO-8859-15`, `ISO-8859-16`, `KOI8-R`, `KOI8-U`, `Shift_JIS`, `UTF-16BE`, `UTF-16LE`, `UTF-32BE`, `UTF-32LE`, `Windows-31J`, `Windows-1250`, `Windows-1251`, `Windows-1252`, `IBM437`, `IBM737`, `IBM775`, `CP850`, `IBM852`, `CP852`, `IBM855`, `CP855`, `IBM857`, `IBM860`, `IBM861`, `IBM862`, `IBM863`, `IBM864`, `IBM865`, `IBM866`, `IBM869`, `Windows-1258`, `GB1988`, `macCentEuro`, `macCroatian`, `macCyrillic`, `macGreek`, `macIceland`, `macRoman`, `macRomania`, `macThai`, `macTurkish`, `macUkraine`, `CP950`, `CP951`, `IBM037`, `stateless-ISO-2022-JP`, `eucJP-ms`, `CP51932`, `EUC-JIS-2004`, `GB12345`, `ISO-2022-JP`, `ISO-2022-JP-2`, `CP50220`, `CP50221`, `Windows-1256`, `Windows-1253`, `Windows-1255`, `Windows-1254`, `TIS-620`, `Windows-874`, `Windows-1257`, `MacJapanese`, `UTF-7`, `UTF8-MAC`, `UTF-16`, `UTF-32`, `UTF8-DoCoMo`, `SJIS-DoCoMo`, `UTF8-KDDI`, `SJIS-KDDI`, `ISO-2022-JP-KDDI`, `stateless-ISO-2022-JP-KDDI`, `UTF8-SoftBank`, `SJIS-SoftBank`, `BINARY`, `CP437`, `CP737`, `CP775`, `IBM850`, `CP857`, `CP860`, `CP861`, `CP862`, `CP863`, `CP864`, `CP865`, `CP866`, `CP869`, `CP1258`, `Big5-HKSCS:2008`, `ebcdic-cp-us`, `eucJP`, `euc-jp-ms`, `EUC-JISX0213`, `eucKR`, `eucTW`, `EUC-CN`, `eucCN`, `CP936`, `ISO2022-JP`, `ISO2022-JP2`, `ISO8859-1`, `ISO8859-2`, `ISO8859-3`, `ISO8859-4`, `ISO8859-5`, `ISO8859-6`, `CP1256`, `ISO8859-7`, `CP1253`, `ISO8859-8`, `CP1255`, `ISO8859-9`, `CP1254`, `ISO8859-10`, `ISO8859-11`, `CP874`, `ISO8859-13`, `CP1257`, `ISO8859-14`, `ISO8859-15`, `ISO8859-16`, `CP878`, `MacJapan`, `ASCII`, `ANSI_X3.4-1968`, `646`, `CP65000`, `CP65001`, `UTF-8-MAC`, `UTF-8-HFS`, `UCS-2BE`, `UCS-4BE`, `UCS-4LE`, `CP932`, `csWindows31J`, `SJIS`, `PCK`, `CP1250`, `CP1251`, `CP1252`, `external`, `locale`
47
+ * Default value is `"UTF-8"`
48
+
49
+ The character encoding used in this codec. Examples include `UTF-8` and
50
+ `CP1252`
51
+
52
+ JSON requires valid `UTF-8` strings, but in some cases, software that
53
+ emits JSON does so in another encoding (nxlog, for example). In
54
+ weird cases like this, you can set the charset setting to the
55
+ actual encoding of the text and logstash will convert it for you.
56
+
57
+ For nxlog users, you'll want to set this to `CP1252`
58
+
59
+ [id="plugins-{type}s-{plugin}-delimiter"]
60
+ ===== `delimiter`
61
+
62
+ * Value type is <<string,string>>
63
+ * Default value is `"\n"`
64
+
65
+ Change the delimiter that separates lines
66
+
67
+
@@ -0,0 +1,84 @@
1
+ # encoding: utf-8
2
+ require "logstash/codecs/base"
3
+ require "logstash/util/charset"
4
+ require "logstash/util/buftok"
5
+ require "logstash/json"
6
+
7
+ # This codec will decode streamed JSON that is not delimited.
8
+ # Encoding will emit a single JSON string ending in a `@delimiter`
9
+
10
+ class LogStash::Codecs::JSONStream < LogStash::Codecs::Base
11
+ config_name "json_stream"
12
+
13
+ config :charset, :validate => ::Encoding.name_list, :default => "UTF-8"
14
+
15
+ # Change the delimiter that separates lines
16
+ config :delimiter, :validate => :string, :default => "\n"
17
+
18
+ public
19
+
20
+ def register
21
+ @converter = LogStash::Util::Charset.new(@charset)
22
+ @converter.logger = @logger
23
+ end
24
+
25
+ def decode(data, &block)
26
+ io = StringIO.new data
27
+
28
+ loop.inject(counter: 0, string: '') do |acc|
29
+ char = io.getc
30
+
31
+ break if char.nil? # EOF
32
+ next acc if acc[:counter].zero? && char != '{' # between objects
33
+
34
+ acc[:string] << char
35
+
36
+ if char == '}' && (acc[:counter] -= 1).zero?
37
+ # ⇓⇓⇓ # CALLBACK, feel free to JSON.parse here
38
+ parse(@converter.convert(acc[:string].gsub(/\p{Space}+/, ' ')), &block)
39
+ next {counter: 0, string: ''} # from scratch
40
+ end
41
+
42
+ acc.tap do |result|
43
+ result[:counter] += 1 if char == '{'
44
+ end
45
+ end
46
+ end
47
+
48
+ def encode(event)
49
+ # Tack on a @delimiter for now because previously most of logstash's JSON
50
+ # outputs emitted one per line, and whitespace is OK in json.
51
+ @on_event.call(event, "#{event.to_json}#{@delimiter}")
52
+ end
53
+
54
+ def flush(&block)
55
+ remainder = @buffer.flush
56
+ if !remainder.empty?
57
+ parse(@converter.convert(remainder), &block)
58
+ end
59
+ end
60
+
61
+ private
62
+
63
+ # from_json_parse uses the Event#from_json method to deserialize and directly produce events
64
+ def from_json_parse(json, &block)
65
+ LogStash::Event.from_json(json).each { |event| yield event }
66
+ rescue LogStash::Json::ParserError => e
67
+ @logger.warn("JSON parse error, original data now in message field", :error => e, :data => json)
68
+ yield LogStash::Event.new("message" => json, "tags" => ["_jsonparsefailure"])
69
+ end
70
+
71
+ # legacy_parse uses the LogStash::Json class to deserialize json
72
+ def legacy_parse(json, &block)
73
+ # ignore empty/blank lines which LogStash::Json#load returns as nil
74
+ o = LogStash::Json.load(json)
75
+ yield(LogStash::Event.new(o)) if o
76
+ rescue LogStash::Json::ParserError => e
77
+ @logger.warn("JSON parse error, original data now in message field", :error => e, :data => json)
78
+ yield LogStash::Event.new("message" => json, "tags" => ["_jsonparsefailure"])
79
+ end
80
+
81
+ # keep compatibility with all v2.x distributions. only in 2.3 will the Event#from_json method be introduced
82
+ # and we need to keep compatibility for all v2 releases.
83
+ alias_method :parse, LogStash::Event.respond_to?(:from_json) ? :from_json_parse : :legacy_parse
84
+ end
@@ -0,0 +1,29 @@
1
+ Gem::Specification.new do |s|
2
+
3
+ s.name = 'logstash-codec-json_stream'
4
+ s.version = '0.0.1'
5
+ s.licenses = ['Apache License (2.0)']
6
+ s.summary = "Reads and writes non JSON Streams"
7
+ s.description = "This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program"
8
+ s.authors = ["Christian Herweg"]
9
+ s.email = 'christian.herweg@gmail.com'
10
+ s.homepage = "https://github.com/cherweg/logstash-codec-json_stream"
11
+ s.require_paths = ["lib"]
12
+
13
+ # Files
14
+ s.files = Dir["lib/**/*","spec/**/*","*.gemspec","*.md","CONTRIBUTORS","Gemfile","LICENSE","NOTICE.TXT", "vendor/jar-dependencies/**/*.jar", "vendor/jar-dependencies/**/*.rb", "VERSION", "docs/**/*"]
15
+
16
+ # Tests
17
+ s.test_files = s.files.grep(%r{^(test|spec|features)/})
18
+
19
+ # Special flag to let us know this is actually a logstash plugin
20
+ s.metadata = { "logstash_plugin" => "true", "logstash_group" => "codec" }
21
+
22
+ # Gem dependencies
23
+ s.add_runtime_dependency "logstash-core-plugin-api", ">= 1.60", "<= 2.99"
24
+
25
+ s.add_runtime_dependency 'logstash-codec-line', '>= 2.1.0'
26
+
27
+ s.add_development_dependency 'logstash-devutils'
28
+ end
29
+
@@ -0,0 +1,237 @@
1
+ # encoding: utf-8
2
+ require "logstash/devutils/rspec/spec_helper"
3
+ require "logstash/codecs/json_lines"
4
+ require "logstash/event"
5
+ require "logstash/json"
6
+ require "insist"
7
+
8
+ describe LogStash::Codecs::JSONLines do
9
+
10
+ let(:codec_options) { {} }
11
+
12
+ shared_examples :codec do
13
+
14
+ context "#decode" do
15
+ it "should return an event from json data" do
16
+ data = {"foo" => "bar", "baz" => {"bah" => ["a","b","c"]}}
17
+ subject.decode(LogStash::Json.dump(data) + "\n") do |event|
18
+ insist { event.is_a? LogStash::Event }
19
+ insist { event.get("foo") } == data["foo"]
20
+ insist { event.get("baz") } == data["baz"]
21
+ insist { event.get("bah") } == data["bah"]
22
+ end
23
+ end
24
+
25
+ it "should return an event from json data when a newline is recieved" do
26
+ data = {"foo" => "bar", "baz" => {"bah" => ["a","b","c"]}}
27
+ subject.decode(LogStash::Json.dump(data)) do |event|
28
+ insist {false}
29
+ end
30
+ subject.decode("\n") do |event|
31
+ insist { event.is_a? LogStash::Event }
32
+ insist { event.get("foo") } == data["foo"]
33
+ insist { event.get("baz") } == data["baz"]
34
+ insist { event.get("bah") } == data["bah"]
35
+ end
36
+ end
37
+
38
+ context "when using custom delimiter" do
39
+ let(:delimiter) { "|" }
40
+ let(:line) { "{\"hey\":1}|{\"hey\":2}|{\"hey\":3}|" }
41
+ let(:codec_options) { { "delimiter" => delimiter } }
42
+
43
+ it "should decode multiple lines separated by the delimiter" do
44
+ result = []
45
+ subject.decode(line) { |event| result << event }
46
+ expect(result.size).to eq(3)
47
+ expect(result[0].get("hey")).to eq(1)
48
+ expect(result[1].get("hey")).to eq(2)
49
+ expect(result[2].get("hey")).to eq(3)
50
+ end
51
+ end
52
+
53
+ context "processing plain text" do
54
+ it "falls back to plain text" do
55
+ decoded = false
56
+ subject.decode("something that isn't json\n") do |event|
57
+ decoded = true
58
+ insist { event.is_a?(LogStash::Event) }
59
+ insist { event.get("message") } == "something that isn't json"
60
+ insist { event.get("tags") }.include?("_jsonparsefailure")
61
+ end
62
+ insist { decoded } == true
63
+ end
64
+ end
65
+
66
+ context "processing weird binary blobs" do
67
+ it "falls back to plain text and doesn't crash (LOGSTASH-1595)" do
68
+ decoded = false
69
+ blob = (128..255).to_a.pack("C*").force_encoding("ASCII-8BIT")
70
+ subject.decode(blob)
71
+ subject.decode("\n") do |event|
72
+ decoded = true
73
+ insist { event.is_a?(LogStash::Event) }
74
+ insist { event.get("message").encoding.to_s } == "UTF-8"
75
+ end
76
+ insist { decoded } == true
77
+ end
78
+ end
79
+
80
+ context "when json could not be parsed" do
81
+ let(:message) { "random_message\n" }
82
+
83
+ it "add the failure tag" do
84
+ subject.decode(message) do |event|
85
+ expect(event).to include "tags"
86
+ end
87
+ end
88
+
89
+ it "uses an array to store the tags" do
90
+ subject.decode(message) do |event|
91
+ expect(event.get('tags')).to be_a Array
92
+ end
93
+ end
94
+
95
+ it "add a json parser failure tag" do
96
+ subject.decode(message) do |event|
97
+ expect(event.get('tags')).to include "_jsonparsefailure"
98
+ end
99
+ end
100
+ end
101
+
102
+ context "blank lines" do
103
+ let(:collector) { Array.new }
104
+
105
+ it "should ignore bare blanks" do
106
+ subject.decode("\n\n") do |event|
107
+ collector.push(event)
108
+ end
109
+ expect(collector.size).to eq(0)
110
+ end
111
+
112
+ it "should ignore in between blank lines" do
113
+ subject.decode("\n{\"a\":1}\n\n{\"b\":2}\n\n") do |event|
114
+ collector.push(event)
115
+ end
116
+ expect(collector.size).to eq(2)
117
+ end
118
+ end
119
+
120
+ end
121
+
122
+ context "#encode" do
123
+ let(:data) { { LogStash::Event::TIMESTAMP => "2015-12-07T11:37:00.000Z", "foo" => "bar", "baz" => {"bah" => ["a","b","c"]}} }
124
+ let(:event) { LogStash::Event.new(data) }
125
+
126
+ it "should return json data" do
127
+ got_event = false
128
+ subject.on_event do |e, d|
129
+ insist { d } == "#{LogStash::Event.new(data).to_json}\n"
130
+ insist { LogStash::Json.load(d)["foo"] } == data["foo"]
131
+ insist { LogStash::Json.load(d)["baz"] } == data["baz"]
132
+ insist { LogStash::Json.load(d)["bah"] } == data["bah"]
133
+ got_event = true
134
+ end
135
+ subject.encode(event)
136
+ insist { got_event }
137
+ end
138
+
139
+ context "when using custom delimiter" do
140
+ let(:delimiter) { "|" }
141
+ let(:codec_options) { { "delimiter" => delimiter } }
142
+
143
+ it "should decode multiple lines separated by the delimiter" do
144
+ subject.on_event do |e, d|
145
+ insist { d } == "#{LogStash::Event.new(data).to_json}#{delimiter}"
146
+ end
147
+ subject.encode(event)
148
+ end
149
+ end
150
+ end
151
+
152
+ context 'reading from a simulated multiline json file without last newline' do
153
+ let(:input) do
154
+ %{{"field": "value1"}
155
+ {"field": "value2"}}
156
+ end
157
+
158
+ let(:collector) { Array.new }
159
+
160
+ it 'should generate one event' do
161
+ subject.decode(input) do |event|
162
+ collector.push(event)
163
+ end
164
+ expect(collector.size).to eq(1)
165
+ expect(collector.first.get('field')).to eq('value1')
166
+ end
167
+ end
168
+
169
+ context 'reading from a simulated multiline json file with last newline' do
170
+ let(:input) do
171
+ %{{"field": "value1"}
172
+ {"field": "value2"}
173
+ }
174
+ end
175
+
176
+ let(:collector) { Array.new }
177
+
178
+ it 'should generate two events' do
179
+ subject.decode(input) do |event|
180
+ collector.push(event)
181
+ end
182
+ expect(collector.size).to eq(2)
183
+ expect(collector.first.get('field')).to eq('value1')
184
+ expect(collector.last.get('field')).to eq('value2')
185
+ end
186
+ end
187
+
188
+ end
189
+
190
+ context "forcing legacy parsing" do
191
+ it_behaves_like :codec do
192
+ subject do
193
+ # register method is called in the constructor
194
+ LogStash::Codecs::JSONLines.new(codec_options)
195
+ end
196
+
197
+ before(:each) do
198
+ # stub codec parse method to force use of the legacy parser.
199
+ # this is very implementation specific but I am not sure how
200
+ # this can be tested otherwise.
201
+ allow(subject).to receive(:parse) do |line, &block|
202
+ subject.send(:legacy_parse, line, &block)
203
+ end
204
+ end
205
+ end
206
+ end
207
+
208
+ context "default parser choice" do
209
+ # here we cannot force the use of the Event#from_json since if this test is run in the
210
+ # legacy context (no Java Event) it will fail but if in the new context, it will be picked up.
211
+ it_behaves_like :codec do
212
+ subject do
213
+ # register method is called in the constructor
214
+ LogStash::Codecs::JSONLines.new(codec_options)
215
+ end
216
+ end
217
+
218
+ context "flush" do
219
+ subject do
220
+ LogStash::Codecs::JSONLines.new(codec_options)
221
+ end
222
+
223
+ let(:input) { "{\"foo\":\"bar\"}" }
224
+
225
+ it "should flush buffered data'" do
226
+ result = []
227
+ subject.decode(input) { |e| result << e }
228
+ expect(result.size).to eq(0)
229
+
230
+ subject.flush { |e| result << e }
231
+ expect(result.size).to eq(1)
232
+
233
+ expect(result[0].get("foo")).to eq("bar")
234
+ end
235
+ end
236
+ end
237
+ end
metadata ADDED
@@ -0,0 +1,104 @@
1
+ --- !ruby/object:Gem::Specification
2
+ name: logstash-codec-json_stream
3
+ version: !ruby/object:Gem::Version
4
+ version: 0.0.1
5
+ platform: ruby
6
+ authors:
7
+ - Christian Herweg
8
+ autorequire:
9
+ bindir: bin
10
+ cert_chain: []
11
+ date: 2018-04-23 00:00:00.000000000 Z
12
+ dependencies:
13
+ - !ruby/object:Gem::Dependency
14
+ requirement: !ruby/object:Gem::Requirement
15
+ requirements:
16
+ - - '>='
17
+ - !ruby/object:Gem::Version
18
+ version: '1.60'
19
+ - - <=
20
+ - !ruby/object:Gem::Version
21
+ version: '2.99'
22
+ name: logstash-core-plugin-api
23
+ prerelease: false
24
+ type: :runtime
25
+ version_requirements: !ruby/object:Gem::Requirement
26
+ requirements:
27
+ - - '>='
28
+ - !ruby/object:Gem::Version
29
+ version: '1.60'
30
+ - - <=
31
+ - !ruby/object:Gem::Version
32
+ version: '2.99'
33
+ - !ruby/object:Gem::Dependency
34
+ requirement: !ruby/object:Gem::Requirement
35
+ requirements:
36
+ - - '>='
37
+ - !ruby/object:Gem::Version
38
+ version: 2.1.0
39
+ name: logstash-codec-line
40
+ prerelease: false
41
+ type: :runtime
42
+ version_requirements: !ruby/object:Gem::Requirement
43
+ requirements:
44
+ - - '>='
45
+ - !ruby/object:Gem::Version
46
+ version: 2.1.0
47
+ - !ruby/object:Gem::Dependency
48
+ requirement: !ruby/object:Gem::Requirement
49
+ requirements:
50
+ - - '>='
51
+ - !ruby/object:Gem::Version
52
+ version: '0'
53
+ name: logstash-devutils
54
+ prerelease: false
55
+ type: :development
56
+ version_requirements: !ruby/object:Gem::Requirement
57
+ requirements:
58
+ - - '>='
59
+ - !ruby/object:Gem::Version
60
+ version: '0'
61
+ description: This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program
62
+ email: christian.herweg@gmail.com
63
+ executables: []
64
+ extensions: []
65
+ extra_rdoc_files: []
66
+ files:
67
+ - CHANGELOG.md
68
+ - CONTRIBUTORS
69
+ - Gemfile
70
+ - LICENSE
71
+ - NOTICE.TXT
72
+ - README.md
73
+ - docs/index.asciidoc
74
+ - lib/logstash/codecs/json_stream.rb
75
+ - logstash-codec-json_stream.gemspec
76
+ - spec/codecs/json_lines_spec.rb
77
+ homepage: https://github.com/cherweg/logstash-codec-json_stream
78
+ licenses:
79
+ - Apache License (2.0)
80
+ metadata:
81
+ logstash_plugin: 'true'
82
+ logstash_group: codec
83
+ post_install_message:
84
+ rdoc_options: []
85
+ require_paths:
86
+ - lib
87
+ required_ruby_version: !ruby/object:Gem::Requirement
88
+ requirements:
89
+ - - '>='
90
+ - !ruby/object:Gem::Version
91
+ version: '0'
92
+ required_rubygems_version: !ruby/object:Gem::Requirement
93
+ requirements:
94
+ - - '>='
95
+ - !ruby/object:Gem::Version
96
+ version: '0'
97
+ requirements: []
98
+ rubyforge_project:
99
+ rubygems_version: 2.4.5
100
+ signing_key:
101
+ specification_version: 4
102
+ summary: Reads and writes non JSON Streams
103
+ test_files:
104
+ - spec/codecs/json_lines_spec.rb