fluent-plugin-jsonish 1.0.2 → 2.0.1

data/README.md

@@ -1,5 +1,8 @@
 # jsonish, nginx_jsonish, nodejs_bunyan, and logstash parser plugins for Fluentd
 
+## Notes
+Some of the functionality implemented by these plugins seems to now be availabe in the standard JSON parser for Fluentd 1.0.  I haven't fully checked how usable it may be for particular purposes, but these plugins may end up being abandoned if it turns out that they are no longer needed.  However, I have updated them so that they do work with Fluentd 1.0.  I haven't checked to ensure that the changes made for Fluentd are back-compatible with previous versions: the 2.x.x versions of this plugin now require Fluentd 1.0.
+
 ## Overview
 The jsonish [parser plugin](http://docs.fluentd.org/articles/parser-plugin-overview) for fluentd.  It subclasses the JSONParser to allow for modifications to be made to input test before it is deserialized.  It subclasses the TimeParser to allow time format specifications using Ruby Time class method names instead of strftime format strings -- in particular, the iso8601 method.
 :
@@ -22,14 +25,16 @@ gem install fluent-plugin-jsonish
 
 ```
 <source>
-  type [tail|tcp|udp|syslog|http] # any input type which takes a format
-  format jsonish
-  maps <mappings for text processing prior to the JSON parse>
-  null_pattern <the pattern used for nulls in the text>
-  null_maps <how the patterns specified by null_pattern should be replaced>
-  message_key <key to use for setting the 'message' key in the record>
-  add_full_message <whether to the record 'full_message' key to the raw input>
-  move_keys <hash in JSON format, defaults to {}>
+  @type [tail|tcp|udp|syslog|http] # any input type which takes a format
+  <parse>
+    @type jsonish
+    maps <mappings for text processing prior to the JSON parse>
+    null_pattern <the pattern used for nulls in the text>
+    null_maps <how the patterns specified by null_pattern should be replaced>
+    message_key <key to use for setting the 'message' key in the record>
+    add_full_message <whether to the record 'full_message' key to the raw input>
+    move_keys <hash in JSON format, defaults to {}>
+  </parse>
 </source>
 ```
 
@@ -50,12 +55,14 @@ As an example, the nginx_jsonish subclasses the jsonish parser which really only
 
 ```
 <source>
-  type tail
-  format jsonish
-  path <nginx access log file>
-  maps ([ "slashes", "nulls" ] automatically prepended to anything set here)
-  message_key (set to "request", by default)
-  [ standard parser or jsonish arguments ]
+  @type tail
+  <parse>
+    @type jsonish
+    path <nginx access log file>
+    maps ([ "slashes", "nulls" ] automatically prepended to anything set here)
+    message_key (set to "request", by default)
+    [ standard parser or jsonish arguments ]
+  </parse>
 </source>
 ```
 
@@ -80,10 +87,33 @@ The fluentd parser configuration for this input is straight-forward:
 
 ```
 <source>
-  type tail
-  format nginx_jsonish
-  path <nginx access log file>
-  [ standard parser or jsonish arguments ]
+  @type tail
+  <parse>
+    @type nginx_jsonish
+    path <nginx access log file>
+    [ standard parser or jsonish arguments ]
+  </parse>
+</source>
+```
+
+With Fluentd 1.0, timestamps with millisecond precision are available.  They are available for use in a nginx log_format specification, but not in an ISo-8601 format -- only by using the "$msec" variable, which is horribly name (it's actually a Unix epoch in seconds, with millisecond precision).  However, there is something strange going on with the time_type "float" where it seems to insist the value for the time_key be a string.  I haven't been able to get that to, but the following does work for nginx access logs with millisecond precision timestamps.
+
+```
+log_format extended  "{ \"time\": \"$msec\", \"proxy_http_x_forwarded_for\": \"$proxy_add_x_forwarded_for\", \"proxy_x_forwarded_host\": \"$host\", \"proxy_x_forwarded_proto\": \"$scheme\", \"proxy_host\": \"$proxy_host\", \"remote_addr\": \"$remote_addr\", \"remote_port\": \"$remote_port\", \"request\": \"$request\", \"request_method\": \"$request_method\", \"request_uri\": \"$request_uri\", \"request_protocol\": \"$server_protocol\", \"http_accept\": \"$http_accept\", \"http_accept_encoding\": \"$http_accept_encoding\", \"http_accept_language\": \"$http_accept_language\", \"http_connection\": \"$http_connection\", \"sent_http_connection\": \"$sent_http_connection\", \"http_host\": \"$http_host\", \"http_user_agent\": \"$http_user_agent\", \"http_x_forwarded_for\": \"$http_x_forwarded_for\", \"body_bytes_sent\": $body_bytes_sent, \"connection_requests\": $connection_requests, \"proxy_internal_body_length\": $proxy_internal_body_length, \"request_length\": $request_length, \"request_time\": $request_time, \"status\": $status, \"upstream_response_time\": [$upstream_response_time], \"upstream_response_length\": [$upstream_response_length], \"upstream_status\": [$upstream_status], \"gzip_ratio\": $gzip_ratio }";
+```
+
+```
+<source>
+  @type tail
+  <parse>
+    @type nginx_jsonish
+    time_type string
+    time_format %s.%L
+    path <nginx access log file>
+    maps ([ "slashes", "nulls" ] automatically prepended to anything set here)
+    message_key (set to "request", by default)
+    [ standard parser or jsonish arguments ]
+  </parse>
 </source>
 ```
 
@@ -94,10 +124,12 @@ The fluentd parser configuration for this input is straight-forward:
 
 ```
 <source>
-  type tail
-  format nodejs_bunyan
-  path <application log file name>
-  [ standard parser or jsonish arguments ]
+  @type tail
+  <parse>
+    @type nodejs_bunyan
+    path <application log file name>
+    [ standard parser or jsonish arguments ]
+  </parse>
 </source>
 ```
 
@@ -108,9 +140,11 @@ The fluentd parser configuration for this input is straight-forward:
 
 ```
 <source>
-  type tail
-  format logstash
-  path <application log file name>
-  [ standard parser or jsonish arguments ]
+  @type tail
+  <parse>
+    @type logstash
+    path <application log file name>
+    [ standard parser or jsonish arguments ]
+  </parse>
 </source>
 ```

data/fluent-plugin-jsonish.gemspec

@@ -10,7 +10,7 @@ Gem::Specification.new do |gem|
   gem.summary       = %q{Input parser for records which require minor text processing before they can be parsed as JSON.  Also allows names of standard Time parser methods to be passed as time_format arguments and sets a reasonable default (iso8601).}
   gem.homepage      = 'https://github.com/bodhi-space/fluent-plugin-jsonish'
   gem.license       = 'Apache-2.0'
-  gem.add_runtime_dependency 'fluentd', '>= 0.10.0'
+  gem.add_runtime_dependency 'fluentd', '>= 0.14.0'
   gem.files         = `git ls-files`.split("\n")
   gem.executables   = gem.files.grep(%r{^bin/}) { |f| File.basename(f) }
   gem.test_files    = gem.files.grep(%r{^(test|spec|features)/})

data/lib/fluent/plugin/parser_jsonish.rb

@@ -14,20 +14,13 @@ module Fluent
       # * '%y-%m-%dT%H:%M:%S%Z' (no fractional seconds)
       # * '%y-%m-%dT%H:%M:%S.%L%Z' (fractional seconds)
 
-      def initialize(time_format)
-
-        if not time_format.nil? and time_format.empty?
-          # Set a reasonable default.
-          time_format = 'iso8601'
-        end
-
-        if not time_format.nil? and not /%/.match(time_format)
-          super(nil)
-
-          @parser = Proc.new { |v| Time.method(time_format).call(v) }
+      def initialize(format)
 
+        if not /%/.match(format)
+          super()
+          @parse = ->(v){ Fluent::EventTime.from_time(Time.method(format).call(v)) }
         else
-          super(time_format)
+          super(format)
         end
       end
     end
@@ -54,27 +47,39 @@ module Fluent
 
       def configure(conf)
 
-        if conf['time_format']
-          # Remove the time_format key before the super call
-          # so the it does as little as possible as possible
-          # (ie. less that we'll have to override).
-          tmp_time_format = conf['time_format']
-          conf.delete('time_format')
+        if conf['time_format'] and not conf['time_format'].nil?
+          # Remove the time_format key before the super call so
+          # the it does as little as possible as possible (ie.
+          # less that we'll have to override).
+          tmp_time_format = conf.delete('time_format')
+          # This has to be set to string when time_format is set.
+          # Deleting it without deleting time_type will leave an
+          # invalid configuration.
+          tmp_time_type = conf.delete('time_type')
         end
 
         super(conf)
 
-        @time_parser = StdFormatTimeParser.new(tmp_time_format)
+        # Overwrite the time parser unless the time_type is set
+        # to something other than string.
+        if not tmp_time_type.nil? and tmp_time_type == 'string'
+            @time_parser = StdFormatTimeParser.new(tmp_time_format)
+            # If these values are not restored, fluent will not
+            # show the value in trace/debug output.
+            conf['time_type'] = tmp_time_type
+            @time_type = tmp_time_type
+            conf['time_format'] = tmp_time_format
+            @time_format = tmp_time_format
+        elsif tmp_time_format.nil?
+            # The v1.0 time parser has a way to use the Time class
+            # method iso8601, but I would still to be able to access
+            # any available Time methods in a generic way.
+            @time_parser = StdFormatTimeParser.new('iso8601')
+            @time_type = 'string'
+            @time_format = 'iso8601'
+        end
         @mutex = Mutex.new
 
-        # This may look stupid (it actually is really stupid),
-        # but this *must* be set back to a non-null string
-        # prior to the return, since the superclass parser
-        # method checks for this and them implements its
-        # own ad hoc parser, in-line.  This is a necessary
-        # kludge to bypass a more egregious kludge.
-        @time_format = 'ignore_me'
-
         @transforms = []
 
         @maps.each do |elem|

metadata

@@ -1,13 +1,13 @@
 --- !ruby/object:Gem::Specification 
 name: fluent-plugin-jsonish
 version: !ruby/object:Gem::Version 
-  hash: 19
+  hash: 13
   prerelease: 
   segments: 
-  - 1
-  - 0
   - 2
-  version: 1.0.2
+  - 0
+  - 1
+  version: 2.0.1
 platform: ruby
 authors: 
 - Alex Yamauchi
@@ -41,7 +41,7 @@ cert_chain:
   2Zk648Ep9HVPKmwoVuB75+xEQw==
   -----END CERTIFICATE-----
 
-date: 2017-08-17 00:00:00 Z
+date: 2018-07-11 00:00:00 Z
 dependencies: 
 - !ruby/object:Gem::Dependency 
   name: fluentd
@@ -51,12 +51,12 @@ dependencies:
     requirements: 
     - - ">="
       - !ruby/object:Gem::Version 
-        hash: 55
+        hash: 39
         segments: 
         - 0
-        - 10
+        - 14
         - 0
-        version: 0.10.0
+        version: 0.14.0
   type: :runtime
   version_requirements: *id001
 description: Input parser for records which require minor text processing before they can be parsed as JSON