logstash-filter-date 3.1.15 → 3.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 0a94493615283f8f0f4b9c38205a13b616f9e5fe3627d5ae4d7f364f2cfae927
-  data.tar.gz: a151b9b52d5358050a1b6e322bfbe85e16f7f58ed95c003d2128b04c30c91676
+  metadata.gz: 8ea85b9a739ee3c2f7074203a7a30d7db4d37615541b126b5372e2d118260d4b
+  data.tar.gz: 106e2c9d67c60fa1ce5eea4ca4e27654aa416a264ed704e06cbdafc5e054b1b4
 SHA512:
-  metadata.gz: 79d841f0d1e42509d03e3791256d597041d1a9b1d258a176ef6833cdcdfdc32640cd33015bebd66ff7e8c0678201813118a2f15127174bc573e93a1713362fca
-  data.tar.gz: cce2db7171e539cfb98b2a897269c4f2269f07cba0cf2a61e2687e90c71a693afb5d9456b55e3f5e271af6806c836b2c53c6a21870efe328d1e5608470c11706
+  metadata.gz: 8051731609e613b09430d809b077ffcf48e3151fcc5ab5affaed09dd8da3f1a8dc7b4c39a90468e8f32599609881900abd9f5c6f4dc3f94b1f891705fd7ccd7b
+  data.tar.gz: e1b8552ede171c1ea0dc77398395c9fdaeb1b79ece9e8dabe7d61c58a94c23f44d712f91950a0adfbb8c1bafce64c4bf7aafea8312c3d2b41e53f726639030a4

data/CHANGELOG.md

@@ -1,3 +1,17 @@
+## 3.2.0
+  - Add `precision` setting to support nanosecond precision timestamps [#165](https://github.com/logstash-plugins/logstash-filter-date/pull/165)
+    - `ms` (default): timestamps are stored with millisecond precision
+      - keeps the same behavior as before for backward compatibility
+      - fractional seconds are truncated to 3 digits
+      - custom parsing formats use `joda-time` library
+    - `ns`: timestamps are stored with nanosecond precision
+      - fractional seconds support up to 9 digits
+      - custom parsing formats use `java.time`
+  - `ISO8601` now accepts up to 9 fractional-second digits
+
+## 3.1.16
+  - Re-packaging the plugin [#163](https://github.com/logstash-plugins/logstash-filter-date/pull/163)
+
 ## 3.1.15
   - Build: review build to be more reliable/portable [#139](https://github.com/logstash-plugins/logstash-filter-date/pull/139)
     * cleaned up Java dependencies 

data/docs/index.asciidoc

@@ -20,23 +20,11 @@ include::{include_path}/plugin_header.asciidoc[]
 
 ==== Description
 
-The date filter is used for parsing dates from fields, and then using that
-date or timestamp as the logstash timestamp for the event.
+The date filter is used for parsing dates from fields, and then using that date or timestamp as the logstash timestamp for the event.
+Timestamp is stored with millisecond precision. Set `precision => "ns"` to preserve nanoseconds.
 
-For example, syslog events usually have timestamps like this:
-[source,ruby]
-    "Apr 17 09:32:01"
-
-You would use the date format `MMM dd HH:mm:ss` to parse this.
-
-The date filter is especially important for sorting events and for
-backfilling old data. If you don't get the date correct in your
-event, then searching for them later will likely sort out of order.
-
-In the absence of this filter, logstash will choose a timestamp based on the
-first time it sees the event (at input time), if the timestamp is not already
-set in the event. For example, with file input, the timestamp is set to the
-time of each read.
+Custom parsing formats use the JVM’s default locale and time zone.
+To override them, configure the `locale` and `timezone` settings.
 
 [id="plugins-{type}s-{plugin}-options"]
 ==== Date Filter Configuration Options
@@ -48,6 +36,7 @@ This plugin supports the following configuration options plus the <<plugins-{typ
 |Setting |Input type|Required
 | <<plugins-{type}s-{plugin}-locale>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-match>> |<<array,array>>|No
+| <<plugins-{type}s-{plugin}-precision>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-tag_on_failure>> |<<array,array>>|No
 | <<plugins-{type}s-{plugin}-target>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-timezone>> |<<string,string>>|No
@@ -89,11 +78,14 @@ If your time field has multiple possible formats, you can do this:
 
 The above will match a syslog (rfc3164) or `iso8601` timestamp.
 
-There are a few special exceptions. The following format literals exist
+By default, the custom formats use the deprecated https://www.joda.org/joda-time/key_format.html[joda-time] library for parsing and timestamp is stored with millisecond precision.
+If you set `precision => "ns"`, parsing is performed using https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/time/format/DateTimeFormatter.html[java.time], and timestamps are stored with nanosecond precision.
+Both libraries use similar syntax, but there are some differences in supported features. For example, `java.time` supports time zone IDs with `VV` while `joda-time` uses `ZZZ`.
+
+Besides custom formats, the following format literals exist
 to help you save time and ensure correctness of date parsing.
 
-* `ISO8601` - should parse any valid ISO8601 timestamp, such as
-  `2011-04-19T03:44:01.103Z`
+* `ISO8601` - should parse any valid ISO8601 timestamp, such as `2011-04-19T03:44:01.123456789Z`
 * `UNIX` - will parse *float or int* value expressing unix time in seconds since epoch like 1326149001.132 as well as 1326149001
 * `UNIX_MS` - will parse **int** value expressing unix time in milliseconds since epoch like 1366125117000
 * `TAI64N` - will parse tai64n time values
@@ -147,10 +139,14 @@ s:: seconds of the minute (60 seconds per minute)
   ss::: two-digit seconds, zero-padded if needed. Example: `00`.
 
 S:: fraction of a second
-  *Maximum precision is milliseconds (`SSS`). Beyond that, zeroes are appended.*
   S::: tenths of a second. Example:  `0` for a subsecond value `012`
   SS::: hundredths of a second. Example:  `01` for a subsecond value `01`
-  SSS::: thousandths of a second. Example:  `012` for a subsecond value `012`
+  SSS::: milliseconds. Example:  `012` for a subsecond value `012`
+  SSSSSS::: microseconds. Example:  `012345`
+  SSSSSSSSS::: nanoseconds. Example:  `012345678`
+
+V:: time-zone ID
+  VV::: time zone ID. Example: `America/Los_Angeles`. Note: This is only supported by `java.time` parsing.
 
 Z:: time zone offset or identity
   Z::: Timezone offset structured as HHmm (hour and minutes offset from Zulu/UTC). Example: `-0700`.
@@ -174,10 +170,23 @@ E:: day of the week (text)
 For non-formatting syntax, you'll need to put single-quote characters around the value. For example, if you were parsing ISO8601 time, "2015-01-01T01:12:23" that little "T" isn't a valid time format, and you want to say "literally, a T", your format would be this: "yyyy-MM-dd'T'HH:mm:ss"
 
 Other less common date units, such as era (G), century \(C), am/pm (a), and # more, can be learned about on the
-http://www.joda.org/joda-time/key_format.html[joda-time documentation].
+https://www.joda.org/joda-time/key_format.html[joda-time documentation].
+
+[id="plugins-{type}s-{plugin}-precision"]
+===== `precision`
+
+  * Value type is <<string,string>>
+  * Valid values are `ms` and `ns`
+  * Default value is `ms`
+
+Controls the sub-second precision of the stored timestamp.
+
+`ms` stores data in millisecond precision. Custom pattern formats use https://www.joda.org/joda-time/key_format.html[joda-time] parsing rules. For example, `yyyy-MM-dd HH:mm:ss ZZZ`.
+
+`ns` stores data in nanosecond precision. Custom pattern formats use https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/time/format/DateTimeFormatter.html[java.time] parsing rules. For example, `yyyy-MM-dd HH:mm:ss VV`.
 
 [id="plugins-{type}s-{plugin}-tag_on_failure"]
-===== `tag_on_failure` 
+===== `tag_on_failure`
 
   * Value type is <<array,array>>
   * Default value is `["_dateparsefailure"]`

data/lib/logstash/filters/date.rb

@@ -57,9 +57,8 @@ class LogStash::Filters::Date < LogStash::Filters::Base
   # There are a few special exceptions. The following format literals exist
   # to help you save time and ensure correctness of date parsing.
   #
-  # * `ISO8601` - should parse any valid ISO8601 timestamp, such as
-  #   `2011-04-19T03:44:01.103Z`
-  # * `UNIX` - will parse *float or int* value expressing unix time in seconds since epoch like 1326149001.132 as well as 1326149001
+  # * `ISO8601` - should parse any valid ISO8601 timestamp with up to 9 fractional-second digits
+  # * `UNIX` - will parse *float or int* value expressing unix time in seconds since epoch like 1326149001.123456789 as well as 1326149001
   # * `UNIX_MS` - will parse **int** value expressing unix time in milliseconds since epoch like 1366125117000
   # * `TAI64N` - will parse tai64n time values
   #
@@ -112,10 +111,11 @@ class LogStash::Filters::Date < LogStash::Filters::Base
   #   ss::: two-digit seconds, zero-padded if needed. Example: `00`.
   #
   # S:: fraction of a second
-  #   *Maximum precision is milliseconds (`SSS`). Beyond that, zeroes are appended.*
   #   S::: tenths of a second. Example:  `0` for a subsecond value `012`
   #   SS::: hundredths of a second. Example:  `01` for a subsecond value `01`
-  #   SSS::: thousandths of a second. Example:  `012` for a subsecond value `012`
+  #   SSS::: milliseconds. Example:  `012` for a subsecond value `012`
+  #   SSSSSS::: microseconds.
+  #   SSSSSSSSS::: nanoseconds.
   #
   # Z:: time zone offset or identity
   #   Z::: Timezone offset structured as HHmm (hour and minutes offset from Zulu/UTC). Example: `-0700`.
@@ -150,6 +150,14 @@ class LogStash::Filters::Date < LogStash::Filters::Base
   # successful match
   config :tag_on_failure, :validate => :array, :default => ["_dateparsefailure"]
 
+  # Controls the sub-second precision of the stored timestamp.
+  #
+  # `"ms"` (default):: Stores in millisecond precision. Custom-pattern formats use Joda-Time parsing rules.
+  # `"ns"`:: Stores in nanosecond precision. Custom-pattern formats use java.time parsing rules.
+  config :precision, :validate => [Java::OrgLogstashFiltersParser::TimestampParserFactory::PRECISION_MS,
+                                   Java::OrgLogstashFiltersParser::TimestampParserFactory::PRECISION_NS],
+                          :default => Java::OrgLogstashFiltersParser::TimestampParserFactory::PRECISION_MS
+
   def register
     # nothing
   end
@@ -179,13 +187,13 @@ class LogStash::Filters::Date < LogStash::Filters::Base
       metric.increment(:failures)
     end
 
-    @datefilter = org.logstash.filters.DateFilter.new(source, @target, @tag_on_failure, success_block, failure_block)
+    @datefilter = org.logstash.filters.DateFilter.new(source, @target, @tag_on_failure, @precision, success_block, failure_block)
 
     @match[1..-1].map do |format|
       @datefilter.accept_filter_config(format, @locale, @timezone)
 
       # Offer a fallback parser such that if the default system Locale is non-english and that no locale is set,
-      # we should try to parse english if the first local parsing fails.:w
+      # we should try to parse english if the first local parsing fails.
       if !@locale && "en" != java.util.Locale.getDefault().getLanguage() && (format.include?("MMM") || format.include?("E"))
         @datefilter.accept_filter_config(format, "en-US", @timezone)
       end

data/logstash-filter-date.gemspec

@@ -1,7 +1,7 @@
 Gem::Specification.new do |s|
 
   s.name            = 'logstash-filter-date'
-  s.version         = '3.1.15'
+  s.version         = '3.2.0'
   s.licenses        = ['Apache License (2.0)']
   s.summary         = "Parses dates from fields to use as the Logstash timestamp for an event"
   s.description     = "This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program"

data/spec/filters/date_spec.rb

@@ -17,62 +17,6 @@ RUBY_ENGINE == "jruby" and describe LogStash::Filters::Date do
     end
   end
 
-  describe "parsing with ISO8601" do
-    config <<-CONFIG
-      filter {
-        date {
-          match => [ "mydate", "ISO8601" ]
-          locale => "en"
-          timezone => "UTC"
-        }
-      }
-    CONFIG
-
-    times = {
-      "2001-01-01T00:00:00-0800"         => "2001-01-01T08:00:00.000Z",
-      "1974-03-02T04:09:09-0800"         => "1974-03-02T12:09:09.000Z",
-      "2010-05-03T08:18:18+00:00"        => "2010-05-03T08:18:18.000Z",
-      "2004-07-04T12:27:27-00:00"        => "2004-07-04T12:27:27.000Z",
-      "2001-09-05T16:36:36+0000"         => "2001-09-05T16:36:36.000Z",
-      "2001-11-06T20:45:45-0000"         => "2001-11-06T20:45:45.000Z",
-      "2001-12-07T23:54:54Z"             => "2001-12-07T23:54:54.000Z",
-
-      # TODO: This test assumes PDT
-      #"2001-01-01T00:00:00.123"          => "2001-01-01T08:00:00.123Z",
-
-      "2010-05-03T08:18:18.123+00:00"    => "2010-05-03T08:18:18.123Z",
-      "2004-07-04T12:27:27.123-04:00"    => "2004-07-04T16:27:27.123Z",
-      "2001-09-05T16:36:36.123+0700"     => "2001-09-05T09:36:36.123Z",
-      "2001-11-06T20:45:45.123-0000"     => "2001-11-06T20:45:45.123Z",
-      "2001-12-07T23:54:54.123Z"         => "2001-12-07T23:54:54.123Z",
-      "2001-12-07T23:54:54,123Z"         => "2001-12-07T23:54:54.123Z",
-
-      #Almost ISO8601 support, with timezone
-
-      "2001-11-06 20:45:45.123-0000"     => "2001-11-06T20:45:45.123Z",
-      "2001-12-07 23:54:54.123Z"         => "2001-12-07T23:54:54.123Z",
-      "2001-12-07 23:54:54,123Z"         => "2001-12-07T23:54:54.123Z",
-
-      #Almost ISO8601 support, without timezone
-
-      "2001-11-06 20:45:45.123"     => "2001-11-06T20:45:45.123Z",
-      "2001-11-06 20:45:45,123"     => "2001-11-06T20:45:45.123Z",
-
-    }
-
-    times.each do |input, output|
-      sample("mydate" => input) do
-        begin
-          insist { subject.get("mydate") } == input
-          insist { subject.get("@timestamp").time } == Time.iso8601(output).utc
-        rescue
-          #require "pry"; binding.pry
-          raise
-        end
-      end
-    end # times.each
-  end
-
   describe "parsing with java SimpleDateFormat syntax" do
     config <<-CONFIG
       filter {
@@ -91,7 +35,7 @@ RUBY_ENGINE == "jruby" and describe LogStash::Filters::Date do
       "Nov 24 01:29:01 -0800" => "#{year}-11-24T09:29:01.000Z",
     }
     times.each do |input, output|
-      sample("mydate" => input) do
+      sample({"mydate" => input}) do
         insist { subject.get("mydate") } == input
         insist { subject.get("@timestamp").time } == Time.iso8601(output).utc
       end
@@ -119,55 +63,17 @@ RUBY_ENGINE == "jruby" and describe LogStash::Filters::Date do
       1478207457 => "2016-11-03T21:10:57.000Z"
     }
     times.each do |input, output|
-      sample("mydate" => input) do
+      sample({"mydate" => input}) do
         insist { subject.get("mydate") } == input
         insist { subject.get("@timestamp").time } == Time.iso8601(output).utc
       end
     end # times.each
 
     #Invalid value should not be evaluated to zero (String#to_i madness)
-    sample("mydate" => "%{bad_value}") do
-      insist { subject.get("mydate") } == "%{bad_value}"
-      insist { subject.get("@timestamp").time } != Time.iso8601("1970-01-01T00:00:00.000Z").utc
-    end
-  end
-
-  describe "parsing microsecond-precise times with UNIX (#213)" do
-    config <<-CONFIG
-      filter {
-        date {
-          match => [ "mydate", "UNIX" ]
-          locale => "en"
-        }
-      }
-    CONFIG
-
-    sample("mydate" => "1350414944.123456") do
-      # Joda time only supports milliseconds :\
-      insist { subject.timestamp.time } == Time.iso8601("2012-10-16T12:15:44.123-07:00").utc
-    end
-
-    #Support float values
-    sample("mydate" => 1350414944.123456) do
-      insist { subject.get("mydate") } == 1350414944.123456
-      insist { subject.get("@timestamp").time } == Time.iso8601("2012-10-16T12:15:44.123-07:00").utc
-    end
-
-    #Invalid value should not be evaluated to zero (String#to_i madness)
-    sample("mydate" => "%{bad_value}") do
+    sample({"mydate" => "%{bad_value}"}) do
       insist { subject.get("mydate") } == "%{bad_value}"
       insist { subject.get("@timestamp").time } != Time.iso8601("1970-01-01T00:00:00.000Z").utc
     end
-
-    # Regression test
-    # Support numeric values that come through the JSON parser. These numbers appear as BigDecimal
-    # instead of Float.
-    sample(LogStash::Json.load('{ "mydate": 1350414944.123456 }')) do
-      # its generally problematic to compare different Floating Point class implementations using equals
-      # because they can't always represent a value exactly.
-      insist { subject.get("mydate") } == BigDecimal.new("1350414944.123456")
-      insist { subject.get("@timestamp").time } == Time.iso8601("2012-10-16T12:15:44.123-07:00").utc
-    end
   end
 
   describe "parsing with UNIX_MS" do
@@ -191,7 +97,7 @@ RUBY_ENGINE == "jruby" and describe LogStash::Filters::Date do
       1000000000123 => "2001-09-09T01:46:40.123Z"
     }
     times.each do |input, output|
-      sample("mydate" => input) do
+      sample({"mydate" => input}) do
         insist { subject.get("mydate") } == input
         insist { subject.get("@timestamp").time } == Time.iso8601(output).utc
       end
@@ -223,7 +129,7 @@ RUBY_ENGINE == "jruby" and describe LogStash::Filters::Date do
       1478207457.456 => "2016-11-03T21:10:57.456Z",
     }
     times.each do |input, output|
-      sample("mydate" => input) do
+      sample({"mydate" => input}) do
         insist { subject.get("mydate") } == input
         insist { subject.get("@timestamp").time } == Time.iso8601(output).utc
       end
@@ -269,14 +175,7 @@ RUBY_ENGINE == "jruby" and describe LogStash::Filters::Date do
       }
     CONFIG
 
-    # Try without leading "@"
-    sample("t" => "4000000050d506482dbdf024") do
-      insist { subject.timestamp.time } == Time.iso8601("2012-12-22T01:00:46.767Z").utc
-    end
-
-    # Should still parse successfully if it's a full tai64n time (with leading
-    # '@')
-    sample("t" => "@4000000050d506482dbdf024") do
+    sample({"t" => "4000000050d506482dbdf024"}) do
       insist { subject.timestamp.time } == Time.iso8601("2012-12-22T01:00:46.767Z").utc
     end
   end
@@ -293,7 +192,7 @@ RUBY_ENGINE == "jruby" and describe LogStash::Filters::Date do
 
     time = "2001-09-09T01:46:40.000Z"
 
-    sample("mydate" => time) do
+    sample({"mydate" => time}) do
       insist { subject.get("mydate") } == time
       insist { subject.get("@timestamp").time } == Time.iso8601(time).utc
     end
@@ -309,7 +208,7 @@ RUBY_ENGINE == "jruby" and describe LogStash::Filters::Date do
       }
     CONFIG
 
-    sample("data" => { "deep" => "2013-01-01T00:00:00.000Z" }) do
+    sample({"data" => { "deep" => "2013-01-01T00:00:00.000Z" }}) do
       insist { subject.get("@timestamp").time } == Time.iso8601("2013-01-01T00:00:00.000Z").utc
     end
   end
@@ -324,7 +223,7 @@ RUBY_ENGINE == "jruby" and describe LogStash::Filters::Date do
       }
     CONFIG
 
-    sample("thedate" => "2013/Apr/21") do
+    sample({"thedate" => "2013/Apr/21"}) do
       expected = Time.iso8601("2013-04-21T00:00:00.000Z").utc
       expect(subject.get("@timestamp").time).not_to eq(expected)
     end
@@ -341,7 +240,7 @@ RUBY_ENGINE == "jruby" and describe LogStash::Filters::Date do
       }
     CONFIG
 
-    sample("thedate" => "2013/04/21") do
+    sample({"thedate" => "2013/04/21"}) do
       expected = Time.iso8601("2013-04-21T00:00:00.000Z").utc
       expect(subject.get("@timestamp").time).to eq(expected)
       insist { subject.get("tags") } == ["tagged"]
@@ -359,7 +258,7 @@ RUBY_ENGINE == "jruby" and describe LogStash::Filters::Date do
       }
     CONFIG
 
-    sample("thedate" => "2013/Apr/21") do
+    sample({"thedate" => "2013/Apr/21"}) do
       expected = Time.iso8601("2013-04-21T00:00:00.000Z").utc
       expect(subject.get("@timestamp").time).not_to eq(expected)
       reject { subject.get("tags") }.include? "tagged"
@@ -377,7 +276,7 @@ RUBY_ENGINE == "jruby" and describe LogStash::Filters::Date do
       }
     CONFIG
 
-    sample("thedate" => "2013/Apr/21") do
+    sample({"thedate" => "2013/Apr/21"}) do
       expected = Time.iso8601("2013-04-21T00:00:00.000Z").utc
       expect(subject.get("@timestamp").time).not_to eq(expected)
       insist { subject.get("tags") }.include? "date_failed"
@@ -401,7 +300,7 @@ RUBY_ENGINE == "jruby" and describe LogStash::Filters::Date do
       "2013 Jun 24 01:29:01" => "2013-06-24T08:29:01.000Z",
     }
     times.each do |input, output|
-      sample("mydate" => input) do
+      sample({"mydate" => input}) do
         insist { subject.get("mydate") } == input
         insist { subject.get("@timestamp").time } == Time.iso8601(output).utc
       end
@@ -425,7 +324,7 @@ RUBY_ENGINE == "jruby" and describe LogStash::Filters::Date do
       "2013 Jun 24 01:29:01" => "2013-06-24T08:29:01.000Z",
     }
     times.each do |input, output|
-      sample("mydate" => input, "mytz" => "America/Los_Angeles") do
+      sample({"mydate" => input, "mytz" => "America/Los_Angeles"}) do
         insist { subject.get("mydate") } == input
         insist { subject.get("@timestamp").time } == Time.iso8601(output).utc
       end
@@ -446,204 +345,21 @@ RUBY_ENGINE == "jruby" and describe LogStash::Filters::Date do
     require 'java'
 
     # Venezuela changed from -4:00 to -4:30 at 03:00 on Sun, 9 Dec 2007
-    sample("mydate" => "2007-12-09T01:00:00", "mytz" => "America/Caracas") do
+    sample({"mydate" => "2007-12-09T01:00:00", "mytz" => "America/Caracas"}) do
       expect(subject.get("mydate")).to eq("2007-12-09T01:00:00")
       expect(subject.get("@timestamp").time).to eq(Time.iso8601("2007-12-09T05:00:00.000Z").utc)
     end
-    sample("mydate" => "2007-12-09T10:00:00", "mytz" => "America/Caracas") do
+    sample({"mydate" => "2007-12-09T10:00:00", "mytz" => "America/Caracas"}) do
       expect(subject.get("mydate")).to eq("2007-12-09T10:00:00")
       expect(subject.get("@timestamp").time).to eq(Time.iso8601("2007-12-09T14:30:00.000Z").utc)
     end
     # Venezuela changed from -4:30 to -4:00 at 02:30 on Sunday, 1 May 2016
-    sample("mydate" => "2016-05-01T08:18:18.123", "mytz" => "America/Caracas") do
+    sample({"mydate" => "2016-05-01T08:18:18.123", "mytz" => "America/Caracas"}) do
       expect(subject.get("mydate")).to eq("2016-05-01T08:18:18.123")
       expect(subject.get("@timestamp").time).to eq(Time.iso8601("2016-05-01T12:18:18.123Z").utc)
     end
   end
 
-  describe "don't fail on next years DST switchover in CET" do
-    config <<-CONFIG
-      filter {
-        date {
-          match => [ "message", "yyyy MMM dd HH:mm:ss" ]
-          locale => "en"
-          timezone => "CET"
-        }
-      }
-    CONFIG
-
-    before(:each) do
-      logstash_time = Time.utc(2016,03,29,23,59,50)
-      allow(Time).to receive(:now).and_return(logstash_time)
-    end
-
-    sample "2016 Mar 26 02:00:37" do
-      insist { subject.get("tags") } != ["_dateparsefailure"]
-      expect(subject.get("@timestamp")).to be_a_logstash_timestamp_equivalent_to("2016-03-26T01:00:37.000Z")
-    end
-  end
-
-  context "Default year handling when parsing with timezone from event" do
-
-    describe "LOGSTASH-34 - Default year should be this year" do
-      config <<-CONFIG
-        filter {
-          date {
-            match => [ "message", "EEE MMM dd HH:mm:ss" ]
-            locale => "en"
-            timezone => "%{mytz}"
-          }
-        }
-      CONFIG
-
-      sample("message" => "Sun Jun 02 20:38:03", "mytz" => "UTC") do
-        insist { subject.get("@timestamp").year } == Time.now.year
-      end
-    end
-
-    describe "fill last year if december events arrive in january" do
-      config <<-CONFIG
-        filter {
-          date {
-            match => [ "message", "MMM dd HH:mm:ss" ]
-            locale => "en"
-            timezone => "%{mytz}"
-          }
-        }
-      CONFIG
-
-      before(:each) do
-        logstash_time = Time.utc(2014,1,1,00,30,50)
-        allow(Time).to receive(:now).and_return(logstash_time)
-        org.logstash.filters.parser.JodaParser.setDefaultClock { org.joda.time.DateTime.new(2014,1,1,00,30,50, org.joda.time.DateTimeZone::UTC ) }
-      end
-
-      sample("message" => "Dec 31 23:59:00", "mytz" => "UTC") do
-        insist { subject.get("@timestamp").year } == 2013
-      end
-    end
-
-    describe "fill next year if january events arrive in december" do
-      config <<-CONFIG
-        filter {
-          date {
-            match => [ "message", "MMM dd HH:mm:ss" ]
-            locale => "en"
-            timezone => "%{mytz}"
-          }
-        }
-      CONFIG
-
-      before(:each) do
-        logstash_time = Time.utc(2013,12,31,23,59,50)
-        allow(Time).to receive(:now).and_return(logstash_time)
-        org.logstash.filters.parser.JodaParser.setDefaultClock { org.joda.time.DateTime.new(2013,12,31,23,59,50, org.joda.time.DateTimeZone::UTC ) }
-      end
-
-      sample( "message" => "Jan 01 01:00:00", "mytz" => "UTC") do
-        insist { subject.get("@timestamp").year } == 2014
-      end
-    end
-
-    describe "do fail on 2016 DST switchover in CET" do
-      # This test tries to parse a time that doesn't exist. '02:00:01' is a time that doesn't exist
-      # because this DST switch goes from 01:59:59 to 03:00:00, skipping 2am entirely. The last Sunday of March in 2016 was 27th.
-      # (Guy Boertje) Fixed the GuessYear logic
-      # Joda has a default year for DateTimeFormat of 2000
-      # meaning that the initial time parsed was Monday 2000-03-27 02:00:01 and the last Sunday of March in 2000 was the 26th
-      # then by adding the year of 2016 creates an invalid time
-      # The parser default now takes the year from the DefaultClock in the JodaParser
-      config <<-CONFIG
-        filter {
-          date {
-            match => [ "message", "MMM dd HH:mm:ss" ]
-            locale => "en"
-            timezone => "CET"
-          }
-        }
-      CONFIG
-
-      before(:each) do
-        logstash_time = Time.utc(2016,03,29,23,59,50)
-        allow(Time).to receive(:now).and_return(logstash_time)
-        org.logstash.filters.parser.JodaParser.setDefaultClock { org.joda.time.DateTime.new(2016,03,29,23,59,50, org.joda.time.DateTimeZone::UTC ) }
-      end
-
-      sample "Mar 27 01:59:59" do
-        expect(subject.get("tags")).to be_nil
-        expect(subject.get("@timestamp")).to be_a_logstash_timestamp_equivalent_to("2016-03-27T00:59:59.000Z")
-      end
-
-      sample "Mar 27 02:00:01" do
-        expect(subject.get("tags")).to eq ["_dateparsefailure"]
-      end
-
-      sample "Mar 27 03:00:01" do
-        expect(subject.get("tags")).to be_nil
-        expect(subject.get("@timestamp")).to be_a_logstash_timestamp_equivalent_to("2016-03-27T01:00:01.000Z")
-      end
-    end
-  end
-
-  describe "LOGSTASH-34 - Default year should be this year" do
-    config <<-CONFIG
-      filter {
-        date {
-          match => [ "message", "EEE MMM dd HH:mm:ss" ]
-          locale => "en"
-        }
-      }
-    CONFIG
-
-    sample "Sun Jun 02 20:38:03" do
-      insist { subject.get("@timestamp").year } == Time.now.year
-    end
-  end
-
-  describe "fill last year if december events arrive in january" do
-    config <<-CONFIG
-      filter {
-        date {
-          match => [ "message", "MMM dd HH:mm:ss" ]
-          locale => "en"
-          timezone => "UTC"
-        }
-      }
-    CONFIG
-
-    before(:each) do
-      logstash_time = Time.utc(2014,1,1,00,30,50)
-      allow(Time).to receive(:now).and_return(logstash_time)
-      org.logstash.filters.parser.JodaParser.setDefaultClock { org.joda.time.DateTime.new(2014,1,1,00,30,50, org.joda.time.DateTimeZone::UTC ) }
-    end
-
-    sample "Dec 31 23:59:00" do
-      insist { subject.get("@timestamp").year } == 2013
-    end
-  end
-
-  describe "fill next year if january events arrive in december" do
-    config <<-CONFIG
-      filter {
-        date {
-          match => [ "message", "MMM dd HH:mm:ss" ]
-          locale => "en"
-          timezone => "UTC"
-        }
-      }
-    CONFIG
-
-    before(:each) do
-      logstash_time = Time.utc(2013,12,31,23,59,50)
-      allow(Time).to receive(:now).and_return(logstash_time)
-      org.logstash.filters.parser.JodaParser.setDefaultClock { org.joda.time.DateTime.new(2013,12,31,15,59,50, org.joda.time.DateTimeZone::UTC ) }
-    end
-
-    sample "Jan 01 01:00:00" do
-      insist { subject.get("@timestamp").year } == 2014
-    end
-  end
-
   describe "Supporting locale only" do
     config <<-CONFIG
       filter {
@@ -703,7 +419,7 @@ RUBY_ENGINE == "jruby" and describe LogStash::Filters::Date do
       }
     CONFIG
 
-    sample("timestamp" => "25/Mar/2013:20:33:56 +0000") do
+    sample({"timestamp" => "25/Mar/2013:20:33:56 +0000"}) do
       insist { subject.get("@timestamp").time } == Time.iso8601("2013-03-25T20:33:56.000Z")
     end
   end
@@ -731,74 +447,6 @@ RUBY_ENGINE == "jruby" and describe LogStash::Filters::Date do
     end
   end
 
-  context "Default year handling when parsing with english fallback parser" do
-
-    around do |example|
-      default = java.util.Locale.getDefault
-      java.util.Locale.setDefault(java.util.Locale.forLanguageTag('fr-FR'))
-      example.run
-      java.util.Locale.setDefault(default)
-    end
-
-    puts "override locale"
-    describe "LOGSTASH-34 - Default year should be this year" do
-      config <<-CONFIG
-        filter {
-          date {
-            match => [ "message", "EEE MMM dd HH:mm:ss" ]
-            timezone => "UTC"
-          }
-        }
-      CONFIG
-
-      sample "Sun Jun 02 20:38:03" do
-        insist { subject.get("@timestamp").year } == Time.now.year
-      end
-    end
-
-    describe "fill last year if december events arrive in january" do
-      config <<-CONFIG
-        filter {
-          date {
-            match => [ "message", "MMM dd HH:mm:ss" ]
-            timezone => "UTC"
-          }
-        }
-      CONFIG
-
-      before(:each) do
-        logstash_time = Time.utc(2014,1,1,00,30,50)
-        allow(Time).to receive(:now).and_return(logstash_time)
-        org.logstash.filters.parser.JodaParser.setDefaultClock { org.joda.time.DateTime.new(2014,1,1,00,30,50, org.joda.time.DateTimeZone::UTC) }
-      end
-
-      sample "Dec 31 23:59:00" do
-        insist { subject.get("@timestamp").year } == 2013
-      end
-    end
-
-    describe "fill next year if january events arrive in december" do
-      config <<-CONFIG
-        filter {
-          date {
-            match => [ "message", "MMM dd HH:mm:ss" ]
-            timezone => "UTC"
-          }
-        }
-      CONFIG
-
-      before(:each) do
-        logstash_time = Time.utc(2013,12,31,23,59,50)
-        allow(Time).to receive(:now).and_return(logstash_time)
-        org.logstash.filters.parser.JodaParser.setDefaultClock { org.joda.time.DateTime.new(2013,12,31,23,59,50, org.joda.time.DateTimeZone::UTC) }
-      end
-
-      sample "Jan 01 01:00:00" do
-        insist { subject.get("@timestamp").year } == 2014
-      end
-    end
-  end
-
   describe "metric counters" do
     subject { described_class.new("match" => [ "message", "yyyy" ]) }
 
@@ -872,4 +520,85 @@ RUBY_ENGINE == "jruby" and describe LogStash::Filters::Date do
       end
     end
   end
+
+  describe "precision => ns, java.time with nanoseconds" do
+    config <<-CONFIG
+      filter {
+        date {
+          match => [ "mydate", "yyyy-MM-dd HH:mm:ss.SSSSSSSSS VV" ]
+          precision => "ns"
+        }
+      }
+    CONFIG
+
+    sample({"mydate" => "2016-11-03 21:10:57.123456789 America/New_York"}) do
+      insist { subject.get("@timestamp").time.to_i } == 1478221857
+      insist { subject.get("@timestamp").time.nsec } == 123456789
+    end
+  end
+
+  describe "precision => ns, ISO8601 with nanoseconds" do
+    config <<-CONFIG
+      filter {
+        date {
+          match => [ "mydate", "ISO8601" ]
+          precision => "ns"
+        }
+      }
+    CONFIG
+
+    sample({"mydate" => "2016-11-03T21:10:57.123456789Z"}) do
+      insist { subject.get("@timestamp").time.to_i } == 1478207457
+      insist { subject.get("@timestamp").time.nsec } == 123456789
+    end
+  end
+
+  describe "precision => ns, UNIX epoch string with nanoseconds" do
+    config <<-CONFIG
+      filter {
+        date {
+          match => [ "mydate", "UNIX" ]
+          precision => "ns"
+        }
+      }
+    CONFIG
+
+    sample({"mydate" => "1478207457.123456789"}) do
+      insist { subject.get("@timestamp").time.to_i } == 1478207457
+      insist { subject.get("@timestamp").time.nsec } == 123456789
+    end
+  end
+
+  describe "precision => ns, TAI64N full nanoseconds" do
+    config <<-CONFIG
+      filter {
+        date {
+          match => [ "mydate", "TAI64N" ]
+          precision => "ns"
+        }
+      }
+    CONFIG
+
+    sample({"mydate" => "4000000050d506482dbdf024"}) do
+      # 0x2dbdf024 = 767422500 ns
+      insist { subject.get("@timestamp").time.to_i } == 1356138046
+      insist { subject.get("@timestamp").time.nsec } == 767422500
+    end
+  end
+
+  describe "precision => ms (default) still truncates to milliseconds" do
+    config <<-CONFIG
+      filter {
+        date {
+          match => [ "mydate", "TAI64N" ]
+          precision => "ms"
+        }
+      }
+    CONFIG
+
+    sample({"mydate" => "4000000050d506482dbdf024"}) do
+      insist { subject.get("@timestamp").time.to_i } == 1356138046
+      insist { subject.get("@timestamp").time.nsec } == 767000000
+    end
+  end
 end

metadata

@@ -1,16 +1,16 @@
 --- !ruby/object:Gem::Specification
 name: logstash-filter-date
 version: !ruby/object:Gem::Version
-  version: 3.1.15
+  version: 3.2.0
 platform: ruby
 authors:
 - Elastic
-autorequire:
 bindir: bin
 cert_chain: []
-date: 2022-06-29 00:00:00.000000000 Z
+date: 2026-03-03 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
+  name: logstash-core-plugin-api
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
@@ -19,9 +19,8 @@ dependencies:
     - - "<="
       - !ruby/object:Gem::Version
         version: '2.99'
-  name: logstash-core-plugin-api
-  prerelease: false
   type: :runtime
+  prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
@@ -31,84 +30,84 @@ dependencies:
       - !ruby/object:Gem::Version
         version: '2.99'
 - !ruby/object:Gem::Dependency
+  name: logstash-input-generator
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
-  name: logstash-input-generator
-  prerelease: false
   type: :development
+  prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
+  name: logstash-codec-json
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
-  name: logstash-codec-json
-  prerelease: false
   type: :development
+  prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
+  name: logstash-output-null
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
-  name: logstash-output-null
-  prerelease: false
   type: :development
+  prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
+  name: logstash-devutils
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
         version: '2.3'
-  name: logstash-devutils
-  prerelease: false
   type: :development
+  prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
         version: '2.3'
 - !ruby/object:Gem::Dependency
+  name: insist
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
-  name: insist
-  prerelease: false
   type: :development
+  prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
+  name: benchmark-ips
   requirement: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
       - !ruby/object:Gem::Version
         version: '0'
-  name: benchmark-ips
-  prerelease: false
   type: :development
+  prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
     - - ">="
@@ -141,7 +140,6 @@ licenses:
 metadata:
   logstash_plugin: 'true'
   logstash_group: filter
-post_install_message:
 rdoc_options: []
 require_paths:
 - lib
@@ -157,8 +155,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubygems_version: 3.1.6
-signing_key:
+rubygems_version: 3.6.3
 specification_version: 4
 summary: Parses dates from fields to use as the Logstash timestamp for an event
 test_files: