logstash-filter-date 3.1.16 → 3.2.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 178ab94368b56ddaf9d595951dd2888c856b752a7d25c412c67ea65d11af1911
-  data.tar.gz: b8fc77811870b4aebc7e1f9f2a75930f89d0cb574f0246fff43374214999aca5
+  metadata.gz: acaae4f07bc5115a72912039e65d94e0036d7cb13eb426f5a6da08728db6e347
+  data.tar.gz: 5b52406d8576e434a23b4b7a9d30ce9ca4699e8ef16640151964067d7f799a58
 SHA512:
-  metadata.gz: 51bd27aadf72fdb8739aad971e7a24ff5bca717360d0dc661ecb02b8c62d4d0933f46e7761d8099bf901f420237536eac2e3afd398a203b34e8da691f7f7d063
-  data.tar.gz: 82a4316970c8571678b4f545221593234decfad7dff7c3846498c5b4b1af7836ae87a07ddd42ea7ed72d6a303e5f9d8bf6b6445e9b43b8b92bd5f6fb50ca3847
+  metadata.gz: 0c6ec611ae7343d02ca09326de16c80d39d6aace82bd7eb159a9ff9e33ff038915dad79cd58760ae83fdeb613e526636cb4274ff948ec8512c96374de43a0ded
+  data.tar.gz: b7c9daffcf5364c87f529b4c7e473b50a0b88498777ed33b95e5012cac7770bf33c2d71f4f34ac0d8d5266b126fabd7f7b9613d3cdbd24767799d18dc520535e

data/CHANGELOG.md

@@ -1,3 +1,17 @@
+## 3.2.1
+  - Fix date syntax docs list formatting [#166](https://github.com/logstash-plugins/logstash-filter-date/pull/166)
+
+## 3.2.0
+  - Add `precision` setting to support nanosecond precision timestamps [#165](https://github.com/logstash-plugins/logstash-filter-date/pull/165)
+    - `ms` (default): timestamps are stored with millisecond precision
+      - keeps the same behavior as before for backward compatibility
+      - fractional seconds are truncated to 3 digits
+      - custom parsing formats use `joda-time` library
+    - `ns`: timestamps are stored with nanosecond precision
+      - fractional seconds support up to 9 digits
+      - custom parsing formats use `java.time`
+  - `ISO8601` now accepts up to 9 fractional-second digits
+
 ## 3.1.16
   - Re-packaging the plugin [#163](https://github.com/logstash-plugins/logstash-filter-date/pull/163)
 

data/docs/index.asciidoc

@@ -20,23 +20,11 @@ include::{include_path}/plugin_header.asciidoc[]
 
 ==== Description
 
-The date filter is used for parsing dates from fields, and then using that
-date or timestamp as the logstash timestamp for the event.
+The date filter is used for parsing dates from fields, and then using that date or timestamp as the logstash timestamp for the event.
+Timestamp is stored with millisecond precision. Set `precision => "ns"` to preserve nanoseconds.
 
-For example, syslog events usually have timestamps like this:
-[source,ruby]
-    "Apr 17 09:32:01"
-
-You would use the date format `MMM dd HH:mm:ss` to parse this.
-
-The date filter is especially important for sorting events and for
-backfilling old data. If you don't get the date correct in your
-event, then searching for them later will likely sort out of order.
-
-In the absence of this filter, logstash will choose a timestamp based on the
-first time it sees the event (at input time), if the timestamp is not already
-set in the event. For example, with file input, the timestamp is set to the
-time of each read.
+Custom parsing formats use the JVM’s default locale and time zone.
+To override them, configure the `locale` and `timezone` settings.
 
 [id="plugins-{type}s-{plugin}-options"]
 ==== Date Filter Configuration Options
@@ -48,6 +36,7 @@ This plugin supports the following configuration options plus the <<plugins-{typ
 |Setting |Input type|Required
 | <<plugins-{type}s-{plugin}-locale>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-match>> |<<array,array>>|No
+| <<plugins-{type}s-{plugin}-precision>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-tag_on_failure>> |<<array,array>>|No
 | <<plugins-{type}s-{plugin}-target>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-timezone>> |<<string,string>>|No
@@ -89,11 +78,14 @@ If your time field has multiple possible formats, you can do this:
 
 The above will match a syslog (rfc3164) or `iso8601` timestamp.
 
-There are a few special exceptions. The following format literals exist
+By default, the custom formats use the deprecated https://www.joda.org/joda-time/key_format.html[joda-time] library for parsing and timestamp is stored with millisecond precision.
+If you set `precision => "ns"`, parsing is performed using https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/time/format/DateTimeFormatter.html[java.time], and timestamps are stored with nanosecond precision.
+Both libraries use similar syntax, but there are some differences in supported features. For example, `java.time` supports time zone IDs with `VV` while `joda-time` uses `ZZZ`.
+
+Besides custom formats, the following format literals exist
 to help you save time and ensure correctness of date parsing.
 
-* `ISO8601` - should parse any valid ISO8601 timestamp, such as
-  `2011-04-19T03:44:01.103Z`
+* `ISO8601` - should parse any valid ISO8601 timestamp, such as `2011-04-19T03:44:01.123456789Z`
 * `UNIX` - will parse *float or int* value expressing unix time in seconds since epoch like 1326149001.132 as well as 1326149001
 * `UNIX_MS` - will parse **int** value expressing unix time in milliseconds since epoch like 1366125117000
 * `TAI64N` - will parse tai64n time values
@@ -119,65 +111,81 @@ indicate the form of that value (2-digit month, full month name, etc).
 
 Here's what you can use to parse dates and times:
 
-[horizontal]
-y:: year
-  yyyy::: full year number. Example: `2015`.
-  yy::: two-digit year. Example: `15` for the year 2015.
+* `y`: year
+** `yyyy`: full year number. Example: `2015`.
+** `yy`: two-digit year. Example: `15` for the year 2015.
+
+* `M`: month of the year
+** `M`: minimal-digit month. Example: `1` for January and `12` for December.
+** `MM`: two-digit month. zero-padded if needed. Example: `01` for January  and `12` for December
+** `MMM`: abbreviated month text. Example: `Jan` for January. Note: The language used depends on your locale. See the `locale` setting for how to change the language.
+** `MMMM`: full month text, Example: `January`. Note: The language used depends on your locale.
 
-M:: month of the year
-  M::: minimal-digit month. Example: `1` for January and `12` for December.
-  MM::: two-digit month. zero-padded if needed. Example: `01` for January  and `12` for December
-  MMM::: abbreviated month text. Example: `Jan` for January. Note: The language used depends on your locale. See the `locale` setting for how to change the language.
-  MMMM::: full month text, Example: `January`. Note: The language used depends on your locale.
+* `d`: day of the month
+** `d`: minimal-digit day. Example: `1` for the 1st of the month.
+** `dd`: two-digit day, zero-padded if needed. Example: `01` for the 1st of the month.
 
-d:: day of the month
-  d::: minimal-digit day. Example: `1` for the 1st of the month.
-  dd::: two-digit day, zero-padded if needed. Example: `01` for the 1st of the month.
+* `H`: hour of the day (24-hour clock)
+** `H`: minimal-digit hour. Example: `0` for midnight.
+** `HH`: two-digit hour, zero-padded if needed. Example: `00` for midnight.
 
-H:: hour of the day (24-hour clock)
-  H::: minimal-digit hour. Example: `0` for midnight.
-  HH::: two-digit hour, zero-padded if needed. Example: `00` for midnight.
+* `m`: minutes of the hour (60 minutes per hour)
+** `m`: minimal-digit minutes. Example: `0`.
+** `mm`: two-digit minutes, zero-padded if needed. Example: `00`.
 
-m:: minutes of the hour (60 minutes per hour)
-  m::: minimal-digit minutes. Example: `0`.
-  mm::: two-digit minutes, zero-padded if needed. Example: `00`.
+* `s`: seconds of the minute (60 seconds per minute)
+** `s`: minimal-digit seconds. Example: `0`.
+** `ss`: two-digit seconds, zero-padded if needed. Example: `00`.
 
-s:: seconds of the minute (60 seconds per minute)
-  s::: minimal-digit seconds. Example: `0`.
-  ss::: two-digit seconds, zero-padded if needed. Example: `00`.
+* `S`: fraction of a second
+** `S`: tenths of a second. Example:  `0` for a subsecond value `012`
+** `SS`: hundredths of a second. Example:  `01` for a subsecond value `01`
+** `SSS`: milliseconds. Example:  `012` for a subsecond value `012`
+** `SSSSSS`: microseconds. Example:  `012345`
+** `SSSSSSSSS`: nanoseconds. Example:  `012345678`
 
-S:: fraction of a second
-  *Maximum precision is milliseconds (`SSS`). Beyond that, zeroes are appended.*
-  S::: tenths of a second. Example:  `0` for a subsecond value `012`
-  SS::: hundredths of a second. Example:  `01` for a subsecond value `01`
-  SSS::: thousandths of a second. Example:  `012` for a subsecond value `012`
+* `V`: time-zone ID
+** `VV`: time zone ID. Example: `America/Los_Angeles`. Note: This is only supported by `java.time` parsing.
 
-Z:: time zone offset or identity
-  Z::: Timezone offset structured as HHmm (hour and minutes offset from Zulu/UTC). Example: `-0700`.
-  ZZ::: Timezone offset structured as HH:mm (colon in between hour and minute offsets). Example: `-07:00`.
-  ZZZ::: Timezone identity. Example: `America/Los_Angeles`. Note: Valid IDs are listed on the http://joda-time.sourceforge.net/timezones.html[Joda.org available time zones page].
+* `Z`: time zone offset or identity
+** `Z`: Timezone offset structured as HHmm (hour and minutes offset from Zulu/UTC). Example: `-0700`.
+** `ZZ`: Timezone offset structured as HH:mm (colon in between hour and minute offsets). Example: `-07:00`.
+** `ZZZ`: Timezone identity. Example: `America/Los_Angeles`. Note: Valid IDs are listed on the http://joda-time.sourceforge.net/timezones.html[Joda.org available time zones page].
 
-z:: time zone names. *Time zone names ('z') cannot be parsed.*
+* `z`: time zone names. *Time zone names (`z`) cannot be parsed.*
 
-w:: week of the year
-  w::: minimal-digit week. Example: `1`.
-  ww::: two-digit week, zero-padded if needed. Example: `01`.
+* `w`: week of the year
+** `w`: minimal-digit week. Example: `1`.
+** `ww`: two-digit week, zero-padded if needed. Example: `01`.
 
-D:: day of the year
+* `D`: day of the year
 
-e:: day of the week (number)
+* `e`: day of the week (number)
 
-E:: day of the week (text)
-  E, EE, EEE::: Abbreviated day of the week. Example:  `Mon`, `Tue`, `Wed`, `Thu`, `Fri`, `Sat`, `Sun`. Note: The actual language of this will depend on your locale.
-  EEEE::: The full text day of the week. Example: `Monday`, `Tuesday`, ... Note: The actual language of this will depend on your locale.
+* `E`: day of the week (text)
+** `E`, `EE`, `EEE`: Abbreviated day of the week. Example:  `Mon`, `Tue`, `Wed`, `Thu`, `Fri`, `Sat`, `Sun`. Note: The actual language of this will depend on your locale.
+** `EEEE`: The full text day of the week. Example: `Monday`, `Tuesday`, ... Note: The actual language of this will depend on your locale.
 
 For non-formatting syntax, you'll need to put single-quote characters around the value. For example, if you were parsing ISO8601 time, "2015-01-01T01:12:23" that little "T" isn't a valid time format, and you want to say "literally, a T", your format would be this: "yyyy-MM-dd'T'HH:mm:ss"
 
 Other less common date units, such as era (G), century \(C), am/pm (a), and # more, can be learned about on the
-http://www.joda.org/joda-time/key_format.html[joda-time documentation].
+https://www.joda.org/joda-time/key_format.html[joda-time documentation].
+
+[id="plugins-{type}s-{plugin}-precision"]
+===== `precision`
+
+  * Value type is <<string,string>>
+  * Valid values are `ms` and `ns`
+  * Default value is `ms`
+
+Controls the sub-second precision of the stored timestamp.
+
+`ms` stores data in millisecond precision. Custom pattern formats use https://www.joda.org/joda-time/key_format.html[joda-time] parsing rules. For example, `yyyy-MM-dd HH:mm:ss ZZZ`.
+
+`ns` stores data in nanosecond precision. Custom pattern formats use https://docs.oracle.com/en/java/javase/11/docs/api/java.base/java/time/format/DateTimeFormatter.html[java.time] parsing rules. For example, `yyyy-MM-dd HH:mm:ss VV`.
 
 [id="plugins-{type}s-{plugin}-tag_on_failure"]
-===== `tag_on_failure` 
+===== `tag_on_failure`
 
   * Value type is <<array,array>>
   * Default value is `["_dateparsefailure"]`

data/lib/logstash/filters/date.rb

@@ -57,9 +57,8 @@ class LogStash::Filters::Date < LogStash::Filters::Base
   # There are a few special exceptions. The following format literals exist
   # to help you save time and ensure correctness of date parsing.
   #
-  # * `ISO8601` - should parse any valid ISO8601 timestamp, such as
-  #   `2011-04-19T03:44:01.103Z`
-  # * `UNIX` - will parse *float or int* value expressing unix time in seconds since epoch like 1326149001.132 as well as 1326149001
+  # * `ISO8601` - should parse any valid ISO8601 timestamp with up to 9 fractional-second digits
+  # * `UNIX` - will parse *float or int* value expressing unix time in seconds since epoch like 1326149001.123456789 as well as 1326149001
   # * `UNIX_MS` - will parse **int** value expressing unix time in milliseconds since epoch like 1366125117000
   # * `TAI64N` - will parse tai64n time values
   #
@@ -84,57 +83,57 @@ class LogStash::Filters::Date < LogStash::Filters::Base
   #
   # Here's what you can use to parse dates and times:
   #
-  # [horizontal]
-  # y:: year
-  #   yyyy::: full year number. Example: `2015`.
-  #   yy::: two-digit year. Example: `15` for the year 2015.
+  # * `y`: year
+  # ** `yyyy`: full year number. Example: `2015`.
+  # ** `yy`: two-digit year. Example: `15` for the year 2015.
   #
-  # M:: month of the year
-  #   M::: minimal-digit month. Example: `1` for January and `12` for December.
-  #   MM::: two-digit month. zero-padded if needed. Example: `01` for January  and `12` for December
-  #   MMM::: abbreviated month text. Example: `Jan` for January. Note: The language used depends on your locale. See the `locale` setting for how to change the language.
-  #   MMMM::: full month text, Example: `January`. Note: The language used depends on your locale.
+  # * `M`: month of the year
+  # ** `M`: minimal-digit month. Example: `1` for January and `12` for December.
+  # ** `MM`: two-digit month. zero-padded if needed. Example: `01` for January  and `12` for December
+  # ** `MMM`: abbreviated month text. Example: `Jan` for January. Note: The language used depends on your locale. See the `locale` setting for how to change the language.
+  # ** `MMMM`: full month text, Example: `January`. Note: The language used depends on your locale.
   #
-  # d:: day of the month
-  #   d::: minimal-digit day. Example: `1` for the 1st of the month.
-  #   dd::: two-digit day, zero-padded if needed. Example: `01` for the 1st of the month.
+  # * `d`: day of the month
+  # ** `d`: minimal-digit day. Example: `1` for the 1st of the month.
+  # ** `dd`: two-digit day, zero-padded if needed. Example: `01` for the 1st of the month.
   #
-  # H:: hour of the day (24-hour clock)
-  #   H::: minimal-digit hour. Example: `0` for midnight.
-  #   HH::: two-digit hour, zero-padded if needed. Example: `00` for midnight.
+  # * `H`: hour of the day (24-hour clock)
+  # ** `H`: minimal-digit hour. Example: `0` for midnight.
+  # ** `HH`: two-digit hour, zero-padded if needed. Example: `00` for midnight.
   #
-  # m:: minutes of the hour (60 minutes per hour)
-  #   m::: minimal-digit minutes. Example: `0`.
-  #   mm::: two-digit minutes, zero-padded if needed. Example: `00`.
+  # * `m`: minutes of the hour (60 minutes per hour)
+  # ** `m`: minimal-digit minutes. Example: `0`.
+  # ** `mm`: two-digit minutes, zero-padded if needed. Example: `00`.
   #
-  # s:: seconds of the minute (60 seconds per minute)
-  #   s::: minimal-digit seconds. Example: `0`.
-  #   ss::: two-digit seconds, zero-padded if needed. Example: `00`.
+  # * `s`: seconds of the minute (60 seconds per minute)
+  # ** `s`: minimal-digit seconds. Example: `0`.
+  # ** `ss`: two-digit seconds, zero-padded if needed. Example: `00`.
   #
-  # S:: fraction of a second
-  #   *Maximum precision is milliseconds (`SSS`). Beyond that, zeroes are appended.*
-  #   S::: tenths of a second. Example:  `0` for a subsecond value `012`
-  #   SS::: hundredths of a second. Example:  `01` for a subsecond value `01`
-  #   SSS::: thousandths of a second. Example:  `012` for a subsecond value `012`
+  # * `S`: fraction of a second
+  # ** `S`: tenths of a second. Example:  `0` for a subsecond value `012`
+  # ** `SS`: hundredths of a second. Example:  `01` for a subsecond value `01`
+  # ** `SSS`: milliseconds. Example:  `012` for a subsecond value `012`
+  # ** `SSSSSS`: microseconds.
+  # ** `SSSSSSSSS`: nanoseconds.
   #
-  # Z:: time zone offset or identity
-  #   Z::: Timezone offset structured as HHmm (hour and minutes offset from Zulu/UTC). Example: `-0700`.
-  #   ZZ::: Timezone offset structured as HH:mm (colon in between hour and minute offsets). Example: `-07:00`.
-  #   ZZZ::: Timezone identity. Example: `America/Los_Angeles`. Note: Valid IDs are listed on the http://joda-time.sourceforge.net/timezones.html[Joda.org available time zones page].
+  # * `Z`: time zone offset or identity
+  # ** `Z`: Timezone offset structured as HHmm (hour and minutes offset from Zulu/UTC). Example: `-0700`.
+  # ** `ZZ`: Timezone offset structured as HH:mm (colon in between hour and minute offsets). Example: `-07:00`.
+  # ** `ZZZ`: Timezone identity. Example: `America/Los_Angeles`. Note: Valid IDs are listed on the http://joda-time.sourceforge.net/timezones.html[Joda.org available time zones page].
   #
-  # z:: time zone names. *Time zone names ('z') cannot be parsed.*
+  # * `z`: time zone names. *Time zone names (`z`) cannot be parsed.*
   #
-  # w:: week of the year
-  #   w::: minimal-digit week. Example: `1`.
-  #   ww::: two-digit week, zero-padded if needed. Example: `01`.
+  # * `w`: week of the year
+  # ** `w`: minimal-digit week. Example: `1`.
+  # ** `ww`: two-digit week, zero-padded if needed. Example: `01`.
   #
-  # D:: day of the year
+  # * `D`: day of the year
   #
-  # e:: day of the week (number)
+  # * `e`: day of the week (number)
   #
-  # E:: day of the week (text)
-  #   E, EE, EEE::: Abbreviated day of the week. Example:  `Mon`, `Tue`, `Wed`, `Thu`, `Fri`, `Sat`, `Sun`. Note: The actual language of this will depend on your locale.
-  #   EEEE::: The full text day of the week. Example: `Monday`, `Tuesday`, ... Note: The actual language of this will depend on your locale.
+  # * `E`: day of the week (text)
+  # ** `E`, `EE`, `EEE`: Abbreviated day of the week. Example:  `Mon`, `Tue`, `Wed`, `Thu`, `Fri`, `Sat`, `Sun`. Note: The actual language of this will depend on your locale.
+  # ** `EEEE`: The full text day of the week. Example: `Monday`, `Tuesday`, ... Note: The actual language of this will depend on your locale.
   #
   # For non-formatting syntax, you'll need to put single-quote characters around the value. For example, if you were parsing ISO8601 time, "2015-01-01T01:12:23" that little "T" isn't a valid time format, and you want to say "literally, a T", your format would be this: "yyyy-MM-dd'T'HH:mm:ss"
   #
@@ -150,6 +149,14 @@ class LogStash::Filters::Date < LogStash::Filters::Base
   # successful match
   config :tag_on_failure, :validate => :array, :default => ["_dateparsefailure"]
 
+  # Controls the sub-second precision of the stored timestamp.
+  #
+  # `"ms"` (default):: Stores in millisecond precision. Custom-pattern formats use Joda-Time parsing rules.
+  # `"ns"`:: Stores in nanosecond precision. Custom-pattern formats use java.time parsing rules.
+  config :precision, :validate => [Java::OrgLogstashFiltersParser::TimestampParserFactory::PRECISION_MS,
+                                   Java::OrgLogstashFiltersParser::TimestampParserFactory::PRECISION_NS],
+                          :default => Java::OrgLogstashFiltersParser::TimestampParserFactory::PRECISION_MS
+
   def register
     # nothing
   end
@@ -179,13 +186,13 @@ class LogStash::Filters::Date < LogStash::Filters::Base
       metric.increment(:failures)
     end
 
-    @datefilter = org.logstash.filters.DateFilter.new(source, @target, @tag_on_failure, success_block, failure_block)
+    @datefilter = org.logstash.filters.DateFilter.new(source, @target, @tag_on_failure, @precision, success_block, failure_block)
 
     @match[1..-1].map do |format|
       @datefilter.accept_filter_config(format, @locale, @timezone)
 
       # Offer a fallback parser such that if the default system Locale is non-english and that no locale is set,
-      # we should try to parse english if the first local parsing fails.:w
+      # we should try to parse english if the first local parsing fails.
       if !@locale && "en" != java.util.Locale.getDefault().getLanguage() && (format.include?("MMM") || format.include?("E"))
         @datefilter.accept_filter_config(format, "en-US", @timezone)
       end

data/logstash-filter-date.gemspec

@@ -1,7 +1,7 @@
 Gem::Specification.new do |s|
 
   s.name            = 'logstash-filter-date'
-  s.version         = '3.1.16'
+  s.version         = '3.2.1'
   s.licenses        = ['Apache License (2.0)']
   s.summary         = "Parses dates from fields to use as the Logstash timestamp for an event"
   s.description     = "This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program"

data/spec/filters/date_spec.rb

@@ -17,62 +17,6 @@ RUBY_ENGINE == "jruby" and describe LogStash::Filters::Date do
     end
   end
 
-  describe "parsing with ISO8601" do
-    config <<-CONFIG
-      filter {
-        date {
-          match => [ "mydate", "ISO8601" ]
-          locale => "en"
-          timezone => "UTC"
-        }
-      }
-    CONFIG
-
-    times = {
-      "2001-01-01T00:00:00-0800"         => "2001-01-01T08:00:00.000Z",
-      "1974-03-02T04:09:09-0800"         => "1974-03-02T12:09:09.000Z",
-      "2010-05-03T08:18:18+00:00"        => "2010-05-03T08:18:18.000Z",
-      "2004-07-04T12:27:27-00:00"        => "2004-07-04T12:27:27.000Z",
-      "2001-09-05T16:36:36+0000"         => "2001-09-05T16:36:36.000Z",
-      "2001-11-06T20:45:45-0000"         => "2001-11-06T20:45:45.000Z",
-      "2001-12-07T23:54:54Z"             => "2001-12-07T23:54:54.000Z",
-
-      # TODO: This test assumes PDT
-      #"2001-01-01T00:00:00.123"          => "2001-01-01T08:00:00.123Z",
-
-      "2010-05-03T08:18:18.123+00:00"    => "2010-05-03T08:18:18.123Z",
-      "2004-07-04T12:27:27.123-04:00"    => "2004-07-04T16:27:27.123Z",
-      "2001-09-05T16:36:36.123+0700"     => "2001-09-05T09:36:36.123Z",
-      "2001-11-06T20:45:45.123-0000"     => "2001-11-06T20:45:45.123Z",
-      "2001-12-07T23:54:54.123Z"         => "2001-12-07T23:54:54.123Z",
-      "2001-12-07T23:54:54,123Z"         => "2001-12-07T23:54:54.123Z",
-
-      #Almost ISO8601 support, with timezone
-
-      "2001-11-06 20:45:45.123-0000"     => "2001-11-06T20:45:45.123Z",
-      "2001-12-07 23:54:54.123Z"         => "2001-12-07T23:54:54.123Z",
-      "2001-12-07 23:54:54,123Z"         => "2001-12-07T23:54:54.123Z",
-
-      #Almost ISO8601 support, without timezone
-
-      "2001-11-06 20:45:45.123"     => "2001-11-06T20:45:45.123Z",
-      "2001-11-06 20:45:45,123"     => "2001-11-06T20:45:45.123Z",
-
-    }
-
-    times.each do |input, output|
-      sample({"mydate" => input}) do
-        begin
-          insist { subject.get("mydate") } == input
-          insist { subject.get("@timestamp").time } == Time.iso8601(output).utc
-        rescue
-          #require "pry"; binding.pry
-          raise
-        end
-      end
-    end # times.each
-  end
-
   describe "parsing with java SimpleDateFormat syntax" do
     config <<-CONFIG
       filter {
@@ -132,44 +76,6 @@ RUBY_ENGINE == "jruby" and describe LogStash::Filters::Date do
     end
   end
 
-  describe "parsing microsecond-precise times with UNIX (#213)" do
-    config <<-CONFIG
-      filter {
-        date {
-          match => [ "mydate", "UNIX" ]
-          locale => "en"
-        }
-      }
-    CONFIG
-
-    sample({"mydate" => "1350414944.123456"}) do
-      # Joda time only supports milliseconds :\
-      insist { subject.timestamp.time } == Time.iso8601("2012-10-16T12:15:44.123-07:00").utc
-    end
-
-    #Support float values
-    sample({"mydate" => 1350414944.123456}) do
-      insist { subject.get("mydate") } == 1350414944.123456
-      insist { subject.get("@timestamp").time } == Time.iso8601("2012-10-16T12:15:44.123-07:00").utc
-    end
-
-    #Invalid value should not be evaluated to zero (String#to_i madness)
-    sample({"mydate" => "%{bad_value}"}) do
-      insist { subject.get("mydate") } == "%{bad_value}"
-      insist { subject.get("@timestamp").time } != Time.iso8601("1970-01-01T00:00:00.000Z").utc
-    end
-
-    # Regression test
-    # Support numeric values that come through the JSON parser. These numbers appear as BigDecimal
-    # instead of Float.
-    sample(LogStash::Json.load('{ "mydate": 1350414944.123456 }')) do
-      # its generally problematic to compare different Floating Point class implementations using equals
-      # because they can't always represent a value exactly.
-      insist { subject.get("mydate") } == BigDecimal.new("1350414944.123456")
-      insist { subject.get("@timestamp").time } == Time.iso8601("2012-10-16T12:15:44.123-07:00").utc
-    end
-  end
-
   describe "parsing with UNIX_MS" do
     config <<-CONFIG
       filter {
@@ -269,16 +175,9 @@ RUBY_ENGINE == "jruby" and describe LogStash::Filters::Date do
       }
     CONFIG
 
-    # Try without leading "@"
     sample({"t" => "4000000050d506482dbdf024"}) do
       insist { subject.timestamp.time } == Time.iso8601("2012-12-22T01:00:46.767Z").utc
     end
-
-    # Should still parse successfully if it's a full tai64n time (with leading
-    # '@')
-    sample({"t" => "@4000000050d506482dbdf024"}) do
-      insist { subject.timestamp.time } == Time.iso8601("2012-12-22T01:00:46.767Z").utc
-    end
   end
 
   describe "accept match config option with hash value (LOGSTASH-735)" do
@@ -461,189 +360,6 @@ RUBY_ENGINE == "jruby" and describe LogStash::Filters::Date do
     end
   end
 
-  describe "don't fail on next years DST switchover in CET" do
-    config <<-CONFIG
-      filter {
-        date {
-          match => [ "message", "yyyy MMM dd HH:mm:ss" ]
-          locale => "en"
-          timezone => "CET"
-        }
-      }
-    CONFIG
-
-    before(:each) do
-      logstash_time = Time.utc(2016,03,29,23,59,50)
-      allow(Time).to receive(:now).and_return(logstash_time)
-    end
-
-    sample "2016 Mar 26 02:00:37" do
-      insist { subject.get("tags") } != ["_dateparsefailure"]
-      expect(subject.get("@timestamp")).to be_a_logstash_timestamp_equivalent_to("2016-03-26T01:00:37.000Z")
-    end
-  end
-
-  context "Default year handling when parsing with timezone from event" do
-
-    describe "LOGSTASH-34 - Default year should be this year" do
-      config <<-CONFIG
-        filter {
-          date {
-            match => [ "message", "EEE MMM dd HH:mm:ss" ]
-            locale => "en"
-            timezone => "%{mytz}"
-          }
-        }
-      CONFIG
-
-      sample({"message" => "Sun Jun 02 20:38:03", "mytz" => "UTC"}) do
-        insist { subject.get("@timestamp").year } == Time.now.year
-      end
-    end
-
-    describe "fill last year if december events arrive in january" do
-      config <<-CONFIG
-        filter {
-          date {
-            match => [ "message", "MMM dd HH:mm:ss" ]
-            locale => "en"
-            timezone => "%{mytz}"
-          }
-        }
-      CONFIG
-
-      before(:each) do
-        logstash_time = Time.utc(2014,1,1,00,30,50)
-        allow(Time).to receive(:now).and_return(logstash_time)
-        org.logstash.filters.parser.JodaParser.setDefaultClock { org.joda.time.DateTime.new(2014,1,1,00,30,50, org.joda.time.DateTimeZone::UTC ) }
-      end
-
-      sample({"message" => "Dec 31 23:59:00", "mytz" => "UTC"}) do
-        insist { subject.get("@timestamp").year } == 2013
-      end
-    end
-
-    describe "fill next year if january events arrive in december" do
-      config <<-CONFIG
-        filter {
-          date {
-            match => [ "message", "MMM dd HH:mm:ss" ]
-            locale => "en"
-            timezone => "%{mytz}"
-          }
-        }
-      CONFIG
-
-      before(:each) do
-        logstash_time = Time.utc(2013,12,31,23,59,50)
-        allow(Time).to receive(:now).and_return(logstash_time)
-        org.logstash.filters.parser.JodaParser.setDefaultClock { org.joda.time.DateTime.new(2013,12,31,23,59,50, org.joda.time.DateTimeZone::UTC ) }
-      end
-
-      sample({"message" => "Jan 01 01:00:00", "mytz" => "UTC"}) do
-        insist { subject.get("@timestamp").year } == 2014
-      end
-    end
-
-    describe "do fail on 2016 DST switchover in CET" do
-      # This test tries to parse a time that doesn't exist. '02:00:01' is a time that doesn't exist
-      # because this DST switch goes from 01:59:59 to 03:00:00, skipping 2am entirely. The last Sunday of March in 2016 was 27th.
-      # (Guy Boertje) Fixed the GuessYear logic
-      # Joda has a default year for DateTimeFormat of 2000
-      # meaning that the initial time parsed was Monday 2000-03-27 02:00:01 and the last Sunday of March in 2000 was the 26th
-      # then by adding the year of 2016 creates an invalid time
-      # The parser default now takes the year from the DefaultClock in the JodaParser
-      config <<-CONFIG
-        filter {
-          date {
-            match => [ "message", "MMM dd HH:mm:ss" ]
-            locale => "en"
-            timezone => "CET"
-          }
-        }
-      CONFIG
-
-      before(:each) do
-        logstash_time = Time.utc(2016,03,29,23,59,50)
-        allow(Time).to receive(:now).and_return(logstash_time)
-        org.logstash.filters.parser.JodaParser.setDefaultClock { org.joda.time.DateTime.new(2016,03,29,23,59,50, org.joda.time.DateTimeZone::UTC ) }
-      end
-
-      sample "Mar 27 01:59:59" do
-        expect(subject.get("tags")).to be_nil
-        expect(subject.get("@timestamp")).to be_a_logstash_timestamp_equivalent_to("2016-03-27T00:59:59.000Z")
-      end
-
-      sample "Mar 27 02:00:01" do
-        expect(subject.get("tags")).to eq ["_dateparsefailure"]
-      end
-
-      sample "Mar 27 03:00:01" do
-        expect(subject.get("tags")).to be_nil
-        expect(subject.get("@timestamp")).to be_a_logstash_timestamp_equivalent_to("2016-03-27T01:00:01.000Z")
-      end
-    end
-  end
-
-  describe "LOGSTASH-34 - Default year should be this year" do
-    config <<-CONFIG
-      filter {
-        date {
-          match => [ "message", "EEE MMM dd HH:mm:ss" ]
-          locale => "en"
-        }
-      }
-    CONFIG
-
-    sample "Sun Jun 02 20:38:03" do
-      insist { subject.get("@timestamp").year } == Time.now.year
-    end
-  end
-
-  describe "fill last year if december events arrive in january" do
-    config <<-CONFIG
-      filter {
-        date {
-          match => [ "message", "MMM dd HH:mm:ss" ]
-          locale => "en"
-          timezone => "UTC"
-        }
-      }
-    CONFIG
-
-    before(:each) do
-      logstash_time = Time.utc(2014,1,1,00,30,50)
-      allow(Time).to receive(:now).and_return(logstash_time)
-      org.logstash.filters.parser.JodaParser.setDefaultClock { org.joda.time.DateTime.new(2014,1,1,00,30,50, org.joda.time.DateTimeZone::UTC ) }
-    end
-
-    sample "Dec 31 23:59:00" do
-      insist { subject.get("@timestamp").year } == 2013
-    end
-  end
-
-  describe "fill next year if january events arrive in december" do
-    config <<-CONFIG
-      filter {
-        date {
-          match => [ "message", "MMM dd HH:mm:ss" ]
-          locale => "en"
-          timezone => "UTC"
-        }
-      }
-    CONFIG
-
-    before(:each) do
-      logstash_time = Time.utc(2013,12,31,23,59,50)
-      allow(Time).to receive(:now).and_return(logstash_time)
-      org.logstash.filters.parser.JodaParser.setDefaultClock { org.joda.time.DateTime.new(2013,12,31,15,59,50, org.joda.time.DateTimeZone::UTC ) }
-    end
-
-    sample "Jan 01 01:00:00" do
-      insist { subject.get("@timestamp").year } == 2014
-    end
-  end
-
   describe "Supporting locale only" do
     config <<-CONFIG
       filter {
@@ -731,74 +447,6 @@ RUBY_ENGINE == "jruby" and describe LogStash::Filters::Date do
     end
   end
 
-  context "Default year handling when parsing with english fallback parser" do
-
-    around do |example|
-      default = java.util.Locale.getDefault
-      java.util.Locale.setDefault(java.util.Locale.forLanguageTag('fr-FR'))
-      example.run
-      java.util.Locale.setDefault(default)
-    end
-
-    puts "override locale"
-    describe "LOGSTASH-34 - Default year should be this year" do
-      config <<-CONFIG
-        filter {
-          date {
-            match => [ "message", "EEE MMM dd HH:mm:ss" ]
-            timezone => "UTC"
-          }
-        }
-      CONFIG
-
-      sample "Sun Jun 02 20:38:03" do
-        insist { subject.get("@timestamp").year } == Time.now.year
-      end
-    end
-
-    describe "fill last year if december events arrive in january" do
-      config <<-CONFIG
-        filter {
-          date {
-            match => [ "message", "MMM dd HH:mm:ss" ]
-            timezone => "UTC"
-          }
-        }
-      CONFIG
-
-      before(:each) do
-        logstash_time = Time.utc(2014,1,1,00,30,50)
-        allow(Time).to receive(:now).and_return(logstash_time)
-        org.logstash.filters.parser.JodaParser.setDefaultClock { org.joda.time.DateTime.new(2014,1,1,00,30,50, org.joda.time.DateTimeZone::UTC) }
-      end
-
-      sample "Dec 31 23:59:00" do
-        insist { subject.get("@timestamp").year } == 2013
-      end
-    end
-
-    describe "fill next year if january events arrive in december" do
-      config <<-CONFIG
-        filter {
-          date {
-            match => [ "message", "MMM dd HH:mm:ss" ]
-            timezone => "UTC"
-          }
-        }
-      CONFIG
-
-      before(:each) do
-        logstash_time = Time.utc(2013,12,31,23,59,50)
-        allow(Time).to receive(:now).and_return(logstash_time)
-        org.logstash.filters.parser.JodaParser.setDefaultClock { org.joda.time.DateTime.new(2013,12,31,23,59,50, org.joda.time.DateTimeZone::UTC) }
-      end
-
-      sample "Jan 01 01:00:00" do
-        insist { subject.get("@timestamp").year } == 2014
-      end
-    end
-  end
-
   describe "metric counters" do
     subject { described_class.new("match" => [ "message", "yyyy" ]) }
 
@@ -872,4 +520,85 @@ RUBY_ENGINE == "jruby" and describe LogStash::Filters::Date do
       end
     end
   end
+
+  describe "precision => ns, java.time with nanoseconds" do
+    config <<-CONFIG
+      filter {
+        date {
+          match => [ "mydate", "yyyy-MM-dd HH:mm:ss.SSSSSSSSS VV" ]
+          precision => "ns"
+        }
+      }
+    CONFIG
+
+    sample({"mydate" => "2016-11-03 21:10:57.123456789 America/New_York"}) do
+      insist { subject.get("@timestamp").time.to_i } == 1478221857
+      insist { subject.get("@timestamp").time.nsec } == 123456789
+    end
+  end
+
+  describe "precision => ns, ISO8601 with nanoseconds" do
+    config <<-CONFIG
+      filter {
+        date {
+          match => [ "mydate", "ISO8601" ]
+          precision => "ns"
+        }
+      }
+    CONFIG
+
+    sample({"mydate" => "2016-11-03T21:10:57.123456789Z"}) do
+      insist { subject.get("@timestamp").time.to_i } == 1478207457
+      insist { subject.get("@timestamp").time.nsec } == 123456789
+    end
+  end
+
+  describe "precision => ns, UNIX epoch string with nanoseconds" do
+    config <<-CONFIG
+      filter {
+        date {
+          match => [ "mydate", "UNIX" ]
+          precision => "ns"
+        }
+      }
+    CONFIG
+
+    sample({"mydate" => "1478207457.123456789"}) do
+      insist { subject.get("@timestamp").time.to_i } == 1478207457
+      insist { subject.get("@timestamp").time.nsec } == 123456789
+    end
+  end
+
+  describe "precision => ns, TAI64N full nanoseconds" do
+    config <<-CONFIG
+      filter {
+        date {
+          match => [ "mydate", "TAI64N" ]
+          precision => "ns"
+        }
+      }
+    CONFIG
+
+    sample({"mydate" => "4000000050d506482dbdf024"}) do
+      # 0x2dbdf024 = 767422500 ns
+      insist { subject.get("@timestamp").time.to_i } == 1356138046
+      insist { subject.get("@timestamp").time.nsec } == 767422500
+    end
+  end
+
+  describe "precision => ms (default) still truncates to milliseconds" do
+    config <<-CONFIG
+      filter {
+        date {
+          match => [ "mydate", "TAI64N" ]
+          precision => "ms"
+        }
+      }
+    CONFIG
+
+    sample({"mydate" => "4000000050d506482dbdf024"}) do
+      insist { subject.get("@timestamp").time.to_i } == 1356138046
+      insist { subject.get("@timestamp").time.nsec } == 767000000
+    end
+  end
 end

data/spec/fixtures/old_date_filter.rb

@@ -92,57 +92,56 @@ class LogStash::Filters::DateRuby < LogStash::Filters::Base
   #
   # Here's what you can use to parse dates and times:
   #
-  # [horizontal]
-  # y:: year
-  #   yyyy::: full year number. Example: `2015`.
-  #   yy::: two-digit year. Example: `15` for the year 2015.
+  # * `y`: year
+  # ** `yyyy`: full year number. Example: `2015`.
+  # ** `yy`: two-digit year. Example: `15` for the year 2015.
   #
-  # M:: month of the year
-  #   M::: minimal-digit month. Example: `1` for January and `12` for December.
-  #   MM::: two-digit month. zero-padded if needed. Example: `01` for January  and `12` for December
-  #   MMM::: abbreviated month text. Example: `Jan` for January. Note: The language used depends on your locale. See the `locale` setting for how to change the language.
-  #   MMMM::: full month text, Example: `January`. Note: The language used depends on your locale.
+  # * `M`: month of the year
+  # ** `M`: minimal-digit month. Example: `1` for January and `12` for December.
+  # ** `MM`: two-digit month. zero-padded if needed. Example: `01` for January  and `12` for December
+  # ** `MMM`: abbreviated month text. Example: `Jan` for January. Note: The language used depends on your locale. See the `locale` setting for how to change the language.
+  # ** `MMMM`: full month text, Example: `January`. Note: The language used depends on your locale.
   #
-  # d:: day of the month
-  #   d::: minimal-digit day. Example: `1` for the 1st of the month.
-  #   dd::: two-digit day, zero-padded if needed. Example: `01` for the 1st of the month.
+  # * `d`: day of the month
+  # ** `d`: minimal-digit day. Example: `1` for the 1st of the month.
+  # ** `dd`: two-digit day, zero-padded if needed. Example: `01` for the 1st of the month.
   #
-  # H:: hour of the day (24-hour clock)
-  #   H::: minimal-digit hour. Example: `0` for midnight.
-  #   HH::: two-digit hour, zero-padded if needed. Example: `00` for midnight.
+  # * `H`: hour of the day (24-hour clock)
+  # ** `H`: minimal-digit hour. Example: `0` for midnight.
+  # ** `HH`: two-digit hour, zero-padded if needed. Example: `00` for midnight.
   #
-  # m:: minutes of the hour (60 minutes per hour)
-  #   m::: minimal-digit minutes. Example: `0`.
-  #   mm::: two-digit minutes, zero-padded if needed. Example: `00`.
+  # * `m`: minutes of the hour (60 minutes per hour)
+  # ** `m`: minimal-digit minutes. Example: `0`.
+  # ** `mm`: two-digit minutes, zero-padded if needed. Example: `00`.
   #
-  # s:: seconds of the minute (60 seconds per minute)
-  #   s::: minimal-digit seconds. Example: `0`.
-  #   ss::: two-digit seconds, zero-padded if needed. Example: `00`.
+  # * `s`: seconds of the minute (60 seconds per minute)
+  # ** `s`: minimal-digit seconds. Example: `0`.
+  # ** `ss`: two-digit seconds, zero-padded if needed. Example: `00`.
   #
-  # S:: fraction of a second
-  #   *Maximum precision is milliseconds (`SSS`). Beyond that, zeroes are appended.*
-  #   S::: tenths of a second. Example:  `0` for a subsecond value `012`
-  #   SS::: hundredths of a second. Example:  `01` for a subsecond value `01`
-  #   SSS::: thousandths of a second. Example:  `012` for a subsecond value `012`
+  # * `S`: fraction of a second
+  # ** Maximum precision is milliseconds (`SSS`). Beyond that, zeroes are appended.
+  # ** `S`: tenths of a second. Example:  `0` for a subsecond value `012`
+  # ** `SS`: hundredths of a second. Example:  `01` for a subsecond value `01`
+  # ** `SSS`: thousandths of a second. Example:  `012` for a subsecond value `012`
   #
-  # Z:: time zone offset or identity
-  #   Z::: Timezone offset structured as HHmm (hour and minutes offset from Zulu/UTC). Example: `-0700`.
-  #   ZZ::: Timezone offset structured as HH:mm (colon in between hour and minute offsets). Example: `-07:00`.
-  #   ZZZ::: Timezone identity. Example: `America/Los_Angeles`. Note: Valid IDs are listed on the http://joda-time.sourceforge.net/timezones.html[Joda.org available time zones page].
+  # * `Z`: time zone offset or identity
+  # ** `Z`: Timezone offset structured as HHmm (hour and minutes offset from Zulu/UTC). Example: `-0700`.
+  # ** `ZZ`: Timezone offset structured as HH:mm (colon in between hour and minute offsets). Example: `-07:00`.
+  # ** `ZZZ`: Timezone identity. Example: `America/Los_Angeles`. Note: Valid IDs are listed on the http://joda-time.sourceforge.net/timezones.html[Joda.org available time zones page].
   #
-  # z:: time zone names. *Time zone names ('z') cannot be parsed.*
+  # * `z`: time zone names. *Time zone names (`z`) cannot be parsed.*
   #
-  # w:: week of the year
-  #   w::: minimal-digit week. Example: `1`.
-  #   ww::: two-digit week, zero-padded if needed. Example: `01`.
+  # * `w`: week of the year
+  # ** `w`: minimal-digit week. Example: `1`.
+  # ** `ww`: two-digit week, zero-padded if needed. Example: `01`.
   #
-  # D:: day of the year
+  # * `D`: day of the year
   #
-  # e:: day of the week (number)
+  # * `e`: day of the week (number)
   #
-  # E:: day of the week (text)
-  #   E, EE, EEE::: Abbreviated day of the week. Example:  `Mon`, `Tue`, `Wed`, `Thu`, `Fri`, `Sat`, `Sun`. Note: The actual language of this will depend on your locale.
-  #   EEEE::: The full text day of the week. Example: `Monday`, `Tuesday`, ... Note: The actual language of this will depend on your locale.
+  # * `E`: day of the week (text)
+  # ** `E`, `EE`, `EEE`: Abbreviated day of the week. Example:  `Mon`, `Tue`, `Wed`, `Thu`, `Fri`, `Sat`, `Sun`. Note: The actual language of this will depend on your locale.
+  # ** `EEEE`: The full text day of the week. Example: `Monday`, `Tuesday`, ... Note: The actual language of this will depend on your locale.
   #
   # For non-formatting syntax, you'll need to put single-quote characters around the value. For example, if you were parsing ISO8601 time, "2015-01-01T01:12:23" that little "T" isn't a valid time format, and you want to say "literally, a T", your format would be this: "yyyy-MM-dd'T'HH:mm:ss"
   #

metadata

@@ -1,13 +1,13 @@
 --- !ruby/object:Gem::Specification
 name: logstash-filter-date
 version: !ruby/object:Gem::Version
-  version: 3.1.16
+  version: 3.2.1
 platform: ruby
 authors:
 - Elastic
 bindir: bin
 cert_chain: []
-date: 2026-01-31 00:00:00.000000000 Z
+date: 2026-05-07 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: logstash-core-plugin-api