pysysops-filter-date 2.0.3

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 4a945e95fa172248c1620a80648b0434789a1b00
+  data.tar.gz: c6946268ff371a5b393edb85f58d1d91ecbc4a1b
+SHA512:
+  metadata.gz: 56372dd4aa79a9e2c6241a6a79ee74e36e1784adf1268dc28bfcb63b60fad75ecc7d894506e37abb33eacd8890a0928586bfb53814894c1ba91b9c50835b355a
+  data.tar.gz: 1eb72298f62f0d8c81de21c69e8aa1c8058aa1a9f5b3c6d6d621602251f3cc74a4f7948d82963628bf19224dc6544a34d5d426424745dcf7112bb647684277a6

data/CHANGELOG.md

@@ -0,0 +1,5 @@
+## 2.0.0
+ - Plugins were updated to follow the new shutdown semantic, this mainly allows Logstash to instruct input plugins to terminate gracefully, 
+   instead of using Thread.raise on the plugins' threads. Ref: https://github.com/elastic/logstash/pull/3895
+ - Dependency on logstash-core update to 2.0
+

data/CONTRIBUTORS

@@ -0,0 +1,32 @@
+The following is a list of people who have contributed ideas, code, bug
+reports, or in general have helped logstash along its way.
+
+Contributors:
+* Aaron Mildenstein (untergeek)
+* Bob Corsaro (dokipen)
+* Christian S. (squiddle)
+* Colin Surprenant (colinsurprenant)
+* Danny Berger (dpb587)
+* James Turnbull (jamtur01)
+* Jason Kendall (coolacid)
+* John E. Vincent (lusis)
+* Jonathan Van Eenwyk (jdve)
+* Jordan Sissel (jordansissel)
+* Kevin O'Connor (kjoconnor)
+* Kurt Hurtado (kurtado)
+* Mike Worth (MikeWorth)
+* Nick Ethier (nickethier)
+* Olivier Le Moal (olivierlemoal)
+* Pete Fritchman (fetep)
+* Philippe Weber (wiibaa)
+* Pier-Hugues Pellerin (ph)
+* Pierre Baillet (octplane)
+* Ralph Meijer (ralphm)
+* Richard Pijnenburg (electrical)
+* Suyog Rao (suyograo)
+* debadair
+
+Note: If you've sent us patches, bug reports, or otherwise contributed to
+Logstash, and you aren't on the list above and want to be, please let us know
+and we'll make sure you're here. Contributions from folks like you are what make
+open source awesome.

data/Gemfile

@@ -0,0 +1,2 @@
+source 'https://rubygems.org'
+gemspec

data/LICENSE

@@ -0,0 +1,13 @@
+Copyright (c) 2012–2015 Elasticsearch <http://www.elastic.co>
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

data/NOTICE.TXT

@@ -0,0 +1,5 @@
+Elasticsearch
+Copyright 2012-2015 Elasticsearch
+
+This product includes software developed by The Apache Software
+Foundation (http://www.apache.org/).

data/README.md

@@ -0,0 +1,89 @@
+# Logstash Plugin
+
+[![Build
+Status](http://build-eu-00.elastic.co/view/LS%20Plugins/view/LS%20Filters/job/logstash-plugin-filter-date-unit/badge/icon)](http://build-eu-00.elastic.co/view/LS%20Plugins/view/LS%20Filters/job/logstash-plugin-filter-date-unit/)
+
+This is a plugin for [Logstash](https://github.com/elastic/logstash).
+
+It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
+
+## Documentation
+
+Logstash provides infrastructure to automatically generate documentation for this plugin. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html. All plugin documentation are placed under one [central location](http://www.elastic.co/guide/en/logstash/current/).
+
+- For formatting code or config example, you can use the asciidoc `[source,ruby]` directive
+- For more asciidoc formatting tips, see the excellent reference here https://github.com/elastic/docs#asciidoc-guide
+
+## Need Help?
+
+Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/logstash discussion forum.
+
+## Developing
+
+### 1. Plugin Developement and Testing
+
+#### Code
+- To get started, you'll need JRuby with the Bundler gem installed.
+
+- Create a new plugin or clone and existing from the GitHub [logstash-plugins](https://github.com/logstash-plugins) organization. We also provide [example plugins](https://github.com/logstash-plugins?query=example).
+
+- Install dependencies
+```sh
+bundle install
+```
+
+#### Test
+
+- Update your dependencies
+
+```sh
+bundle install
+```
+
+- Run tests
+
+```sh
+bundle exec rspec
+```
+
+### 2. Running your unpublished Plugin in Logstash
+
+#### 2.1 Run in a local Logstash clone
+
+- Edit Logstash `Gemfile` and add the local plugin path, for example:
+```ruby
+gem "logstash-filter-awesome", :path => "/your/local/logstash-filter-awesome"
+```
+- Install plugin
+```sh
+bin/plugin install --no-verify
+```
+- Run Logstash with your plugin
+```sh
+bin/logstash -e 'filter {awesome {}}'
+```
+At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.
+
+#### 2.2 Run in an installed Logstash
+
+You can use the same **2.1** method to run your plugin in an installed Logstash by editing its `Gemfile` and pointing the `:path` to your local plugin development directory or you can build the gem and install it using:
+
+- Build your plugin gem
+```sh
+gem build logstash-filter-awesome.gemspec
+```
+- Install the plugin from the Logstash home
+```sh
+bin/plugin install /your/local/plugin/logstash-filter-awesome.gem
+```
+- Start Logstash and proceed to test the plugin
+
+## Contributing
+
+All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.
+
+Programming is not a required skill. Whatever you've seen about open source and maintainers or community members  saying "send patches or die" - you will not see that here.
+
+It is more important to the community that you are able to contribute.
+
+For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file.

data/lib/logstash/filters/date.rb

@@ -0,0 +1,296 @@
+# encoding: utf-8
+require "logstash/filters/base"
+require "logstash/namespace"
+require "logstash/timestamp"
+
+# The date filter is used for parsing dates from fields, and then using that
+# date or timestamp as the logstash timestamp for the event.
+#
+# For example, syslog events usually have timestamps like this:
+# [source,ruby]
+#     "Apr 17 09:32:01"
+#
+# You would use the date format `MMM dd HH:mm:ss` to parse this.
+#
+# The date filter is especially important for sorting events and for
+# backfilling old data. If you don't get the date correct in your
+# event, then searching for them later will likely sort out of order.
+#
+# In the absence of this filter, logstash will choose a timestamp based on the
+# first time it sees the event (at input time), if the timestamp is not already
+# set in the event. For example, with file input, the timestamp is set to the
+# time of each read.
+class LogStash::Filters::Date < LogStash::Filters::Base
+  if RUBY_ENGINE == "jruby"
+    JavaException = java.lang.Exception
+    UTC = org.joda.time.DateTimeZone.forID("UTC")
+  end
+
+  config_name "date"
+
+  # Specify a time zone canonical ID to be used for date parsing.
+  # The valid IDs are listed on the http://joda-time.sourceforge.net/timezones.html[Joda.org available time zones page].
+  # This is useful in case the time zone cannot be extracted from the value,
+  # and is not the platform default.
+  # If this is not specified the platform default will be used.
+  # Canonical ID is good as it takes care of daylight saving time for you
+  # For example, `America/Los_Angeles` or `Europe/Paris` are valid IDs.
+  # This field can be dynamic and include parts of the event using the `%{field}` syntax
+  config :timezone, :validate => :string
+
+  # Specify a locale to be used for date parsing using either IETF-BCP47 or POSIX language tag.
+  # Simple examples are `en`,`en-US` for BCP47 or `en_US` for POSIX.
+  #
+  # The locale is mostly necessary to be set for parsing month names (pattern with `MMM`) and
+  # weekday names (pattern with `EEE`).
+  #
+  # If not specified, the platform default will be used but for non-english platform default
+  # an english parser will also be used as a fallback mechanism.
+  config :locale, :validate => :string
+
+  # The date formats allowed are anything allowed by Joda-Time (java time
+  # library). You can see the docs for this format here:
+  #
+  # http://joda-time.sourceforge.net/apidocs/org/joda/time/format/DateTimeFormat.html[joda.time.format.DateTimeFormat]
+  #
+  # An array with field name first, and format patterns following, `[ field,
+  # formats... ]`
+  #
+  # If your time field has multiple possible formats, you can do this:
+  # [source,ruby]
+  #     match => [ "logdate", "MMM dd YYY HH:mm:ss",
+  #               "MMM  d YYY HH:mm:ss", "ISO8601" ]
+  #
+  # The above will match a syslog (rfc3164) or `iso8601` timestamp.
+  #
+  # There are a few special exceptions. The following format literals exist
+  # to help you save time and ensure correctness of date parsing.
+  #
+  # * `ISO8601` - should parse any valid ISO8601 timestamp, such as
+  #   `2011-04-19T03:44:01.103Z`
+  # * `UNIX` - will parse *float or int* value expressing unix time in seconds since epoch like 1326149001.132 as well as 1326149001
+  # * `UNIX_MS` - will parse **int** value expressing unix time in milliseconds since epoch like 1366125117000
+  # * `TAI64N` - will parse tai64n time values
+  #
+  # For example, if you have a field `logdate`, with a value that looks like
+  # `Aug 13 2010 00:03:44`, you would use this configuration:
+  # [source,ruby]
+  #     filter {
+  #       date {
+  #         match => [ "logdate", "MMM dd YYYY HH:mm:ss" ]
+  #       }
+  #     }
+  #
+  # If your field is nested in your structure, you can use the nested
+  # syntax `[foo][bar]` to match its value. For more information, please refer to
+  # <<logstash-config-field-references>>
+  config :match, :validate => :array, :default => []
+
+  # Store the matching timestamp into the given target field.  If not provided,
+  # default to updating the `@timestamp` field of the event.
+  config :target, :validate => :string, :default => "@timestamp"
+
+  # Append values to the `tags` field when there has been no
+  # successful match
+  config :tag_on_failure, :validate => :array, :default => ["_dateparsefailure"]
+
+  # LOGSTASH-34
+  DATEPATTERNS = %w{ y d H m s S }
+
+  public
+  def initialize(config = {})
+    super
+
+    @parsers = Hash.new { |h,k| h[k] = [] }
+  end # def initialize
+
+  public
+  def register
+    require "java"
+    if @match.length < 2
+      raise LogStash::ConfigurationError, I18n.t("logstash.agent.configuration.invalid_plugin_register",
+        :plugin => "filter", :type => "date",
+        :error => "The match setting should contains first a field name and at least one date format, current value is #{@match}")
+    end
+
+    locale = nil
+    if @locale
+      if @locale.include? '_'
+        @logger.warn("Date filter now use BCP47 format for locale, replacing underscore with dash")
+        @locale.gsub!('_','-')
+      end
+      locale = java.util.Locale.forLanguageTag(@locale)
+    end
+
+    @sprintf_timezone = @timezone && !@timezone.index("%{").nil?
+
+    setupMatcher(@config["match"].shift, locale, @config["match"] )
+  end
+
+  def setupMatcher(field, locale, value)
+    value.each do |format|
+      parsers = []
+      case format
+        when "ISO8601"
+          iso_parser = org.joda.time.format.ISODateTimeFormat.dateTimeParser
+          if @timezone && !@sprintf_timezone
+            iso_parser = iso_parser.withZone(org.joda.time.DateTimeZone.forID(@timezone))
+          else
+            iso_parser = iso_parser.withOffsetParsed
+          end
+          parsers << lambda { |date| iso_parser.parseMillis(date) }
+          #Fall back solution of almost ISO8601 date-time
+          almostISOparsers = [
+            org.joda.time.format.DateTimeFormat.forPattern("yyyy-MM-dd HH:mm:ss.SSSZ").getParser(),
+            org.joda.time.format.DateTimeFormat.forPattern("yyyy-MM-dd HH:mm:ss.SSS").getParser(),
+            org.joda.time.format.DateTimeFormat.forPattern("yyyy-MM-dd HH:mm:ss,SSSZ").getParser(),
+            org.joda.time.format.DateTimeFormat.forPattern("yyyy-MM-dd HH:mm:ss,SSS").getParser()
+          ].to_java(org.joda.time.format.DateTimeParser)
+          joda_parser = org.joda.time.format.DateTimeFormatterBuilder.new.append( nil, almostISOparsers ).toFormatter()
+          if @timezone && !@sprintf_timezone
+            joda_parser = joda_parser.withZone(org.joda.time.DateTimeZone.forID(@timezone))
+          else
+            joda_parser = joda_parser.withOffsetParsed
+          end
+          parsers << lambda { |date| joda_parser.parseMillis(date) }
+        when "UNIX" # unix epoch
+          parsers << lambda do |date|
+            raise "Invalid UNIX epoch value '#{date}'" unless /^\d+(?:\.\d+)?$/ === date || date.is_a?(Numeric)
+            (date.to_f * 1000).to_i
+          end
+        when "UNIX_MS" # unix epoch in ms
+          parsers << lambda do |date|
+            raise "Invalid UNIX epoch value '#{date}'" unless /^\d+$/ === date || date.is_a?(Numeric)
+            date.to_i
+          end
+        when "UNIX_NANO" # unix epoch in nanoseconds
+          parsers << lambda do |date|
+            raise "Invalid UNIX epoch value '#{date}'" unless /^\d+$/ === date || date.is_a?(Numeric)
+            (date.to_i / 1000000).to_i
+          end
+        when "TAI64N" # TAI64 with nanoseconds, -10000 accounts for leap seconds
+          parsers << lambda do |date|
+            # Skip leading "@" if it is present (common in tai64n times)
+            date = date[1..-1] if date[0, 1] == "@"
+            return (date[1..15].hex * 1000 - 10000)+(date[16..23].hex/1000000)
+          end
+        else
+          begin
+            format_has_year = format.match(/y|Y/)
+            joda_parser = org.joda.time.format.DateTimeFormat.forPattern(format)
+            if @timezone && !@sprintf_timezone
+              joda_parser = joda_parser.withZone(org.joda.time.DateTimeZone.forID(@timezone))
+            else
+              joda_parser = joda_parser.withOffsetParsed
+            end
+            if locale
+              joda_parser = joda_parser.withLocale(locale)
+            end
+            if @sprintf_timezone
+              parsers << lambda { |date , tz|
+                joda_parser.withZone(org.joda.time.DateTimeZone.forID(tz)).parseMillis(date)
+              }
+            end
+            parsers << lambda do |date|
+              return joda_parser.parseMillis(date) if format_has_year
+              now = Time.now
+              now_month = now.month
+              result = joda_parser.parseDateTime(date)
+              event_month = result.month_of_year.get
+
+              if (event_month == now_month)
+                result.with_year(now.year)
+              elsif (event_month == 12 && now_month == 1)
+                result.with_year(now.year-1)
+              elsif (event_month == 1 && now_month == 12)
+                result.with_year(now.year+1)
+              else
+                result.with_year(now.year)
+              end.get_millis
+            end
+
+            #Include a fallback parser to english when default locale is non-english
+            if !locale &&
+              "en" != java.util.Locale.getDefault().getLanguage() &&
+              (format.include?("MMM") || format.include?("E"))
+              en_joda_parser = joda_parser.withLocale(java.util.Locale.forLanguageTag('en-US'))
+              parsers << lambda { |date| en_joda_parser.parseMillis(date) }
+            end
+          rescue JavaException => e
+            raise LogStash::ConfigurationError, I18n.t("logstash.agent.configuration.invalid_plugin_register",
+              :plugin => "filter", :type => "date",
+              :error => "#{e.message} for pattern '#{format}'")
+          end
+      end
+
+      @logger.debug("Adding type with date config", :type => @type,
+                    :field => field, :format => format)
+      @parsers[field] << {
+        :parser => parsers,
+        :format => format
+      }
+    end
+  end
+
+  # def register
+
+  public
+  def filter(event)
+    @logger.debug? && @logger.debug("Date filter: received event", :type => event["type"])
+
+    @parsers.each do |field, fieldparsers|
+      @logger.debug? && @logger.debug("Date filter looking for field",
+                                      :type => event["type"], :field => field)
+      next unless event.include?(field)
+
+      fieldvalues = event[field]
+      fieldvalues = [fieldvalues] if !fieldvalues.is_a?(Array)
+      fieldvalues.each do |value|
+        next if value.nil?
+        begin
+          epochmillis = nil
+          success = false
+          last_exception = RuntimeError.new "Unknown"
+          fieldparsers.each do |parserconfig|
+            parserconfig[:parser].each do |parser|
+              begin
+                if @sprintf_timezone
+                  epochmillis = parser.call(value, event.sprintf(@timezone))
+                else
+                  epochmillis = parser.call(value)
+                end
+                success = true
+                break # success
+              rescue StandardError, JavaException => e
+                last_exception = e
+              end
+            end # parserconfig[:parser].each
+            break if success
+          end # fieldparsers.each
+
+          raise last_exception unless success
+
+          # Convert joda DateTime to a ruby Time
+          event[@target] = LogStash::Timestamp.at(epochmillis / 1000, (epochmillis % 1000) * 1000)
+
+          @logger.debug? && @logger.debug("Date parsing done", :value => value, :timestamp => event[@target])
+          filter_matched(event)
+        rescue StandardError, JavaException => e
+          @logger.warn("Failed parsing date from field", :field => field,
+                       :value => value, :exception => e.message,
+                       :config_parsers => fieldparsers.collect {|x| x[:format]}.join(','),
+                       :config_locale => @locale ? @locale : "default="+java.util.Locale.getDefault().toString()
+                       )
+          # Tag this event if we can't parse it. We can use this later to
+          # reparse+reindex logs if we improve the patterns given.
+          @tag_on_failure.each do |tag|
+            event["tags"] ||= []
+            event["tags"] << tag unless event["tags"].include?(tag)
+          end
+        end # begin
+      end # fieldvalue.each
+    end # @parsers.each
+
+    return event
+  end # def filter
+end # class LogStash::Filters::Date

data/logstash-filter-date.gemspec

@@ -0,0 +1,28 @@
+Gem::Specification.new do |s|
+
+  s.name            = 'pysysops-filter-date'
+  s.version         = '2.0.3'
+  s.licenses        = ['Apache License (2.0)']
+  s.summary         = "The date filter is used for parsing dates from fields, and then using that date or timestamp as the logstash timestamp for the event."
+  s.description     = "This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not a stand-alone program"
+  s.authors         = ["Elastic"]
+  s.email           = 'info@elastic.co'
+  s.homepage        = "http://www.elastic.co/guide/en/logstash/current/index.html"
+  s.require_paths = ["lib"]
+
+  # Files
+  s.files = Dir['lib/**/*','spec/**/*','vendor/**/*','*.gemspec','*.md','CONTRIBUTORS','Gemfile','LICENSE','NOTICE.TXT']
+
+  # Tests
+  s.test_files = s.files.grep(%r{^(test|spec|features)/})
+
+  # Special flag to let us know this is actually a logstash plugin
+  s.metadata = { "logstash_plugin" => "true", "logstash_group" => "filter" }
+
+  # Gem dependencies
+  s.add_runtime_dependency "logstash-core", ">= 2.0.0.beta2", "< 3.0.0"
+  s.add_runtime_dependency 'logstash-input-generator'
+  s.add_runtime_dependency 'logstash-codec-json'
+  s.add_runtime_dependency 'logstash-output-null'
+  s.add_development_dependency 'logstash-devutils'
+end

data/spec/filters/date_spec.rb

@@ -0,0 +1,551 @@
+require "logstash/devutils/rspec/spec_helper"
+require "logstash/filters/date"
+
+puts "Skipping date performance tests because this ruby is not jruby" if RUBY_ENGINE != "jruby"
+RUBY_ENGINE == "jruby" and describe LogStash::Filters::Date do
+
+  describe "giving an invalid match config, raise a configuration error" do
+    config <<-CONFIG
+      filter {
+        date {
+          match => [ "mydate"]
+          locale => "en"
+        }
+      }
+    CONFIG
+
+    sample "not_really_important" do
+      insist {subject}.raises LogStash::ConfigurationError
+    end
+
+  end
+
+  describe "parsing with ISO8601" do
+    config <<-CONFIG
+      filter {
+        date {
+          match => [ "mydate", "ISO8601" ]
+          locale => "en"
+          timezone => "UTC"
+        }
+      }
+    CONFIG
+
+    times = {
+      "2001-01-01T00:00:00-0800"         => "2001-01-01T08:00:00.000Z",
+      "1974-03-02T04:09:09-0800"         => "1974-03-02T12:09:09.000Z",
+      "2010-05-03T08:18:18+00:00"        => "2010-05-03T08:18:18.000Z",
+      "2004-07-04T12:27:27-00:00"        => "2004-07-04T12:27:27.000Z",
+      "2001-09-05T16:36:36+0000"         => "2001-09-05T16:36:36.000Z",
+      "2001-11-06T20:45:45-0000"         => "2001-11-06T20:45:45.000Z",
+      "2001-12-07T23:54:54Z"             => "2001-12-07T23:54:54.000Z",
+
+      # TODO: This test assumes PDT
+      #"2001-01-01T00:00:00.123"          => "2001-01-01T08:00:00.123Z",
+
+      "2010-05-03T08:18:18.123+00:00"    => "2010-05-03T08:18:18.123Z",
+      "2004-07-04T12:27:27.123-04:00"    => "2004-07-04T16:27:27.123Z",
+      "2001-09-05T16:36:36.123+0700"     => "2001-09-05T09:36:36.123Z",
+      "2001-11-06T20:45:45.123-0000"     => "2001-11-06T20:45:45.123Z",
+      "2001-12-07T23:54:54.123Z"         => "2001-12-07T23:54:54.123Z",
+      "2001-12-07T23:54:54,123Z"         => "2001-12-07T23:54:54.123Z",
+
+      #Almost ISO8601 support, with timezone
+
+      "2001-11-06 20:45:45.123-0000"     => "2001-11-06T20:45:45.123Z",
+      "2001-12-07 23:54:54.123Z"         => "2001-12-07T23:54:54.123Z",
+      "2001-12-07 23:54:54,123Z"         => "2001-12-07T23:54:54.123Z",
+
+      #Almost ISO8601 support, without timezone
+
+      "2001-11-06 20:45:45.123"     => "2001-11-06T20:45:45.123Z",
+      "2001-11-06 20:45:45,123"     => "2001-11-06T20:45:45.123Z",
+
+    }
+
+    times.each do |input, output|
+      sample("mydate" => input) do
+        begin
+          insist { subject["mydate"] } == input
+          insist { subject["@timestamp"].time } == Time.iso8601(output).utc
+        rescue
+          #require "pry"; binding.pry
+          raise
+        end
+      end
+    end # times.each
+  end
+
+  describe "parsing with java SimpleDateFormat syntax" do
+    config <<-CONFIG
+      filter {
+        date {
+          match => [ "mydate", "MMM dd HH:mm:ss Z" ]
+          locale => "en"
+        }
+      }
+    CONFIG
+
+    now = Time.now
+    year = now.year
+    require 'java'
+
+    times = {
+      "Nov 24 01:29:01 -0800" => "#{year}-11-24T09:29:01.000Z",
+    }
+    times.each do |input, output|
+      sample("mydate" => input) do
+        insist { subject["mydate"] } == input
+        insist { subject["@timestamp"].time } == Time.iso8601(output).utc
+      end
+    end # times.each
+  end
+
+  describe "parsing with UNIX" do
+    config <<-CONFIG
+      filter {
+        date {
+          match => [ "mydate", "UNIX" ]
+          locale => "en"
+        }
+      }
+    CONFIG
+
+    times = {
+      "0"          => "1970-01-01T00:00:00.000Z",
+      "1000000000" => "2001-09-09T01:46:40.000Z",
+
+      # LOGSTASH-279 - sometimes the field is a number.
+      0          => "1970-01-01T00:00:00.000Z",
+      1000000000 => "2001-09-09T01:46:40.000Z"
+    }
+    times.each do |input, output|
+      sample("mydate" => input) do
+        insist { subject["mydate"] } == input
+        insist { subject["@timestamp"].time } == Time.iso8601(output).utc
+      end
+    end # times.each
+
+    #Invalid value should not be evaluated to zero (String#to_i madness)
+    sample("mydate" => "%{bad_value}") do
+      insist { subject["mydate"] } == "%{bad_value}"
+      insist { subject["@timestamp"] } != Time.iso8601("1970-01-01T00:00:00.000Z").utc
+    end
+  end
+
+  describe "parsing microsecond-precise times with UNIX (#213)" do
+    config <<-CONFIG
+      filter {
+        date {
+          match => [ "mydate", "UNIX" ]
+          locale => "en"
+        }
+      }
+    CONFIG
+
+    sample("mydate" => "1350414944.123456") do
+      # Joda time only supports milliseconds :\
+      insist { subject.timestamp.time } == Time.iso8601("2012-10-16T12:15:44.123-07:00").utc
+    end
+
+    #Support float values
+    sample("mydate" => 1350414944.123456) do
+      insist { subject["mydate"] } == 1350414944.123456
+      insist { subject["@timestamp"].time } == Time.iso8601("2012-10-16T12:15:44.123-07:00").utc
+    end
+
+    #Invalid value should not be evaluated to zero (String#to_i madness)
+    sample("mydate" => "%{bad_value}") do
+      insist { subject["mydate"] } == "%{bad_value}"
+      insist { subject["@timestamp"] } != Time.iso8601("1970-01-01T00:00:00.000Z").utc
+    end
+  end
+
+  describe "parsing with UNIX_MS" do
+    config <<-CONFIG
+      filter {
+        date {
+          match => [ "mydate", "UNIX_MS" ]
+          locale => "en"
+        }
+      }
+    CONFIG
+
+    times = {
+      "0"          => "1970-01-01T00:00:00.000Z",
+      "456"          => "1970-01-01T00:00:00.456Z",
+      "1000000000123" => "2001-09-09T01:46:40.123Z",
+
+      # LOGSTASH-279 - sometimes the field is a number.
+      0          => "1970-01-01T00:00:00.000Z",
+      456          => "1970-01-01T00:00:00.456Z",
+      1000000000123 => "2001-09-09T01:46:40.123Z"
+    }
+    times.each do |input, output|
+      sample("mydate" => input) do
+        insist { subject["mydate"] } == input
+        insist { subject["@timestamp"].time } == Time.iso8601(output)
+      end
+    end # times.each
+  end
+
+  describe "parsing with UNIX_NANO" do
+    config <<-CONFIG
+      filter {
+        date {
+          match => [ "mydate", "UNIX_NANO" ]
+          locale => "en"
+        }
+      }
+    CONFIG
+
+    times = {
+      "0"          => "1970-01-01T00:00:00.000Z",
+      "4569999"          => "1970-01-01T00:00:00.004Z",
+      "456999936"          => "1970-01-01T00:00:00.456Z",
+      "1000000000123999936" => "2001-09-09T01:46:40.123Z",
+
+      # LOGSTASH-279 - sometimes the field is a number.
+      0          => "1970-01-01T00:00:00.000Z",
+      4569999         => "1970-01-01T00:00:00.004Z",
+      456999936          => "1970-01-01T00:00:00.456Z",
+      1000000000123999936 => "2001-09-09T01:46:40.123Z"
+    }
+    times.each do |input, output|
+      sample("mydate" => input) do
+        insist { subject["mydate"] } == input
+        insist { subject["@timestamp"].time } == Time.iso8601(output)
+      end
+    end # times.each
+  end
+
+  describe "failed parses should not cause a failure (LOGSTASH-641)" do
+    config <<-'CONFIG'
+      input {
+        generator {
+          lines => [
+            '{ "mydate": "this will not parse" }',
+            '{ }'
+          ]
+          codec => json
+          type => foo
+          count => 1
+        }
+      }
+      filter {
+        date {
+          match => [ "mydate", "MMM  d HH:mm:ss", "MMM dd HH:mm:ss" ]
+          locale => "en"
+        }
+      }
+      output {
+        null { }
+      }
+    CONFIG
+
+    agent do
+      # nothing to do, if this crashes it's an error..
+    end
+  end
+
+  describe "TAI64N support" do
+    config <<-'CONFIG'
+      filter {
+        date {
+          match => [ "t",  TAI64N ]
+          locale => "en"
+        }
+      }
+    CONFIG
+
+    # Try without leading "@"
+    sample("t" => "4000000050d506482dbdf024") do
+      insist { subject.timestamp.time } == Time.iso8601("2012-12-22T01:00:46.767Z").utc
+    end
+
+    # Should still parse successfully if it's a full tai64n time (with leading
+    # '@')
+    sample("t" => "@4000000050d506482dbdf024") do
+      insist { subject.timestamp.time } == Time.iso8601("2012-12-22T01:00:46.767Z").utc
+    end
+  end
+
+  describe "accept match config option with hash value (LOGSTASH-735)" do
+    config <<-CONFIG
+      filter {
+        date {
+          match => [ "mydate", "ISO8601" ]
+          locale => "en"
+        }
+      }
+    CONFIG
+
+    time = "2001-09-09T01:46:40.000Z"
+
+    sample("mydate" => time) do
+      insist { subject["mydate"] } == time
+      insist { subject["@timestamp"].time } == Time.iso8601(time).utc
+    end
+  end
+
+  describe "support deep nested field access" do
+    config <<-CONFIG
+      filter {
+        date {
+          match => [ "[data][deep]", "ISO8601" ]
+          locale => "en"
+        }
+      }
+    CONFIG
+
+    sample("data" => { "deep" => "2013-01-01T00:00:00.000Z" }) do
+      insist { subject["@timestamp"].time } == Time.iso8601("2013-01-01T00:00:00.000Z").utc
+    end
+  end
+
+  describe "failing to parse should not throw an exception" do
+    config <<-CONFIG
+      filter {
+        date {
+          match => [ "thedate", "yyyy/MM/dd" ]
+          locale => "en"
+        }
+      }
+    CONFIG
+
+    sample("thedate" => "2013/Apr/21") do
+      insist { subject["@timestamp"] } != "2013-04-21T00:00:00.000Z"
+    end
+  end
+
+   describe "success to parse should apply on_success config(add_tag,add_field...)" do
+    config <<-CONFIG
+      filter {
+        date {
+          match => [ "thedate", "yyyy/MM/dd" ]
+          add_tag => "tagged"
+        }
+      }
+    CONFIG
+
+    sample("thedate" => "2013/04/21") do
+      insist { subject["@timestamp"] } != "2013-04-21T00:00:00.000Z"
+      insist { subject["tags"] } == ["tagged"]
+    end
+  end
+
+   describe "failing to parse should not apply on_success config(add_tag,add_field...)" do
+    config <<-CONFIG
+      filter {
+        date {
+          match => [ "thedate", "yyyy/MM/dd" ]
+          add_tag => "tagged"
+        }
+      }
+    CONFIG
+
+    sample("thedate" => "2013/Apr/21") do
+      insist { subject["@timestamp"] } != "2013-04-21T00:00:00.000Z"
+      reject { subject["tags"] }.include? "tagged"
+    end
+  end
+
+  describe "failing to parse should apply tag_on_failure" do
+    config <<-CONFIG
+      filter {
+        date {
+          match => [ "thedate", "yyyy/MM/dd" ]
+          tag_on_failure => ["date_failed"]
+        }
+      }
+    CONFIG
+
+    sample("thedate" => "2013/Apr/21") do
+      insist { subject["@timestamp"] } != "2013-04-21T00:00:00.000Z"
+      insist { subject["tags"] }.include? "date_failed"
+    end
+  end
+
+  describe "parsing with timezone parameter" do
+    config <<-CONFIG
+      filter {
+        date {
+          match => ["mydate", "yyyy MMM dd HH:mm:ss"]
+          locale => "en"
+          timezone => "America/Los_Angeles"
+        }
+      }
+    CONFIG
+
+    require 'java'
+    times = {
+      "2013 Nov 24 01:29:01" => "2013-11-24T09:29:01.000Z",
+      "2013 Jun 24 01:29:01" => "2013-06-24T08:29:01.000Z",
+    }
+    times.each do |input, output|
+      sample("mydate" => input) do
+        insist { subject["mydate"] } == input
+        insist { subject["@timestamp"].time } == Time.iso8601(output).utc
+      end
+    end # times.each
+  end
+
+  describe "parsing with timezone from event" do
+    config <<-CONFIG
+      filter {
+        date {
+          match => ["mydate", "yyyy MMM dd HH:mm:ss"]
+          locale => "en"
+          timezone => "%{mytz}"
+        }
+      }
+    CONFIG
+
+    require 'java'
+    times = {
+      "2013 Nov 24 01:29:01" => "2013-11-24T09:29:01.000Z",
+      "2013 Jun 24 01:29:01" => "2013-06-24T08:29:01.000Z",
+    }
+    times.each do |input, output|
+      sample("mydate" => input, "mytz" => "America/Los_Angeles") do
+        insist { subject["mydate"] } == input
+        insist { subject["@timestamp"].time } == Time.iso8601(output).utc
+      end
+    end # times.each
+  end
+
+  describe "LOGSTASH-34 - Default year should be this year" do
+    config <<-CONFIG
+      filter {
+        date {
+          match => [ "message", "EEE MMM dd HH:mm:ss" ]
+          locale => "en"
+        }
+      }
+    CONFIG
+
+    sample "Sun Jun 02 20:38:03" do
+      insist { subject["@timestamp"].year } == Time.now.year
+    end
+  end
+
+  describe "fill last year if december events arrive in january" do
+    config <<-CONFIG
+      filter {
+        date {
+          match => [ "message", "MMM dd HH:mm:ss" ]
+          locale => "en"
+          timezone => "UTC"
+        }
+      }
+    CONFIG
+
+    sample "Dec 31 23:59:00" do
+      logstash_time = Time.utc(2014,1,1,00,30,50)
+      expect(Time).to receive(:now).twice.and_return(logstash_time)
+      insist { subject["@timestamp"].year } == 2013
+    end
+  end
+
+  describe "fill next year if january events arrive in december" do
+    config <<-CONFIG
+      filter {
+        date {
+          match => [ "message", "MMM dd HH:mm:ss" ]
+          locale => "en"
+          timezone => "UTC"
+        }
+      }
+    CONFIG
+
+    sample "Jan 01 01:00:00" do
+      logstash_time = Time.utc(2013,12,31,23,59,50)
+      expect(Time).to receive(:now).twice.and_return(logstash_time)
+      insist { subject["@timestamp"].year } == 2014
+    end
+  end
+
+  describe "Supporting locale only" do
+    config <<-CONFIG
+      filter {
+        date {
+          match => [ "message", "dd MMMM yyyy" ]
+          locale => "fr"
+          timezone => "UTC"
+        }
+      }
+    CONFIG
+
+    sample "14 juillet 1789" do
+      insist { subject["@timestamp"].time } == Time.iso8601("1789-07-14T00:00:00.000Z").utc
+    end
+  end
+
+  describe "Supporting locale+country in BCP47" do
+    config <<-CONFIG
+      filter {
+        date {
+          match => [ "message", "dd MMMM yyyy" ]
+          locale => "fr-FR"
+          timezone => "UTC"
+        }
+      }
+    CONFIG
+
+    sample "14 juillet 1789" do
+      insist { subject["@timestamp"].time } == Time.iso8601("1789-07-14T00:00:00.000Z").utc
+    end
+  end
+
+  describe "Supporting locale+country in POSIX (internally replace '_' by '-')" do
+    config <<-CONFIG
+      filter {
+        date {
+          match => [ "message", "dd MMMM yyyy" ]
+          locale => "fr_FR"
+          timezone => "UTC"
+        }
+      }
+    CONFIG
+
+    sample "14 juillet 1789" do
+      insist { subject["@timestamp"].time } == Time.iso8601("1789-07-14T00:00:00.000Z").utc
+    end
+  end
+
+  describe "http dates" do
+
+    config <<-'CONFIG'
+      filter {
+        date {
+          match => [ "timestamp", "dd/MMM/yyyy:HH:mm:ss Z" ]
+          locale => "en"
+        }
+      }
+    CONFIG
+
+    sample("timestamp" => "25/Mar/2013:20:33:56 +0000") do
+      insist { subject["@timestamp"].time } == Time.iso8601("2013-03-25T20:33:56.000Z")
+    end
+  end
+
+  describe "Support fallback to english for non-english default locale" do
+    default_locale = java.util.Locale.getDefault()
+    #Override default locale with non-english
+    java.util.Locale.setDefault(java.util.Locale.forLanguageTag('fr-FR'))
+    config <<-CONFIG
+      filter {
+        date {
+          match => [ "message", "dd MMMM yyyy" ]
+          timezone => "UTC"
+        }
+      }
+    CONFIG
+
+    sample "01 September 2014" do
+      insist { subject["@timestamp"].time } == Time.iso8601("2014-09-01T00:00:00.000Z").utc
+    end
+    #Restore default locale
+    java.util.Locale.setDefault(default_locale)
+  end
+end

metadata

@@ -0,0 +1,131 @@
+--- !ruby/object:Gem::Specification
+name: pysysops-filter-date
+version: !ruby/object:Gem::Version
+  version: 2.0.3
+platform: ruby
+authors:
+- Elastic
+autorequire:
+bindir: bin
+cert_chain: []
+date: 2015-12-02 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: 2.0.0.beta2
+    - - <
+      - !ruby/object:Gem::Version
+        version: 3.0.0
+  name: logstash-core
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: 2.0.0.beta2
+    - - <
+      - !ruby/object:Gem::Version
+        version: 3.0.0
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  name: logstash-input-generator
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  name: logstash-codec-json
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  name: logstash-output-null
+  prerelease: false
+  type: :runtime
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  name: logstash-devutils
+  prerelease: false
+  type: :development
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+description: This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not a stand-alone program
+email: info@elastic.co
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- CONTRIBUTORS
+- Gemfile
+- LICENSE
+- NOTICE.TXT
+- README.md
+- lib/logstash/filters/date.rb
+- logstash-filter-date.gemspec
+- spec/filters/date_spec.rb
+homepage: http://www.elastic.co/guide/en/logstash/current/index.html
+licenses:
+- Apache License (2.0)
+metadata:
+  logstash_plugin: 'true'
+  logstash_group: filter
+post_install_message:
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project:
+rubygems_version: 2.4.5
+signing_key:
+specification_version: 4
+summary: The date filter is used for parsing dates from fields, and then using that date or timestamp as the logstash timestamp for the event.
+test_files:
+- spec/filters/date_spec.rb