logstash-input-elf-se 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: b13ec6985e00a56f332af3df61d47279cd5c0964
+  data.tar.gz: a91038100b51d6e6f4247a3ca40c0affc3b586f5
+SHA512:
+  metadata.gz: c989d2e3d6fd0c4f61f88034b6124a1459fcbeec89e339d10b815e5c02b867c195c8047f6c410a95b051aa5919fe765a23f7e928d7f7e3959c0cbb8b0593abfa
+  data.tar.gz: 1943274c8b29246d09f39d46029a9378afd287f5d96e82decb2dc693e56f11baa2907c9fabdfd2741c0d6e701b75087a35ec78b45a940fbfcc4f469db5914482

data/CHANGELOG.md

@@ -0,0 +1,2 @@
+## 0.1.0
+  - Plugin created with the logstash plugin generator

data/CONTRIBUTORS

@@ -0,0 +1,10 @@
+The following is a list of people who have contributed ideas, code, bug
+reports, or in general have helped logstash along its way.
+
+Contributors:
+* Sid - siddharatha.n@gmail.com
+
+Note: If you've sent us patches, bug reports, or otherwise contributed to
+Logstash, and you aren't on the list above and want to be, please let us know
+and we'll make sure you're here. Contributions from folks like you are what make
+open source awesome.

data/DEVELOPER.md

@@ -0,0 +1,2 @@
+# logstash-input-elf-se
+Example input plugin. This should help bootstrap your effort to write your own input plugin!

data/Gemfile

@@ -0,0 +1,3 @@
+source 'https://rubygems.org'
+gemspec
+

data/LICENSE

@@ -0,0 +1,11 @@
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

data/README.md

@@ -0,0 +1,86 @@
+# Logstash Plugin
+
+This is a plugin for [Logstash](https://github.com/elastic/logstash).
+
+It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
+
+## Documentation
+
+Logstash provides infrastructure to automatically generate documentation for this plugin. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html. All plugin documentation are placed under one [central location](http://www.elastic.co/guide/en/logstash/current/).
+
+- For formatting code or config example, you can use the asciidoc `[source,ruby]` directive
+- For more asciidoc formatting tips, see the excellent reference here https://github.com/elastic/docs#asciidoc-guide
+
+## Need Help?
+
+Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/logstash discussion forum.
+
+## Developing
+
+### 1. Plugin Developement and Testing
+
+#### Code
+- To get started, you'll need JRuby with the Bundler gem installed.
+
+- Create a new plugin or clone and existing from the GitHub [logstash-plugins](https://github.com/logstash-plugins) organization. We also provide [example plugins](https://github.com/logstash-plugins?query=example).
+
+- Install dependencies
+```sh
+bundle install
+```
+
+#### Test
+
+- Update your dependencies
+
+```sh
+bundle install
+```
+
+- Run tests
+
+```sh
+bundle exec rspec
+```
+
+### 2. Running your unpublished Plugin in Logstash
+
+#### 2.1 Run in a local Logstash clone
+
+- Edit Logstash `Gemfile` and add the local plugin path, for example:
+```ruby
+gem "logstash-filter-awesome", :path => "/your/local/logstash-filter-awesome"
+```
+- Install plugin
+```sh
+bin/logstash-plugin install --no-verify
+```
+- Run Logstash with your plugin
+```sh
+bin/logstash -e 'filter {awesome {}}'
+```
+At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.
+
+#### 2.2 Run in an installed Logstash
+
+You can use the same **2.1** method to run your plugin in an installed Logstash by editing its `Gemfile` and pointing the `:path` to your local plugin development directory or you can build the gem and install it using:
+
+- Build your plugin gem
+```sh
+gem build logstash-filter-awesome.gemspec
+```
+- Install the plugin from the Logstash home
+```sh
+bin/logstash-plugin install /your/local/plugin/logstash-filter-awesome.gem
+```
+- Start Logstash and proceed to test the plugin
+
+## Contributing
+
+All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.
+
+Programming is not a required skill. Whatever you've seen about open source and maintainers or community members  saying "send patches or die" - you will not see that here.
+
+It is more important to the community that you are able to contribute.
+
+For more information about contributing, see the [CONTRIBUTING](https://github.com/elastic/logstash/blob/master/CONTRIBUTING.md) file.

data/lib/logstash/inputs/elf-se.rb

@@ -0,0 +1,150 @@
+# encoding: utf-8
+require 'logstash/inputs/base'
+require 'logstash/namespace'
+require_relative 'sfdc_elf/client_with_streaming_support'
+require_relative 'sfdc_elf/queue_util'
+require_relative 'sfdc_elf/state_persistor'
+require_relative 'sfdc_elf/scheduler'
+
+# This plugin enables Salesforce customers to load EventLogFile(ELF) data from their Force.com orgs. The plugin will
+# handle downloading ELF CSV file, parsing them, and handling any schema changes transparently.
+class LogStash::Inputs::SfdcElf < LogStash::Inputs::Base
+  LOG_KEY        = 'SFDC'
+  RETRY_ATTEMPTS = 3
+
+  config_name 'sfdc_elf'
+  default :codec, 'plain'
+
+  # Username to your Force.com organization.
+  config :username, validate: :string, required: true
+
+  # Password to your Force.com organization.
+  config :password, validate: :password, required: true
+
+  # Client id to your Force.com organization.
+  config :client_id, validate: :password, required: true
+
+  # Client secret to your Force.com organization.
+  config :client_secret, validate: :password, required: true
+
+  # The host to use for OAuth2 authentication.
+  config :host, validate: :string, default: 'login.salesforce.com'
+
+
+  config :eventtypesstring, validate: :string, default: "'ApexExecution','ApexSoap','API','BulkApi','Dashboard','LightningError','LightningInteraction','LightningPageView','LightningPerformance','LoginAs','Login','Logout','MetadataApiOperation','Report','RestApi','URI','VisualforceRequest','WaveChange','WaveInteraction','WavePerformance'"
+
+  # Only needed when your Force.com organization requires it.
+  # Security token to you Force.com organization, can be found in  My Settings > Personal > Reset My Security Token.
+  # Then it will take you to "Reset My Security Token" page, and click on the "Reset Security Token" button. The token
+  # will be emailed to you.
+  config :security_token, validate: :password, default: ''
+
+  # The path to be use to store the .sfdc_info_logstash state persistor file. You set the path like so, `~/SomeDirectory` Paths must be
+  # absolute and cannot be relative.
+  config :path, validate: :string, default: Dir.home
+
+  # Specify how often the plugin should grab new data in terms of minutes.
+  config :poll_interval_in_minutes, validate: [*1..(24 * 60)], default: (24 * 60)
+
+
+  # The first part of logstash pipeline is register, where all instance variables are initialized.
+
+  public
+  def register
+    # Initialize the client.
+    @client = ClientWithStreamingSupport.new
+    @client.client_id     = @client_id.value
+    @client.client_secret = @client_secret.value
+    @client.host          = @host
+    @client.version       = '46.0'
+
+    # Authenticate the client
+    @logger.info("#{LOG_KEY}: tyring to authenticate client")
+    @client.retryable_authenticate(username: @username,
+                                   password: @password.value + @security_token.value,
+                                   retry_attempts: RETRY_ATTEMPTS)
+    @logger.info("#{LOG_KEY}: authenticating succeeded")
+
+    # Save org id to distinguish between multiple orgs.
+    @org_id = @client.query('select id from Organization')[0]['Id']
+
+    # Set up time interval for forever while loop.
+    @poll_interval_in_seconds = @poll_interval_in_minutes * 60
+
+    # Handel the @path config passed by the user. If path does not exist then set @path to home directory.
+    verify_path
+
+    # Handel parsing the data into event objects and enqueue it to the queue.
+    @queue_util = QueueUtil.new
+
+    # Handel when to schedule the next process based on the @poll_interval_in_hours config.
+    @scheduler = Scheduler.new(@poll_interval_in_seconds)
+
+    # Handel state of the plugin based on the read and writes of LogDates to the .sdfc_info_logstash file.
+    @state_persistor = StatePersistor.new(@path, @org_id)
+
+    # Grab the last indexed log date.
+    @last_indexed_log_date = @state_persistor.get_last_indexed_log_date
+    @logger.info("#{LOG_KEY}: @last_indexed_log_date =  #{@last_indexed_log_date}")
+  end  # def register
+
+
+
+
+  # The second stage of Logstash pipeline is run, where it expects to parse your data into event objects and then pass
+  # it into the queue to be used in the rest of the pipeline.
+
+  public
+  def run(queue)
+    @scheduler.schedule do
+      # Line for readable log statements.
+      @logger.info('---------------------------------------------------')
+
+      # Grab a list of SObjects, specifically EventLogFiles.
+      
+      soql_expr= "SELECT Id, EventType, Logfile, LogDate, LogFileLength, LogFileFieldTypes
+      FROM EventLogFile
+      WHERE LogDate > #{@last_indexed_log_date} and EventType in (#{@eventtypesstring}) and Sequence>0 and Interval='Hourly'  ORDER BY LogDate ASC"   
+
+      query_result_list = @client.retryable_query(username: @username,
+                                                  password: @password.value + @security_token.value,
+                                                  retry_attempts: RETRY_ATTEMPTS,
+                                                  soql_expr: soql_expr)
+
+      @logger.info("#{LOG_KEY}: query result size = #{query_result_list.size}")
+
+      if !query_result_list.empty?
+        # query_result_list is in ascending order based on the LogDate, so grab the last one of the list and save the
+        # LogDate to @last_read_log_date and .sfdc_info_logstash
+        @last_indexed_log_date = query_result_list.last.LogDate.strftime('%FT%T.%LZ')
+
+        # TODO: grab tempfiles here!!
+
+        # Overwrite the .sfdc_info_logstash file with the @last_read_log_date.
+        # Note: we currently do not support deduplication, but will implement it soon.
+        # TODO: need to implement deduplication
+        # TODO: might have to move this after enqueue_events(), in case of a crash in between.
+        # TODO: can do all @state_persistor calls after the if statement
+        @state_persistor.update_last_indexed_log_date(@last_indexed_log_date)
+
+        # Creates events from query_result_list, then simply append the events to the queue.
+        @queue_util.enqueue_events(query_result_list, queue, @client)
+      end
+    end # do loop
+  end # def run
+
+
+
+
+  # Handel the @path variable passed by the user. If path does not exist then set @path to home directory.
+
+  private
+  def verify_path
+    # Check if the path exist, if not then set @path to home directory.
+    unless File.directory?(@path)
+      @logger.warn("#{LOG_KEY}: provided path does not exist or is invalid. path=#{@path}")
+      @path = Dir.home
+    end
+    @logger.info("#{LOG_KEY}: path = #{@path}")
+  end
+end # class LogStash::inputs::File

data/lib/logstash/inputs/sfdc_elf/client_with_streaming_support.rb

@@ -0,0 +1,61 @@
+# encoding: utf-8
+require 'databasedotcom'
+
+# This class subclasses Databasedotcom Client object and added steaming
+# downloading and retryable authentication and retryable query.
+class ClientWithStreamingSupport < Databasedotcom::Client
+  # Constants
+  # LOG_KEY        = 'SFDC - ClientWithStreamingSupport'
+  #
+  # def initialize
+  #   @logger = Cabin::Channel.get(LogStash)
+  # end
+
+  def streaming_download(path, output_stream)
+    connection = Net::HTTP.new(URI.parse(instance_url).host, 443)
+    connection.use_ssl = true
+    encoded_path = URI.escape(path)
+
+    req = Net::HTTP::Get.new(encoded_path, 'Authorization' => "OAuth #{oauth_token}")
+    connection.request(req) do |response|
+      raise SalesForceError.new(response) unless response.is_a?(Net::HTTPSuccess)
+      response.read_body do |chunk|
+        output_stream.write chunk
+      end
+    end
+  end
+
+  # This helper method is called whenever we need to initaialize the client
+  # object or whenever the client token expires. It will attempt 3 times
+  # with a 30 second delay between each retry. On the 3th try, if it fails the
+  # exception will be raised.
+
+  def retryable_authenticate(options = {})
+    1.upto(options[:retry_attempts]) do |count|
+      begin
+        # If exception is not thrown, then break out of loop.
+        authenticate(username: options[:username], password: options[:password])
+        break
+      rescue StandardError => e
+        # Sleep for 30 seconds 2 times. On the 3th time if it fails raise the exception without sleeping.
+        if (count == options[:retry_attempts])
+          raise e
+        else
+          sleep(30)
+        end
+      end
+    end
+  end # def authenticate
+
+
+  def retryable_query(options = {})
+    query(options[:soql_expr])
+  rescue Databasedotcom::SalesForceError => e
+    # Session has expired. Force user logout, then re-authenticate.
+    if e.message == 'Session expired or invalid'
+      retryable_authenticate(options)
+    else
+      raise e
+    end
+  end # def retryable_query
+end # ClientWithStreamingSupport

data/lib/logstash/inputs/sfdc_elf/queue_util.rb

@@ -0,0 +1,176 @@
+# encoding: utf-8
+require 'csv'
+require 'resolv'
+
+# Handel parsing data into event objects and then enqueue all of the events to the queue.
+class QueueUtil
+  # Constants
+  LOG_KEY        = 'SFDC - QueueUtil'
+  SEPARATOR      = ','
+  QUOTE_CHAR     = '"'
+
+  # Zip up the tempfile, which is a CSV file, and the field types, so that when parsing the CSV file we can accurately
+  # convert each field to its respective type. Like Integers and Booleans.
+  EventLogFile = Struct.new(:field_types, :temp_file, :event_type)
+
+
+  def initialize
+    @logger = Cabin::Channel.get(LogStash)
+  end
+
+
+
+
+  # Given a list of query result's, iterate through it and grab the CSV file associated with it. Then parse the CSV file
+  # line by line and generating the event object for it. Then enqueue it.
+
+  public
+  def enqueue_events(query_result_list, queue, client)
+    @logger.info("#{LOG_KEY}: enqueue events")
+
+    # Grab a list of Tempfiles that contains CSV file data.
+    event_log_file_records = get_event_log_file_records(query_result_list, client)
+
+    # Iterate though each record.
+    event_log_file_records.each do |elf|
+      begin
+        # Create local variable to simplify & make code more readable.
+        tmp = elf.temp_file
+
+        # Get the schema from the first line in the tempfile. It will be in CSV format so we parse it, and it will
+        # return an array.
+        schema = CSV.parse_line(tmp.readline, col_sep: SEPARATOR, quote_char: QUOTE_CHAR)
+
+        # Loop through tempfile, line by line.
+        tmp.each_line do |line|
+          # Parse the current line, it will return an string array.
+          string_array = CSV.parse_line(line, col_sep: SEPARATOR, quote_char: QUOTE_CHAR)
+
+          # Convert the string array into its corresponding type array.
+          data = string_to_type_array(string_array, elf.field_types)
+
+          # create_event will return a event object.
+          queue << create_event(schema, data, elf.event_type)
+        end
+      ensure
+        # Close tmp file and unlink it, doing this will delete the actual tempfile.
+        tmp.close
+        tmp.unlink
+      end
+    end # do loop, tempfile_list
+  end # def create_event_list
+
+
+
+
+  # Convert the given string array to its corresponding type array and return it.
+
+  private
+  def string_to_type_array(string_array, field_types)
+    data = []
+
+    field_types.each_with_index do |type, i|
+      case type
+        when 'Number'
+          data[i] = (string_array[i].empty?) ? nil : string_array[i].to_f
+        when 'Boolean'
+          data[i] = (string_array[i].empty?) ? nil : (string_array[i] == '0')
+        when 'IP'
+          data[i] = valid_ip(string_array[i]) ? string_array[i] : nil
+        else # 'String', 'Id', 'EscapedString', 'Set'
+          data[i] = (string_array[i].empty?) ? nil : string_array[i]
+      end
+    end # do loop
+
+    data
+  end # convert_string_to_type
+
+
+
+  # Check if the given ip address is truely an ip address.
+
+  private
+  def valid_ip(ip)
+    ip =~ Resolv::IPv4::Regex ? true : false
+  end
+
+
+
+
+  # Bases on the schema and data, we create the event object. At any point if the data is nil we simply dont add
+  # the data to the event object. Special handling is needed when the schema 'TIMESTAMP' occurs, then the data
+  # associated with it needs to be converted into a LogStash::Timestamp.
+
+  private
+  def create_event(schema, data, event_type)
+    # Initaialize event to be used. @timestamp and @version is automatically added
+    event = LogStash::Event.new
+
+    # Add column data pair to event.
+    data.each_index do |i|
+      # Grab current key.
+      schema_name = schema[i]
+
+      # Handle when field_name is 'TIMESTAMP', Change the @timestamp field to the actual time on the CSV file,
+      # but convert it to iso8601.
+      if schema_name == 'TIMESTAMP'
+        epochmillis = DateTime.parse(data[i]).to_time.to_f
+        event.timestamp = LogStash::Timestamp.at(epochmillis)
+      end
+
+      # Allow Elasticsearch index's have to types set to EventType.
+      event['type'] = event_type.downcase
+
+      # Add the schema data pair to event object.
+      if data[i] != nil
+        event[schema_name] = data[i]
+      end
+    end
+
+    # Return the event
+    event
+  end # def create_event
+
+
+
+
+  # This helper method takes as input a list/collection of SObjects which each contains a path to their respective CSV
+  # files. The path is stored in the LogFile field. Using that path, we are able to grab the actual CSV file via
+  # @client.http_get method.
+  #
+  # After grabbing the CSV file we then store them using the standard Tempfile library. Tempfile will create a unique
+  # file each time using 'sfdc_elf_tempfile' as the prefix and finally we will be returning a list of Tempfile object,
+  # where the user can read the Tempfile and then close it and unlink it, which will delete the file.
+
+  public
+  def get_event_log_file_records(query_result_list, client)
+    @logger.info("#{LOG_KEY}: generating tempfile list")
+    result = []
+    query_result_list.each do |event_log_file|
+      # Get the path of the CSV file from the LogFile field, then stream the data to the .write method of the Tempfile
+      tmp = Tempfile.new('sfdc_elf_tempfile')
+      client.streaming_download(event_log_file.LogFile, tmp)
+
+      # Flushing will write the buffer into the Tempfile itself.
+      tmp.flush
+
+      # Rewind will move the file pointer from the end to the beginning of the file, so that users can simple
+      # call the Read method.
+      tmp.rewind
+
+      # Append the EventLogFile object into the result list
+      field_types = event_log_file.LogFileFieldTypes.split(',')
+      result << EventLogFile.new(field_types, tmp, event_log_file.EventType)
+
+      # Log the info from event_log_file object.
+      @logger.info("  #{LOG_KEY}: Id = #{event_log_file.Id}")
+      @logger.info("  #{LOG_KEY}: EventType = #{event_log_file.EventType}")
+      @logger.info("  #{LOG_KEY}: LogFile = #{event_log_file.LogFile}")
+      @logger.info("  #{LOG_KEY}: LogDate = #{event_log_file.LogDate}")
+      @logger.info("  #{LOG_KEY}: LogFileLength = #{event_log_file.LogFileLength}")
+      @logger.info("  #{LOG_KEY}: LogFileFieldTypes = #{event_log_file.LogFileFieldTypes}")
+      @logger.info('  ......................................')
+    end
+    result
+  end # def get_event_log_file_records
+end # QueueUtil

data/lib/logstash/inputs/sfdc_elf/scheduler.rb

@@ -0,0 +1,73 @@
+# encoding: utf-8
+
+# Handel when to schedule the next process based on the poll interval specified. The poll interval provided has to be
+# in seconds.
+class Scheduler
+  LOG_KEY = 'SFDC - Scheduler'
+
+  def initialize(poll_interval_in_seconds)
+    @logger = Cabin::Channel.get(LogStash)
+    @poll_interval_in_seconds = poll_interval_in_seconds
+  end
+
+
+
+
+  # In a forever loop, run the block provided then sleep based on the poll interval and repeat.
+
+  public
+  def schedule(&block)
+    # Grab the current time and one @interval to it so that the while loop knows when it need to compute again.
+    next_schedule_time = Time.now + @poll_interval_in_seconds
+
+    # sleep until start time
+    loop do
+      block.call
+
+      # Depending on the next_schedule_time and the time taking the compute the code above,
+      # sleep this loop and adjust the next_schedule_time.
+      @logger.info("#{LOG_KEY}: next_schedule_time = #{next_schedule_time}")
+      next_schedule_time = stall_schedule(next_schedule_time)
+    end
+  end
+
+
+
+
+  # Given as input the next schedule time, stall_schedule() will decide if we need to sleep until the next
+  # schedule time or skip sleeping because of missing the next schedule time.
+  #
+  # For both examples, the time interval is 1 hour.
+  # Example 1:
+  # started time       = 1:00pm
+  # next_schedule_time = 2:00pm
+  # current_time       = 1:30pm
+  # In this example you will need to sleep for 30 mins, so you will be on schedule.
+  #
+  # Example 2:
+  # started time       = 1:00pm
+  # next_schedule_time = 2:00pm
+  # current_time       = 2:30pm
+  # In this example you will not be allowed to sleep, and will proceed to compute again since you missed the
+  # schedule time.
+
+  public
+  def stall_schedule(next_schedule_time)
+    current_time = Time.now
+    @logger.info("#{LOG_KEY}: time before sleep  = #{current_time}")
+
+    # Example 2 case from above.
+    if current_time > next_schedule_time
+      @logger.info("#{LOG_KEY}: missed next schedule time, proceeding to next task without sleeping")
+      next_schedule_time += @poll_interval_in_seconds while current_time > next_schedule_time
+
+      # Example 1 case from above.
+    else
+      @logger.info("#{LOG_KEY}: sleeping for #{(next_schedule_time - current_time)} seconds")
+      sleep(next_schedule_time - current_time)
+      next_schedule_time += @poll_interval_in_seconds
+    end
+    @logger.info("#{LOG_KEY} time after sleep   = #{Time.now}")
+    next_schedule_time
+  end # def determine_loop_stall
+end # Scheduler

data/lib/logstash/inputs/sfdc_elf/state_persistor.rb

@@ -0,0 +1,49 @@
+# encoding: utf-8
+
+# Handel what the next procedure should be based on the .sfdc_info_logstash file. States proceed via reading and
+# writing LogDates to the .sfdc_info_logstash file.
+class StatePersistor
+  LOG_KEY        = 'SFDC - StatePersistor'
+  FILE_PREFIX    = 'sfdc_info_logstash'
+  DEFAULT_TIME   = '0001-01-01T00:00:00Z'
+
+  def initialize(base_path, org_id)
+    @logger = Cabin::Channel.get(LogStash)
+    @path_with_file_name = "#{base_path}/.#{FILE_PREFIX}_#{org_id}"
+  end
+
+
+  # Read the last indexed LogDate from .sfdc_info_logstash file and return it. If the .sfdc_info_logstash file does
+  # not exist then create the file and write DEFAULT_TIME to it using update_last_indexed_log_date() method.
+
+  public
+  def get_last_indexed_log_date
+    # Read from .sfdc_info_logstash if it exists, otherwise load @last_read_log_date with DEFAULT_TIME.
+    if File.exist?(@path_with_file_name)
+      # Load last read LogDate from .sfdc_info_logstash.
+      @logger.info("#{LOG_KEY}: .#{@path_with_file_name} does exist, read and return the time on it.")
+      File.read(@path_with_file_name)
+    else
+      # Load default time to ensure getting all possible EventLogFiles from oldest to current. Also
+      # create .sfdc_info_logstash file
+      @logger.info("#{LOG_KEY}: .sfdc_info_logstash does not exist and loaded DEFAULT_TIME to @last_read_instant")
+      update_last_indexed_log_date(DEFAULT_TIME)
+      DEFAULT_TIME
+    end
+  end
+
+
+
+
+  # Take as input a date sting that is in iso8601 format, then overwrite .sfdc_info_logstash with the date string,
+  # because of the 'w' flag used with the File class.
+
+  public
+  def update_last_indexed_log_date(date)
+    @logger.info("#{LOG_KEY}: overwriting #{@path_with_file_name} with #{date}")
+    f = File.open(@path_with_file_name, 'w')
+    f.write(date)
+    f.flush
+    f.close
+  end
+end

data/logstash-input-elf-se.gemspec

@@ -0,0 +1,27 @@
+Gem::Specification.new do |s|
+  s.name          = 'logstash-input-elf-se'
+  s.version       = '0.1.0'
+  s.licenses      = ['Apache-2.0']
+  s.summary       = 'A Logstash plugin the receives events from Salesforce EventLogFile'
+  s.description   = 'This gem is a logstash plugin required to be installed on top of the Logstash core pipeline
+                       using $LS_HOME/bin/plugin install gemname. This gem is not a stand-alone program , changes made are quite specific to schneider electric, we removed some logtypes to support our cause'
+  s.homepage      = 'https://github.com/siddharatha/logstash-input-elf-se'
+  s.authors       = ['Sid']
+  s.email         = 'siddharatha.n@gmail.com'
+  s.require_paths = ['lib']
+
+  # Files
+  s.files = Dir['lib/**/*','spec/**/*','vendor/**/*','*.gemspec','*.md','CONTRIBUTORS','Gemfile','LICENSE','NOTICE.TXT']
+   # Tests
+  s.test_files = s.files.grep(%r{^(test|spec|features)/})
+
+  # Special flag to let us know this is actually a logstash plugin
+  s.metadata = { "logstash_plugin" => "true", "logstash_group" => "input" }
+
+  # Gem dependencies
+  s.add_runtime_dependency "logstash-core-plugin-api", "~> 2.0"
+  s.add_runtime_dependency 'logstash-codec-plain'
+  s.add_runtime_dependency 'stud', '>= 0.0.22'
+  s.add_runtime_dependency 'databasedotcom', '~> 1.3', '>= 1.3.3'
+  s.add_development_dependency 'logstash-devutils', '>= 0.0.16'  
+end

data/spec/inputs/elf-se_spec.rb

@@ -0,0 +1,11 @@
+# encoding: utf-8
+require "logstash/devutils/rspec/spec_helper"
+require "logstash/inputs/elf-se"
+
+describe LogStash::Inputs::ElfSe do
+
+  it_behaves_like "an interruptible input plugin" do
+    let(:config) { { "interval" => 100 } }
+  end
+
+end

metadata

@@ -0,0 +1,137 @@
+--- !ruby/object:Gem::Specification
+name: logstash-input-elf-se
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Sid
+autorequire: 
+bindir: bin
+cert_chain: []
+date: 2019-10-23 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: logstash-core-plugin-api
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.0'
+- !ruby/object:Gem::Dependency
+  name: logstash-codec-plain
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '0'
+- !ruby/object:Gem::Dependency
+  name: stud
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 0.0.22
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 0.0.22
+- !ruby/object:Gem::Dependency
+  name: databasedotcom
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.3'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 1.3.3
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.3'
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 1.3.3
+- !ruby/object:Gem::Dependency
+  name: logstash-devutils
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 0.0.16
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: 0.0.16
+description: |-
+  This gem is a logstash plugin required to be installed on top of the Logstash core pipeline
+                         using $LS_HOME/bin/plugin install gemname. This gem is not a stand-alone program , changes made are quite specific to schneider electric, we removed some logtypes to support our cause
+email: siddharatha.n@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- CONTRIBUTORS
+- DEVELOPER.md
+- Gemfile
+- LICENSE
+- README.md
+- lib/logstash/inputs/elf-se.rb
+- lib/logstash/inputs/sfdc_elf/client_with_streaming_support.rb
+- lib/logstash/inputs/sfdc_elf/queue_util.rb
+- lib/logstash/inputs/sfdc_elf/scheduler.rb
+- lib/logstash/inputs/sfdc_elf/state_persistor.rb
+- logstash-input-elf-se.gemspec
+- spec/inputs/elf-se_spec.rb
+homepage: https://github.com/siddharatha/logstash-input-elf-se
+licenses:
+- Apache-2.0
+metadata:
+  logstash_plugin: 'true'
+  logstash_group: input
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.5.2.3
+signing_key: 
+specification_version: 4
+summary: A Logstash plugin the receives events from Salesforce EventLogFile
+test_files:
+- spec/inputs/elf-se_spec.rb