logstash-input-salesforce 3.2.1 → 3.3.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 991a872d0042a97aeb4f9d7d27d6cc25fad15a62771a8a6c6b7eae9d427d55a8
-  data.tar.gz: 525f6d7d6796b4670dcd47faabc7dbbbd641355b763626693e5be7edfc059107
+  metadata.gz: 9c35a5c5144e60a5958fefebef36dfe5eb74a9b833e56dc513639f4e56d8e476
+  data.tar.gz: ff39efdf9da888ad2668e91fa1afcf7ac8c99ebb56579f675b9efa27387154fa
 SHA512:
-  metadata.gz: aca114e8b09b4895136cbd398d5134e7ef2adf85d1a2e067fe7a5986940f7ec1bdf27293b07da43c95db946b873bd8cbabd30d5e2c62550bb96775f9bd15d0a2
-  data.tar.gz: b5f752607e86f8b9873d3c8e45477dd24007d5f97cbbd3305645608fcf185d088a7f478f7175cd4f288495f6d61e184aa038a75e1f0919c19e6516756e6e17eb
+  metadata.gz: 3237d6346f270425502e9d1ec134b976bbb750149ecc2c6814647ae19a4e5a14f9ad9f59e201ce2c8fa354230ab4e848a6d16872ee9cb6df502f79b9ab39d58e
+  data.tar.gz: ed6170f0cb50898d9431f9eacc1d5f6661785dfcde4877ffdc3ead944e1bff15846274bdf60af0e89dbda8704dd59a3a4297fe5ae8c5725f7515c94fc0ca71e1

data/CHANGELOG.md

@@ -1,3 +1,9 @@
+## 3.3.0
+  - Added `timeout` configuration to override the connect/read timeout for REST requests to Salesforce
+  - Added `interval` configuration to run the plugin continuously, querying records and publishing events 
+    for them at a set interval instead of running once and quitting
+  - Added incremental data loading feature, controlled by configuration options `tracking_field`, `tracking_field_value_file`, and `changed_data_filter`
+
 ## 3.2.1
   - Changes sensitive configs type to Password for better protection from leaks in debug logs. [#35](https://github.com/logstash-plugins/logstash-input-salesforce/pull/35)
 

data/README.md

@@ -1,9 +1,34 @@
-# Logstash Plugin
+# Logstash Salesforce input Plugin
+
+This Logstash input plugin allows you to query Salesforce using SOQL and puts the results
+into Logstash, one row per event. You can configure it to pull entire sObjects or only
+specific fields.
 
 This is a plugin for [Logstash](https://github.com/elasticsearch/logstash).
 
 It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
 
+## How to use
+
+Add the input plugin to your Logstash pipeline definition.
+
+This example queries all the Salesforce Opportunities and publishes an event for each opportunity found:
+
+```ruby
+input {
+  salesforce {
+    client_id => 'OAUTH CLIENT ID FROM YOUR SFDC APP'
+    client_secret => 'OAUTH CLIENT SECRET FROM YOUR SFDC APP'
+    username => 'email@example.com'
+    password => 'super-secret'
+    security_token => 'SECURITY TOKEN FOR THIS USER'
+    sfdc_object_name => 'Opportunity'
+  }
+}
+```
+
+For more examples and an explanation of all configuration options, see https://www.elastic.co/docs/reference/logstash/plugins/plugins-inputs-salesforce.
+
 ## Documentation
 
 Logstash provides infrastructure to automatically generate documentation for this plugin. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html. All plugin documentation are placed under one [central location](http://www.elasticsearch.org/guide/en/logstash/current/).
@@ -17,39 +42,55 @@ Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/log
 
 ## Developing
 
-### 1. Plugin Developement and Testing
+### 1. Plugin Development and Testing
 
 #### Code
-- To get started, you'll need JRuby with the Bundler gem installed.
 
-- Create a new plugin or clone and existing from the GitHub [logstash-plugins](https://github.com/logstash-plugins) organization. We also provide [example plugins](https://github.com/logstash-plugins?query=example).
+- To get started, you'll need JRuby with the Bundler gem installed. We strongly recommend
+  using a Ruby Version Manager such as `rvm` to install JRuby. If you're using a JRuby installed
+  by Homebrew on macOS, replace the `bundle` command with `jbundle` in all examples in this
+  document: Homebrew renames the JRuby binaries so that they don't clash with those from the system
+  (C) Ruby that ships with macOS.
+
+- Clone the plugin code from the GitHub [logstash-plugins/logstash-input-salesforce](https://github.com/logstash-plugins/logstash-input-salesforce) repository.
 
 - Install dependencies
 ```sh
-bundle install
+bundle install --path=vendor/bundle
 ```
 
+- Download a source release of the Logstash version you're targeting 
+  (e.g. https://github.com/elastic/logstash/archive/refs/tags/v8.14.3.zip) and
+  extract (unzip) it to a local directory.
+
 #### Test
 
 - Update your dependencies
 
 ```sh
-bundle install
+bundle install --path=vendor/bundle
 ```
 
 - Run tests
 
 ```sh
+export LOGSTASH_PATH=<path to logstash source>
+export LOGSTASH_SOURCE=1
 bundle exec rspec
 ```
 
+If you get an error like `Could not find logstash-core-plugin-api-2.1.16-java, logstash-core-8.14.3-java in locally 
+installed gems`, double check that you've set and exported the `LOGSTASH_PATH` and `LOGSTASH_SOURCE` environment
+variables as explained in the previous section, and that the `LOGSTASH_PATH` points to an unzipped Logstash source 
+distribution.
+
 ### 2. Running your unpublished Plugin in Logstash
 
 #### 2.1 Run in a local Logstash clone
 
 - Edit Logstash `Gemfile` and add the local plugin path, for example:
 ```ruby
-gem "logstash-filter-awesome", :path => "/your/local/logstash-filter-awesome"
+gem "logstash-input-salesforce", :path => "/your/local/logstash-input-salesforce"
 ```
 - Install plugin
 ```sh
@@ -62,7 +103,7 @@ bin/plugin install --no-verify
 ```
 - Run Logstash with your plugin
 ```sh
-bin/logstash -e 'filter {awesome {}}'
+bin/logstash -e 'input { salesforce { ... } }'
 ```
 At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.
 
@@ -72,7 +113,7 @@ You can use the same **2.1** method to run your plugin in an installed Logstash
 
 - Build your plugin gem
 ```sh
-gem build logstash-filter-awesome.gemspec
+gem build logstash-input-salesforce.gemspec
 ```
 - Install the plugin from the Logstash home
 ```sh

data/docs/index.asciidoc

@@ -25,14 +25,15 @@ This Logstash input plugin allows you to query Salesforce using SOQL and puts th
 into Logstash, one row per event. You can configure it to pull entire sObjects or only
 specific fields.
 
-NOTE: This input plugin will stop after all the results of the query are processed and will
-need to be re-run to fetch new results. It does not utilize the streaming API.
+NOTE: By default, this input plugin will stop after all the results of the query are processed and will
+need to be re-run to fetch new results. It does not utilize the streaming API. However, by setting the `interval`
+configuration option you can configure the plugin to automatically run at a set frequency.
 
-In order to use this plugin, you will need to create a new SFDC Application using
-oauth. More details can be found here:
+In order to use this plugin, you will need to create a new Salesforce Connected App with OAuth enabled.
+More details can be found here:
 https://help.salesforce.com/apex/HTViewHelpDoc?id=connected_app_create.htm
 
-You will also need a username, password, and security token for your salesforce instance.
+You will also need a username, password, and security token for your Salesforce instance.
 More details for generating a token can be found here:
 https://help.salesforce.com/apex/HTViewHelpDoc?id=user_security_token.htm
 
@@ -41,7 +42,7 @@ that will be used in the SOQL query.
 
 ==== HTTP proxy
 
-If your infrastructure uses a HTTP proxy, you can set the `SALESFORCE_PROXY_URI` environment variable with the desired URI value (e.g `export SALESFORCE_PROXY_URI="http://proxy.example.com:123"`).
+If your infrastructure uses an HTTP proxy, you can set the `SALESFORCE_PROXY_URI` environment variable with the desired URI value (e.g `export SALESFORCE_PROXY_URI="http://proxy.example.com:123"`).
 
 ==== Example
 This example prints all the Salesforce Opportunities to standard out
@@ -75,15 +76,20 @@ This plugin supports the following configuration options plus the <<plugins-{typ
 |=======================================================================
 |Setting |Input type|Required
 | <<plugins-{type}s-{plugin}-api_version>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-changed_data_filter>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-client_id>> |<<string,string>>|Yes
 | <<plugins-{type}s-{plugin}-client_secret>> |<<password,password>>|Yes
+| <<plugins-{type}s-{plugin}-interval>> |<<number,number>>|No
 | <<plugins-{type}s-{plugin}-password>> |<<password,password>>|Yes
 | <<plugins-{type}s-{plugin}-security_token>> |<<password,password>>|Yes
 | <<plugins-{type}s-{plugin}-sfdc_fields>> |<<array,array>>|No
 | <<plugins-{type}s-{plugin}-sfdc_filters>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-sfdc_instance_url>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-sfdc_object_name>> |<<string,string>>|Yes
+| <<plugins-{type}s-{plugin}-timeout>> |<<number,number>>|No
 | <<plugins-{type}s-{plugin}-to_underscores>> |<<boolean,boolean>>|No
+| <<plugins-{type}s-{plugin}-tracking_field>> |<<string,string>>|No
+| <<plugins-{type}s-{plugin}-tracking_field_value_file>> |<<string,string>>|No
 | <<plugins-{type}s-{plugin}-use_test_sandbox>> |<<boolean,boolean>>|No
 | <<plugins-{type}s-{plugin}-use_tooling_api>> |<<boolean,boolean>>|No
 | <<plugins-{type}s-{plugin}-username>> |<<string,string>>|Yes
@@ -101,7 +107,32 @@ input plugins.
   * There is no default value for this setting.
 
 By default, this uses the default Restforce API version.
-To override this, set this to something like "32.0" for example
+To override this, set this to something like "32.0" for example.
+
+[id="plugins-{type}s-{plugin}-changed_data_filter"]
+===== `changed_data_filter`
+
+* Value type is <<string,string>>
+  * There is no default value for this setting.
+
+The filter to add to the Salesforce query when a previous tracking field value
+was read from the <<plugins-{type}s-{plugin}-tracking_field_value_file>>.
+The string can (and should) contain a placeholder `%+{last_tracking_field_value}+` that
+will be substituted with the actual value read from the <<plugins-{type}s-{plugin}-tracking_field_value_file>>.
+
+This clause is combined with any <<plugins-{type}s-{plugin}-sfdc_filters>>
+clause that is configured using the `AND` operator.
+
+The value should be properly quoted according to the SOQL rules for the field
+type.
+
+**Examples:**
+
+[source,ruby]
+"changed_data_filter" => "Number > '%{last_tracking_field_value}'"
+
+[source,ruby]
+"changed_data_filter" => "LastModifiedDate >= %{last_tracking_field_value}"
 
 [id="plugins-{type}s-{plugin}-client_id"]
 ===== `client_id` 
@@ -110,10 +141,10 @@ To override this, set this to something like "32.0" for example
   * Value type is <<string,string>>
   * There is no default value for this setting.
 
-Consumer Key for authentication. You must set up a new SFDC
-connected app with oath to use this output. More information
+Consumer Key for authentication. You must set up a new Salesforce
+connected app with OAuth enabled to use this plugin. More information
 can be found here:
-https://help.salesforce.com/apex/HTViewHelpDoc?id=connected_app_create.htm
+https://help.salesforce.com/apex/HTViewHelpDoc?id=connected_app_create.htm.
 
 [id="plugins-{type}s-{plugin}-client_secret"]
 ===== `client_secret` 
@@ -122,30 +153,47 @@ https://help.salesforce.com/apex/HTViewHelpDoc?id=connected_app_create.htm
   * Value type is <<password,password>>
   * There is no default value for this setting.
 
-Consumer Secret from your oauth enabled connected app
+Consumer secret from your OAuth enabled connected app.
+
+[id="plugins-{type}s-{plugin}-interval"]
+===== `interval`
+
+  * Value type is <<number,number>>
+  * There is no default value for this setting.
+
+The interval in seconds between each run of the plugin.
+
+If specified, the plugin only terminates when it receives the stop
+signal from Logstash, e.g. when you press Ctrl-C when running interactively,
+or when the process receives a TERM signal. It will query and publish
+events for all results, then sleep until `interval` seconds from the start
+of the previous run of the plugin have passed. If the plugin ran for longer
+than `interval` seconds, it will run again immediately.
+
+If this property is not specified or is set to -1, the plugin will run once and then exit.
 
 [id="plugins-{type}s-{plugin}-password"]
-===== `password` 
+===== `password`
 
   * This is a required setting.
   * Value type is <<password,password>>
   * There is no default value for this setting.
 
-The password used to login to sfdc
+The password used to log in to Salesforce.
 
 [id="plugins-{type}s-{plugin}-security_token"]
-===== `security_token` 
+===== `security_token`
 
-  * This is a required setting.
-  * Value type is <<password,password>>
-  * There is no default value for this setting.
+* This is a required setting.
+* Value type is <<password,password>>
+* There is no default value for this setting.
 
 The security token for this account. For more information about
-generting a security token, see:
-https://help.salesforce.com/apex/HTViewHelpDoc?id=user_security_token.htm
+generating a security token, see:
+https://help.salesforce.com/apex/HTViewHelpDoc?id=user_security_token.htm.
 
 [id="plugins-{type}s-{plugin}-sfdc_fields"]
-===== `sfdc_fields` 
+===== `sfdc_fields`
 
   * Value type is <<array,array>>
   * Default value is `[]`
@@ -159,9 +207,9 @@ If this is empty, all fields are returned.
   * Value type is <<string,string>>
   * Default value is `""`
 
-These options will be added to the WHERE clause in the
+These options will be added to the `WHERE` clause in the
 SOQL statement. Additional fields can be filtered on by
-adding field1 = value1 AND field2 = value2 AND...
+adding `field1 = value1 AND field2 = value2 AND...`.
 
 [id="plugins-{type}s-{plugin}-sfdc_instance_url"]
 ===== `sfdc_instance_url`
@@ -183,7 +231,18 @@ but not both to configure the url to which the plugin connects to.
   * Value type is <<string,string>>
   * There is no default value for this setting.
 
-The name of the salesforce object you are creating or updating
+The name of the Salesforce object you are creating or updating.
+
+[id="plugins-{type}s-{plugin}-timeout"]
+===== `timeout`
+
+* Value type is <<number,number>>
+* Default value is `60`
+
+The timeout to apply to REST API calls to Salesforce, in seconds. If
+a connection to Salesforce cannot be made in this time, an error occurs.
+If it takes longer than the timeout for a block of data (e.g. query results) to be
+read, an error occurs.
 
 [id="plugins-{type}s-{plugin}-to_underscores"]
 ===== `to_underscores` 
@@ -191,7 +250,59 @@ The name of the salesforce object you are creating or updating
   * Value type is <<boolean,boolean>>
   * Default value is `false`
 
-Setting this to true will convert SFDC's NamedFields__c to named_fields__c
+Setting this to true will convert Salesforce's `++NamedFields__c++` to `++named_fields__c++`.
+
+[id="plugins-{type}s-{plugin}-tracking_field"]
+===== `tracking_field`
+
+  * Value type is <<string,string>>
+  * There is no default value for this setting.
+
+The field to track for incremental data loads. This field will
+be used in an `ORDER BY ... ASC` clause that is added to the Salesforce query.
+This field _should_ also be used in the <<plugins-{type}s-{plugin}-changed_data_filter>> clause
+to actually achieve incremental loading of data.
+
+The last value (which is the highest value if the query sorts by this field ascending)
+value for this field will be saved to the file at the path configured by
+<<plugins-{type}s-{plugin}-tracking_field_value_file>>, if specified.
+
+This field should ideally be strictly ascending for new records. An
+autonumber field is ideal for this.
+
+The standard `LastModifiedDate` field can be used, but since it is not _strictly_
+ascending (multiple records can have the same `LastModifiedDate`, the
+<<plugins-{type}s-{plugin}-changed_data_filter>> should account for this by using the `>=`
+operator, and duplicates should be expected.
+
+Note that Salesforce does not guarantee that the standard `Id` field has ascending
+values for new records (https://developer.salesforce.com/docs/atlas.en-us.apexcode.meta/apexcode/apex_testing_best_practices.htm).
+Therefore, using `Id` as tracking field risks missing records and is not recommended.
+
+If this field is not already included in the <<plugins-{type}s-{plugin}-sfdc_fields>>,
+it is added.
+
+[id="plugins-{type}s-{plugin}-tracking_field_value_file"]
+===== `tracking_field_value_file`
+
+* Value type is <<string,string>>
+* There is no default value for this setting.
+
+The full path to the file from which the latest tracking field value from the previous
+plugin invocation will be read, and to which the new latest tracking field value will be
+written after the current plugin invocation.
+
+This keeps persistent track of the last seen value of the tracking field used for incremental
+loading of data.
+
+The file should be readable and writable by the Logstash process.
+
+If the file exists and a <<plugins-{type}s-{plugin}-changed_data_filter>> is configured,
+a changed data filter clause is added to the query (and combined with any <<plugins-{type}s-{plugin}-sfdc_filters>>
+clause that is configured using the `AND` operator).
+
+If the result set is not empty, the value for `tracking_field` from the last row is
+written to the file.
 
 [id="plugins-{type}s-{plugin}-use_test_sandbox"]
 ===== `use_test_sandbox` 
@@ -200,7 +311,7 @@ Setting this to true will convert SFDC's NamedFields__c to named_fields__c
   * Default value is `false`
 
 Set this to true to connect to a sandbox sfdc instance
-logging in through test.salesforce.com
+logging in through test.salesforce.com.
 
 Use either this or the `sfdc_instance_url` configuration option
 but not both to configure the url to which the plugin connects to.
@@ -225,9 +336,9 @@ of elements of sfdc flows) and security health check risks.
   * Value type is <<string,string>>
   * There is no default value for this setting.
 
-A valid salesforce user name, usually your email address.
+A valid Salesforce username, usually your email address.
 Used for authentication and will be the user all objects
-are created or modified by
+are created or modified by.
 
 
 

data/lib/logstash/inputs/salesforce.rb

@@ -2,6 +2,7 @@
 require "logstash/inputs/base"
 require "logstash/namespace"
 require "time"
+require "stud/interval"
 
 # This Logstash input plugin allows you to query Salesforce using SOQL and puts the results
 # into Logstash, one row per event. You can configure it to pull entire sObjects or only
@@ -55,46 +56,84 @@ class LogStash::Inputs::Salesforce < LogStash::Inputs::Base
   # See https://developer.salesforce.com/docs/atlas.en-us.api_tooling.meta/api_tooling
   # for more details about the Tooling API
   config :use_tooling_api, :validate => :boolean, :default => false
+
   # Set this to true to connect to a sandbox sfdc instance
   # logging in through test.salesforce.com
   config :use_test_sandbox, :validate => :boolean, :default => false
+
   # Set this to the instance url of the sfdc instance you want
   # to connect to already during login. If you have configured
   # a MyDomain in your sfdc instance you would provide
   # <mydomain>.my.salesforce.com here.
   config :sfdc_instance_url, :validate => :string, :required => false
+
   # By default, this uses the default Restforce API version.
   # To override this, set this to something like "32.0" for example
   config :api_version, :validate => :string, :required => false
+
   # Consumer Key for authentication. You must set up a new SFDC
   # connected app with oath to use this output. More information
   # can be found here:
   # https://help.salesforce.com/apex/HTViewHelpDoc?id=connected_app_create.htm
   config :client_id, :validate => :string, :required => true
+
   # Consumer Secret from your oauth enabled connected app
   config :client_secret, :validate => :password, :required => true
+
   # A valid salesforce user name, usually your email address.
   # Used for authentication and will be the user all objects
   # are created or modified by
   config :username, :validate => :string, :required => true
+
   # The password used to login to sfdc
   config :password, :validate => :password, :required => true
+
   # The security token for this account. For more information about
   # generting a security token, see:
   # https://help.salesforce.com/apex/HTViewHelpDoc?id=user_security_token.htm
   config :security_token, :validate => :password, :required => true
+
   # The name of the salesforce object you are creating or updating
   config :sfdc_object_name, :validate => :string, :required => true
+
   # These are the field names to return in the Salesforce query
   # If this is empty, all fields are returned.
   config :sfdc_fields, :validate => :array, :default => []
+
   # These options will be added to the WHERE clause in the
   # SOQL statement. Additional fields can be filtered on by
   # adding field1 = value1 AND field2 = value2 AND...
   config :sfdc_filters, :validate => :string, :default => ""
+
+  # RESTForce request timeout in seconds.
+  config :timeout, :validate => :number, :default => 60, :required => false
+
   # Setting this to true will convert SFDC's NamedFields__c to named_fields__c
   config :to_underscores, :validate => :boolean, :default => false
 
+  # File that stores the tracking field's latest value. This is read before querying data to interpolate
+  # the tracking field value into the incremental_filter, and the latest value of the tracking field is written
+  # to it after all the query results have been read.
+  config :tracking_field_value_file, :validate => :string, :required => false
+
+  # Filter clause to use for incremental retrieval and indexing of data that has changed since the last invodation 
+  # of the plugin. This is combined with sfdc_filters using the AND operator, if tracking_field_value_path exists. 
+  # String interpolation is applied to replace "%{last_tracking_field_value}" in this string with the value read 
+  # from tracking_field_value_file. This would usually be something like "tracking_field > '%{last_tracking_field_value}'" 
+  # where tracking_field is the API name of the actual tracking field set using the tracking_field configuration property, 
+  # e.g. LastModifiedDate
+  config :changed_data_filter, :validate => :string, :required => false
+
+  # The field from which the last value will be stored in the tracking_field_value_file and interpolated
+  # for "%{last_tracking_field_value}" in the changed_data_filter expression. This field will also be used in an ORDER BY
+  # clause added to the query, with sorting done ascending, so that the last value in the results is also the
+  # highest.
+  config :tracking_field, :validate => :string, :required => false
+
+  # Interval to run the command. Value is in seconds. If no interval is given,
+  # this plugin only fetches data once.
+  config :interval, :validate => :number, :required => false, :default => -1
+
   public
   def register
     require 'restforce'
@@ -105,26 +144,57 @@ class LogStash::Inputs::Salesforce < LogStash::Inputs::Base
 
   public
   def run(queue)
-    results = client.query(get_query())
-    if results && results.first
-      results.each do |result|
-        event = LogStash::Event.new()
-        decorate(event)
-        @sfdc_fields.each do |field|
-          field_type = @sfdc_field_types[field]
-          value = result.send(field)
-          event_key = @to_underscores ? underscore(field) : field
-          if not value.nil?
-            case field_type
-            when 'datetime', 'date'
-              event.set(event_key, format_time(value))
-            else
-              event.set(event_key, value)
+    while !stop?
+      start = Time.now
+      results = client.query(get_query())
+      latest_tracking_field_value = nil
+      if results && results.first
+        results.each do |result|
+          event = LogStash::Event.new()
+          decorate(event)
+          @sfdc_fields.each do |field|
+            field_type = @sfdc_field_types[field]
+            value = result.send(field)
+            event_key = @to_underscores ? underscore(field) : field
+            unless value.nil?
+              case field_type
+              when 'datetime', 'date'
+                event.set(event_key, format_time(value))
+              else
+                event.set(event_key, value)
+              end
             end
           end
+          queue << event
+          unless @tracking_field.nil?
+            latest_tracking_field_value = result[@tracking_field]
+          end
+        end # loop sObjects
+      end
+
+      unless @tracking_field_value_file.nil?
+        unless latest_tracking_field_value.nil?
+          @logger.debug("Writing latest tracking field value " + latest_tracking_field_value + " to " + @tracking_field_value_file)
+          File.write(@tracking_field_value_file, latest_tracking_field_value)
+        else
+          @logger.debug("No tracking field value found in result, not updating " + @tracking_field_value_file)
         end
-        queue << event
       end
+
+      if @interval == -1
+        break
+      else
+        duration = Time.now - start
+        # Sleep for the remainder of the interval, or 0 if the duration ran
+        # longer than the interval.
+        sleeptime = [0, @interval - duration].max
+        if sleeptime == 0
+          @logger.warn("Execution ran longer than the interval. Skipping sleep.",
+                       :duration => duration,
+                       :interval => @interval)
+        end
+        Stud.stoppable_sleep(sleeptime) { stop? }
+      end  # end interval check
     end
   end # def run
 
@@ -144,7 +214,8 @@ class LogStash::Inputs::Salesforce < LogStash::Inputs::Base
       :password       => @password.value,
       :security_token => @security_token.value,
       :client_id      => @client_id,
-      :client_secret  => @client_secret.value
+      :client_secret  => @client_secret.value,
+      :timeout        => @timeout
     }
     # configure the endpoint to which restforce connects to for authentication
     if @sfdc_instance_url && @use_test_sandbox
@@ -160,15 +231,43 @@ class LogStash::Inputs::Salesforce < LogStash::Inputs::Base
 
   private
   def get_query()
-    query = ["SELECT",@sfdc_fields.join(','),
-             "FROM",@sfdc_object_name]
-    query << ["WHERE",@sfdc_filters] unless @sfdc_filters.empty?
-    query << "ORDER BY LastModifiedDate DESC" if @sfdc_fields.include?('LastModifiedDate')
+    sfdc_fields = @sfdc_fields.dup
+    unless @tracking_field.nil?
+      unless sfdc_fields.include?(@tracking_field)
+        sfdc_fields << [@tracking_field]
+      end
+    end
+    query = ["SELECT", sfdc_fields.join(','),
+             "FROM", @sfdc_object_name]
+    where = []
+    unless @sfdc_filters.empty?
+      append_to_where_clause(@sfdc_filters, where)
+    end
+    unless @changed_data_filter.nil?
+      if File.exist?(@tracking_field_value_file)
+        last_tracking_field_value = File.read(@tracking_field_value_file)
+        changed_data_filter_interpolated = @changed_data_filter % { :last_tracking_field_value => last_tracking_field_value }
+        append_to_where_clause(changed_data_filter_interpolated, where)
+      end
+    end
+    query << where
+    unless @tracking_field.nil?
+      query << ["ORDER BY", @tracking_field, "ASC"]
+    end
     query_str = query.flatten.join(" ")
     @logger.debug? && @logger.debug("SFDC Query", :query => query_str)
     return query_str
   end
 
+  def append_to_where_clause(changed_data_filter_interpolated, where)
+    if where.empty?
+      where << ["WHERE"]
+    else
+      where << ["AND"]
+    end
+    where << [changed_data_filter_interpolated]
+  end
+
   private
   def get_field_types(obj_desc)
     field_types = {}

data/logstash-input-salesforce.gemspec

@@ -1,7 +1,7 @@
 Gem::Specification.new do |s|
   s.name = 'logstash-input-salesforce'
-  s.version         = '3.2.1'
-  s.licenses = ['Apache License (2.0)']
+  s.version         = '3.3.0'
+  s.licenses = ['apache-2.0']
   s.summary = "Creates events based on a Salesforce SOQL query"
   s.description     = "This gem is a Logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/logstash-plugin install gemname. This gem is not a stand-alone program"
   s.authors = ["Russ Savage"]