logstash-input-salesforce 0.1.1

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 76ea7e2676f417fdf939152847cc94bec98d4df0
+  data.tar.gz: 98057efaa75fbda1370ff6a341d32a28d065f8b1
+SHA512:
+  metadata.gz: 58c0067521a8709ab646b2acd5250505c8afd83f48d4c48852b6d5d8f9a163b0ccdc71db93cff7479f38987dea2ed601589586974fede3875535531fad70bb13
+  data.tar.gz: 4c1a7c87ccd8fdef2fdd63f28eddc10a7924c74baa7dcdf8048f4879bca9f4e2bae766d6f4e0e81c0929d16d98ff89e9fcd8a56adba29c2330d13b036a44f957

data/DEVELOPER.md

@@ -0,0 +1,2 @@
+# logstash-input-salesforce
+This input will query objects from salesforce and turn each record into a message

data/Gemfile

@@ -0,0 +1,2 @@
+source 'https://rubygems.org'
+gemspec

data/LICENSE

@@ -0,0 +1,13 @@
+Copyright (c) 2012–2015 Elasticsearch <http://www.elastic.co>
+
+Licensed under the Apache License, Version 2.0 (the "License");
+you may not use this file except in compliance with the License.
+You may obtain a copy of the License at
+
+    http://www.apache.org/licenses/LICENSE-2.0
+
+Unless required by applicable law or agreed to in writing, software
+distributed under the License is distributed on an "AS IS" BASIS,
+WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
+See the License for the specific language governing permissions and
+limitations under the License.

data/NOTICE.TXT

@@ -0,0 +1,5 @@
+Elasticsearch
+Copyright 2012-2015 Elasticsearch
+
+This product includes software developed by The Apache Software
+Foundation (http://www.apache.org/).

data/README.md

@@ -0,0 +1,86 @@
+# Logstash Plugin
+
+This is a plugin for [Logstash](https://github.com/elasticsearch/logstash).
+
+It is fully free and fully open source. The license is Apache 2.0, meaning you are pretty much free to use it however you want in whatever way.
+
+## Documentation
+
+Logstash provides infrastructure to automatically generate documentation for this plugin. We use the asciidoc format to write documentation so any comments in the source code will be first converted into asciidoc and then into html. All plugin documentation are placed under one [central location](http://www.elasticsearch.org/guide/en/logstash/current/).
+
+- For formatting code or config example, you can use the asciidoc `[source,ruby]` directive
+- For more asciidoc formatting tips, see the excellent reference here https://github.com/elasticsearch/docs#asciidoc-guide
+
+## Need Help?
+
+Need help? Try #logstash on freenode IRC or the https://discuss.elastic.co/c/logstash discussion forum.
+
+## Developing
+
+### 1. Plugin Developement and Testing
+
+#### Code
+- To get started, you'll need JRuby with the Bundler gem installed.
+
+- Create a new plugin or clone and existing from the GitHub [logstash-plugins](https://github.com/logstash-plugins) organization. We also provide [example plugins](https://github.com/logstash-plugins?query=example).
+
+- Install dependencies
+```sh
+bundle install
+```
+
+#### Test
+
+- Update your dependencies
+
+```sh
+bundle install
+```
+
+- Run tests
+
+```sh
+bundle exec rspec
+```
+
+### 2. Running your unpublished Plugin in Logstash
+
+#### 2.1 Run in a local Logstash clone
+
+- Edit Logstash `Gemfile` and add the local plugin path, for example:
+```ruby
+gem "logstash-filter-awesome", :path => "/your/local/logstash-filter-awesome"
+```
+- Install plugin
+```sh
+bin/plugin install --no-verify
+```
+- Run Logstash with your plugin
+```sh
+bin/logstash -e 'filter {awesome {}}'
+```
+At this point any modifications to the plugin code will be applied to this local Logstash setup. After modifying the plugin, simply rerun Logstash.
+
+#### 2.2 Run in an installed Logstash
+
+You can use the same **2.1** method to run your plugin in an installed Logstash by editing its `Gemfile` and pointing the `:path` to your local plugin development directory or you can build the gem and install it using:
+
+- Build your plugin gem
+```sh
+gem build logstash-filter-awesome.gemspec
+```
+- Install the plugin from the Logstash home
+```sh
+bin/plugin install /your/local/plugin/logstash-filter-awesome.gem
+```
+- Start Logstash and proceed to test the plugin
+
+## Contributing
+
+All contributions are welcome: ideas, patches, documentation, bug reports, complaints, and even something you drew up on a napkin.
+
+Programming is not a required skill. Whatever you've seen about open source and maintainers or community members  saying "send patches or die" - you will not see that here.
+
+It is more important to the community that you are able to contribute.
+
+For more information about contributing, see the [CONTRIBUTING](https://github.com/elasticsearch/logstash/blob/master/CONTRIBUTING.md) file.

data/Rakefile

@@ -0,0 +1 @@
+require "logstash/devutils/rake"

data/lib/logstash/inputs/salesforce.rb

@@ -0,0 +1,176 @@
+# encoding: utf-8
+require "logstash/inputs/base"
+require "logstash/namespace"
+
+# This Logstash input plugin allows you to query Salesforce using SOQL and puts the results
+# into Logstash, one row per event. You can configure it to pull entire sObjects or only
+# specific fields.
+#
+# NOTE: This input plugin will stop after all the results of the query are processed and will
+# need to be re-run to fetch new results. It does not utilize the streaming API.
+#
+# In order to use this plugin, you will need to create a new SFDC Application using
+# oauth. More details can be found here:
+# https://help.salesforce.com/apex/HTViewHelpDoc?id=connected_app_create.htm
+#
+# You will also need a username, password, and security token for your salesforce instance.
+# More details for generating a token can be found here:
+# https://help.salesforce.com/apex/HTViewHelpDoc?id=user_security_token.htm
+#
+# In addition to specifying an sObject, you can also supply a list of API fields
+# that will be used in the SOQL query.
+#
+# ==== Example
+# This example prints all the Salesforce Opportunities to standard out
+#
+# [source,ruby]
+# ----------------------------------
+# input {
+#   salesforce {
+#     client_id => 'OAUTH CLIENT ID FROM YOUR SFDC APP'
+#     client_secret => 'OAUTH CLIENT SECRET FROM YOUR SFDC APP'
+#     username => 'email@example.com'
+#     password => 'super-secret'
+#     security_token => 'SECURITY TOKEN FOR THIS USER'
+#     sfdc_object_name => 'Opportunity'
+#   }
+# }
+#
+# output {
+#   stdout {
+#     codec => rubydebug
+#   }
+# }
+# ----------------------------------
+
+class LogStash::Inputs::Salesforce < LogStash::Inputs::Base
+
+  config_name "salesforce"
+  default :codec, "plain" #not used
+
+  # Set this to true to connect to a sandbox sfdc instance
+  # logging in through test.salesforce.com
+  config :use_test_sandbox, :validate => :boolean, :default => false
+  # By default, this uses the default Restforce API version.
+  # To override this, set this to something like "32.0" for example
+  config :api_version, :validate => :string, :required => false
+  # Consumer Key for authentication. You must set up a new SFDC
+  # connected app with oath to use this output. More information
+  # can be found here:
+  # https://help.salesforce.com/apex/HTViewHelpDoc?id=connected_app_create.htm
+  config :client_id, :validate => :string, :required => true
+  # Consumer Secret from your oauth enabled connected app
+  config :client_secret, :validate => :string, :required => true
+  # A valid salesforce user name, usually your email address.
+  # Used for authentication and will be the user all objects
+  # are created or modified by
+  config :username, :validate => :string, :required => true
+  # The password used to login to sfdc
+  config :password, :validate => :string, :required => true
+  # The security token for this account. For more information about
+  # generting a security token, see:
+  # https://help.salesforce.com/apex/HTViewHelpDoc?id=user_security_token.htm
+  config :security_token, :validate => :string, :required => true
+  # The name of the salesforce object you are creating or updating
+  config :sfdc_object_name, :validate => :string, :required => true
+  # These are the field names to return in the Salesforce query
+  # If this is empty, all fields are returned.
+  config :sfdc_fields, :validate => :array, :default => []
+  # These options will be added to the WHERE clause in the
+  # SOQL statement. Additional fields can be filtered on by
+  # adding field1 = value1 AND field2 = value2 AND...
+  config :sfdc_filters, :validate => :string, :default => ""
+  # Setting this to true will convert SFDC's NamedFields__c to named_fields__c
+  config :to_underscores, :validate => :boolean, :default => false
+
+  public
+  def register
+    require 'restforce'
+    obj_desc = client.describe(@sfdc_object_name)
+    @sfdc_field_types = get_field_types(obj_desc)
+    @sfdc_fields = get_all_fields if @sfdc_fields.empty?
+  end # def register
+
+  public
+  def run(queue)
+    results = client.query(get_query())
+    if results && results.first
+      results.each do |result|
+        event = LogStash::Event.new()
+        decorate(event)
+        @sfdc_fields.each do |field|
+          field_type = @sfdc_field_types[field]
+          value = result.send(field)
+          event_key = @to_underscores ? underscore(field) : field
+          if not value.nil?
+            case field_type
+            when 'datetime'
+              event[event_key] = LogStash::Timestamp.parse(value)
+            when 'date'
+              event[event_key] = LogStash::Timestamp.parse(value)
+            else
+              event[event_key] = value
+            end
+          end
+        end
+        queue << event
+      end
+    end
+  end # def run
+
+  private
+  def client
+    @client ||= Restforce.new client_options
+  end
+
+  private
+  def client_options
+    options = {
+      :username       => @username,
+      :password       => @password,
+      :security_token => @security_token,
+      :client_id      => @client_id,
+      :client_secret  => @client_secret
+    }
+    options.merge!({ :host => "test.salesforce.com" }) if @use_test_sandbox
+    options.merge!({ :api_version => @api_version }) if @api_version
+    return options
+  end
+
+  private
+  def get_query()
+    query = ["SELECT",@sfdc_fields.join(','),
+             "FROM",@sfdc_object_name]
+    query << ["WHERE",@sfdc_filters] unless @sfdc_filters.empty?
+    query << "ORDER BY LastModifiedDate DESC" if @sfdc_fields.include?('LastModifiedDate')
+    query_str = query.flatten.join(" ")
+    @logger.debug? && @logger.debug("SFDC Query", :query => query_str)
+    return query_str
+  end
+
+  private
+  def get_field_types(obj_desc)
+    field_types = {}
+    obj_desc.fields.each do |f|
+      field_types[f.name] = f.type
+    end
+    @logger.debug? && @logger.debug("Field types", :field_types => field_types.to_s)
+    return field_types
+  end
+
+  private
+  def get_all_fields
+    return @sfdc_field_types.keys
+  end
+
+  private
+  # From http://stackoverflow.com/a/1509957/4701287
+  def underscore(camel_cased_word)
+    camel_cased_word.to_s.gsub(/::/, '/').
+       gsub(/([A-Z]+)([A-Z][a-z])/,'\1_\2').
+       gsub(/([a-z\d])([A-Z])/,'\1_\2').
+       tr("-", "_").
+       downcase
+  end
+
+end # class LogStash::Inputs::Salesforce

data/logstash-input-salesforce.gemspec

@@ -0,0 +1,28 @@
+Gem::Specification.new do |s|
+  s.name = 'logstash-input-salesforce'
+  s.version         = '0.1.1'
+  s.licenses = ['Apache License (2.0)']
+  s.summary = "Pulls objects from Salesforce into Logstash"
+  s.description = "This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not a stand-alone program"
+  s.authors = ["Russ Savage"]
+  s.email = 'russ@elastic.co'
+  s.homepage = "http://www.elastic.co/guide/en/logstash/current/index.html"
+  s.require_paths = ["lib"]
+
+  # Files
+  s.files = `git ls-files`.split($\)
+   # Tests
+  s.test_files = s.files.grep(%r{^(test|spec|features)/})
+
+  # Special flag to let us know this is actually a logstash plugin
+  s.metadata = { "logstash_plugin" => "true", "logstash_group" => "input" }
+
+  # Gem dependencies
+  s.add_runtime_dependency "logstash-core", '>= 1.4.0', '< 2.0.0'
+  s.add_runtime_dependency 'logstash-codec-plain'
+  s.add_runtime_dependency 'restforce'
+  s.add_development_dependency 'logstash-devutils'
+  s.add_development_dependency 'vcr'
+  s.add_development_dependency 'webmock'
+  s.add_development_dependency 'json'
+end

data/spec/fixtures/vcr_cassettes/describe_lead_object.yml

@@ -0,0 +1,87 @@
+---
+http_interactions:
+- request:
+    method: post
+    uri: https://login.salesforce.com/services/oauth2/token
+    body:
+      encoding: US-ASCII
+      string: grant_type=password&client_id=xxxx&client_secret=xxxx&username=xxxx&password=xxxx
+    headers:
+      User-Agent:
+      - Faraday v0.9.1
+      Content-Type:
+      - application/x-www-form-urlencoded
+      Accept:
+      - '*/*'
+  response:
+    status:
+      code: 200
+      message: OK
+    headers:
+      Date:
+      - Thu, 27 Aug 2015 21:31:28 GMT
+      Set-Cookie:
+      - BrowserId=xxxx;Path=/;Domain=.salesforce.com;Expires=Mon, 26-Oct-2015 21:31:28 GMT
+      Expires:
+      - Thu, 01 Jan 1970 00:00:00 GMT
+      Pragma:
+      - no-cache
+      Cache-Control:
+      - no-cache, no-store
+      Content-Type:
+      - application/json;charset=UTF-8
+      Transfer-Encoding:
+      - chunked
+    body:
+      encoding: US-ASCII
+      string: '{"id":"https://login.salesforce.com/id/xxxx/xxxx","issued_at":"1440711088469","token_type":"Bearer","instance_url":"https://eu2.salesforce.com","signature":"xxxx","access_token":"xxxx"}'
+    http_version:
+  recorded_at: Thu, 27 Aug 2015 21:31:28 GMT
+- request:
+    method: get
+    uri: https://eu2.salesforce.com/services/data/v26.0/sobjects/Lead/describe
+    body:
+      encoding: US-ASCII
+      string: ''
+    headers:
+      User-Agent:
+      - Faraday v0.9.1
+      Authorization:
+      - OAuth xxxx
+      Accept-Encoding:
+      - gzip;q=1.0,deflate;q=0.6,identity;q=0.3
+      Accept:
+      - '*/*'
+  response:
+    status:
+      code: 200
+      message: OK
+    headers:
+      Date:
+      - Thu, 27 Aug 2015 21:31:29 GMT
+      Set-Cookie:
+      - BrowserId=xxxx;Path=/;Domain=.salesforce.com;Expires=Mon, 26-Oct-2015 21:31:29 GMT
+      Expires:
+      - Thu, 01 Jan 1970 00:00:00 GMT
+      Sforce-Limit-Info:
+      - api-usage=204568/451000
+      Org.eclipse.jetty.server.include.etag:
+      - f3049665
+      Last-Modified:
+      - Thu, 27 Aug 2015 18:14:47 GMT
+      Content-Type:
+      - application/json;charset=UTF-8
+      Etag:
+      - f304966-gzip"
+      Transfer-Encoding:
+      - chunked
+    body:
+      encoding: UTF-8
+      string: '{"activateable":false,"childRelationships":[],"createable":true,"custom":false,"customSetting":false,"deletable":true,"deprecatedAndHidden":false,"feedEnabled":true,"fields":[{"autoNumber":false,"byteLength":18,"calculated":false,"calculatedFormula":null,"cascadeDelete":false,"caseSensitive":false,"controllerName":null,"createable":false,"custom":false,"defaultValue":null,"defaultValueFormula":null,"defaultedOnCreate":true,"dependentPicklist":false,"deprecatedAndHidden":false,"digits":0,"displayLocationInDecimal":false,"externalId":false,"filterable":true,"groupable":true,"htmlFormatted":false,"idLookup":true,"inlineHelpText":null,"label":"Lead
+        ID","length":18,"name":"Id","nameField":false,"namePointing":false,"nillable":false,"permissionable":false,"picklistValues":[],"precision":0,"referenceTo":[],"relationshipName":null,"relationshipOrder":null,"restrictedDelete":false,"restrictedPicklist":false,"scale":0,"soapType":"tns:ID","sortable":true,"type":"id","unique":false,"updateable":false,"writeRequiresMasterRead":false},{"autoNumber":false,"byteLength":0,"calculated":false,"calculatedFormula":null,"cascadeDelete":false,"caseSensitive":false,"controllerName":null,"createable":false,"custom":false,"defaultValue":null,"defaultValueFormula":null,"defaultedOnCreate":true,"dependentPicklist":false,"deprecatedAndHidden":false,"digits":0,"displayLocationInDecimal":false,"externalId":false,"filterable":true,"groupable":true,"htmlFormatted":false,"idLookup":false,"inlineHelpText":null,"label":"Deleted","length":0,"name":"IsDeleted","nameField":false,"namePointing":false,"nillable":false,"permissionable":false,"picklistValues":[],"precision":0,"referenceTo":[],"relationshipName":null,"relationshipOrder":null,"restrictedDelete":false,"restrictedPicklist":false,"scale":0,"soapType":"xsd:boolean","sortable":true,"type":"boolean","unique":false,"updateable":false,"writeRequiresMasterRead":false},{"autoNumber":false,"byteLength":240,"calculated":false,"calculatedFormula":null,"cascadeDelete":false,"caseSensitive":false,"controllerName":null,"createable":true,"custom":false,"defaultValue":null,"defaultValueFormula":null,"defaultedOnCreate":false,"dependentPicklist":false,"deprecatedAndHidden":false,"digits":0,"displayLocationInDecimal":false,"externalId":false,"filterable":true,"groupable":true,"htmlFormatted":false,"idLookup":false,"inlineHelpText":null,"label":"Last
+        Name","length":80,"name":"LastName","nameField":false,"namePointing":false,"nillable":false,"permissionable":false,"picklistValues":[],"precision":0,"referenceTo":[],"relationshipName":null,"relationshipOrder":null,"restrictedDelete":false,"restrictedPicklist":false,"scale":0,"soapType":"xsd:string","sortable":true,"type":"string","unique":false,"updateable":true,"writeRequiresMasterRead":false},{"autoNumber":false,"byteLength":120,"calculated":false,"calculatedFormula":null,"cascadeDelete":false,"caseSensitive":false,"controllerName":null,"createable":true,"custom":false,"defaultValue":null,"defaultValueFormula":null,"defaultedOnCreate":false,"dependentPicklist":false,"deprecatedAndHidden":false,"digits":0,"displayLocationInDecimal":false,"externalId":false,"filterable":true,"groupable":true,"htmlFormatted":false,"idLookup":false,"inlineHelpText":null,"label":"First
+        Name","length":40,"name":"FirstName","nameField":false,"namePointing":false,"nillable":true,"permissionable":false,"picklistValues":[],"precision":0,"referenceTo":[],"relationshipName":null,"relationshipOrder":null,"restrictedDelete":false,"restrictedPicklist":false,"scale":0,"soapType":"xsd:string","sortable":true,"type":"string","unique":false,"updateable":true,"writeRequiresMasterRead":false},{"autoNumber":false,"byteLength":120,"calculated":false,"calculatedFormula":null,"cascadeDelete":false,"caseSensitive":false,"controllerName":null,"createable":true,"custom":false,"defaultValue":null,"defaultValueFormula":null,"defaultedOnCreate":false,"dependentPicklist":false,"deprecatedAndHidden":false,"digits":0,"displayLocationInDecimal":false,"externalId":false,"filterable":true,"groupable":true,"htmlFormatted":false,"idLookup":false,"inlineHelpText":null,"label":"Salutation","length":40,"name":"Salutation","nameField":false,"namePointing":false,"nillable":true,"permissionable":false,"picklistValues":[{"active":true,"defaultValue":false,"label":"Mr.","validFor":null,"value":"Mr."},{"active":true,"defaultValue":false,"label":"Ms.","validFor":null,"value":"Ms."},{"active":true,"defaultValue":false,"label":"Mrs.","validFor":null,"value":"Mrs."},{"active":true,"defaultValue":false,"label":"Dr.","validFor":null,"value":"Dr."},{"active":true,"defaultValue":false,"label":"Prof.","validFor":null,"value":"Prof."}],"precision":0,"referenceTo":[],"relationshipName":null,"relationshipOrder":null,"restrictedDelete":false,"restrictedPicklist":false,"scale":0,"soapType":"xsd:string","sortable":true,"type":"picklist","unique":false,"updateable":true,"writeRequiresMasterRead":false}],"keyPrefix":"00Q","label":"Lead","labelPlural":"Leads","layoutable":true,"listviewable":null,"lookupLayoutable":null,"mergeable":true,"name":"Lead","queryable":true,"recordTypeInfos":[{"available":true,"defaultRecordTypeMapping":true,"name":"Marketing","recordTypeId":"xxxx"},{"available":true,"defaultRecordTypeMapping":false,"name":"Partner
+        Deal","recordTypeId":"xxxx"},{"available":true,"defaultRecordTypeMapping":false,"name":"Sales","recordTypeId":"xxxx"},{"available":true,"defaultRecordTypeMapping":false,"name":"Master","recordTypeId":"xxxx"}],"replicateable":true,"retrieveable":true,"searchLayoutable":null,"searchable":true,"triggerable":true,"undeletable":true,"updateable":true,"urls":{"uiEditTemplate":"https://xxx.salesforce.com/{ID}/e","sobject":"/services/data/v26.0/sobjects/Lead","uiDetailTemplate":"https://xxx.salesforce.com/{ID}","describe":"/services/data/v26.0/sobjects/Lead/describe","rowTemplate":"/services/data/v26.0/sobjects/Lead/{ID}","uiNewRecord":"https://xxx.salesforce.com/00Q/e"}}'
+    http_version:
+  recorded_at: Thu, 27 Aug 2015 21:31:30 GMT
+recorded_with: VCR 2.9.3

data/spec/fixtures/vcr_cassettes/load_some_lead_objects.yml

@@ -0,0 +1,124 @@
+---
+http_interactions:
+- request:
+    method: post
+    uri: https://login.salesforce.com/services/oauth2/token
+    body:
+      encoding: US-ASCII
+      string: grant_type=password&client_id=xxxx&client_secret=xxxx&username=xxxx&password=xxxx
+    headers:
+      User-Agent:
+      - Faraday v0.9.1
+      Content-Type:
+      - application/x-www-form-urlencoded
+      Accept:
+      - '*/*'
+  response:
+    status:
+      code: 200
+      message: OK
+    headers:
+      Date:
+      - Thu, 27 Aug 2015 23:57:42 GMT
+      Set-Cookie:
+      - BrowserId=xxxx;Path=/;Domain=.salesforce.com;Expires=Mon, 26-Oct-2015 23:57:42 GMT
+      Expires:
+      - Thu, 01 Jan 1970 00:00:00 GMT
+      Pragma:
+      - no-cache
+      Cache-Control:
+      - no-cache, no-store
+      Content-Type:
+      - application/json;charset=UTF-8
+      Transfer-Encoding:
+      - chunked
+    body:
+      encoding: US-ASCII
+      string: '{"id":"https://login.salesforce.com/id/xxxx/xxxx","issued_at":"1440719862904","token_type":"Bearer","instance_url":"https://eu2.salesforce.com","signature":"xxxx","access_token":"xxxx"}'
+    http_version:
+  recorded_at: Thu, 27 Aug 2015 23:57:43 GMT
+- request:
+    method: get
+    uri: https://eu2.salesforce.com/services/data/v26.0/sobjects/Lead/describe
+    body:
+      encoding: US-ASCII
+      string: ''
+    headers:
+      User-Agent:
+      - Faraday v0.9.1
+      Authorization:
+      - OAuth xxxx
+      Accept-Encoding:
+      - gzip;q=1.0,deflate;q=0.6,identity;q=0.3
+      Accept:
+      - '*/*'
+  response:
+    status:
+      code: 200
+      message: OK
+    headers:
+      Date:
+      - Thu, 27 Aug 2015 23:57:44 GMT
+      Set-Cookie:
+      - BrowserId=xxxx;Path=/;Domain=.salesforce.com;Expires=Mon, 26-Oct-2015 23:57:44 GMT
+      Expires:
+      - Thu, 01 Jan 1970 00:00:00 GMT
+      Sforce-Limit-Info:
+      - api-usage=211068/451000
+      Org.eclipse.jetty.server.include.etag:
+      - 5fb54cb6
+      Last-Modified:
+      - Thu, 27 Aug 2015 22:36:55 GMT
+      Content-Type:
+      - application/json;charset=UTF-8
+      Etag:
+      - 5fb54cb-gzip"
+      Transfer-Encoding:
+      - chunked
+    body:
+      encoding: UTF-8
+      string: '{"activateable":false,"childRelationships":[],"createable":true,"custom":false,"customSetting":false,"deletable":true,"deprecatedAndHidden":false,"feedEnabled":true,"fields":[{"autoNumber":false,"byteLength":18,"calculated":false,"calculatedFormula":null,"cascadeDelete":false,"caseSensitive":false,"controllerName":null,"createable":false,"custom":false,"defaultValue":null,"defaultValueFormula":null,"defaultedOnCreate":true,"dependentPicklist":false,"deprecatedAndHidden":false,"digits":0,"displayLocationInDecimal":false,"externalId":false,"filterable":true,"groupable":true,"htmlFormatted":false,"idLookup":true,"inlineHelpText":null,"label":"Lead
+        ID","length":18,"name":"Id","nameField":false,"namePointing":false,"nillable":false,"permissionable":false,"picklistValues":[],"precision":0,"referenceTo":[],"relationshipName":null,"relationshipOrder":null,"restrictedDelete":false,"restrictedPicklist":false,"scale":0,"soapType":"tns:ID","sortable":true,"type":"id","unique":false,"updateable":false,"writeRequiresMasterRead":false},{"autoNumber":false,"byteLength":0,"calculated":false,"calculatedFormula":null,"cascadeDelete":false,"caseSensitive":false,"controllerName":null,"createable":false,"custom":false,"defaultValue":null,"defaultValueFormula":null,"defaultedOnCreate":true,"dependentPicklist":false,"deprecatedAndHidden":false,"digits":0,"displayLocationInDecimal":false,"externalId":false,"filterable":true,"groupable":true,"htmlFormatted":false,"idLookup":false,"inlineHelpText":null,"label":"Deleted","length":0,"name":"IsDeleted","nameField":false,"namePointing":false,"nillable":false,"permissionable":false,"picklistValues":[],"precision":0,"referenceTo":[],"relationshipName":null,"relationshipOrder":null,"restrictedDelete":false,"restrictedPicklist":false,"scale":0,"soapType":"xsd:boolean","sortable":true,"type":"boolean","unique":false,"updateable":false,"writeRequiresMasterRead":false},{"autoNumber":false,"byteLength":240,"calculated":false,"calculatedFormula":null,"cascadeDelete":false,"caseSensitive":false,"controllerName":null,"createable":true,"custom":false,"defaultValue":null,"defaultValueFormula":null,"defaultedOnCreate":false,"dependentPicklist":false,"deprecatedAndHidden":false,"digits":0,"displayLocationInDecimal":false,"externalId":false,"filterable":true,"groupable":true,"htmlFormatted":false,"idLookup":false,"inlineHelpText":null,"label":"Last
+        Name","length":80,"name":"LastName","nameField":false,"namePointing":false,"nillable":false,"permissionable":false,"picklistValues":[],"precision":0,"referenceTo":[],"relationshipName":null,"relationshipOrder":null,"restrictedDelete":false,"restrictedPicklist":false,"scale":0,"soapType":"xsd:string","sortable":true,"type":"string","unique":false,"updateable":true,"writeRequiresMasterRead":false},{"autoNumber":false,"byteLength":120,"calculated":false,"calculatedFormula":null,"cascadeDelete":false,"caseSensitive":false,"controllerName":null,"createable":true,"custom":false,"defaultValue":null,"defaultValueFormula":null,"defaultedOnCreate":false,"dependentPicklist":false,"deprecatedAndHidden":false,"digits":0,"displayLocationInDecimal":false,"externalId":false,"filterable":true,"groupable":true,"htmlFormatted":false,"idLookup":false,"inlineHelpText":null,"label":"First
+        Name","length":40,"name":"FirstName","nameField":false,"namePointing":false,"nillable":true,"permissionable":false,"picklistValues":[],"precision":0,"referenceTo":[],"relationshipName":null,"relationshipOrder":null,"restrictedDelete":false,"restrictedPicklist":false,"scale":0,"soapType":"xsd:string","sortable":true,"type":"string","unique":false,"updateable":true,"writeRequiresMasterRead":false},{"autoNumber":false,"byteLength":120,"calculated":false,"calculatedFormula":null,"cascadeDelete":false,"caseSensitive":false,"controllerName":null,"createable":true,"custom":false,"defaultValue":null,"defaultValueFormula":null,"defaultedOnCreate":false,"dependentPicklist":false,"deprecatedAndHidden":false,"digits":0,"displayLocationInDecimal":false,"externalId":false,"filterable":true,"groupable":true,"htmlFormatted":false,"idLookup":false,"inlineHelpText":null,"label":"Salutation","length":40,"name":"Salutation","nameField":false,"namePointing":false,"nillable":true,"permissionable":false,"picklistValues":[{"active":true,"defaultValue":false,"label":"Mr.","validFor":null,"value":"Mr."},{"active":true,"defaultValue":false,"label":"Ms.","validFor":null,"value":"Ms."},{"active":true,"defaultValue":false,"label":"Mrs.","validFor":null,"value":"Mrs."},{"active":true,"defaultValue":false,"label":"Dr.","validFor":null,"value":"Dr."},{"active":true,"defaultValue":false,"label":"Prof.","validFor":null,"value":"Prof."}],"precision":0,"referenceTo":[],"relationshipName":null,"relationshipOrder":null,"restrictedDelete":false,"restrictedPicklist":false,"scale":0,"soapType":"xsd:string","sortable":true,"type":"picklist","unique":false,"updateable":true,"writeRequiresMasterRead":false}],"keyPrefix":"00Q","label":"Lead","labelPlural":"Leads","layoutable":true,"listviewable":null,"lookupLayoutable":null,"mergeable":true,"name":"Lead","queryable":true,"recordTypeInfos":[{"available":true,"defaultRecordTypeMapping":true,"name":"Marketing","recordTypeId":"xxxx"},{"available":true,"defaultRecordTypeMapping":false,"name":"Partner
+        Deal","recordTypeId":"xxxx"},{"available":true,"defaultRecordTypeMapping":false,"name":"Sales","recordTypeId":"xxxx"},{"available":true,"defaultRecordTypeMapping":false,"name":"Master","recordTypeId":"xxxx"}],"replicateable":true,"retrieveable":true,"searchLayoutable":null,"searchable":true,"triggerable":true,"undeletable":true,"updateable":true,"urls":{"uiEditTemplate":"https://xxx.salesforce.com/{ID}/e","sobject":"/services/data/v26.0/sobjects/Lead","uiDetailTemplate":"https://xxx.salesforce.com/{ID}","describe":"/services/data/v26.0/sobjects/Lead/describe","rowTemplate":"/services/data/v26.0/sobjects/Lead/{ID}","uiNewRecord":"https://xxx.salesforce.com/00Q/e"}}'
+    http_version:
+  recorded_at: Thu, 27 Aug 2015 23:57:44 GMT
+- request:
+    method: get
+    uri: https://eu2.salesforce.com/services/data/v26.0/query?q=SELECT%20Id,IsDeleted,LastName,FirstName,Salutation%20FROM%20Lead%20WHERE%20Email%20LIKE%20'%25@elastic.co'
+    body:
+      encoding: US-ASCII
+      string: ''
+    headers:
+      User-Agent:
+      - Faraday v0.9.1
+      Authorization:
+      - OAuth xxx
+      Accept-Encoding:
+      - gzip;q=1.0,deflate;q=0.6,identity;q=0.3
+      Accept:
+      - '*/*'
+  response:
+    status:
+      code: 200
+      message: OK
+    headers:
+      Date:
+      - Thu, 27 Aug 2015 23:57:45 GMT
+      Set-Cookie:
+      - BrowserId=xxxx;Path=/;Domain=.salesforce.com;Expires=Mon, 26-Oct-2015 23:57:45 GMT
+      Expires:
+      - Thu, 01 Jan 1970 00:00:00 GMT
+      Sforce-Limit-Info:
+      - api-usage=211063/451000
+      Content-Type:
+      - application/json;charset=UTF-8
+      Transfer-Encoding:
+      - chunked
+    body:
+      encoding: UTF-8
+      string: '{"totalSize":3,"done":true,"records":[{"attributes":{"type":"Lead","url":"/services/data/v26.0/sobjects/Lead/xxxx"},"Id":"xxxx","IsDeleted":false,"LastName":"Katz","FirstName":"Aaron","Salutation":"Mr."},{"attributes":{"type":"Lead","url":"/services/data/v26.0/sobjects/Lead/xxxx"},"Id":"xxxx","IsDeleted":false,"LastName":"Grand","FirstName":"Adrien","Salutation":"Dr."},{"attributes":{"type":"Lead","url":"/services/data/v26.0/sobjects/Lead/xxxx"},"Id":"xxxx","IsDeleted":false,"LastName":"Hardy","FirstName":"Alan","Salutation":"Overlord"}]}'
+    http_version:
+  recorded_at: Thu, 27 Aug 2015 23:57:45 GMT
+recorded_with: VCR 2.9.3

data/spec/inputs/salesforce_spec.rb

@@ -0,0 +1,123 @@
+require "logstash/devutils/rspec/spec_helper"
+require "logstash/inputs/salesforce"
+require "vcr"
+require 'json'
+
+RSpec.describe LogStash::Inputs::Salesforce do
+  describe "inputs/salesforce" do
+    let(:options) do
+      {
+        "client_id" => "",
+        "client_secret" => "",
+        "username" => "",
+        "password" => "",
+        "security_token" => "",
+        "sfdc_object_name" => ""
+      }
+    end
+    let(:input) { LogStash::Inputs::Salesforce.new(options) }
+    subject { input }
+
+    it "should convert to lowercase with underscores" do
+      camel_cased_words = ['CleanStatus','CreatedBy',
+                           'of_Open_Opp_related_account__c',
+                           'USAField','ABCD_ABCDE_Threshold__c']
+      underscore_words = ['clean_status','created_by',
+                          'of_open_opp_related_account__c',
+                          'usa_field','abcd_abcde_threshold__c']
+
+      camel_cased_words.zip(underscore_words).each do |c,u|
+        expect(subject.send(:underscore,c)).to eq(u)
+      end
+    end
+
+    context "add fields and filters" do
+      let(:options) do
+        {
+          "client_id" => "",
+          "client_secret" => "",
+          "username" => "",
+          "password" => "",
+          "security_token" => "",
+          "sfdc_object_name" => "Lead",
+          "sfdc_fields" => ["Something"]
+        }
+      end
+      let(:input) { LogStash::Inputs::Salesforce.new(options) }
+
+      it "should build a query" do
+        expect(subject.send(:get_query)).to eq('SELECT Something FROM Lead')
+      end
+    end
+
+    context "describe Lead object" do
+      VCR.configure do |config|
+        config.cassette_library_dir = File.join(File.dirname(__FILE__), '..', 'fixtures', 'vcr_cassettes')
+        config.hook_into :webmock
+        config.before_record do |i|
+          if i.response.body.encoding.to_s == 'ASCII-8BIT'
+            # required because sfdc doesn't send back the content encoding and it
+            # confuses the yaml parser
+            json_body = JSON.load(i.response.body.encode("ASCII-8BIT").force_encoding("utf-8"))
+            i.response.body = json_body.to_json
+            i.response.update_content_length_header
+          end
+        end
+      end
+      let(:options) do
+        {
+          "client_id" => "",
+          "client_secret" => "",
+          "username" => "",
+          "password" => "",
+          "security_token" => "",
+          "sfdc_object_name" => "Lead"
+        }
+      end
+      let(:input) { LogStash::Inputs::Salesforce.new(options) }
+      let(:expected_fields_result) { ["Id", "IsDeleted",
+                                      "LastName", "FirstName", "Salutation"] }
+      let(:expected_types_result) { [["FirstName", "string"],
+                                     ["Id", "id"],
+                                     ["IsDeleted", "boolean"],
+                                     ["LastName", "string"],
+                                     ["Salutation", "picklist"]] }
+      subject { input }
+      it "loads the Lead object fields" do
+        VCR.use_cassette("describe_lead_object",:decode_compressed_response => true) do
+          subject.register
+          expect(subject.instance_variable_get(:@sfdc_field_types)).to match_array(expected_types_result)
+          expect(subject.instance_variable_get(:@sfdc_fields)).to match_array(expected_fields_result)
+        end
+      end
+      context "load Lead objects" do
+        let(:options) do
+          {
+            "client_id" => "",
+            "client_secret" => "",
+            "username" => "",
+            "password" => "",
+            "security_token" => "",
+            "sfdc_object_name" => "Lead",
+            "sfdc_fields" => ["Id", "IsDeleted", "LastName", "FirstName", "Salutation"],
+            "sfdc_filters" => "Email LIKE '%@elastic.co'"
+          }
+        end
+        let(:input) { LogStash::Inputs::Salesforce.new(options) }
+        subject { input }
+        let(:queue) { [] }
+        it "loads some lead records" do
+          VCR.use_cassette("load some lead objects",:decode_compressed_response => true) do
+            subject.register
+            subject.run(queue)
+            expect(queue.length).to eq(3)
+            e = queue.pop
+            expected_fields_result.each do |f|
+              expect(e.to_hash).to include(f)
+            end
+          end
+        end
+      end
+    end
+  end
+end

data/spec/spec_helper.rb

@@ -0,0 +1,25 @@
+# This file was generated by the `rspec --init` command. Conventionally, all
+# specs live under a `spec` directory, which RSpec adds to the `$LOAD_PATH`.
+# The generated `.rspec` file contains `--require spec_helper` which will cause this
+# file to always be loaded, without a need to explicitly require it in any files.
+#
+# Given that it is always loaded, you are encouraged to keep this file as
+# light-weight as possible. Requiring heavyweight dependencies from this file
+# will add to the boot time of your test suite on EVERY test run, even for an
+# individual file that may not need all of that loaded. Instead, consider making
+# a separate helper file that requires the additional dependencies and performs
+# the additional setup, and require it from the spec files that actually need it.
+#
+# The `.rspec` file also contains a few flags that are not defaults but that
+# users commonly want.
+#
+# See http://rubydoc.info/gems/rspec-core/RSpec/Core/Configuration
+RSpec.configure do |config|
+  config.expect_with :rspec do |expectations|
+    expectations.include_chain_clauses_in_custom_matcher_descriptions = true
+  end
+
+  config.mock_with :rspec do |mocks|
+    mocks.verify_partial_doubles = true
+  end
+end

metadata

@@ -0,0 +1,166 @@
+--- !ruby/object:Gem::Specification
+name: logstash-input-salesforce
+version: !ruby/object:Gem::Version
+  version: 0.1.1
+platform: ruby
+authors:
+- Russ Savage
+autorequire:
+bindir: bin
+cert_chain: []
+date: 2015-09-10 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: logstash-core
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: 1.4.0
+    - - <
+      - !ruby/object:Gem::Version
+        version: 2.0.0
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: 1.4.0
+    - - <
+      - !ruby/object:Gem::Version
+        version: 2.0.0
+  prerelease: false
+  type: :runtime
+- !ruby/object:Gem::Dependency
+  name: logstash-codec-plain
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  prerelease: false
+  type: :runtime
+- !ruby/object:Gem::Dependency
+  name: restforce
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  prerelease: false
+  type: :runtime
+- !ruby/object:Gem::Dependency
+  name: logstash-devutils
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  prerelease: false
+  type: :development
+- !ruby/object:Gem::Dependency
+  name: vcr
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  prerelease: false
+  type: :development
+- !ruby/object:Gem::Dependency
+  name: webmock
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  prerelease: false
+  type: :development
+- !ruby/object:Gem::Dependency
+  name: json
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - '>='
+      - !ruby/object:Gem::Version
+        version: '0'
+  prerelease: false
+  type: :development
+description: This gem is a logstash plugin required to be installed on top of the Logstash core pipeline using $LS_HOME/bin/plugin install gemname. This gem is not a stand-alone program
+email: russ@elastic.co
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- CHANGELOG.md
+- DEVELOPER.md
+- Gemfile
+- LICENSE
+- NOTICE.TXT
+- README.md
+- Rakefile
+- lib/logstash/inputs/salesforce.rb
+- logstash-input-salesforce.gemspec
+- spec/fixtures/vcr_cassettes/describe_lead_object.yml
+- spec/fixtures/vcr_cassettes/load_some_lead_objects.yml
+- spec/inputs/salesforce_spec.rb
+- spec/spec_helper.rb
+homepage: http://www.elastic.co/guide/en/logstash/current/index.html
+licenses:
+- Apache License (2.0)
+metadata:
+  logstash_plugin: 'true'
+  logstash_group: input
+post_install_message:
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - '>='
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project:
+rubygems_version: 2.1.9
+signing_key:
+specification_version: 4
+summary: Pulls objects from Salesforce into Logstash
+test_files:
+- spec/fixtures/vcr_cassettes/describe_lead_object.yml
+- spec/fixtures/vcr_cassettes/load_some_lead_objects.yml
+- spec/inputs/salesforce_spec.rb
+- spec/spec_helper.rb