salesforcebulk 1.0.0

data/.gitignore

@@ -0,0 +1,4 @@
+*.gem
+.bundle
+Gemfile.lock
+pkg/*

data/.rbenv-version

@@ -0,0 +1 @@
+1.9.3-p194

data/Gemfile

@@ -0,0 +1,2 @@
+source "http://rubygems.org"
+gemspec

data/README.md

@@ -0,0 +1,143 @@
+# SalesforceBulk
+
+## Overview
+
+Salesforce Bulk is a simple Ruby gem for connecting to and using the [Salesforce Bulk API](http://www.salesforce.com/us/developer/docs/api_asynch/index.htm). This is a rewrite of Jorge Valdivia's salesforce_bulk gem with unit tests and full API capability.
+
+## Installation
+
+Install SalesforceBulk from RubyGems:
+
+    gem install salesforcebulk
+
+Or include it in your project's `Gemfile` with Bundler:
+
+    gem 'salesforcebulk'
+
+## Contribute
+
+To contribute, fork this repo, create a topic branch, make changes, then send a pull request. Pull requests without accompanying tests will *not* be accepted. To run tests in your fork, just do:
+
+    bundle install
+    rake
+
+## Configuration and Initialization
+
+### Basic Configuration
+
+    require 'salesforce_bulk'
+    
+    client = SalesforceBulk::Client.new(username: 'MyUsername', password: 'MyPassword', token: 'MySecurityToken')
+    client.authenticate
+
+Optional keys include host (default: login.salesforce.com), version (default: 24.0) and debugging (default: false).
+
+### Configuring from a YAML file
+
+The optional keys mentioned in the Basic Configuration section can also be used here.
+
+    ---
+    username: MyUsername
+    password: MyPassword
+    token: MySecurityToken
+
+Then in a Ruby script:
+
+    require 'salesforce_bulk'
+    
+    client = SalesforceBulk::Client.new("config/salesforce_bulk.yml")
+    client.authenticate
+
+## Usage Examples
+
+An important note about the data in any of the examples: each hash in a data set must have the same set of keys. If you need to have logic to not include certain values simply specify a nil value for a key.
+
+### Basic Overall Example
+
+    data1 = [{:Name__c => 'Test 1'}, {:Name__c => 'Test 2'}]
+    data2 = [{:Name__c => 'Test 3'}, {:Name__c => 'Test 4'}]
+    
+    job = client.add_job(:insert, :MyObject__c)
+    
+    # easily add multiple batches to a job
+    batch = client.add_batch(job.id, data1)
+    batch = client.add_batch(job.id, data2)
+    
+    job = client.close_job(job.id) # or use the abort_job(id) method
+
+### Adding a Job
+
+When adding a job you can specify the following operations for the first argument:
+- :delete
+- :insert
+- :update
+- :upsert
+- :query
+
+When using the :upsert operation you must specify an external ID field name:
+
+    job = client.add_job(:upsert, :MyObject__c, :external_id_field_name => :MyId__c)
+
+For any operation you should be able to specify a concurrency mode. The default is Parallel. The other choice is Serial.
+
+    job = client.add_job(:upsert, :MyObject__c, :concurrency_mode => :Serial, :external_id_field_name => :MyId__c)
+
+### Retrieving Job Information (e.g. Status)
+
+    job = client.job_info(jobId) # returns a Job object
+    
+    puts "Job #{job.id} is closed." if job.closed? # other: open?, aborted?
+
+### Retrieving Info for all Batches
+
+    batches = client.batch_info_list(jobId) # returns an Array of Batch objects
+    
+    batches.each do |batch|
+      puts "Batch #{batch.id} failed." if batch.failed? # other: completed?, failed?, in_progress?, queued?
+    end
+
+### Retrieving Info for a single Batch
+
+    batch = client.batch_info(jobId, batchId) # returns a Batch object
+    
+    puts "Batch #{batch.id} is in progress." if batch.in_progress?
+
+### Retrieving Batch Results (for Delete, Insert, Update and Upsert)
+
+To verify that a batch completed successfully or failed call the `batch_info` or `batch_info_list` methods first, otherwise if you call `batch_result` without verifying and the batch failed the method will raise an error.
+
+The object returned from the following example only applies to the operations: delete, insert, update and upsert. Query results are handled differently.
+
+    results = client.batch_result(jobId, batchId) # returns an Array of BatchResult objects
+    
+    results.each do |result|
+      puts "Item #{result.id} had an error of: #{result.error}" if result.error?
+    end
+
+### Retrieving Query based Batch Results
+
+To verify that a batch completed successfully or failed call the `batch_info` or `batch_info_list` methods first, otherwise if you call `batch_result` without verifying and the batch failed the method will raise an error.
+
+Query results are handled differently as the response will not contain the full result set. You'll have to page through sets if you added multiple batches to a job.
+
+    # returns a QueryResultCollection object (an Array)
+    results = client.batch_result(jobId, batchId)
+    
+    while results.any?
+      
+      # Assuming query was: SELECT Id, Name, CustomField__c FROM Account
+      results.each do |result|
+        puts result[:Id], result[:Name], result[:CustomField__c]
+      end
+      
+      puts "Another set is available." if results.next?
+      
+      results.next
+      
+    end
+
+Note: By reviewing the API docs and response format my understanding was that the API would return multiple results sets for a single batch if the query was to large but this does not seem to be the case in my live testing. It seems to be capped at 10000 records (as it when inserting data) but I haven't been able to verify through the documentation. If you know anything about that your input is appreciated. In the meantime the gem was built to support multiple result sets for a query batch but seems that will change which will simplify that method.
+
+## Copyright
+
+Copyright (c) 2012 Javier Julio.

data/Rakefile

@@ -0,0 +1,22 @@
+require 'bundler'
+Bundler::GemHelper.install_tasks
+
+require 'rake/testtask'
+Rake::TestTask.new(:test) do |test|
+  test.libs << 'lib' << 'test'
+  test.pattern = 'test/**/test_*.rb'
+  test.verbose = true
+end
+
+task :default => :test
+
+namespace :doc do
+  require 'rdoc/task'
+  require File.expand_path('../lib/salesforce_bulk/version', __FILE__)
+  RDoc::Task.new do |rdoc|
+    rdoc.rdoc_dir = 'rdoc'
+    rdoc.title = "SalesforceBulk #{SalesforceBulk::VERSION}"
+    rdoc.main = 'README.md'
+    rdoc.rdoc_files.include('README.md', 'LICENSE.md', 'lib/**/*.rb')
+  end
+end

data/lib/salesforce_bulk.rb

@@ -0,0 +1,15 @@
+require 'net/https'
+require 'xmlsimple'
+require 'csv'
+require 'active_support'
+require 'active_support/core_ext/object/blank'
+require 'active_support/core_ext/hash/keys'
+require 'salesforce_bulk/version'
+require 'salesforce_bulk/core_extensions/string'
+require 'salesforce_bulk/salesforce_error'
+require 'salesforce_bulk/client'
+require 'salesforce_bulk/job'
+require 'salesforce_bulk/batch'
+require 'salesforce_bulk/batch_result'
+require 'salesforce_bulk/batch_result_collection'
+require 'salesforce_bulk/query_result_collection'

data/lib/salesforce_bulk/batch.rb

@@ -0,0 +1,50 @@
+module SalesforceBulk
+  class Batch
+    
+    attr_accessor :apex_processing_time
+    attr_accessor :api_active_processing_time
+    attr_accessor :completed_at
+    attr_accessor :created_at
+    attr_accessor :failed_records
+    attr_accessor :id
+    attr_accessor :job_id
+    attr_accessor :processed_records
+    attr_accessor :state
+    attr_accessor :total_processing_time
+    
+    def self.new_from_xml(data)
+      batch = self.new
+      batch.id = data['id']
+      batch.job_id = data['jobId']
+      batch.state = data['state']
+      batch.created_at = DateTime.parse(data['createdDate'])
+      batch.completed_at = DateTime.parse(data['systemModstamp'])
+      batch.processed_records = data['numberRecordsProcessed'].to_i
+      batch.failed_records = data['numberRecordsFailed'].to_i
+      batch.total_processing_time = data['totalProcessingTime'].to_i
+      batch.api_active_processing_time = data['apiActiveProcessingTime'].to_i
+      batch.apex_processing_time = data['apex_processing_time'].to_i
+      batch
+    end
+    
+    def in_progress?
+      state? 'InProgress'
+    end
+    
+    def queued?
+      state? 'Queued'
+    end
+    
+    def completed?
+      state? 'Completed'
+    end
+    
+    def failed?
+      state? 'Failed'
+    end
+    
+    def state?(value)
+      self.state.present? && self.state.casecmp(value) == 0
+    end
+  end
+end

data/lib/salesforce_bulk/batch_result.rb

@@ -0,0 +1,39 @@
+module SalesforceBulk
+  class BatchResult
+    
+    # A boolean indicating if record was created. If updated value is false.
+    attr_accessor :created
+    
+    # The error message.
+    attr_accessor :error
+    
+    # The record's unique id.
+    attr_accessor :id
+    
+    # If record was created successfully. If false then an error message is provided. 
+    attr_accessor :success
+    
+    def initialize(id, success, created, error)
+      @id = id
+      @success = success
+      @created = created
+      @error = error
+    end
+    
+    def error?
+      error.present?
+    end
+    
+    def created?
+      created
+    end
+    
+    def successful?
+      success
+    end
+    
+    def updated?
+      !created && success
+    end
+  end
+end

data/lib/salesforce_bulk/batch_result_collection.rb

@@ -0,0 +1,29 @@
+module SalesforceBulk
+  class BatchResultCollection < Array
+    
+    attr_reader :batch_id
+    attr_reader :job_id
+    
+    def initialize(job_id, batch_id)
+      @job_id = job_id
+      @batch_id = batch_id
+    end
+    
+    def any_failures?
+      self.any? { |result| result.error.length > 0 }
+    end
+    
+    def failed
+      self.select { |result| result.error.length > 0 }
+    end
+    
+    def completed
+      self.select { |result| result.success }
+    end
+    
+    def created
+      self.select { |result| result.success && result.created }
+    end
+    
+  end
+end

data/lib/salesforce_bulk/client.rb

@@ -0,0 +1,254 @@
+module SalesforceBulk
+  # Interface for operating the Salesforce Bulk REST API
+  class Client
+    # If true, print API debugging information to stdout. Defaults to false.
+    attr_accessor :debugging
+    
+    # The host to use for authentication. Defaults to login.salesforce.com.
+    attr_accessor :host
+    
+    # The instance host to use for API calls. Determined from login response.
+    attr_accessor :instance_host
+    
+    # The Salesforce password
+    attr_accessor :password
+    
+    # The Salesforce security token
+    attr_accessor :token
+    
+    # The Salesforce username
+    attr_accessor :username
+    
+    # The API version the client is using. Defaults to 24.0.
+    attr_accessor :version
+    
+    def initialize(options={})
+      if options.is_a?(String)
+        options = YAML.load_file(options)
+        options.symbolize_keys!
+      end
+      
+      options = {:debugging => false, :host => 'login.salesforce.com', :version => 24.0}.merge(options)
+      
+      options.assert_valid_keys(:username, :password, :token, :debugging, :host, :version)
+      
+      self.username = options[:username]
+      self.password = "#{options[:password]}#{options[:token]}"
+      self.token = options[:token]
+      self.debugging = options[:debugging]
+      self.host = options[:host]
+      self.version = options[:version]
+      
+      @api_path_prefix = "/services/async/#{self.version}/"
+      @valid_operations = [:delete, :insert, :update, :upsert, :query]
+      @valid_concurrency_modes = ['Parallel', 'Serial']
+    end
+    
+    def authenticate
+      xml = '<?xml version="1.0" encoding="utf-8"?>'
+      xml += '<env:Envelope xmlns:xsd="http://www.w3.org/2001/XMLSchema"'
+      xml += ' xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"'
+      xml += ' xmlns:env="http://schemas.xmlsoap.org/soap/envelope/">'
+      xml += "<env:Body>"
+      xml += '<n1:login xmlns:n1="urn:partner.soap.sforce.com">'
+      xml += "<n1:username>#{self.username}</n1:username>"
+      xml += "<n1:password>#{self.password}</n1:password>"
+      xml += "</n1:login>"
+      xml += "</env:Body>"
+      xml += "</env:Envelope>"
+      
+      response = http_post("/services/Soap/u/#{self.version}", xml, 'Content-Type' => 'text/xml', 'SOAPAction' => 'login')
+      
+      data = XmlSimple.xml_in(response.body, :ForceArray => false)
+      result = data['Body']['loginResponse']['result']
+      
+      @session_id = result['sessionId']
+      
+      self.instance_host = "#{instance_id(result['serverUrl'])}.salesforce.com"
+    end
+    
+    def abort_job(jobId)
+      xml = '<?xml version="1.0" encoding="utf-8"?>'
+      xml += '<jobInfo xmlns="http://www.force.com/2009/06/asyncapi/dataload">'
+      xml += "<state>Aborted</state>"
+      xml += "</jobInfo>"
+      
+      response = http_post("job/#{jobId}", xml)
+      data = XmlSimple.xml_in(response.body, :ForceArray => false)
+      Job.new_from_xml(data)
+    end
+    
+    def add_batch(jobId, data)
+      body = data
+      
+      if data.is_a?(Array)
+        raise ArgumentError, "Data set exceeds 10000 record limit by #{data.length - 10000}" if data.length > 10000
+        
+        keys = data.first.keys
+        body = keys.to_csv
+        
+        data.each do |item|
+          item_values = keys.map { |key| item[key] }
+          body += item_values.to_csv
+        end
+      end
+      
+      # Despite the content for a query operation batch being plain text we 
+      # still have to specify CSV content type per API docs.
+      response = http_post("job/#{jobId}/batch", body, "Content-Type" => "text/csv; charset=UTF-8")
+      result = XmlSimple.xml_in(response.body, 'ForceArray' => false)
+      Batch.new_from_xml(result)
+    end
+    
+    def add_job(operation, sobject, options={})
+      operation = operation.to_sym.downcase
+      
+      raise ArgumentError.new("Invalid operation: #{operation}") unless @valid_operations.include?(operation)
+      
+      options.assert_valid_keys(:external_id_field_name, :concurrency_mode)
+      
+      if options[:concurrency_mode]
+        concurrency_mode = options[:concurrency_mode].capitalize
+        raise ArgumentError.new("Invalid concurrency mode: #{concurrency_mode}") unless @valid_concurrency_modes.include?(concurrency_mode)
+      end
+      
+      xml = '<?xml version="1.0" encoding="utf-8"?>'
+      xml += '<jobInfo xmlns="http://www.force.com/2009/06/asyncapi/dataload">'
+      xml += "<operation>#{operation}</operation>"
+      xml += "<object>#{sobject}</object>"
+      xml += "<externalIdFieldName>#{options[:external_id_field_name]}</externalIdFieldName>" if options[:external_id_field_name]
+      xml += "<concurrencyMode>#{options[:concurrency_mode]}</concurrencyMode>" if options[:concurrency_mode]
+      xml += "<contentType>CSV</contentType>"
+      xml += "</jobInfo>"
+      
+      response = http_post("job", xml)
+      data = XmlSimple.xml_in(response.body, :ForceArray => false)
+      job = Job.new_from_xml(data)
+    end
+    
+    def batch_info_list(jobId)
+      response = http_get("job/#{jobId}/batch")
+      result = XmlSimple.xml_in(response.body, 'ForceArray' => false)
+      
+      if result['batchInfo'].is_a?(Array)
+        result['batchInfo'].collect do |info|
+          Batch.new_from_xml(info)
+        end
+      else
+        [Batch.new_from_xml(result['batchInfo'])]
+      end
+    end
+    
+    def batch_info(jobId, batchId)
+      response = http_get("job/#{jobId}/batch/#{batchId}")
+      result = XmlSimple.xml_in(response.body, 'ForceArray' => false)
+      Batch.new_from_xml(result)
+    end
+    
+    def batch_result(jobId, batchId)
+      response = http_get("job/#{jobId}/batch/#{batchId}/result")
+      
+      if response.body =~ /<.*?>/m
+        result = XmlSimple.xml_in(response.body)
+        
+        if result['result'].present?
+          results = query_result(jobId, batchId, result['result'].first)
+          
+          collection = QueryResultCollection.new(self, jobId, batchId, result['result'].first, result['result'])
+          collection.replace(results)
+        end
+      else
+        result = BatchResultCollection.new(jobId, batchId)
+        
+        CSV.parse(response.body, :headers => true) do |row|
+          result << BatchResult.new(row[0], row[1].to_b, row[2].to_b, row[3])
+        end
+        
+        result
+      end
+    end
+    
+    def query_result(job_id, batch_id, result_id)
+      headers = {"Content-Type" => "text/csv; charset=UTF-8"}
+      response = http_get("job/#{job_id}/batch/#{batch_id}/result/#{result_id}", headers)
+      
+      lines = response.body.lines.to_a
+      headers = CSV.parse_line(lines.shift).collect { |header| header.to_sym }
+      
+      result = []
+      
+      #CSV.parse(lines.join, :headers => headers, :converters => [:all, lambda{|s| s.to_b if s.kind_of? String }]) do |row|
+      CSV.parse(lines.join, :headers => headers) do |row|
+        result << Hash[row.headers.zip(row.fields)]
+      end
+      
+      result
+    end
+    
+    def close_job(jobId)
+      xml = '<?xml version="1.0" encoding="utf-8"?>'
+      xml += '<jobInfo xmlns="http://www.force.com/2009/06/asyncapi/dataload">'
+      xml += "<state>Closed</state>"
+      xml += "</jobInfo>"
+      
+      response = http_post("job/#{jobId}", xml)
+      data = XmlSimple.xml_in(response.body, :ForceArray => false)
+      Job.new_from_xml(data)
+    end
+    
+    def job_info(jobId)
+      response = http_get("job/#{jobId}")
+      data = XmlSimple.xml_in(response.body, :ForceArray => false)
+      Job.new_from_xml(data)
+    end
+    
+    def http_post(path, body, headers={})
+      host = self.host
+      
+      headers = {'Content-Type' => 'application/xml'}.merge(headers)
+      
+      if @session_id
+        headers['X-SFDC-Session'] = @session_id
+        host = self.instance_host
+        path = "#{@api_path_prefix}#{path}"
+      end
+      
+      response = https_request(host).post(path, body, headers)
+      
+      if response.is_a?(Net::HTTPSuccess)
+        response
+      else
+        raise SalesforceError.new(response)
+      end
+    end
+    
+    def http_get(path, headers={})
+      path = "#{@api_path_prefix}#{path}"
+      
+      headers = {'Content-Type' => 'application/xml'}.merge(headers)
+      
+      if @session_id
+        headers['X-SFDC-Session'] = @session_id
+      end
+      
+      response = https_request(self.instance_host).get(path, headers)
+      
+      if response.is_a?(Net::HTTPSuccess)
+        response
+      else
+        raise SalesforceError.new(response)
+      end
+    end
+    
+    def https_request(host)
+      req = Net::HTTP.new(host, 443)
+      req.use_ssl = true
+      req.verify_mode = OpenSSL::SSL::VERIFY_NONE
+      req
+    end
+    
+    def instance_id(url)
+      url.match(/:\/\/([a-zA-Z0-9-]{2,}).salesforce/)[1]
+    end
+  end
+end