salesforce_bulk_api 1.1.0 → 1.3.0

data/LICENCE

@@ -1,6 +1,6 @@
 The MIT License (MIT)
 
-Copyright (c) 2015 Yatish Mehta
+Copyright (c) 2025 Yatish Mehta
 
 Permission is hereby granted, free of charge, to any person obtaining a copy
 of this software and associated documentation files (the "Software"), to deal

data/README.md

@@ -1,141 +1,496 @@
 # Salesforce-Bulk-Api
+
 [![Gem Version](https://badge.fury.io/rb/salesforce_bulk_api.png)](http://badge.fury.io/rb/salesforce_bulk_api)
 
+## Table of Contents
+
+- [Overview](#overview)
+- [Installation](#installation)
+- [Authentication](#authentication)
+- [Usage](#usage)
+  - [Basic Operations](#basic-operations)
+  - [Method Parameters](#method-parameters)
+  - [Getting Results](#getting-results)
+  - [Null Value Handling](#null-value-handling)
+  - [Job Management](#job-management)
+  - [Batch Operations](#batch-operations)
+  - [Event Listening](#event-listening)
+  - [API Call Throttling](#api-call-throttling)
+  - [Monitoring and Counters](#monitoring-and-counters)
+- [Error Handling](#error-handling)
+- [Advanced Features](#advanced-features)
+- [Contributing](#contributing)
+- [License](#license)
+
 ## Overview
 
-`SalesforceBulkApi` is a ruby wrapper for the Salesforce Bulk API.
-It is rewritten from [salesforce_bulk](https://github.com/jorgevaldivia/salesforce_bulk).
-It adds some missing features of `salesforce_bulk`.
+`SalesforceBulkApi` is a Ruby wrapper for the Salesforce Bulk API. It is rewritten from [salesforce_bulk](https://github.com/jorgevaldivia/salesforce_bulk) and adds several missing features, making it easier to perform bulk operations with Salesforce from Ruby applications.
 
-## How to use
+Key features:
+- Support for all Bulk API operations (create, update, upsert, delete, query)
+- Comprehensive error handling
+- Job and batch status monitoring
+- Event listening for job lifecycle
+- API call throttling and monitoring
+- Performance optimized string concatenation for large batches
 
-Using this gem is simple and straight forward.
+## Installation
 
-### Install
+Add this line to your application's Gemfile:
 
-   `gem install salesforce_bulk_api`
+```ruby
+gem 'salesforce_bulk_api'
+```
 
-or add it to your Gemfile
+And then execute:
 
-   `gem salesforce_bulk_api`
+```
+bundle install
+```
+
+Or install it directly:
 
-### Authenticate
+```
+gem install salesforce_bulk_api
+```
 
-You can authenticate with Salesforce using two gems, `databasedotcom` & `restforce`.
+## Authentication
 
-Please check the documentation of the respective gems to learn how to authenticate with Salesforce
+You can authenticate with Salesforce using either `databasedotcom` or `restforce` gems. Both support various authentication methods including username/password, OmniAuth, and OAuth2.
 
-[Databasedotcom](https://github.com/heroku/databasedotcom)
-[Restforce](https://github.com/ejholmes/restforce)
+Please refer to the documentation of these gems for detailed authentication options:
 
-You can use username password combo, OmniAuth, Oauth2
-You can use as many records possible in the Array. Governor limits are taken care of inside the gem.
+- [Databasedotcom](https://github.com/heroku/databasedotcom)
+- [Restforce](https://github.com/ejholmes/restforce)
+
+### Authentication Examples
+
+#### Using Databasedotcom:
 
 ```ruby
 require 'salesforce_bulk_api'
 
 client = Databasedotcom::Client.new(
-  :client_id =>  SFDC_APP_CONFIG["client_id"],
-  :client_secret => SFDC_APP_CONFIG["client_secret"]
+  client_id: SFDC_APP_CONFIG["client_id"],
+  client_secret: SFDC_APP_CONFIG["client_secret"]
 )
 client.authenticate(
-  :token => " ",
-  :instance_url => "http://na1.salesforce.com"
+  token: " ",
+  instance_url: "http://na1.salesforce.com"
 )
 
 salesforce = SalesforceBulkApi::Api.new(client)
 ```
 
-OR
+#### Using Restforce:
 
 ```ruby
 require 'salesforce_bulk_api'
+
 client = Restforce.new(
-  username:       SFDC_APP_CONFIG['SFDC_USERNAME'],
-  password:       SFDC_APP_CONFIG['SFDC_PASSWORD'],
+  username: SFDC_APP_CONFIG['SFDC_USERNAME'],
+  password: SFDC_APP_CONFIG['SFDC_PASSWORD'],
   security_token: SFDC_APP_CONFIG['SFDC_SECURITY_TOKEN'],
-  client_id:      SFDC_APP_CONFIG['SFDC_CLIENT_ID'],
-  client_secret:  SFDC_APP_CONFIG['SFDC_CLIENT_SECRET'].to_i,
-  host:           SFDC_APP_CONFIG['SFDC_HOST']
+  client_id: SFDC_APP_CONFIG['SFDC_CLIENT_ID'],
+  client_secret: SFDC_APP_CONFIG['SFDC_CLIENT_SECRET'],
+  host: SFDC_APP_CONFIG['SFDC_HOST']
 )
 client.authenticate!
 
 salesforce = SalesforceBulkApi::Api.new(client)
 ```
 
-### Sample operations:
+## Usage
+
+### Basic Operations
+
+#### Create/Insert Records
 
 ```ruby
-# Insert/Create
-# Add as many fields per record as needed.
-new_account = Hash["name" => "Test Account", "type" => "Other"]
-records_to_insert = Array.new
-# You can add as many records as you want here, just keep in mind that Salesforce has governor limits.
-records_to_insert.push(new_account)
+new_account = { "name" => "Test Account", "type" => "Other" }
+records_to_insert = [new_account]
+
+# Basic usage
 result = salesforce.create("Account", records_to_insert)
-puts "result is: #{result.inspect}"
 
-# Update
-updated_account = Hash["name" => "Test Account -- Updated", id => "a00A0001009zA2m"] # Nearly identical to an insert, but we need to pass the salesforce id.
-records_to_update = Array.new
-records_to_update.push(updated_account)
+# With response and custom batch size
+result = salesforce.create("Account", records_to_insert, true, false, [], 5000)
+```
+
+#### Update Records
+
+```ruby
+updated_account = { "name" => "Test Account -- Updated", "id" => "a00A0001009zA2m" }
+records_to_update = [updated_account]
+
+# Basic usage
 salesforce.update("Account", records_to_update)
 
-# Upsert
-upserted_account = Hash["name" => "Test Account -- Upserted", "External_Field_Name" => "123456"] # Fields to be updated. External field must be included
-records_to_upsert = Array.new
-records_to_upsert.push(upserted_account)
-salesforce.upsert("Account", records_to_upsert, "External_Field_Name") # Note that upsert accepts an extra parameter for the external field name
+# With null handling
+salesforce.update("Account", records_to_update, true, true, ["Phone"])
+```
+
+#### Upsert Records
+
+```ruby
+upserted_account = { "name" => "Test Account -- Upserted", "External_Field_Name" => "123456" }
+records_to_upsert = [upserted_account]
+
+# Basic usage
+salesforce.upsert("Account", records_to_upsert, "External_Field_Name")
 
-# Delete
-deleted_account = Hash["id" => "a00A0001009zA2m"] # We only specify the id of the records to delete
-records_to_delete = Array.new
-records_to_delete.push(deleted_account)
+# With all options
+result = salesforce.upsert("Account", records_to_upsert, "External_Field_Name", true, false, [], 10000, 3600)
+```
+
+#### Delete Records
+
+```ruby
+deleted_account = { "id" => "a00A0001009zA2m" }
+records_to_delete = [deleted_account]
+
+# Basic usage
 salesforce.delete("Account", records_to_delete)
 
-# Query
-res = salesforce.query("Account", "select id, name, createddate from Account limit 3") # We just need to pass the sobject name and the query string
+# With response
+result = salesforce.delete("Account", records_to_delete, true)
+```
+
+#### Query Records
+
+```ruby
+result = salesforce.query("Account", "SELECT id, name, createddate FROM Account LIMIT 3")
+puts "Records found: #{result["batches"][0]["response"].length}"
 ```
 
-### Helpful methods:
+### Method Parameters
+
+All bulk operation methods support additional parameters for fine-tuned control:
+
+#### Complete Method Signatures:
 
 ```ruby
-# Check status of a job via #job_from_id
-job = salesforce.job_from_id('a00A0001009zA2m') # Returns a SalesforceBulkApi::Job instance
-puts "status is: #{job.check_job_status.inspect}"
+# CREATE
+salesforce.create(sobject, records, get_response=false, send_nulls=false, no_null_list=[], batch_size=10000, timeout=1500)
+
+# UPDATE  
+salesforce.update(sobject, records, get_response=false, send_nulls=false, no_null_list=[], batch_size=10000, timeout=1500)
+
+# UPSERT
+salesforce.upsert(sobject, records, external_field, get_response=false, send_nulls=false, no_null_list=[], batch_size=10000, timeout=1500)
+
+# DELETE
+salesforce.delete(sobject, records, get_response=false, batch_size=10000, timeout=1500)
+
+# QUERY
+salesforce.query(sobject, query_string, batch_size=10000, timeout=1500)
 ```
 
-### Listening to events:
+#### Parameter Descriptions:
+
+- **`get_response`** (Boolean): Whether to return batch processing results (default: false)
+- **`send_nulls`** (Boolean): Whether to send null/empty values to Salesforce (default: false)
+- **`no_null_list`** (Array): Fields to exclude from null value handling when `send_nulls` is true
+- **`batch_size`** (Integer): Number of records per batch (default: 10000, max: 10000)
+- **`timeout`** (Integer): Timeout in seconds for job completion (default: 1500)
+
+### Getting Results
+
+When `get_response` is set to true, you'll receive detailed results:
+
+```ruby
+result = salesforce.create("Account", records, true)
+
+# Access job information
+puts "Job ID: #{result['job_id']}"
+puts "Job state: #{result['state']}"
+
+# Access batch results
+result["batches"].each_with_index do |batch, index|
+  puts "Batch #{index + 1}:"
+  puts "  State: #{batch['state'][0]}"
+  puts "  Records processed: #{batch['numberRecordsProcessed'][0]}"
+  
+  if batch["response"]
+    batch["response"].each do |record|
+      if record["success"] == ["true"]
+        puts "  ✓ Success: #{record['id'][0]}"
+      else
+        puts "  ✗ Error: #{record['errors'][0]['message'][0]}"
+      end
+    end
+  end
+end
+```
+
+### Null Value Handling
+
+Control how null and empty values are handled:
+
+```ruby
+records = [
+  { "Id" => "001...", "Name" => "Test", "Phone" => "", "Website" => nil }
+]
+
+# Send nulls for empty/nil fields, except for Phone
+result = salesforce.update("Account", records, true, true, ["Phone"])
+
+# This will:
+# - Set Website to NULL in Salesforce (because it's nil)
+# - Leave Phone unchanged (because it's in no_null_list)
+# - Update Name normally
+```
+
+### Job Management
+
+#### Get Job by ID
+
+```ruby
+job = salesforce.job_from_id('750A0000001234567')
+status = job.check_job_status
+puts "Job state: #{status['state'][0]}"
+puts "Batches total: #{status['numberBatchesTotal'][0]}"
+```
+
+#### Check Job Status
+
+```ruby
+job = salesforce.job_from_id(job_id)
+status = job.check_job_status
+
+puts "Job Information:"
+puts "  State: #{status['state'][0]}"
+puts "  Object: #{status['object'][0]}"
+puts "  Operation: #{status['operation'][0]}"
+puts "  Total Batches: #{status['numberBatchesTotal'][0]}"
+puts "  Completed Batches: #{status['numberBatchesCompleted'][0]}"
+puts "  Failed Batches: #{status['numberBatchesFailed'][0]}"
+```
+
+### Batch Operations
+
+#### Check Batch Status
+
+```ruby
+job = salesforce.job_from_id(job_id)
+batch_status = job.check_batch_status(batch_id)
+
+puts "Batch Information:"
+puts "  State: #{batch_status['state'][0]}"
+puts "  Records Processed: #{batch_status['numberRecordsProcessed'][0]}"
+puts "  Records Failed: #{batch_status['numberRecordsFailed'][0]}"
+```
+
+#### Retrieve Batch Records
+
+```ruby
+job = salesforce.job_from_id(job_id)
+records = job.get_batch_records(batch_id)
+
+puts "Batch Records:"
+records.each do |record|
+  puts "  #{record.inspect}"
+end
+```
+
+#### Get Batch Results
+
+```ruby
+job = salesforce.job_from_id(job_id)
+results = job.get_batch_result(batch_id)
+
+results.each do |result|
+  if result["success"] == ["true"]
+    puts "Success: Record ID #{result['id'][0]}"
+  else
+    puts "Failed: #{result['errors'][0]['message'][0]}"
+  end
+end
+```
+
+### Event Listening
+
+Listen for job creation events:
 
 ```ruby
-# A job is created
-# Useful when you need to store the job_id before any work begins, then if you fail during a complex load scenario, you can wait for your
-# previous job(s) to finish.
 salesforce.on_job_created do |job|
-  puts "Job #{job.job_id} created!"
+  puts "Job #{job.job_id} created for #{job.operation} on #{job.sobject}!"
+  
+  # You can perform additional operations here
+  # like logging, notifications, etc.
 end
+
+# Now when you create/update/etc, the listener will be called
+result = salesforce.create("Account", records)
 ```
 
-### Fetching records from a batch
+### API Call Throttling
+
+Control the frequency of status checks to avoid hitting API limits:
+
+```ruby
+# Set status check interval to 30 seconds (default is 5 seconds)
+salesforce.connection.set_status_throttle(30)
+
+# Check current throttle setting
+puts "Current throttle: #{salesforce.connection.get_status_throttle} seconds"
+```
+
+### Monitoring and Counters
+
+Track API usage and operations:
 
 ```ruby
-job_id = 'l02A0231009Za8m'
-batch_id = 'H24a0708089zA2J'
-salesforce.get_batch_records(job_id, batch_id)
-# => [{"Id"=>["RECORD_ID_1"], "AField__c"=>["123123"]},
-      {"Id"=>["RECORD_ID_2"], "AField__c"=>["123123"]},
-      {"Id"=>["RECORD_ID_3"], "AField__c"=>["123123"]}]
+# Get operation counters
+counters = salesforce.counters
+puts "API Usage: #{counters}"
+# => {:http_get=>15, :http_post=>8, :upsert=>2, :update=>1, :create=>3, :delete=>0, :query=>2}
+
+# Reset counters
+salesforce.reset_counters
+```
+
+## Error Handling
 
+The gem provides comprehensive error handling:
+
+```ruby
+begin
+  result = salesforce.create("Account", records, true)
+  
+  # Check for batch-level errors
+  result["batches"].each do |batch|
+    if batch["state"][0] == "Failed"
+      puts "Batch failed: #{batch["stateMessage"][0]}"
+    end
+  end
+  
+rescue SalesforceBulkApi::Job::SalesforceException => e
+  puts "Salesforce API error: #{e.message}"
+  # Handle API-level errors (invalid objects, fields, etc.)
+  
+rescue SalesforceBulkApi::JobTimeout => e
+  puts "Job timed out: #{e.message}"
+  # Handle timeout errors - job took longer than specified timeout
+  
+rescue => e
+  puts "Unexpected error: #{e.message}"
+  # Handle other errors (network issues, authentication, etc.)
+end
 ```
 
-### Throttling API calls:
+### Common Error Scenarios
 
 ```ruby
-# By default, this gem (and maybe your app driving it) will query job/batch statuses at an unbounded rate.  We
-# can fix that, e.g.:
-salesforce.connection.set_status_throttle(30) # only check status of individual jobs/batches every 30 seconds
+# Invalid field names
+begin
+  records = [{ "InvalidField__c" => "value" }]
+  salesforce.create("Account", records, true)
+rescue SalesforceBulkApi::Job::SalesforceException => e
+  puts "Field error: #{e.message}"
+end
+
+# Malformed record IDs
+begin
+  records = [{ "Id" => "invalid_id" }]
+  salesforce.update("Account", records, true)
+rescue => e
+  # This might not raise immediately - check batch results
+  result = salesforce.update("Account", records, true)
+  failed_records = result["batches"][0]["response"].select { |r| r["success"] == ["false"] }
+  failed_records.each { |r| puts "Failed: #{r['errors'][0]['message'][0]}" }
+end
+```
+
+## Advanced Features
+
+### Relationship Fields
+
+You can work with relationship fields using dot notation:
+
+```ruby
+# Create records with relationship data
+records = [
+  {
+    "Name" => "Test Account",
+    "Parent.Name" => "Parent Account Name",
+    "Owner.Email" => "owner@example.com"
+  }
+]
+
+result = salesforce.create("Account", records, true)
+```
+
+### Special Data Types
+
+The gem automatically handles various data types:
+
+```ruby
+records = [
+  {
+    "Name" => "Test Account",
+    "AnnualRevenue" => 1000000,                    # Numbers
+    "IsActive__c" => true,                         # Booleans  
+    "LastModifiedDate" => Time.now,                # Timestamps (converted to ISO8601)
+    "Description" => "Text with <special> chars"   # XML encoding handled automatically
+  }
+]
+```
+
+### Large Dataset Handling
+
+For large datasets, the gem automatically handles batching:
+
+```ruby
+# This will be automatically split into multiple batches of 10,000 records each
+large_dataset = (1..50000).map { |i| { "Name" => "Account #{i}" } }
+
+result = salesforce.create("Account", large_dataset, true, false, [], 10000, 7200) # 2 hour timeout
+puts "Created #{result['batches'].length} batches"
+```
+
+### Custom Batch Sizes
+
+Optimize for your use case:
+
+```ruby
+# Smaller batches for complex records
+complex_records = [...]
+salesforce.create("CustomObject__c", complex_records, true, false, [], 2000)
+
+# Larger batches for simple records (up to 10,000)
+simple_records = [...]
+salesforce.create("Account", simple_records, true, false, [], 10000)
+```
+
+## Contributing
+
+We welcome contributions to improve this gem. Feel free to:
+
+1. Fork the repository
+2. Create your feature branch (`git checkout -b feature/amazing-feature`)
+3. Commit your changes (`git commit -am 'Add some amazing feature'`)
+4. Push to the branch (`git push origin feature/amazing-feature`)
+5. Create a new Pull Request
+
+### Development Setup
+
+```bash
+git clone https://github.com/yatish27/salesforce_bulk_api.git
+cd salesforce_bulk_api
+bundle install
+
+# Copy environment template
+cp .env.sample .env
+# Edit .env with your Salesforce credentials
+
+# Run tests
+bundle exec rspec
+
+# Run RuboCop
+bundle exec rubocop
 ```
 
-## Contribute
+## License
 
-Feel to fork and send Pull request
+This project is licensed under the MIT License, Copyright (c) 2025 - see the [LICENCE](LICENCE) file for details.

data/Rakefile

@@ -1,3 +1,3 @@
-require 'rspec/core/rake_task'
-task :default => :spec
-RSpec::Core::RakeTask.new
+require "rspec/core/rake_task"
+task default: :spec
+RSpec::Core::RakeTask.new

data/lib/salesforce_bulk_api/concerns/throttling.rb

@@ -1,6 +1,5 @@
 module SalesforceBulkApi::Concerns
   module Throttling
-
     def throttles
       @throttles.dup
     end
@@ -48,13 +47,12 @@ module SalesforceBulkApi::Concerns
       @limits
     end
 
-    def throttle(details={})
+    def throttle(details = {})
       (@throttles || []).each do |callback|
         args = [details]
         args = args[0..callback.arity]
         callback.call(*args)
       end
     end
-
   end
 end

data/lib/salesforce_bulk_api/connection.rb

@@ -1,20 +1,20 @@
-require 'timeout'
+require "timeout"
 
 module SalesforceBulkApi
   class Connection
     include Concerns::Throttling
 
-    LOGIN_HOST = 'login.salesforce.com'
+    LOGIN_HOST = "login.salesforce.com"
 
     def initialize(api_version, client)
       @client = client
       @api_version = api_version
       @path_prefix = "/services/async/#{@api_version}/"
 
-      login()
+      login
     end
 
-    def login()
+    def login
       client_type = @client.class.to_s
       case client_type
       when "Restforce::Data::Client"
@@ -24,14 +24,14 @@ module SalesforceBulkApi
         @session_id = @client.oauth_token
         @server_url = @client.instance_url
       end
-      @instance = parse_instance()
+      @instance = parse_instance
       @instance_host = "#{@instance}.salesforce.com"
     end
 
     def post_xml(host, path, xml, headers)
-      host = host || @instance_host
+      host ||= @instance_host
       if host != LOGIN_HOST # Not login, need to add session id to header
-        headers['X-SFDC-Session'] = @session_id
+        headers["X-SFDC-Session"] = @session_id
         path = "#{@path_prefix}#{path}"
       end
       i = 0
@@ -52,10 +52,10 @@ module SalesforceBulkApi
     end
 
     def get_request(host, path, headers)
-      host = host || @instance_host
+      host ||= @instance_host
       path = "#{@path_prefix}#{path}"
       if host != LOGIN_HOST # Not login, need to add session id to header
-        headers['X-SFDC-Session'] = @session_id;
+        headers["X-SFDC-Session"] = @session_id
       end
 
       count :get
@@ -80,19 +80,17 @@ module SalesforceBulkApi
     private
 
     def get_counters
-      @counters ||= Hash.new(0)
+      @counters ||= Hash.new { |hash, key| hash[key] = 0 }
     end
 
     def count(http_method)
       get_counters[http_method] += 1
     end
 
-    def parse_instance()
-      @instance = @server_url.match(/https:\/\/[a-z]{2}[0-9]{1,2}\./).to_s.gsub("https://","").split(".")[0]
+    def parse_instance
+      @instance = @server_url.match(/https:\/\/[a-z]{2}[0-9]{1,2}\./).to_s.gsub("https://", "").split(".")[0]
       @instance = @server_url.split(".salesforce.com")[0].split("://")[1] if @instance.nil? || @instance.empty?
       @instance
     end
-
   end
-
 end