salesforce_bulk_api 1.2.0 → 1.3.0

data/README.md

@@ -9,10 +9,16 @@
 - [Authentication](#authentication)
 - [Usage](#usage)
   - [Basic Operations](#basic-operations)
+  - [Method Parameters](#method-parameters)
+  - [Getting Results](#getting-results)
+  - [Null Value Handling](#null-value-handling)
   - [Job Management](#job-management)
+  - [Batch Operations](#batch-operations)
   - [Event Listening](#event-listening)
-  - [Retrieving Batch Records](#retrieving-batch-records)
   - [API Call Throttling](#api-call-throttling)
+  - [Monitoring and Counters](#monitoring-and-counters)
+- [Error Handling](#error-handling)
+- [Advanced Features](#advanced-features)
 - [Contributing](#contributing)
 - [License](#license)
 
@@ -20,6 +26,14 @@
 
 `SalesforceBulkApi` is a Ruby wrapper for the Salesforce Bulk API. It is rewritten from [salesforce_bulk](https://github.com/jorgevaldivia/salesforce_bulk) and adds several missing features, making it easier to perform bulk operations with Salesforce from Ruby applications.
 
+Key features:
+- Support for all Bulk API operations (create, update, upsert, delete, query)
+- Comprehensive error handling
+- Job and batch status monitoring
+- Event listening for job lifecycle
+- API call throttling and monitoring
+- Performance optimized string concatenation for large batches
+
 ## Installation
 
 Add this line to your application's Gemfile:
@@ -95,8 +109,12 @@ salesforce = SalesforceBulkApi::Api.new(client)
 ```ruby
 new_account = { "name" => "Test Account", "type" => "Other" }
 records_to_insert = [new_account]
+
+# Basic usage
 result = salesforce.create("Account", records_to_insert)
-puts "Result: #{result.inspect}"
+
+# With response and custom batch size
+result = salesforce.create("Account", records_to_insert, true, false, [], 5000)
 ```
 
 #### Update Records
@@ -104,7 +122,12 @@ puts "Result: #{result.inspect}"
 ```ruby
 updated_account = { "name" => "Test Account -- Updated", "id" => "a00A0001009zA2m" }
 records_to_update = [updated_account]
+
+# Basic usage
 salesforce.update("Account", records_to_update)
+
+# With null handling
+salesforce.update("Account", records_to_update, true, true, ["Phone"])
 ```
 
 #### Upsert Records
@@ -112,7 +135,12 @@ salesforce.update("Account", records_to_update)
 ```ruby
 upserted_account = { "name" => "Test Account -- Upserted", "External_Field_Name" => "123456" }
 records_to_upsert = [upserted_account]
+
+# Basic usage
 salesforce.upsert("Account", records_to_upsert, "External_Field_Name")
+
+# With all options
+result = salesforce.upsert("Account", records_to_upsert, "External_Field_Name", true, false, [], 10000, 3600)
 ```
 
 #### Delete Records
@@ -120,51 +148,319 @@ salesforce.upsert("Account", records_to_upsert, "External_Field_Name")
 ```ruby
 deleted_account = { "id" => "a00A0001009zA2m" }
 records_to_delete = [deleted_account]
+
+# Basic usage
 salesforce.delete("Account", records_to_delete)
+
+# With response
+result = salesforce.delete("Account", records_to_delete, true)
 ```
 
 #### Query Records
 
 ```ruby
-res = salesforce.query("Account", "SELECT id, name, createddate FROM Account LIMIT 3")
+result = salesforce.query("Account", "SELECT id, name, createddate FROM Account LIMIT 3")
+puts "Records found: #{result["batches"][0]["response"].length}"
+```
+
+### Method Parameters
+
+All bulk operation methods support additional parameters for fine-tuned control:
+
+#### Complete Method Signatures:
+
+```ruby
+# CREATE
+salesforce.create(sobject, records, get_response=false, send_nulls=false, no_null_list=[], batch_size=10000, timeout=1500)
+
+# UPDATE  
+salesforce.update(sobject, records, get_response=false, send_nulls=false, no_null_list=[], batch_size=10000, timeout=1500)
+
+# UPSERT
+salesforce.upsert(sobject, records, external_field, get_response=false, send_nulls=false, no_null_list=[], batch_size=10000, timeout=1500)
+
+# DELETE
+salesforce.delete(sobject, records, get_response=false, batch_size=10000, timeout=1500)
+
+# QUERY
+salesforce.query(sobject, query_string, batch_size=10000, timeout=1500)
+```
+
+#### Parameter Descriptions:
+
+- **`get_response`** (Boolean): Whether to return batch processing results (default: false)
+- **`send_nulls`** (Boolean): Whether to send null/empty values to Salesforce (default: false)
+- **`no_null_list`** (Array): Fields to exclude from null value handling when `send_nulls` is true
+- **`batch_size`** (Integer): Number of records per batch (default: 10000, max: 10000)
+- **`timeout`** (Integer): Timeout in seconds for job completion (default: 1500)
+
+### Getting Results
+
+When `get_response` is set to true, you'll receive detailed results:
+
+```ruby
+result = salesforce.create("Account", records, true)
+
+# Access job information
+puts "Job ID: #{result['job_id']}"
+puts "Job state: #{result['state']}"
+
+# Access batch results
+result["batches"].each_with_index do |batch, index|
+  puts "Batch #{index + 1}:"
+  puts "  State: #{batch['state'][0]}"
+  puts "  Records processed: #{batch['numberRecordsProcessed'][0]}"
+  
+  if batch["response"]
+    batch["response"].each do |record|
+      if record["success"] == ["true"]
+        puts "  ✓ Success: #{record['id'][0]}"
+      else
+        puts "  ✗ Error: #{record['errors'][0]['message'][0]}"
+      end
+    end
+  end
+end
+```
+
+### Null Value Handling
+
+Control how null and empty values are handled:
+
+```ruby
+records = [
+  { "Id" => "001...", "Name" => "Test", "Phone" => "", "Website" => nil }
+]
+
+# Send nulls for empty/nil fields, except for Phone
+result = salesforce.update("Account", records, true, true, ["Phone"])
+
+# This will:
+# - Set Website to NULL in Salesforce (because it's nil)
+# - Leave Phone unchanged (because it's in no_null_list)
+# - Update Name normally
 ```
 
 ### Job Management
 
-You can check the status of a job using its ID:
+#### Get Job by ID
 
 ```ruby
-job = salesforce.job_from_id('a00A0001009zA2m')
-puts "Status: #{job.check_job_status.inspect}"
+job = salesforce.job_from_id('750A0000001234567')
+status = job.check_job_status
+puts "Job state: #{status['state'][0]}"
+puts "Batches total: #{status['numberBatchesTotal'][0]}"
 ```
 
-### Event Listening
+#### Check Job Status
+
+```ruby
+job = salesforce.job_from_id(job_id)
+status = job.check_job_status
+
+puts "Job Information:"
+puts "  State: #{status['state'][0]}"
+puts "  Object: #{status['object'][0]}"
+puts "  Operation: #{status['operation'][0]}"
+puts "  Total Batches: #{status['numberBatchesTotal'][0]}"
+puts "  Completed Batches: #{status['numberBatchesCompleted'][0]}"
+puts "  Failed Batches: #{status['numberBatchesFailed'][0]}"
+```
+
+### Batch Operations
 
-You can listen for job creation events:
+#### Check Batch Status
 
 ```ruby
-salesforce.on_job_created do |job|
-  puts "Job #{job.job_id} created!"
+job = salesforce.job_from_id(job_id)
+batch_status = job.check_batch_status(batch_id)
+
+puts "Batch Information:"
+puts "  State: #{batch_status['state'][0]}"
+puts "  Records Processed: #{batch_status['numberRecordsProcessed'][0]}"
+puts "  Records Failed: #{batch_status['numberRecordsFailed'][0]}"
+```
+
+#### Retrieve Batch Records
+
+```ruby
+job = salesforce.job_from_id(job_id)
+records = job.get_batch_records(batch_id)
+
+puts "Batch Records:"
+records.each do |record|
+  puts "  #{record.inspect}"
 end
 ```
 
-### Retrieving Batch Records
+#### Get Batch Results
 
-Fetch records from a specific batch in a job:
+```ruby
+job = salesforce.job_from_id(job_id)
+results = job.get_batch_result(batch_id)
+
+results.each do |result|
+  if result["success"] == ["true"]
+    puts "Success: Record ID #{result['id'][0]}"
+  else
+    puts "Failed: #{result['errors'][0]['message'][0]}"
+  end
+end
+```
+
+### Event Listening
+
+Listen for job creation events:
 
 ```ruby
-job_id = 'l02A0231009Za8m'
-batch_id = 'H24a0708089zA2J'
-records = salesforce.get_batch_records(job_id, batch_id)
+salesforce.on_job_created do |job|
+  puts "Job #{job.job_id} created for #{job.operation} on #{job.sobject}!"
+  
+  # You can perform additional operations here
+  # like logging, notifications, etc.
+end
+
+# Now when you create/update/etc, the listener will be called
+result = salesforce.create("Account", records)
 ```
 
 ### API Call Throttling
 
-You can control how frequently status checks are performed:
+Control the frequency of status checks to avoid hitting API limits:
 
 ```ruby
-# Set status check interval to 30 seconds
+# Set status check interval to 30 seconds (default is 5 seconds)
 salesforce.connection.set_status_throttle(30)
+
+# Check current throttle setting
+puts "Current throttle: #{salesforce.connection.get_status_throttle} seconds"
+```
+
+### Monitoring and Counters
+
+Track API usage and operations:
+
+```ruby
+# Get operation counters
+counters = salesforce.counters
+puts "API Usage: #{counters}"
+# => {:http_get=>15, :http_post=>8, :upsert=>2, :update=>1, :create=>3, :delete=>0, :query=>2}
+
+# Reset counters
+salesforce.reset_counters
+```
+
+## Error Handling
+
+The gem provides comprehensive error handling:
+
+```ruby
+begin
+  result = salesforce.create("Account", records, true)
+  
+  # Check for batch-level errors
+  result["batches"].each do |batch|
+    if batch["state"][0] == "Failed"
+      puts "Batch failed: #{batch["stateMessage"][0]}"
+    end
+  end
+  
+rescue SalesforceBulkApi::Job::SalesforceException => e
+  puts "Salesforce API error: #{e.message}"
+  # Handle API-level errors (invalid objects, fields, etc.)
+  
+rescue SalesforceBulkApi::JobTimeout => e
+  puts "Job timed out: #{e.message}"
+  # Handle timeout errors - job took longer than specified timeout
+  
+rescue => e
+  puts "Unexpected error: #{e.message}"
+  # Handle other errors (network issues, authentication, etc.)
+end
+```
+
+### Common Error Scenarios
+
+```ruby
+# Invalid field names
+begin
+  records = [{ "InvalidField__c" => "value" }]
+  salesforce.create("Account", records, true)
+rescue SalesforceBulkApi::Job::SalesforceException => e
+  puts "Field error: #{e.message}"
+end
+
+# Malformed record IDs
+begin
+  records = [{ "Id" => "invalid_id" }]
+  salesforce.update("Account", records, true)
+rescue => e
+  # This might not raise immediately - check batch results
+  result = salesforce.update("Account", records, true)
+  failed_records = result["batches"][0]["response"].select { |r| r["success"] == ["false"] }
+  failed_records.each { |r| puts "Failed: #{r['errors'][0]['message'][0]}" }
+end
+```
+
+## Advanced Features
+
+### Relationship Fields
+
+You can work with relationship fields using dot notation:
+
+```ruby
+# Create records with relationship data
+records = [
+  {
+    "Name" => "Test Account",
+    "Parent.Name" => "Parent Account Name",
+    "Owner.Email" => "owner@example.com"
+  }
+]
+
+result = salesforce.create("Account", records, true)
+```
+
+### Special Data Types
+
+The gem automatically handles various data types:
+
+```ruby
+records = [
+  {
+    "Name" => "Test Account",
+    "AnnualRevenue" => 1000000,                    # Numbers
+    "IsActive__c" => true,                         # Booleans  
+    "LastModifiedDate" => Time.now,                # Timestamps (converted to ISO8601)
+    "Description" => "Text with <special> chars"   # XML encoding handled automatically
+  }
+]
+```
+
+### Large Dataset Handling
+
+For large datasets, the gem automatically handles batching:
+
+```ruby
+# This will be automatically split into multiple batches of 10,000 records each
+large_dataset = (1..50000).map { |i| { "Name" => "Account #{i}" } }
+
+result = salesforce.create("Account", large_dataset, true, false, [], 10000, 7200) # 2 hour timeout
+puts "Created #{result['batches'].length} batches"
+```
+
+### Custom Batch Sizes
+
+Optimize for your use case:
+
+```ruby
+# Smaller batches for complex records
+complex_records = [...]
+salesforce.create("CustomObject__c", complex_records, true, false, [], 2000)
+
+# Larger batches for simple records (up to 10,000)
+simple_records = [...]
+salesforce.create("Account", simple_records, true, false, [], 10000)
 ```
 
 ## Contributing
@@ -177,6 +473,24 @@ We welcome contributions to improve this gem. Feel free to:
 4. Push to the branch (`git push origin feature/amazing-feature`)
 5. Create a new Pull Request
 
+### Development Setup
+
+```bash
+git clone https://github.com/yatish27/salesforce_bulk_api.git
+cd salesforce_bulk_api
+bundle install
+
+# Copy environment template
+cp .env.sample .env
+# Edit .env with your Salesforce credentials
+
+# Run tests
+bundle exec rspec
+
+# Run RuboCop
+bundle exec rubocop
+```
+
 ## License
 
-This project is licensed under the MIT License, Copyright (c) 2025 - see the [LICENCE](LICENCE) file for details.
+This project is licensed under the MIT License, Copyright (c) 2025 - see the [LICENCE](LICENCE) file for details.

data/lib/salesforce_bulk_api/concerns/throttling.rb

@@ -19,10 +19,10 @@ module SalesforceBulkApi::Concerns
         key = extract_constraint_key_from(details, throttle_by_keys)
         last_request = limit_log[key]
 
-        if last_request && only_if.call(details)
+        if !last_request.nil? && only_if.call(details)
           seconds_since_last_request = Time.now.to_f - last_request.to_f
           need_to_wait_seconds = limit_seconds - seconds_since_last_request
-          sleep(need_to_wait_seconds) if need_to_wait_seconds.positive?
+          sleep(need_to_wait_seconds) if need_to_wait_seconds > 0
         end
 
         limit_log[key] = Time.now
@@ -32,17 +32,26 @@ module SalesforceBulkApi::Concerns
     private
 
     def extract_constraint_key_from(details, throttle_by_keys)
-      throttle_by_keys.each_with_object({}) { |k, hash| hash[k] = details[k] }
+      hash = {}
+      throttle_by_keys.each { |k| hash[k] = details[k] }
+      hash
     end
 
     def get_limit_log(prune_older_than)
-      @limits ||= {}
-      @limits.delete_if { |_, v| v < prune_older_than }
+      @limits ||= Hash.new(0)
+
+      @limits.delete_if do |k, v|
+        v < prune_older_than
+      end
+
+      @limits
     end
 
     def throttle(details = {})
       (@throttles || []).each do |callback|
-        callback.call(details)
+        args = [details]
+        args = args[0..callback.arity]
+        callback.call(*args)
       end
     end
   end

data/lib/salesforce_bulk_api/connection.rb

@@ -1,92 +1,96 @@
 require "timeout"
-require "net/https"
 
 module SalesforceBulkApi
   class Connection
     include Concerns::Throttling
 
-    LOGIN_HOST = "login.salesforce.com".freeze
-
-    attr_reader :session_id, :server_url, :instance, :instance_host
+    LOGIN_HOST = "login.salesforce.com"
 
     def initialize(api_version, client)
       @client = client
       @api_version = api_version
       @path_prefix = "/services/async/#{@api_version}/"
-      @counters = Hash.new(0)
 
       login
     end
 
-    def post_xml(host, path, xml, headers)
-      host ||= @instance_host
-      headers["X-SFDC-Session"] = @session_id unless host == LOGIN_HOST
-      path = "#{@path_prefix}#{path}" unless host == LOGIN_HOST
-
-      perform_request(:post, host, path, xml, headers)
-    end
-
-    def get_request(host, path, headers)
-      host ||= @instance_host
-      path = "#{@path_prefix}#{path}"
-      headers["X-SFDC-Session"] = @session_id unless host == LOGIN_HOST
-
-      perform_request(:get, host, path, nil, headers)
-    end
-
-    def counters
-      {
-        get: @counters[:get],
-        post: @counters[:post]
-      }
-    end
-
-    private
-
     def login
       client_type = @client.class.to_s
-      @session_id, @server_url = if client_type == "Restforce::Data::Client"
-        [@client.options[:oauth_token], @client.options[:instance_url]]
+      case client_type
+      when "Restforce::Data::Client"
+        @session_id = @client.options[:oauth_token]
+        @server_url = @client.options[:instance_url]
       else
-        [@client.oauth_token, @client.instance_url]
+        @session_id = @client.oauth_token
+        @server_url = @client.instance_url
       end
       @instance = parse_instance
       @instance_host = "#{@instance}.salesforce.com"
     end
 
-    def perform_request(method, host, path, body, headers)
-      retries = 0
+    def post_xml(host, path, xml, headers)
+      host ||= @instance_host
+      if host != LOGIN_HOST # Not login, need to add session id to header
+        headers["X-SFDC-Session"] = @session_id
+        path = "#{@path_prefix}#{path}"
+      end
+      i = 0
       begin
-        count(method)
-        throttle(http_method: method, path: path)
-        response = https(host).public_send(method, path, body, headers)
-        response.body
-      rescue => e
-        retries += 1
-        if retries < 3
-          puts "Request fail #{retries}: Retrying #{path}"
+        count :post
+        throttle(http_method: :post, path: path)
+        https(host).post(path, xml, headers).body
+      rescue
+        i += 1
+        if i < 3
+          puts "Request fail #{i}: Retrying #{path}"
           retry
         else
           puts "FATAL: Request to #{path} failed three times."
-          raise e
+          raise
         end
       end
     end
 
-    def https(host)
-      Net::HTTP.new(host, 443).tap do |http|
-        http.use_ssl = true
-        http.verify_mode = OpenSSL::SSL::VERIFY_NONE
+    def get_request(host, path, headers)
+      host ||= @instance_host
+      path = "#{@path_prefix}#{path}"
+      if host != LOGIN_HOST # Not login, need to add session id to header
+        headers["X-SFDC-Session"] = @session_id
       end
+
+      count :get
+      throttle(http_method: :get, path: path)
+      https(host).get(path, headers).body
+    end
+
+    def https(host)
+      req = Net::HTTP.new(host, 443)
+      req.use_ssl = true
+      req.verify_mode = OpenSSL::SSL::VERIFY_NONE
+      req
+    end
+
+    def counters
+      {
+        get: get_counters[:get],
+        post: get_counters[:post]
+      }
+    end
+
+    private
+
+    def get_counters
+      @counters ||= Hash.new { |hash, key| hash[key] = 0 }
     end
 
     def count(http_method)
-      @counters[http_method] += 1
+      get_counters[http_method] += 1
     end
 
     def parse_instance
-      instance = @server_url.match(%r{https://([a-z]{2}[0-9]{1,2})\.})&.captures&.first
-      instance || @server_url.split(".salesforce.com").first.split("://").last
+      @instance = @server_url.match(/https:\/\/[a-z]{2}[0-9]{1,2}\./).to_s.gsub("https://", "").split(".")[0]
+      @instance = @server_url.split(".salesforce.com")[0].split("://")[1] if @instance.nil? || @instance.empty?
+      @instance
     end
   end
 end