salesforce_bulk_query 0.1.6 → 0.2.0

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA1:
-  metadata.gz: 93410e3eaddd8559f8127d9cad6ed1223093da53
-  data.tar.gz: f24d761b7d5e39c67efc54eb99c9e9006a7b3330
+  metadata.gz: 3c1805666d306731c5d9118b5e231ae9940fc92f
+  data.tar.gz: 347fc0ada770fd4ae8b7bcaaf32d6b3b14516a9e
 SHA512:
-  metadata.gz: d849ee9f3525ecd0a92c57d4c997c7158d7b830892b0aaa221b9b38d7e55d0042d08dd19a020f1729d7deea37df9790d1e81bf4dab6bf15042cf2197d5501033
-  data.tar.gz: 11c53de51a4c6cbe7541e63cf2f236900b559d12b464e287c79c7b800db1d13ff1ff6ae097a21030c59cb6cb95d24c2512844c459728b73a51aed6b8aba380dd
+  metadata.gz: 38d588dea39c3f5ce3d7bb230d8af27729c236ebc9a597d58566af64f8f8dee3f2cc459cf1f893e4e344bf8a53a9e2e0f4bd89d3f9e79a28e0d34719d81bc0e7
+  data.tar.gz: 1fd675272be999224f69cef8a6dbc593e00d0bf9d9879d9e600228b42687969279a2960dc29380365877454d93bea831e5662ecba3172f5df78291b5c65c04dd

data/README.md

@@ -77,7 +77,7 @@ end
 
 ## How it works
 
-The library uses the [Salesforce Bulk API](https://www.salesforce.com/us/developer/docs/api_asynch/index_Left.htm#CSHID=asynch_api_bulk_query.htm|StartTopic=Content%2Fasynch_api_bulk_query.htm|SkinName=webhelp). The given query is divided into 15 subqueries, according to the [limits](http://www.salesforce.com/us/developer/docs/api_asynchpre/Content/asynch_api_concepts_limits.htm#batch_proc_time_title). Each subquery is an interval based on the CreatedDate Salesforce field. The limits are passed to the API in SOQL queries. Subqueries are sent to the API as batches and added to a job. 
+The library uses the [Salesforce Bulk API](https://www.salesforce.com/us/developer/docs/api_asynch/index_Left.htm#CSHID=asynch_api_bulk_query.htm|StartTopic=Content%2Fasynch_api_bulk_query.htm|SkinName=webhelp). The given query is divided into 15 subqueries, according to the [limits](http://www.salesforce.com/us/developer/docs/api_asynchpre/Content/asynch_api_concepts_limits.htm#batch_proc_time_title). Each subquery is an interval based on the CreatedDate Salesforce field (date field can be customized). The limits are passed to the API in SOQL queries. Subqueries are sent to the API as batches and added to a job. 
 
 The first interval starts with the date the first Salesforce object was created, we query Salesforce REST API for that. If this query times out, we use a constant. The last interval ends a few minutes before now to avoid consistency issues. Custom start and end can be passed - see Options.
 
@@ -87,13 +87,14 @@ CSV results are downloaded by chunks, so that we don't run into memory related i
 
 ## Options
 There are a few optional settings you can pass to the `Api` methods:
-* `api_version`: which Salesforce api version should be used
-* `logger`: where logs should go
-* `filename_prefix`: prefix applied to csv files
-* `directory_path`: custom direcotory path for CSVs, if omitted, a new temp directory is created
-* `check_interval`: how often the results should be checked in secs. 
-* `time_limit`: maximum time the query can take. If this time limit is exceeded, available results are downloaded and the list of subqueries that didn't finished is returned. In seconds. The limti should be understood as limit for waiting. When the limit is reached the function downloads data that is ready which can take some additonal time. If no limit is given the query runs until it finishes
-* `created_from`, `created_to`: limits for the CreatedDate field. Note that queries can't contain any WHERE statements as we're doing some manipulations to create subqueries and we don't want things to get too difficult. So this is the way to limit the query yourself. The format is like `"1999-01-01T00:00:00.000Z"`
+* `api_version`: Which Salesforce api version should be used
+* `logger`: Where logs should go, see the example.
+* `filename_prefix`: Prefix applied to csv files.
+* `directory_path`: Custom direcotory path for CSVs, if omitted, a new temp directory is created.
+* `check_interval`: How often the results should be checked in secs. 
+* `time_limit`: Maximum time the query can take. If this time limit is exceeded, available results are downloaded and the list of subqueries that didn't finished is returned. In seconds. The limti should be understood as limit for waiting. When the limit is reached the function downloads data that is ready which can take some additonal time. If no limit is given the query runs until it finishes.
+* `date_field`: Salesforce date field that will be used for splitting the query, and optionally limiting the results. Must be a date field on the queried sobject. Default is `CreatedDate`
+* `date_from`, `date_to`: limits for the `date_field` (`CreatedDate` by default). Note that queries can't contain any WHERE statements as we're doing some manipulations to create subqueries and we don't want things to get too difficult. So this is the way to limit the query yourself. The format is like `"1999-01-01T00:00:00.000Z"`
 * `single_batch`: If true, the queries are not divided into subqueries as described above. Instead one batch job is created with the given query. This is faster for small amount of data, but will fail with a timeout if you have a lot of data. 
 
 See specs for exact usage.
@@ -119,10 +120,14 @@ If you're using Restforce as a client (which you probably are) and you want to d
 
 ## Notes
 
-Query (user given) -> Job (Salesforce construct that encapsulates 15 batches) -> Batch (1 SOQL with CreatedDate constraints)
+Query (user given) -> Job (Salesforce construct that encapsulates 15 batches) -> Batch (1 SOQL with Date constraints)
 
 At the beginning the query is divided into 15 subqueries and put into a single job. When one of the subqueries fails, a new job with 15 subqueries is created, the range of the failed query is divided into 15 sub-subqueries.
 
+## Change policy
+
+The gem is trying to follow [semantic versioning](http://semver.org/). All  methods and options described in this readme document are considered public API. If any of these change, at least the minor version is bumped. Methods and their params not described in this document can change even in patches.
+
 ## Running tests locally
 Travis CI is set up for this repository to make sure all the tests are passing with each commit.
 

data/lib/salesforce_bulk_query/batch.rb

@@ -12,6 +12,7 @@ module SalesforceBulkQuery
       @soql = options[:soql]
       @job_id = options[:job_id]
       @connection = options[:connection]
+      @date_field = options[:date_field] or fail "date_field must be given when creating a batch"
       @start = options[:start]
       @stop = options[:stop]
       @logger = options[:logger]
@@ -77,7 +78,7 @@ module SalesforceBulkQuery
     end
 
     def get_filename
-      return "#{@sobject}_#{@start}_#{@stop}_#{@batch_id}.csv"
+      return "#{@sobject}_#{@date_field}_#{@start}_#{@stop}_#{@batch_id}.csv"
     end
 
     def get_result(options={})
@@ -116,7 +117,7 @@ module SalesforceBulkQuery
     end
 
     def verify
-      api_count = @connection.query_count(@sobject, @start, @stop)
+      api_count = @connection.query_count(@sobject, @date_field, @start, @stop)
       # if we weren't able to get the count, fail.
       if api_count.nil?
         return @verification = false

data/lib/salesforce_bulk_query/connection.rb

@@ -112,9 +112,9 @@ module SalesforceBulkQuery
       end
     end
 
-    def query_count(sobject, from, to)
+    def query_count(sobject, date_field, from, to)
       # do it with retries, if it doesn't succeed, return nil, don't fail.
-      soql = "SELECT COUNT() FROM #{sobject} WHERE CreatedDate >= #{from} AND CreatedDate < #{to}"
+      soql = "SELECT COUNT() FROM #{sobject} WHERE #{date_field} >= #{from} AND #{date_field} < #{to}"
       begin
         with_retries do
           q = @client.query(soql)
@@ -135,4 +135,4 @@ module SalesforceBulkQuery
       }
     end
   end
-end
+end

data/lib/salesforce_bulk_query/job.rb

@@ -16,6 +16,7 @@ module SalesforceBulkQuery
       @connection = connection
       @logger = options[:logger]
       @job_time_limit = options[:job_time_limit] || JOB_TIME_LIMIT
+      @date_field = options[:date_field] or fail "date_field must be given when creating a batch"
 
       # all batches (static)
       @batches = []
@@ -43,7 +44,7 @@ module SalesforceBulkQuery
     end
 
     def get_extended_soql(soql, from, to)
-      return "#{soql} WHERE CreatedDate >= #{from} AND CreatedDate < #{to}"
+      return "#{soql} WHERE #{@date_field} >= #{from} AND #{@date_field} < #{to}"
     end
 
     def generate_batches(soql, start, stop, single_batch=false)
@@ -88,7 +89,8 @@ module SalesforceBulkQuery
         :connection => @connection,
         :start => options[:start],
         :stop => options[:stop],
-        :logger => @logger
+        :logger => @logger,
+        :date_field => @date_field
       )
       batch.create
 

data/lib/salesforce_bulk_query/query.rb

@@ -6,18 +6,21 @@ module SalesforceBulkQuery
   # Abstraction of a single user-given query. It contains multiple jobs, is tied to a specific connection
   class Query
 
-    # if no created_to is given we use the current time with this offset
+    # if no date_to is given we use the current time with this offset
     # subtracted (to make sure the freshest changes that can be inconsistent
     # aren't there) It's in minutes
     OFFSET_FROM_NOW = 10
 
+    DEFAULT_DATE_FIELD = 'CreatedDate'
+
     def initialize(sobject, soql, connection, options={})
       @sobject = sobject
       @soql = soql
       @connection = connection
       @logger = options[:logger]
-      @created_from = options[:created_from]
-      @created_to = options[:created_to]
+      @date_field = options[:date_field] || DEFAULT_DATE_FIELD
+      @date_from = options[:date_from]
+      @date_to = options[:date_to]
       @single_batch = options[:single_batch]
 
       # jobs currently running
@@ -41,31 +44,23 @@ module SalesforceBulkQuery
     def start(options={})
       # order by and where not allowed
       if (!@single_batch) && (@soql =~ /WHERE/i || @soql =~ /ORDER BY/i)
-        raise "You can't have WHERE or ORDER BY in your soql. If you want to download just specific date range use created_from / created_to"
+        raise "You can't have WHERE or ORDER BY in your soql. If you want to download just specific date range use date_from / date_to"
       end
 
       # create the first job
-      job = SalesforceBulkQuery::Job.new(@sobject, @connection, {:logger => @logger}.merge(options))
+      job = SalesforceBulkQuery::Job.new(
+        @sobject,
+        @connection,
+        {:logger => @logger, :date_field => @date_field}.merge(options)
+      )
       job.create_job
 
       # get the date when it should start
-      if @created_from
-        min_created = @created_from
-      else
-        # get the date when the first was created
-        min_created = nil
-        begin
-          min_created_resp = @connection.client.query("SELECT CreatedDate FROM #{@sobject} ORDER BY CreatedDate LIMIT 1")
-          min_created_resp.each {|s| min_created = s[:CreatedDate]}
-        rescue Faraday::Error::TimeoutError => e
-          @logger.warn "Timeout getting the oldest object for #{@sobject}. Error: #{e}. Using the default value" if @logger
-          min_created = DEFAULT_MIN_CREATED
-        end
-      end
+      min_date = get_min_date
 
       # generate intervals
-      start = DateTime.parse(min_created)
-      stop = @created_to ? DateTime.parse(@created_to) : DateTime.now - Rational(options[:offset_from_now] || OFFSET_FROM_NOW, 1440)
+      start = DateTime.parse(min_date)
+      stop = @date_to ? DateTime.parse(@date_to) : DateTime.now - Rational(options[:offset_from_now] || OFFSET_FROM_NOW, 1440)
       job.generate_batches(@soql, start, stop, @single_batch)
 
       job.close_job
@@ -112,7 +107,11 @@ module SalesforceBulkQuery
         to_split.each do |batch|
           # for each unfinished batch create a new job and add it to new jobs
           @logger.info "The following subquery didn't end in time / failed verification: #{batch.soql}. Dividing into multiple and running again" if @logger
-          new_job = SalesforceBulkQuery::Job.new(@sobject, @connection, {:logger => @logger}.merge(options))
+          new_job = SalesforceBulkQuery::Job.new(
+            @sobject,
+            @connection,
+            {:logger => @logger, :date_field => @date_field}.merge(options)
+          )
           new_job.create_job
           new_job.generate_batches(@soql, batch.start, batch.stop)
           new_job.close_job
@@ -153,5 +152,26 @@ module SalesforceBulkQuery
         :jobs_done => @jobs_done.map { |j| j.job_id }
       }
     end
+
+    private
+
+    def get_min_date
+      if @date_from
+        return @date_from
+      end
+
+      # get the date when the first was created
+      min_created = nil
+      begin
+        min_created_resp = @connection.client.query("SELECT #{@date_field} FROM #{@sobject} ORDER BY #{@date_field} LIMIT 1")
+        min_created_resp.each {|s| min_created = s[@date_field.to_sym]}
+      rescue Faraday::Error::TimeoutError => e
+        @logger.warn "Timeout getting the oldest object for #{@sobject}. Error: #{e}. Using the default value" if @logger
+        min_created = DEFAULT_MIN_CREATED
+      rescue Faraday::Error::ClientError => e
+        fail ArgumentError, "Error when trying to get the oldest record according to #{@date_field}, looks like #{@date_field} is not on #{@sobject}. Original error: #{e}\n #{e.message} \n #{e.backtrace} "
+      end
+      min_created
+    end
   end
 end

data/lib/salesforce_bulk_query/version.rb

@@ -1,3 +1,3 @@
 module SalesforceBulkQuery
-  VERSION = '0.1.6'
+  VERSION = '0.2.0'
 end

data/new-version.sh

@@ -0,0 +1,22 @@
+#!/bin/sh
+
+# get the new version
+VERSION=`bundle exec ruby <<-EORUBY
+
+  require 'salesforce_bulk_query'
+  puts SalesforceBulkQuery::VERSION
+
+EORUBY`
+
+# create tag and push it
+TAG="v$VERSION"
+git tag $TAG
+git push origin $TAG
+
+# build and push the gem
+gem build salesforce_bulk_query.gemspec
+gem push "salesforce_bulk_query-$VERSION.gem"
+
+# update the gem after a few secs
+wait 30
+gem update salesforce_bulk_query

data/spec/salesforce_bulk_query_spec.rb

@@ -104,15 +104,17 @@ describe SalesforceBulkQuery do
         from = "#{frm}T00:00:00.000Z"
         t = "2020-01-01"
         to = "#{t}T00:00:00.000Z"
+        field = 'SystemModstamp'
         result = @api.query(
           "Account",
           "SELECT Id, Name, Industry, Type FROM Account",
           :check_interval => 30,
           :directory_path => tmp,
-          :created_from => from,
-          :created_to => to,
+          :date_from => from,
+          :date_to => to,
           :single_batch => true,
-          :count_lines => true
+          :count_lines => true,
+          :date_field => field
         )
 
         result[:filenames].should have(1).items
@@ -132,7 +134,25 @@ describe SalesforceBulkQuery do
         filename.should match(tmp)
         filename.should match(frm)
         filename.should match(t)
+        filename.should match(field)
+      end
+    end
+    context "when you give it a bad date_field" do
+      it "fails with argument error with no from date" do
+        expect{@api.query(@entity, "SELECT Id, CreatedDate FROM #{@entity}", :date_field => 'SomethingInvalid')}.to raise_error(ArgumentError)
       end
+      it "fails with argument error with given from date" do
+        from = "2000-01-01T00:00:00.000Z"
+        expect{
+          @api.query(
+            @entity,
+            "SELECT Id, CreatedDate FROM #{@entity}",
+            :date_field => 'SomethingInvalid',
+            :date_from => from
+          )
+        }.to raise_error(ArgumentError)
+      end
+
     end
     context "when you give it a short time limit" do
       it "downloads some stuff is unfinished" do

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: salesforce_bulk_query
 version: !ruby/object:Gem::Version
-  version: 0.1.6
+  version: 0.2.0
 platform: ruby
 authors:
 - Petr Cvengros
@@ -174,6 +174,7 @@ files:
 - lib/salesforce_bulk_query/query.rb
 - lib/salesforce_bulk_query/utils.rb
 - lib/salesforce_bulk_query/version.rb
+- new-version.sh
 - salesforce_bulk_query.gemspec
 - spec/salesforce_bulk_query_spec.rb
 - spec/spec_helper.rb