rhoconnect-adapters 1.0.5 → 1.0.6

data/.gitignore

@@ -2,3 +2,5 @@
 .bundle
 Gemfile.lock
 pkg/*
+TODO
+*~

data/CHANGELOG

@@ -1,3 +1,8 @@
+## 1.0.6 (11-28-2012)
+* Rework adapter method 'query' be able to paginate into (large) result sets for oracle, sugar, salesforce vendors
+* Updated salesforce SOAP service version to v24.0 (it provides API paginate into backend data)
+* Fixed not working specs under Linux  
+
 ## 1.0.5 
 * support for rhoconnect >= 3.2
 
@@ -22,4 +27,4 @@
 ## 1.0.0
 * Integration of Salesforce, MsDynamics, Sugar backends
 * #15178091 - Integration of OracleCRM backend, specs for generator
-* #14613211 - Initial creation of the rhoconnect-adapters
+* #14613211 - Initial creation of the rhoconnect-adapters

data/README.md

@@ -123,7 +123,7 @@ These are:
 - **:msdynamics_ticket_url:** <msdynamics_web_services_integration_url> - substitute the default URL with your MsDynamics account URL.
 
 For every source adapter based on CRM object there is a corresponding *'vendor/msdynamics/settings/\<CRM\-object\-name\>.yml'*
-file containing the descriptions for the Sugar CRM object.
+file containing the descriptions for the MsDynamics object.
 Every CRM object file has the following entries:
 
 	Query_Fields: hash of the objects's fields 
@@ -199,7 +199,7 @@ In the file `settings.yml` you'll find the entries that you must customize befor
 These are:
 
 - **:salesforce_login_url:** <salesforce_soap_login_url> - Currently, this parameter is pre-defined to 
-`https://login.salesforce.com/services/Soap/c/22.0` for simple SOAP web service authentication. 
+`https://login.salesforce.com/services/Soap/c/24.0` for simple SOAP web service authentication.
 Rhoconnect-adapters is not using `OAuth2` scheme at this point.
 
 For every source adapter based on CRM object there is a corresponding *'vendor/salesforce/settings/\<CRM\-object\-name\>.yml'*
@@ -229,6 +229,73 @@ different fields. For custom adapters, you need to fill this file with relevant
 from the Salesforce documentation and then later used to fill the Query_Fields setting. Alternatively, user can customize the adapter and obtain 
 the list of fields using the `/sobjects/<CRM-object-name>/describe/` API.
 
+## If you have large dataset on backend
+If your application's model has a lot of data, when you might wanted to paginate through backend service data instead of fetching everything in one call. RhoConnect API provides 
+[ `stash_result`](http://docs.rhomobile.com/rhoconnect/source-adapters#source-adapter-api) method to do it.
+
+For example, straightforward way to query data in Salesforce CRM adapter is  
+
+    :::ruby
+	def query(params=nil)
+  	  @result = {}
+  	  fieldquery = ""
+  	  @fields.each do |element_name, element_def|
+        fieldquery << ",#{element_name}"
+  	  end
+      fieldquery[0] = " "
+  
+      querystr = "SELECT #{fieldquery} FROM #{crm_object}"
+      requesturl = @resturl + "/query/?q=" + CGI::escape(querystr)
+      raw_data = RestClient.get(requesturl, @restheaders)
+      parsed_data = JSON.parse raw_data
+  
+      if parsed_data['done']
+        parsed_data["records"].each do |record|
+          record_hash = {}
+          @fields.each do |element_name, element_def|
+            record_hash[element_name] = record[element_name]
+          end
+          @result[record['Id']] = record_hash
+        end
+      else
+        # TODO: queryMore
+      end
+      @result
+    end 
+
+In the case, if you wanted paginate through backend data the same `query` method should be rewritten as 
+
+    :::ruby
+    def query(params=nil)
+      fieldquery = ""
+      @fields.each do |element_name, element_def|
+        fieldquery << ",#{element_name}"
+      end
+      fieldquery[0] = " "
+
+      # Paginate into (large) result sets staring with offset = 0 and page_sz = 100
+      offset, page_sz = 0, 100
+      loop do
+        querystr = "SELECT #{fieldquery} from #{crm_object} limit #{page_sz} offset #{offset}"
+        requesturl = @resturl + "/query/?q=" + CGI::escape(querystr)
+        raw_data = RestClient.get(requesturl, @restheaders)
+        parsed_data = JSON.parse raw_data
+
+        @result ||= {}
+        parsed_data["records"].each do |record|
+          record_hash = {}
+          @fields.each do |element_name, element_def|
+            record_hash[element_name] = record[element_name]
+          end
+          @result[record['Id']] = record_hash
+        end
+        stash_result # => @result is nil now
+        break if parsed_data['done']
+        offset += page_sz
+      end
+    end
+
+`rhoconnect-adapters` gem provides reference implementation of paginating in `query` method for Sugar, Salesforce, and OracleOnDemand adapters.
 
 ## Running the CRM Application
 Once your Rhoconnect application is customized and ready to run, you can start it like any other Rhoconnect app.
@@ -240,4 +307,4 @@ Type the following command in the CRM application's root directory:
 ## Meta
 Created and maintained by Rhomobile Inc.
 
-Released under the [MIT License](http://www.opensource.org/licenses/mit-license.php).
+Released under the [MIT License](http://www.opensource.org/licenses/mit-license.php).

data/generators/crm/vendor/ms_dynamics/adapter.rb

@@ -43,7 +43,7 @@ module RhoconnectAdapters
         def get_object_settings
           return @object_settings if @object_settings
           begin
-            @object_settings = RhoconnectAdapters::CRM::Field.load_file(File.join(ROOT_PATH,'vendor','ms_dynamics','settings',"#{crm_object.downcase}.yml"))
+            @object_settings = RhoconnectAdapters::CRM::Field.load_file(File.join(ROOT_PATH,'vendor','ms_dynamics','settings',"#{crm_object}.yml"))
           rescue Exception => e
             puts "Error opening CRMObjects settings file: #{e}"
             puts e.backtrace.join("\n")
@@ -89,14 +89,7 @@ module RhoconnectAdapters
         end
       
         def query(params=nil)
-          # TODO: Query your backend data source and assign the records 
-          # to a nested hash structure called @result. For example:
-          # @result = { 
-          #   "1"=>{"name"=>"Acme", "industry"=>"Electronics"},
-          #   "2"=>{"name"=>"Best", "industry"=>"Software"}
-          # }
-          @result = {}
-        
+          @result = {}        
           attributes = []
           # strip out artificial 'attrtype' fields
           fields.each do |key, val|
@@ -298,4 +291,4 @@ module RhoconnectAdapters
       end
     end
   end
-end
+end

data/generators/crm/vendor/ms_dynamics/spec/application_spec.rb

@@ -10,13 +10,14 @@ describe "Application" do
     
     it "should authenticate" do 
       auth_info = RhoconnectAdapters::CRM::MsDynamics.load_auth_info(@test_user)
+      auth_info.should_not be_nil
       now = DateTime.now
       should_be_between(auth_info['wlid_expires'],now,(now+(60 * 60 * 24)))
       should_be_between(auth_info['crm_ticket_expires'],now,(now+(60 * 60 * 24)))
     end
     
     it "should not authenticate with wrong credentials" do
-      Application.should_receive(:warn).once.with('Can\'t authenticate user wrong_user: #<RuntimeError: RhoconnectAdapters::CRM::MsDynamics::WlidService error w/ IssueTicket: Authentication Failure (0x80041034): The specified member name is either invalid or empty.&#13;>')
+      Application.should_receive(:warn).once.with('Can\'t authenticate user wrong_user: #<RuntimeError: RhoconnectAdapters::CRM::MsDynamics::WlidService error w/ IssueTicket: Authentication Failure (0x80041034): The specified member name is either invalid or empty.&#xD;>')    
       Application.authenticate('wrong_user','wrong_password',nil).should be_false
     end
   end

data/generators/crm/vendor/oracle_on_demand/adapter.rb

@@ -180,41 +180,65 @@ module RhoconnectAdapters
         end  
 
         def query(params=nil)
-          # TODO: Query your backend data source and assign the records 
-          # to a nested hash structure called @result. For example:
-          # @result = { 
-          #   "1"=>{"name"=>"Acme", "industry"=>"Electronics"},
-          #   "2"=>{"name"=>"Best", "industry"=>"Software"}
-          # }
-          @result = {}
+          #
+          # Straightforward way to query data. Dot not fit for large result sets.
+          #
+          # @result = {}
+          # fetch_more = 'true'
+          # start_row = 0
+          # begin 
+          #   soap_body = "<wsdl:ListOf#{crm_object} recordcountneeded=\"true\" pagesize=\"100\" startrownum=\"#{start_row.to_s}\">
+          #     <wsdl:#{crm_object} searchspec=\"\">
+          #       #{Adapter.get_columns(fields)}
+          #     </wsdl:#{crm_object}>
+          #   </wsdl:ListOf#{crm_object}>"
+          # 
+          #   query_results = execute_soap_action('QueryPage', soap_body)
+          #   fetch_more = query_results['lastpage'] == 'true' ? false : true;
+          # 
+          #   query_results.children.each do |record|
+          #     if record.name == "#{crm_object}"
+          #        id_field = RhoconnectAdapters::SoapService.select_node_text(record, "#{crm_object}doc:Id")
+          #        converted_record = {}
+          #        # grab only the allowed fields 
+          #        fields.each do |element_name,element_def|
+          #          converted_record[element_name] = RhoconnectAdapters::SoapService.select_node_text(record, "#{crm_object}doc:#{element_name}")
+          #        end
+          #        @result[id_field] = converted_record
+          #     end
+          #   end
+          #   start_row = @result.size
+          # end while fetch_more
+          # @result
+                              
+          # Use stash_result instead of accumulating output in @result in every loop
           fetch_more = 'true'
           start_row = 0
           begin 
-          
             soap_body = "<wsdl:ListOf#{crm_object} recordcountneeded=\"true\" pagesize=\"100\" startrownum=\"#{start_row.to_s}\">
-              <wsdl:#{crm_object} searchspec=\"\">
-                #{Adapter.get_columns(fields)}
-              </wsdl:#{crm_object}>
+               <wsdl:#{crm_object} searchspec=\"\">
+                 #{Adapter.get_columns(fields)}
+               </wsdl:#{crm_object}>
             </wsdl:ListOf#{crm_object}>"
-
             query_results = execute_soap_action('QueryPage', soap_body)
             fetch_more = query_results['lastpage'] == 'true' ? false : true;
-          
+
+            @result ||= {}
             query_results.children.each do |record|
               if record.name == "#{crm_object}"
-                 id_field = RhoconnectAdapters::SoapService.select_node_text(record, "#{crm_object}doc:Id")
-                 converted_record = {}
-                 # grab only the allowed fields 
-                 fields.each do |element_name,element_def|
-                   converted_record[element_name] = RhoconnectAdapters::SoapService.select_node_text(record, "#{crm_object}doc:#{element_name}")
-                 end
-                 @result[id_field] = converted_record
-               end
-             end
-             start_row = @result.size
-           end while fetch_more
-           @result
-         end
+                id_field = RhoconnectAdapters::SoapService.select_node_text(record, "#{crm_object}doc:Id")
+                converted_record = {}
+                # grab only the allowed fields
+                fields.each do |element_name,element_def|
+                  converted_record[element_name] = RhoconnectAdapters::SoapService.select_node_text(record, "#{crm_object}doc:#{element_name}")
+                end
+                @result[id_field] = converted_record
+              end
+            end
+            stash_result
+            start_row = @result.size
+          end while fetch_more
+        end
           
         def sync
           # Manipulate @result before it is saved, or save it 
@@ -418,4 +442,4 @@ module RhoconnectAdapters
       end
     end
   end
-end
+end

data/generators/crm/vendor/salesforce/adapter.rb

@@ -33,7 +33,7 @@ module RhoconnectAdapters
         def get_object_settings
           return @object_settings if @object_settings
           begin
-            @object_settings = RhoconnectAdapters::CRM::Field.load_file(File.join(ROOT_PATH,'vendor','salesforce','settings',"#{crm_object.downcase}.yml"))
+            @object_settings = RhoconnectAdapters::CRM::Field.load_file(File.join(ROOT_PATH,'vendor','salesforce','settings',"#{crm_object}.yml"))
           rescue Exception => e
             puts "Error opening CRMObjects settings file: #{e}"
             puts e.backtrace.join("\n")
@@ -105,13 +105,33 @@ module RhoconnectAdapters
         end
       
         def query(params=nil)
-          # TODO: Query your backend data source and assign the records 
-          # to a nested hash structure called @result. For example:
-          # @result = { 
-          #   "1"=>{"name"=>"Acme", "industry"=>"Electronics"},
-          #   "2"=>{"name"=>"Best", "industry"=>"Software"}
-          # }
-          @result = {}
+          #
+          # Straightforward way to query data. Dot not fit for large result sets.
+          #
+          # @result = {}
+          # fieldquery = ""
+          # @fields.each do |element_name, element_def|
+          #   fieldquery << ",#{element_name}"
+          # end
+          # fieldquery[0] = " "
+          #
+          # querystr = "SELECT #{fieldquery} FROM #{crm_object}"
+          # requesturl = @resturl + "/query/?q=" + CGI::escape(querystr)
+          # raw_data = RestClient.get(requesturl, @restheaders)
+          # parsed_data = JSON.parse raw_data
+          #
+          # if parsed_data['done']
+          #   parsed_data["records"].each do |record|
+          #     record_hash = {}
+          #     @fields.each do |element_name, element_def|
+          #       record_hash[element_name] = record[element_name]
+          #     end
+          #     @result[record['Id']] = record_hash
+          #   end
+          # else
+          #   # TODO: queryMore
+          # end
+          # @result
 
           fieldquery = ""
           @fields.each do |element_name, element_def|
@@ -119,23 +139,28 @@ module RhoconnectAdapters
           end
           fieldquery[0] = " "
 
-          querystr = "SELECT #{fieldquery} from #{crm_object}"
+          # Paginate into (large) result sets staring with offset = 0 and page_sz = 100
+          offset, page_sz = 0, 100
+          loop do
+            querystr = "SELECT #{fieldquery} from #{crm_object} limit #{page_sz} offset #{offset}"
+            requesturl = @resturl + "/query/?q=" + CGI::escape(querystr)
+            raw_data = RestClient.get(requesturl, @restheaders)
+            parsed_data = JSON.parse raw_data
 
-          requesturl = @resturl + "/query/?q=" + CGI::escape(querystr)
-          raw_data = RestClient.get(requesturl, @restheaders)
-          parsed_data = JSON.parse raw_data
-
-          parsed_data["records"].each do |record|
-            record_hash = {}
-            @fields.each do |element_name, element_def|
-              record_hash[element_name] = record[element_name]
+            @result ||= {}
+            parsed_data["records"].each do |record|
+              record_hash = {}
+              @fields.each do |element_name, element_def|
+                record_hash[element_name] = record[element_name]
+              end
+              @result[record['Id']] = record_hash
             end
-            @result[record['Id']] = record_hash
+            stash_result # => @result is nil now
+            break if parsed_data['done']
+            offset += page_sz
           end
-          
-          @result
         end
-      
+
         def metadata
           # define the metadata
           show_fields = []
@@ -316,4 +341,4 @@ module RhoconnectAdapters
       end
     end
   end
-end
+end

data/generators/crm/vendor/salesforce/application.rb

@@ -29,7 +29,7 @@ module RhoconnectAdapters
               # here, requestUrl is formatted for SOAP requests
               # and we removing Soap part for it (which is suffix after services)
               # since we will user REST after that
-              endpoint_url = res_hash['serverUrl'].split('services')[0] + 'services/data/v22.0'
+              endpoint_url = res_hash['serverUrl'].split('services')[0] + 'services/data/v24.0'
               
               # store password to be used by SourceAdaptors
               Store.put_value("#{username}:session_id", session_id)

data/generators/crm/vendor/sugar/adapter.rb

@@ -104,12 +104,31 @@ module RhoconnectAdapters
         end
 
         def query(params=nil)
-          @result = {}
+          #
+          # Straightforward way to query data. Dot not fit for large result sets.
+          #
+          # @result = {}
+          # conditions = {:conditions=>{}}
+          # conditions[:conditions][:assigned_user_id] = @namespace.current_user.id
+          # results = get_results(conditions)
+          # @result = create_result_hash(results)
+
+          # Use stash_result to paginate into result sets
           conditions = {:conditions=>{}}
           conditions[:conditions][:assigned_user_id] = @namespace.current_user.id
-          results = get_results(conditions)
-        
-          @result = create_result_hash(results)
+          offset, page_sz = 0, 100
+          conditions[:limit] = page_sz
+
+          loop do
+            conditions[:offset] = offset
+            results = get_results(conditions)
+            @result = create_result_hash(results)
+            rec_num = @result.size
+            stash_result
+
+            break if rec_num < page_sz
+            offset += page_sz
+          end
         end
       
         def metadata

data/lib/rhoconnect-adapters/version.rb

@@ -1,3 +1,3 @@
 module RhoconnectAdapters
-  VERSION = "1.0.5"
+  VERSION = "1.0.6"
 end

data/rhoconnect-adapters.gemspec

@@ -27,10 +27,10 @@ Gem::Specification.new do |s|
   ]
 
   s.add_dependency('bundler', '~> 1.0')  
-  s.add_dependency('rhoconnect', '>= 3.2')
+  s.add_dependency('rhoconnect', '>= 3.4')
   s.add_dependency('activesupport', '>= 3.0.9')
   s.add_dependency('i18n', '>= 0.6.0')
-  s.add_dependency('rake', '~> 0.9.2.2')
-  s.add_dependency('nokogiri', '>= 1.4.6')
+  s.add_dependency('rake', '~> 10.0')
+  s.add_dependency('nokogiri', '~> 1.5')
   s.add_dependency('templater', '~> 1.0.0')
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: rhoconnect-adapters
 version: !ruby/object:Gem::Version
-  version: 1.0.5
+  version: 1.0.6
   prerelease: 
 platform: ruby
 authors:
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2012-07-19 00:00:00.000000000 Z
+date: 2012-12-14 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: bundler
@@ -34,7 +34,7 @@ dependencies:
     requirements:
     - - ! '>='
       - !ruby/object:Gem::Version
-        version: '3.2'
+        version: '3.4'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
@@ -42,7 +42,7 @@ dependencies:
     requirements:
     - - ! '>='
       - !ruby/object:Gem::Version
-        version: '3.2'
+        version: '3.4'
 - !ruby/object:Gem::Dependency
   name: activesupport
   requirement: !ruby/object:Gem::Requirement
@@ -82,7 +82,7 @@ dependencies:
     requirements:
     - - ~>
       - !ruby/object:Gem::Version
-        version: 0.9.2.2
+        version: '10.0'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
@@ -90,23 +90,23 @@ dependencies:
     requirements:
     - - ~>
       - !ruby/object:Gem::Version
-        version: 0.9.2.2
+        version: '10.0'
 - !ruby/object:Gem::Dependency
   name: nokogiri
   requirement: !ruby/object:Gem::Requirement
     none: false
     requirements:
-    - - ! '>='
+    - - ~>
       - !ruby/object:Gem::Version
-        version: 1.4.6
+        version: '1.5'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     none: false
     requirements:
-    - - ! '>='
+    - - ~>
       - !ruby/object:Gem::Version
-        version: 1.4.6
+        version: '1.5'
 - !ruby/object:Gem::Dependency
   name: templater
   requirement: !ruby/object:Gem::Requirement
@@ -250,7 +250,7 @@ required_ruby_version: !ruby/object:Gem::Requirement
       version: '0'
       segments:
       - 0
-      hash: 2546656225608389328
+      hash: 1998545148818093268
 required_rubygems_version: !ruby/object:Gem::Requirement
   none: false
   requirements:
@@ -259,7 +259,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
       version: '0'
 requirements: []
 rubyforge_project: 
-rubygems_version: 1.8.24
+rubygems_version: 1.8.23
 signing_key: 
 specification_version: 3
 summary: Rhoconnect adapters