rhoconnect 3.0.0.beta1

data/doc/async-jobs.txt

@@ -0,0 +1,69 @@
+Adapter Jobs with Resque
+===
+
+RhoConnect uses the [resque](http://github.com/defunkt/resque) library for managing and executing asynchronous jobs.  Using async jobs are especially useful when your application requires processing large amounts of data, pinging devices using the built-in push support, or you just want to perform some task that will take a long period of time.
+
+## Async Job Types
+RhoConnect supports three job types: BulkDataJob, PingJob, SourceJob
+
+### BulkDataJob
+[Bulk Data](/rhoconnect/bulk-sync) jobs are used by RhoConnect to prepare the [bulk sync data files](/rhoconnect/bulk-sync#data-files).  These jobs are automatically queued by the RhoConnect application, so it is not necessary to queue them manually.
+
+### PingJob
+Ping jobs are used to execute a [device ping](/rhoconnect/push).  A PingJob can be queued from anywhere in your RhoConnect application (i.e. at the end of another job or adapter query method).
+
+Example: 
+	
+	:::ruby
+	Resque.enqueue(
+	  PingJob,
+	  {
+	    "user_id" => current_user.login,
+	    "message" => "New products available!", 
+	    "badge" => 1, 
+	    "sources" => ['Product']
+	  }
+	)
+
+### SourceJob
+Source jobs are used to execute source adapter queues asynchronously. 
+
+To make your source adapters always run asynchronously, define a queue in `settings/settings.yml`:
+
+	:::yaml
+	Product:
+	  :queue: product
+
+Then run a resque worker so the job will execute.
+
+Example that queues a new source adapter job manually:
+	
+	:::ruby
+	Resque.enqueue(
+	 SourceJob,
+	 :query,
+	 APP_NAME,
+	 'Joe',
+	 'some-client-id
+	)
+
+## Running Async Jobs
+Running RhoConnect jobs requires one or more [resque](http://github.com/defunkt/resque) workers running.  For example, to start one resque worker listening for all job queues:
+
+	:::term
+	$ cd <your rhoconnect app>
+	$ QUEUE=* rake resque:work
+
+To start multiple workers(in this case 5):
+	
+	:::term
+	$ cd <your rhoconnect app>
+	$ QUEUE=* COUNT=5 rake resque:workers
+	
+For more information on the options available when running resque workers, please refer to the [resque readme](https://github.com/defunkt/resque#readme).
+
+If you want to run resque workers in daemon mode, we recommend using [nohup](http://en.wikipedia.org/wiki/Nohup), [god](http://github.com/defunkt/resque/tree/master/examples/god/), or [monit](http://github.com/defunkt/resque/tree/master/examples/monit):
+
+	:::term
+	$ mkdir log/; touch log/resque_worker_QUEUE.log # if it doesn't exist
+  	$ nohup QUEUE=* rake resque:work & &> log/resque_worker_QUEUE.log

data/doc/authentication.txt

@@ -0,0 +1,76 @@
+Device Authentication in RhoConnect
+===
+
+The generated RhoConnect application code includes a file at the root called "application.rb" which contains an "authenticate" hook.  This method is called when a device first connects to the RhoConnect application and provides username/password credentials.
+
+## Authentication Workflow
+	
+The following diagram shows how this authenticate method is called:
+
+<img src="https://img.skitch.com/20110111-g2ujk785gak2kfgfrspq1w8u8j.png"/>
+
+## Example
+If your backend service requires authentication, simply add code to the authenticate method and return true if authentication was successful or false to deny access to the application from this device.  For example:
+
+	:::ruby
+	def authenticate(username, password, session)
+	  success = false 
+	  # ... connect to backend using API and authenticate ... 
+	  result = do_custom_authentication(username,password) 
+	  if result.code == 200
+	    # save the data for later use in the source adapter
+	    Store.put_value("username:#{username}:token",result.body)
+		success = true
+	  end
+	  return success
+	end
+
+## Changing Login
+If your actual user name is different than the username that is input into the client, you can change the username by returning a string instead of true if authentication was successful.
+
+  :::ruby
+	def authenticate(username, password, session)
+	  success = false 
+	  # ... connect to backend using API and authenticate ... 
+	  result = do_custom_authentication(username,password) 
+	  if result.code == 200
+	    new_user = custom_user_mapping(username)
+	    # save the data for later use in the source adapter
+	    Store.put_value("username:#{new_user}:token",result.body)
+		  success = new_user
+	  end
+	  return success
+	end
+  
+
+## LDAP Example
+The code sample above demonstrates authentication by calling a method you would implement called 'do_custom_authentication' that returns an HTTP response object, however the actual implementation is left as an exercise.  If your application needs to connect to LDAP, you can use the following sample code to perform simple authentication:
+
+	:::ruby
+	require 'net/ldap'
+
+    def authenticate(username, password, session)
+      success = false 
+      ldap = Net::LDAP.new
+      ldap.host = "localhost"
+      ldap.port = 389
+      ldap.auth "cn=#{username},dc=example,dc=com", password
+
+      # we only need to bind to verify successful login
+      success = true if ldap.bind
+
+      return success
+    end
+	
+Where the domain component "dc=example,dc=com" is replaced by your domain.  For example, the `ldap.auth` code above using "rhomobile.com" would be:
+
+ 	:::ruby
+	ldap.auth "cn=#{username},dc=rhomobile,dc=com", password
+
+This example uses the [net-ldap](https://github.com/RoryO/ruby-net-ldap) gem.  To install this gem, run the following:
+
+	:::term
+	$ [sudo] gem install net-ldap
+	
+For more details about LDAP, please visit [http://www.openldap.org/](http://www.openldap.org/).
+

data/doc/benchmarks.txt

@@ -0,0 +1,168 @@
+RhoConnect Benchmarks
+===
+
+Note, these numbers reflect synchronization performance using the rhoconnect bench utility (see the rhoconnect/bench folder for more information).  
+We will be adding more results as we test across different server environments.
+
+
+Servers configuration (Amazon EC2 'm1.small' instances) :
+
+* CPU Units - 1 ECU
+
+* CPU Cores - 1 Core
+
+* Memory - 1.7GB
+
+* Platform - Ubuntu (2.6.38-8-virtual)
+
+
+Environment (instance #1):
+
+* Nginx 1.0.0
+
+* Ruby 1.8.7 (2011-02-18 patchlevel 334) [i686-linux], Ruby Enterprise Edition 2011.03
+ 
+* Passenger 3.0.7 (mod_rails)
+
+* Redis 2.2.11
+
+* Rhoconnect 3.0.0 
+
+Nginx configuration file:
+
+<pre>
+user  www-data;
+worker_processes  4;
+
+pid /var/run/nginx.pid;
+
+events {
+    worker_connections  1024;
+}
+
+http {
+    passenger_root /usr/local/lib/ruby/gems/1.8/gems/passenger-3.0.7;
+    passenger_ruby /usr/local/bin/ruby;
+    passenger_max_pool_size 20;
+    include       mime.types;
+    default_type  application/octet-stream;
+
+    sendfile    on;
+    tcp_nopush  on;
+    tcp_nodelay on;
+
+    keepalive_timeout  65;
+
+    gzip  on;
+
+    server {
+        listen 80;
+        server_name  ec2-50-18-72-186.us-west-1.compute.amazonaws.com;
+        root  /var/www/benchapp/public;   # Bench application #1
+        # root  /var/www/blobapp/public;   # Bench application #2
+        passenger_enabled on;
+        passenger_min_instances 6;
+    }
+}
+</pre>
+
+
+Environment (instance #2):
+
+* Ruby 1.8.7 (2011-02-18 patchlevel 334) [i686-linux], Ruby Enterprise Edition 2011.03
+
+* Rhoconnect 3.0.0 code with bench scripts
+ 
+
+Server instance #1 is running rhoconnect bench applications, 
+the 2nd one executes rhoconnect bench scripts. 
+
+## Bench application #1
+* Simulates creating multiple objects
+
+* 100 concurrent users
+
+* 1 device per user
+
+* 20 object dataset
+
+* each object has 7 attributes 
+
+* 904 total HTTP requests
+
+<pre>
+Request clientcreate   : min: 0.0069, max: 1.6010, avg: 0.3404
+Request ack-cud        : min: 0.0339, max: 3.2250, avg: 0.5910
+Request clientlogin    : min: 0.0063, max: 0.8138, avg: 0.1216
+Request get-cud        : min: 0.0315, max: 3.0044, avg: 0.3988
+Request create-object  : min: 0.0791, max: 4.4959, avg: 1.3521
+Throughput(req/s)      : 7.13629650622702
+Throughput(req/min)    : 428.177790373621
+</pre>
+
+## Bench application #2
+* Simulates creating multiple objects
+
+* 100 concurrent users
+
+* 1 device per user
+
+* 10 object dataset
+
+* each object has variable number of attributes (from 10 to 60) + image
+
+* 904 total HTTP requests
+
+<pre>
+Request clientcreate   : min: 0.0067, max: 2.9286, avg: 0.6870
+Request ack-cud        : min: 0.0473, max: 5.5015, avg: 1.6219
+Request clientlogin    : min: 0.0061, max: 3.2326, avg: 0.4687
+Request get-cud        : min: 0.0382, max: 7.2202, avg: 1.0975
+Request create-object  : min: 0.3765, max: 11.1387, avg: 5.1424
+Throughput(req/s)  : 7.22833609137099
+Throughput(req/min): 433.700165482259
+</pre>
+
+## Prepare bench applications for running on server #1
+
+Go to rhoconnect/bench directory and configure benchapp application: 
+
+<pre>
+sudo cp -R benchapp /var/www/
+cd /var/www/
+sudo mkdir benchapp/public
+sudo chown -R www-data:www-data benchapp/
+</pre>
+
+Do the same for blobapp bench: 
+<pre>
+sudo cp -R blobapp /var/www/
+cd /var/www/
+sudo mkdir blobapp/public
+sudo chown -R www-data:www-data blobapp/
+</pre>
+
+Edit <b>Nginx</b> configuration file, set server_name and root properties to public DNS server name 
+and bench application public folder respectively. After restarting server make sure that bench application is
+running by visiting its web concole, i.e. : 
+<pre>
+http://server_dns_public_name/console/ 
+</pre>
+
+## How to execute bench scripts on server #2
+Before running bench scripts you need configure bench application settings properly. 
+
+Edit rhoconnect/bench/benchapp/settings/settings.yml and set redis and syncserver properties:
+<pre>
+:development:
+  :licensefile: settings/license.key
+  :redis: server_ip_address:6379
+  :syncserver: http://server_dns_public_name:80/api/application/
+</pre>
+
+Do the same for blobapp settings.yml file.
+
+Then from rhoconnect/becnh directory execute run_cud_script.sh and run_blob_script.sh scripts respectively.
+
+ 
+

data/doc/blob-sync.txt

@@ -0,0 +1,130 @@
+Binary Objects and Blob Sync
+===
+Synchronizing images or binary objects between RhoConnect and the client is declared by having a 'blob attribute' on the [Rhom model](/rhodes/rhom).  This instructs the client to download the blob attribute's value (in this case a url) separately from the rest of the data and save it to the device file system.
+
+**NOTE: The example below shows how to synchronize an image created by the device's camera, but could be applied to any arbitrary files on the device.**
+
+## Synchronizing from Device to RhoConnect
+Suppose you have a [Rhom model](/rhodes/rhom) called "Image" and you have a [camera callback function](/rhodes/device-caps#camera) defined in the Image model's controller:
+
+	:::ruby
+	def camera_callback
+	  if @params['status'] == 'ok'
+	    #create image record in the DB
+	    image = Image.new({'image_uri'=>@params['image_uri']})
+	    image.save
+	    SyncEngine.dosync
+	  end  
+	  #reply on the callback
+	  render :action => :ok, :layout => false
+	end
+	
+When this is executed in a Rhodes application:
+
+* This callback triggers a sync after saving saving an image.  
+
+* Then Rhodes will push the image file to the RhoConnect application by HTTP POST.
+
+* `application.rb#store_blob` is called for optional pre-processing of the file.  This is useful in cases where you want to preserve the filename that Rhodes sent.  
+
+The following example stores the `:filename` argument as another object attribute.
+
+	:::ruby
+	def store_blob(obj,field_name,blob)
+      obj['filename'] = blob[:filename]
+      super
+    end
+
+* The RhoConnect source adapter tied to this Rhom model processes the blob in the `create` method.
+
+The following example takes the blob and stores it in S3:
+
+	:::ruby
+	def create(create_hash,blob)
+	  # this would be a blob attribute, points to a tempfile 
+	  # containing the raw post data
+	  name = create_hash["image_uri"] 
+	  
+	  # filename we saved in application.rb#store_blob method
+	  basename = create_hash["filename"]
+	  S3Object.store(
+	    basename,
+	    open(name),
+	    @bucket,
+	    :access => :public_read
+	  )
+	  basename #=> object id here is the filename
+	end
+
+In this example, we store the image blob to s3 so we can query it later.  See the "image" source adapter that comes with [the rhodes-samples-server](https://github.com/rhomobile/rhodes-samples-server/blob/master/sources/image.rb).
+
+## Query for Blobs in the Source Adapter
+
+In your query call, you should store the URL to images in the blob attribute so that the device can fetch the image when it synchronizes.
+
+	:::ruby
+	def query(params=nil)
+	  @result={}
+	  Bucket.find(@bucket).objects.each do |entry|
+	    new_item = {
+		  'image_uri-rhoblob' => @baseurl+@bucket+'/'+File.basename(entry.key)
+	    }
+	    @result[entry.key] = new_item
+	  end
+	  @result
+	end
+
+Here we save the image URL to an attribute `image_uri-rhoblob`.  This will correspond to the `image_uri` attribute in our Rhom model above.
+
+**NOTE: In order for Rhodes to automatically recognize a blob attribute, the attribute name must end with "-rhoblob".  In the above example, `image_uri` was our Rhom attribute, so our source adapter assigned `image_uri-rhoblob` with the blob URL.**
+
+A complete image source adapter that stores files in S3 might look like:
+
+	:::ruby
+	require 'aws/s3'
+	class Image < SourceAdapter
+	  include AWS::S3
+
+
+	  def initialize(source,credential)
+	    AWS::S3::Base.establish_connection!(
+	      :access_key_id      => ENV['AMAZON_ACCESS_KEY_ID'],
+	      :secret_access_key  => ENV['AMAZON_SECRET_ACCESS_KEY']
+	    )
+	    @baseurl = 'http://s3.amazonaws.com/'
+	    @bucket = 'rhodes-samples-images'
+	    super(source,credential)
+	  end
+
+	  def query(params=nil)
+	    @result={}
+	    Bucket.find(@bucket).objects.each do |entry|
+	      new_item = {
+		    'image_uri-rhoblob' => @baseurl+@bucket+'/'+File.basename(entry.key)
+	      }
+	      @result[entry.key] = new_item
+	    end
+	    @result
+	  end
+
+	  def create(create_hash)
+	    puts "Creating: #{create_hash.inspect}"
+	    name = create_hash["image_uri"]
+	    basename = create_hash["filename"]
+	    S3Object.store(
+	      basename,
+	      open(name),
+	      @bucket,
+	      :access => :public_read
+	    )
+	    basename
+	  end
+
+	  def delete(delete_hash)
+	    puts "Removing: #{delete_hash.inspect}"
+	    S3Object.delete delete_hash['id'], @bucket
+	  end
+	end
+
+## Rhodes Sample Code
+See controller and view in the `/app/Blob` directory of the [System API Samples application](http://github.com/rhomobile/rhodes-system-api-samples/tree/master) for a complete Rhodes code sample.

data/doc/bulk-sync.txt

@@ -0,0 +1,102 @@
+Bulk Sync
+===
+During the initial sync process in your applcation, you may need to deliver a large number of objects. This may result in many megabytes of data being sent to the device. In this case, incremental synchronization may take a while to complete since it uses JSON as the wire format. To significantly speed up this process you may use RhoConnect's "bulk sync" feature. 
+
+## Bulk Sync Process
+
+If and when the Rhodes application issues a request to get bulk data, the RhoConnect application will do following:
+
+* Queue a [resque background job](http://github.com/defunkt/resque) which will run the bulk sync process.
+
+* Execute the ['login,'query','logoff' methods](/rhoconnect/source-adapters#source-adapter-api) of all source adapters matching the [requested data partition](/rhoconnect/source-adapters#data-partitioning) to get latest set of objects from the backend service
+
+* Generate appropriate database files for the device - hsqldb for Blackberry and sqlite for the rest of them. RhoConnect will generate sqlite files for iOS, Android, and Windows Mobile clients and hsqldb files for Blackberry (if enabled).
+ 
+* The Rhodes application will poll RhoConnect for the bulk sync job to finish. Once the job is finished, the files are generated and Rhodes will download and install them to the device. 
+
+**NOTE: Be cautious using this in your application - any previous set of objects on the device will be replaced with the latest set from the backend service.**
+
+Once the bulk data sync process is complete for a source, you may continue to synchronize sources using the default incremental sync process. However, the default sync process may be disabled for a source by setting its `sync_type` to 'bulk_sync_only'.  This will save space in your redis database if you don't plan on using incremental sync for this source.
+
+For example, your settings/settings.yml might have:
+
+	:::yaml
+	:sources:
+	  Product:
+	    poll_interval: 300
+	  Customer:
+		poll_interval: 300
+		sync_type: bulk_sync_only
+
+## Setup
+
+### Enabling Bulk Sync in Rhodes
+In the device rhodes application, setting `bulksync_state` to 0 in rhoconfig.txt will trigger bulk sync next time sync is performed:
+
+	bulksync_state = 0
+
+After application executed bulk sync it will set `bulksync_state` to 1 and next time sync will happen, it will not execute bulk sync anymore. You may force bulk sync on the next sync cycle by setting `bulksync_state` to 0 again using ruby in your application:
+	
+	:::ruby
+	Rho::RhoConfig.bulksync_state = '0'
+
+### Enabling BlackBerry Bulk Sync
+If your application will be performing bulk sync for BlackBerry devices, you will need to install the open source Rhomobile HSQLDB tool [hsqldata.jar](http://github.com/rhomobile/hsqldata) in your RhoConnect application.
+  
+1. Download [hsqldata](http://github.com/rhomobile/hsqldata/zipball/0.0.2).
+
+2. Build and install hsqldata.jar into your RhoConnect application:
+  		
+		:::term
+		$ cd hsqldata
+		$ ant
+	  	$ mkdir <path-to-rhoconnect-app>/vendor
+	  	$ cp bin/hsqldata.jar <path-to-rhoconnect-app>/vendor/
+
+3. Next set the `:blackberry_bulk_sync:` flag in settings/settings.yml:
+
+		:::yaml
+		:development:
+  		  :licensefile: settings/license.key
+  		  :redis: localhost:6379
+  		  :syncserver: http://localhost:9292/application/
+  		  :blackberry_bulk_sync: true
+
+## Data Files
+
+By default, bulk sync data files will be stored in the application's root folder, under a 'data' directory.  When bulk data is requested by the Rhodes application, RhoConnect will return a link to the file if the `bulk_sync_poll_interval` has not expired.  If the file is expired, RhoConnect will return a wait command to Rhodes and enqueue another [resque job](http://github.com/defunkt/resque) to generate new bulk data files.  By default, the `bulk_sync_poll_interval` is set to 1 hour (3600) seconds and is configurable in `settings.yml`:
+
+	:::yaml
+	:development: 
+  	  :redis: localhost:6379
+  	  :licensefile: settings/license.key
+      :syncserver: http://localhost:9292/application/
+      :bulk_sync_poll_interval: 3600
+
+## Using Bulk Sync with Fixed Schema Models
+
+If your Rhom model is a [fixed schema model](/rhodes/rhom#fixed-schema) and you use bulk sync, you can control the bulk sync schema at runtime in your source adapter by implementing the 'schema' method.  
+
+**NOTE: This method must return a JSON string.  This JSON string is sent to the Rhodes application and used to dynamically build the database schema.**
+
+An example schema method is as follows:
+
+	:::ruby
+  	def schema
+      {
+        'version' => '1.0',
+        'property' => {
+          'name' => 'string',
+          'brand' => 'string',
+          'price' => 'string',
+          'image_url_cropped' => 'blob,overwrite',
+          'image_url' => 'blob'
+        },
+        'index' => {
+          'by_name_brand' => 'name,brand'
+        },
+        'unique_index' => {
+          'by_price' => 'price'
+        }
+      }.to_json
+    end