condo 0.0.1 → 1.0.0

data/README.textile

@@ -1,11 +1,12 @@
-h1. Condominios
+h1. Condominios aka Condo
 
-A Rails plugin that makes direct uploads to multiple cloud storage providers easy.
+A "Rails plugin":http://guides.rubyonrails.org/plugins.html and "AngularJS application":http://angularjs.org/ that makes direct uploads to multiple cloud storage providers easy.
 Only supports "XMLHttpRequest Level 2":http://en.wikipedia.org/wiki/XMLHttpRequest capable browsers and cloud providers that have a "RESTful API":http://en.wikipedia.org/wiki/Representational_state_transfer with "CORS":http://en.wikipedia.org/wiki/Cross-origin_resource_sharing support.
 
 Why compromise?
 
-Get started now: @gem install condo@
+Get started now: @gem install condo@ or checkout the "example application":https://github.com/cotag/condo_example
+Also see our "github pages site":http://cotag.github.com/Condominios/
 
 
 h2. License
@@ -13,8 +14,117 @@ h2. License
 GNU Lesser General Public License v3 (LGPL version 3)
 
 
+h2. Concept
+
+Condominios was created to provide direct to cloud uploads using standards based browser technology. However it is not limited to that use case.
+The API is RESTful, providing an abstraction layer and signed URLs that can be utilised in native (mobile) applications.
+
+The main advantages are:
+* Off-loads processing to client machines
+* Better guarantees against upload corruption
+** file hashing on the client side instead of an intermediary where it probably won't be hashed either
+* Upload results are guaranteed if the cloud provider provides atomic operations
+** user is always aware of any failures in the process
+* Detailed progress and control over the upload
+ 
+This has numerous advantages over traditional Form Data style post uploads too.
+* Progress bars
+* Resumability when uploading large files
+
+
+Support for all major browsers and IE10.
+* Tested in Firefox 4, Safari 6 and Chromes latest stable
+
+
 h2. Usage
 
-Coming Soon!
+h3. Terms
+
+* Residence == the current storage provider
+* Resident == the current user
+
+
+h3. Quick Start
+
+See the "example application":https://github.com/cotag/condo_example which implements the steps below on an otherwise blank rails app.
+
+# Add the following to your rails application gemfile:
+#* @gem 'condo'@
+#* @gem 'condo_active_record'@ (more backends coming soon)
+#* @gem 'condo_interface'@ (optional)
+# Run migrations
+#* @rake railties:install:migrations FROM=condo_active_record@
+#* @rake db:migrate@
+# Create an initialiser for any default residencies. (details further down)
+# Create controllers that will be used as Condo endpoints
+#* Typically @rails g controller Uploads@
+#* Add the resource to your routes
+# At the top of the new controller add the following line to the class: @include Condo@
+#* This creates the following public methods at run time: new, create, edit, update, destroy implementing the API
+#* The following protected methods are also generated: set_residence, current_residence, current_resident, current_upload
+# You are encouraged to use standard filters to authenticate users and set the residence (if this is dynamic) + implement index / show if desired
+# You must implement the following call-backs:
+#* resident_id - this should provide a unique identifier for the current user, used for authorisation
+#* upload_complete - provides the upload information for storage in the greater application logic. Return true if successful.
+#* destroy_upload - provides the upload information so that a scheduled task can be created to clean up the upload. Return true if successfully scheduled.
+#** This should be done in the background using something like "Fog":http://fog.io/ Can't trust the client
+
+
+If you are using "Condo Interface":https://github.com/cotag/condo_interface then you may want to do the following:
+# Create an index for your controller @def index; end@
+# Create an index.html.erb in your view with:
+#* @<div class="uploads-container" data-ng-app="CondoUploader"><%= render "condo_interface/upload" %></div>@
+
+Alternative you could load an AngularJS template linking to <%= asset_path('templates/_upload.html') %>
+
+
+h3. Defining Static Residencies
+
+If you are creating an application that only communicates with one or two storage providers or accounts then this is the simplest way to get started.
+In an initialiser (<-- I'm Australian) do the following:
+
+<pre><code class="ruby">
+Condo::Configuration.add_residence(:AmazonS3, {
+	:access_id => ENV['S3_KEY'],
+	:secret_key => ENV['S3_SECRET']
+	# :location => 'us-west-1'	# or 'ap-southeast-1' etc (see http://docs.amazonwebservices.com/general/latest/gr/rande.html#s3_region)
+								# Defaults to 'us-east-1' or US Standard - not required for Google
+	# :namespace => :admin_resident	# Allows you to assign different defaults to different controllers
+									# Controller must have the following line 'set_namespace :admin_resident'
+})
+
+</code></pre>
+
+The first residence to be defined in a namespace will be the default. To change the residence for the current request use @set_residence(:name, :location)@ - location is optional
+Currently available residencies:
+* :AmazonS3
+* :GoogleCloudStorage
+* :RackspaceCloudFiles
+
+
+You can also define a dynamic residence each request (maybe clients provided you with access information for their storage provider)
+
+<pre><code class="ruby">
+set_residence(:AmazonS3, {
+	:access_id => user.s3_key,
+	:secret_key => user.s3_secret,
+	:dynamic => true	# Otherwise the same as add_residence
+});
+	
+
+</code></pre>
+
+
+h3. Callbacks
+
+These are pretty well defined "here":https://github.com/cotag/condo_example/blob/master/app/controllers/uploads_controller.rb
+
+
+h2. TODO::
 
+# Write tests... So many tests
+# Create a wiki describing things in more detail
+# Implement API for more residencies
+# Sign other useful requests (bucket listings with search etc)
+#* For Dropbox or Megaupload style applications
 

data/app/assets/javascripts/condo/amazon.js

@@ -60,7 +60,10 @@
 			var self = this,
 				strategy = null,
 				part_size = 5242880,			// Multi-part uploads should be bigger then this
+				pausing = false,
 				defaultError = function(reason) {
+					self.error = !pausing;
+					pausing = false;
 					self.pause(reason);
 				},
 
@@ -210,9 +213,7 @@
 									set_part(data, result);
 								}, defaultError);
 						
-						}, function(reason){
-							self.pause(reason);
-						});	// END BUILD_REQUEST
+						}, defaultError);	// END BUILD_REQUEST
 						
 					} else {
 						//
@@ -307,6 +308,15 @@
 			this.message = 'pending';
 			this.name = file.name;
 			this.size = file.size;
+			this.error = false;
+			
+			
+			//
+			// File path is optional (amazon supports paths as part of the key name)
+			//	http://docs.amazonwebservices.com/AmazonS3/2006-03-01/dev/ListingKeysHierarchy.html
+			//
+			if(!!file.dir_path)
+				this.path = file.dir_path;
 			
 			
 			//
@@ -318,6 +328,9 @@
 			
 			this.start = function(){
 				if(strategy == null) {	// We need to create the upload
+					self.error = false;
+					pausing = false;
+					
 					//
 					// Update part size if required
 					//
@@ -346,13 +359,13 @@
 								}
 							}, defaultError);
 						
-					}, function(reason){
-						self.pause(reason);
-					});	// END BUILD_REQUEST
+					}, defaultError);	// END BUILD_REQUEST
 					
 					
 				} else if (this.state == PAUSED) {				// We need to resume the upload if it is paused
 					this.message = null;
+					self.error = false;
+					pausing = false;
 					strategy.resume();
 				}
 			};
@@ -360,6 +373,7 @@
 			this.pause = function(reason) {
 				if(strategy != null && this.state == UPLOADING) {	// Check if the upload is uploading
 					this.state = PAUSED;
+					pausing = true;
 					strategy.pause();
 				} else if (this.state <= STARTED) {
 					this.state = PAUSED;

data/app/assets/javascripts/condo/controller.js

@@ -19,10 +19,10 @@
 (function (factory) {
 	if (typeof define === 'function' && define.amd) {
 		// AMD
-		define(['jquery', 'condo_uploader'], factory);
+		define('condo_controller', ['jquery', 'condo_uploader'], factory);
 	} else {
 		// Browser globals
-		factory(jQuery, window.CondoUploader);
+		window.CondoController = factory(jQuery, window.CondoUploader);
 	}
 }(function ($, uploads, undefined) {
 	'use strict';
@@ -58,11 +58,15 @@
 		
 		
 		
-	}]).controller('UploadsCtrl', ['$scope', 'Condo.Api', 'Condo.Broadcast', function($scope, api, broadcaster) {
+	}]).controller('Condo.Controller', ['$scope', 'Condo.Api', 'Condo.Broadcast', function($scope, api, broadcaster) {
 		
 		$scope.uploads = [];
+		$scope.upload_count = 0;
 		$scope.endpoint = '/uploads';	// Default, the directive can overwrite this
+		
 		$scope.autostart = true;
+		$scope.ignore_errors = true;		// Continue to autostart after an error
+		$scope.parallelism = 1;				// number of uploads at once
 		
 		
 		$scope.add = function(files) {
@@ -71,6 +75,11 @@
 				ret = 0;		// We only want to check for auto-start after the files have been added
 			
 			for (; i < length; i += 1) {
+				if(files[i].size <= 0 || files[i].type == '')
+					continue;
+				
+				$scope.upload_count += 1;
+				
 				api.check_provider($scope.endpoint, files[i]).then(function(upload){
 					ret += 1;
 					$scope.uploads.push(upload);
@@ -78,6 +87,8 @@
 						$scope.check_autostart();
 				}, function(failure) {
 					
+					$scope.upload_count -= 1;
+					
 					ret += 1;
 					if(ret == length)
 						$scope.check_autostart();
@@ -104,6 +115,7 @@
 			for (var i = 0, length = $scope.uploads.length; i < length; i += 1) {
 				if($scope.uploads[i] === upload) {
 					$scope.uploads.splice(i, 1);
+					$scope.upload_count -= 1;
 					break;
 				}
 			}
@@ -127,28 +139,57 @@
 		});
 		
 		
+		//
+		// Autostart more uploads as this is bumped up
+		//
+		$scope.$watch('parallelism', function(newValue, oldValue) {
+			if(newValue > oldValue)
+				$scope.check_autostart();
+		});
+		
+		
 		$scope.check_autostart = function() {
 			//
 			// Check if any uploads have been started already
 			//	If there are no active uploads we'll auto-start
 			//
+			// PENDING = 0,
+			// STARTED = 1,
+			// PAUSED = 2,
+			// UPLOADING = 3,
+			// COMPLETED = 4,
+			// ABORTED = 5
+			//
 			if ($scope.autostart) {
 				var shouldStart = true,
-					state, i, length;
+					state, i, length, started = 0;
 					
 				for (i = 0, length = $scope.uploads.length; i < length; i += 1) {
 					state = $scope.uploads[i].state;
-					if (state > 0 && state < 4) {
-						shouldStart = false;
-						break;
+					
+					//
+					// Count started uploads (that don't have errors if we are ignoring errors)
+					//	Up until we've reached our parallel limit, then stop
+					//
+					if (state > 0 && state < 4 && !($scope.uploads[i].error && $scope.ignore_errors)) {
+						started += 1;
+						if(started >= $scope.parallelism) {
+							shouldStart = false;
+							break;
+						}
 					}
 				}
 				
 				if (shouldStart) {
+					started = $scope.parallelism - started;		// How many can we start
+					
 					for (i = 0; i < length; i += 1) {
 						if ($scope.uploads[i].state == 0) {
 							$scope.uploads[i].start();
-							break;
+							
+							started -= 1;
+							if(started <= 0)	// Break if we can't start anymore
+								break;
 						}
 					}
 				}
@@ -158,5 +199,9 @@
 	}]);
 	
 	
+	//
+	// Anonymous function return
+	//
+	return uploads;
 	
 }));

data/app/assets/javascripts/condo/google.js

@@ -60,7 +60,10 @@
 			var self = this,
 				strategy = null,
 				part_size = 1048576,	// This is the amount of the file we read into memory as we are building the hash (1mb)
+				pausing = false,
 				defaultError = function(reason) {
+					self.error = !pausing;
+					pausing = false;
 					self.pause(reason);
 				},
 
@@ -200,6 +203,7 @@
 			this.message = 'pending';
 			this.name = file.name;
 			this.size = file.size;
+			this.error = false;
 			
 			
 			//
@@ -212,6 +216,8 @@
 			this.start = function(){
 				if(strategy == null) {	// We need to create the upload
 					
+					this.error = false;
+					pausing = false;
 					this.message = null;
 					this.state = STARTED;
 					strategy = {};			// This function shouldn't be called twice so we need a state
@@ -229,12 +235,12 @@
 								}
 							}, defaultError);
 						
-					}, function(reason){
-						self.pause(reason);
-					});	// END BUILD_REQUEST
+					}, defaultError);	// END BUILD_REQUEST
 					
 					
 				} else if (this.state == PAUSED) {				// We need to resume the upload if it is paused
+					this.error = false;
+					pausing = false;
 					this.message = null;
 					strategy.resume();
 				}
@@ -243,6 +249,7 @@
 			this.pause = function(reason) {
 				if(strategy != null && this.state == UPLOADING) {	// Check if the upload is uploading
 					this.state = PAUSED;
+					pausing = true;
 					strategy.pause();
 				} else if (this.state <= STARTED) {
 					this.state = PAUSED;

data/app/assets/javascripts/condo/rackspace.js

@@ -44,7 +44,10 @@
 			var self = this,
 				strategy = null,
 				part_size = 2097152,			// Multi-part uploads should be bigger then this
+				pausing = false,
 				defaultError = function(reason) {
+					self.error = !pausing;
+					pausing = false;
 					self.pause(reason);
 				},
 
@@ -184,9 +187,7 @@
 								set_part(data, result);
 							}, defaultError);
 						
-						}, function(reason){
-							self.pause(reason);
-						});	// END BUILD_REQUEST
+						}, defaultError);	// END BUILD_REQUEST
 						
 					} else {
 						//
@@ -248,6 +249,7 @@
 			this.message = 'pending';
 			this.name = file.name;
 			this.size = file.size;
+			this.error = false;
 			
 			
 			//
@@ -260,6 +262,8 @@
 			this.start = function(){
 				if(strategy == null) {	// We need to create the upload
 					
+					pausing = false;
+					this.error = false;
 					this.message = null;
 					this.state = STARTED;
 					strategy = {};			// This function shouldn't be called twice so we need a state (TODO:: fix this)
@@ -277,12 +281,13 @@
 								}
 							}, defaultError);
 						
-					}, function(reason){
-						self.pause(reason);
-					});	// END BUILD_REQUEST
+					}, defaultError);	// END BUILD_REQUEST
 					
 					
 				} else if (this.state == PAUSED) {				// We need to resume the upload if it is paused
+					
+					pausing = false;
+					this.error = false;
 					this.message = null;
 					strategy.resume();
 				}
@@ -291,6 +296,7 @@
 			this.pause = function(reason) {
 				if(strategy != null && this.state == UPLOADING) {	// Check if the upload is uploading
 					this.state = PAUSED;
+					pausing = true;
 					strategy.pause();
 				} else if (this.state <= STARTED) {
 					this.state = PAUSED;

data/app/assets/javascripts/condo/uploader.js

@@ -251,6 +251,9 @@
 				params['file_size'] = the_file.size;
 				params['file_name'] = the_file.name;
 				
+				if(!!the_file.dir_path)
+					params['file_path'] = the_file.dir_path;
+				
 				return $http({
 					method: 'GET',
 					url: api_endpoint + '/new',

data/lib/condo.rb

@@ -29,6 +29,7 @@ module Condo
 				@upload ||= {}
 				@upload[:file_size] = params[:file_size].to_i
 				@upload[:file_name] = (instance_eval &@@callbacks[:sanitize_filename])
+				@upload[:file_path] = (instance_eval &@@callbacks[:sanitize_filepath]) if params[:file_path]
 				
 				valid, errors = instance_eval &@@callbacks[:pre_validation]		# Ensure the upload request is valid before uploading
 				
@@ -56,6 +57,7 @@ module Condo
 				@upload[:file_size] = params[:file_size].to_i
 				@upload[:file_id] = params[:file_id]
 				@upload[:file_name] = (instance_eval &@@callbacks[:sanitize_filename])
+				@upload[:file_path] = (instance_eval &@@callbacks[:sanitize_filepath]) if params[:file_path]
 				
 				upload = condo_backend.check_exists({
 					:user_id => resident,
@@ -66,7 +68,7 @@ module Condo
 				
 				if upload.present?
 					residence = set_residence(upload.provider_name, {
-						:provider_location => upload.provider_location,
+						:location => upload.provider_location,
 						:upload => upload
 					})
 					

data/lib/condo/configuration.rb

@@ -8,7 +8,13 @@ module Condo
 		@@callbacks = {
 			#:resident_id		# Must be defined by the including class
 			:bucket_name => proc {"#{Rails.application.class.parent_name}#{instance_eval @@callbacks[:resident_id]}"},
-			:object_key => proc {params[:file_name]},
+			:object_key => proc {
+				if params[:file_path]
+					params[:file_path] + params[:file_name]
+				else
+					params[:file_name]
+				end
+			},
 			:object_options => proc {{:permissions => :private}},
 			:pre_validation => proc {true},	# To respond with errors use: lambda {return false, {:errors => {:param_name => 'wtf are you doing?'}}}
 			:sanitize_filename => proc {
@@ -16,6 +22,11 @@ module Condo
 					filename.gsub!(/^.*(\\|\/)/, '')	# get only the filename (just in case)
 					filename.gsub!(/[^\w\.\-]/,'_')		# replace all non alphanumeric or periods with underscore
 				end
+			},
+			:sanitize_filepath => proc {
+				params[:file_path].tap do |filepath|
+					filepath.gsub!(/[^\w\.\-\/]/,'_')		# replace all non alphanumeric or periods with underscore
+				end
 			}
 			#:upload_complete	# Must be defined by the including class
 			#:destroy_upload	# the actual delete should be done by the application
@@ -76,7 +87,7 @@ module Condo
 					@@locations[namespace][name][options[:location].to_sym] = res
 				else
 					@@locations[namespace][name][:default] = res
-					@@locations[namespace][name][res.location] = res
+					@@locations[namespace][name][res.location.to_sym] = res
 				end
 			end
 		end

data/lib/condo/version.rb

@@ -1,3 +1,3 @@
 module Condo
-	VERSION = "0.0.1"
+	VERSION = "1.0.0"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: condo
 version: !ruby/object:Gem::Version
-  version: 0.0.1
+  version: 1.0.0
   prerelease: 
 platform: ruby
 authors:
@@ -9,7 +9,7 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2012-10-15 00:00:00.000000000 Z
+date: 2012-11-08 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rails