offroad 0.0.1

data/README.rdoc

@@ -0,0 +1,29 @@
+*NOTE*: This gem is a work in progress. It does work, but it's not especially convenient, yet.
+Better documentation and scripts will follow in time.
+
+*NOTE*: Currently the gem is only compatible with Rails 2.
+
+== Overview
+
+This gem allows users without Internet access to still use your app. It requires that your application can
+be thought of in the following way:
+
+Your site is broken down into a set of "groups" or "organizations". Each group has their own distinct set of
+records, which are editable by only a distinct set of a users. Besides that, there are global records that can
+only be edited by overall site administrators, though they can possibly be viewed by regular users.
+
+In order to make offline use of your app possible, Offroad will generate an installer bundle, which is an
+OS-specific executable that can install Ruby, Rails, and also a launcher which controls a local instance of
+WEBrick and manages imports and exports. The installer and launcher will be visually labelled everywhere with
+the name of the Rails app itself, making its function obvious to users. This launcher can generate up-mirror
+files, which are transported on a USB thumbdrive or some other medium to an online system, where they can be
+uploaded to the main app site. After having done an upload, the user is prompted to download a down-mirror
+file which contains any changes to the global records, and bring this back to the offline system, which
+completes the sneakernet round-trip.
+
+Rather than sending all data, only globally accessible records and updates to the app itself are transferred
+in the down-mirror (except for the first down-mirror, which also contains the initial set of records for the
+group). Furthermore, the offline version of the app is considered be the absolutely canonical source of that
+group's records; really, this is not a "sync", it's a "mirror". This means that no-one on the online site can
+make changes to an offline group's records, not even the administrator; this is enforced by Offroad at a
+low level.

data/Rakefile

@@ -0,0 +1,75 @@
+require 'rubygems'
+require 'rake'
+require 'rake/testtask'
+require 'rake/rdoctask'
+require 'rake/gempackagetask'
+
+def common_test_settings(t)
+  t.libs << 'lib'
+  t.libs << 'test'
+  t.pattern = 'test/**/*_test.rb'
+  t.verbose = true
+end
+
+desc 'Default: run unit and functional tests.'
+task :default => :test
+
+desc 'Test Offroad'
+Rake::TestTask.new(:test) do |t|
+  common_test_settings(t)
+end
+
+desc 'Generate documentation for Offroad.'
+Rake::RDocTask.new(:rdoc) do |rdoc|
+  rdoc.rdoc_dir = 'rdoc'
+  rdoc.title    = 'Offroad'
+  rdoc.options << '--line-numbers' << '--inline-source'
+  rdoc.rdoc_files.include('README')
+  rdoc.rdoc_files.include('lib/**/*.rb')
+end
+
+begin
+  require 'rcov/rcovtask'
+  
+  Rcov::RcovTask.new(:rcov) do |t|
+    common_test_settings(t)
+    t.pattern = 'test/unit/*_test.rb' # Don't care about coverage added by functional tests
+    t.rcov_opts << '-o coverage -x "/ruby/,/gems/,/test/,/migrate/"'
+  end
+rescue LoadError
+  # Rcov wasn't available
+end
+
+begin
+  require 'ruby-prof/task'
+  
+  RubyProf::ProfileTask.new(:profile) do |t|
+    common_test_settings(t)
+    t.output_dir = "#{File.dirname(__FILE__)}/profile"
+    t.printer = :call_tree
+    t.min_percent = 10
+  end
+rescue LoadError
+  # Ruby-prof wasn't available
+end
+
+require 'lib/version'
+gemspec = Gem::Specification.new do |s|
+  s.name         = "offroad"
+  s.version      = Offroad::VERSION
+  s.authors      = ["David Mike Simon"]
+  s.email        = "david.mike.simon@gmail.com"
+  s.homepage     = "http://github.com/DavidMikeSimon/offroad"
+  s.summary      = "Manages off-Internet instances of a Rails app"
+  s.description  = "Offroad manages offline instances of a Rails app on computers without Internet access. The online and offline instances can communicate via mirror files, transported by the user via thumbdrive, burned CD, etc."
+
+  s.files        = `git ls-files .`.split("\n") - [".gitignore"]
+  s.platform     = Gem::Platform::RUBY
+  s.require_path = 'lib'
+  s.rubyforge_project = '[none]'
+
+  s.add_dependency('ar-extensions')
+end
+
+Rake::GemPackageTask.new(gemspec) do |pkg|
+end

data/TODO

@@ -0,0 +1,42 @@
+High priority:
+- Creating/updating the installer (ideally, git pull on target platform, one rake task, then git push)
+- Add a generate task to initialize config files in a Rails app
+- Documentation
+- Resolve issue of too-long index names
+
+Low priority:
+- Make unit tests against bugs fixed by 7f8415 and 31455d
+  - Try to come up with some more convenient way of building unit-test-specific model classes
+- Make sure ModelStates don't continue to stick around and cause trouble even after a class no longer acts_as_offroadable
+- Get README into the auto-generated gem rdocs
+- Get the test suite to stop generating a log file, it's annoying
+- Use finer-grained exceptions to allow for better error testing
+- Don't allow group_single records to reference non-single group records, since group_single "ownership" can change
+- Don't assume "id" as primary key name, always look it up with the appropriate ActiveRecord method
+- Allow offline app to relinquish ownership by setting a belongs_to field to 0 or NULL
+- Figure out why Hobo assumes all fields non-editable (something I'm doing in the patched-in updatable check...)
+- Assign a random number to group state when it is created, only accept mirror files with matching random number
+  - This is to prevent confusion if a group is made offline, then online, then offline again
+- Also assign a random number to the online app, and only accept mirror files which match this number as well
+  - Prevents multiple installations of an app from reading each others' mirror files
+- Use read transactions for mirror write operations (is it already doing this?)
+- The launcher should keep a log file
+- Include recent log lines (for both Rails and the launcher) in generated up-mirror files, for debugging purposes
+- Use rails logger to note activity
+- When mirror data version is confirmed, delete all sendable record state entries for deletions older than min(version) over all group_states
+- Try to gather some kind of machine identifier to put in a offline-owned group_state column
+- Try streaming out with large data sets and make sure it is sufficiently speedy, and is actually streaming (i.e. no Content-Length header)
+- Change mime-type of mirror files being downloaded to something that doesn't allow it to be accidentally viewed in-browser
+- Use md5sum on the entire cargo file instead of just each individual part, to protect against corruption outside the actual data segments
+- Allow app to optionally specify batch sizes on a model-by-model basis
+- Mirror file imports in the down mirror app should happen through GUI, and GUI should delete files once succesfully imported
+- If offline app attempts to import a down mirror file and it doesn't work, it should:
+  - Delete the down mirror file as usual, but...
+  - Explain that it wasn't imported and that another down mirror file should be downloaded
+- Figure out which activerecord methods skip the save callbacks, and wrap them to also update the mirror version
+  - Maybe methods for which this isn't practical should raise an error when called...
+- Load all the other informational values in GroupState
+- App creates a record then deletes it then sends mirror; does a deletion srs still get sent?
+  - For any fix to this: make sure that if app updates record then deletes it then sends mirror that deletion gets sent
+  - Maybe that means we need both create_version and update_version on srs?
+- Put the CargoStreamer into its own gem, it might be handy for other stuff

data/lib/app/models/offroad/group_state.rb

@@ -0,0 +1,85 @@
+module Offroad
+  private
+  
+  class GroupState < ActiveRecord::Base
+    set_table_name "offroad_group_states"
+    
+    validates_presence_of :app_group_id
+    
+    def validate
+      errors.add_to_base "Cannot find associated app group record" unless app_group
+    end
+
+    def app_group
+      Offroad::group_base_model.find_by_id(app_group_id)
+    end
+    
+    has_many :received_record_states, :class_name => "::Offroad::ReceivedRecordState", :dependent => :delete_all
+    
+    named_scope :for_group, lambda { |group| { :conditions => {
+      :app_group_id => valid_group_record?(group) ? group.id : 0
+    } } }
+    
+    def before_create
+      if Offroad::app_offline?
+        # FIXME : Fill in last_installation_at, launcher_version, app_version, etc
+        self.operating_system ||= RUBY_PLATFORM
+      end
+      
+      self.confirmed_group_data_version ||= 1
+      
+      # When first setting a group offline at online app, assume it will start out with at least current global data.
+      # It should, since that's the earliest version that could be loaded into the initial down mirror file.
+      self.confirmed_global_data_version ||= Offroad::app_online? ? SystemState::current_mirror_version : 1
+    end
+    
+    def update_from_remote_group_state!(remote_gs)
+      versioning_columns = [
+        'confirmed_global_data_version',
+        'confirmed_group_data_version'
+      ]
+      
+      online_owned_columns = [
+        'last_installer_downloaded_at',
+        'last_down_mirror_created_at',
+        'last_up_mirror_loaded_at'
+      ]
+      
+      offline_owned_columns = [
+        'last_installation_at',
+        'last_down_mirror_loaded_at',
+        'last_up_mirror_created_at',
+        'launcher_version',
+        'app_version',
+        'operating_system'
+      ]
+      
+      # Copy in values from columns owned by the remote environment that created remote_gs
+      (Offroad::app_offline? ? online_owned_columns : offline_owned_columns).each do |col|
+        self.send("#{col}=", remote_gs.send(col))
+      end
+      
+      # If the remote side says they have a newer version of something than we currently think they have, update
+      versioning_columns.each do |col|
+        self.send("#{col}=", [self.send(col), remote_gs.send(col)].max)
+      end
+      
+      save!
+    end
+    
+    def self.safe_to_load_from_cargo_stream?
+      true
+    end
+    
+    def self.note_group_destroyed(group)
+      rec = find_by_app_group_id(group.id)
+      rec.destroy
+    end
+    
+    private
+    
+    def self.valid_group_record?(rec)
+      rec.class.respond_to?(:offroad_group_base?) && rec.class.offroad_group_base?
+    end
+  end
+end

data/lib/app/models/offroad/mirror_info.rb

@@ -0,0 +1,53 @@
+module Offroad
+  private
+  
+  # Non-database model representing general information attached to any mirror file
+  # Based on the pattern found here: http://stackoverflow.com/questions/315850/rails-model-without-database
+  class MirrorInfo < ActiveRecord::Base
+    self.abstract_class = true
+    
+    def self.columns
+      @columns ||= []
+    end
+    
+    [
+      [:created_at, :datetime],
+      [:online_site, :string],
+      [:app, :string],
+      [:app_mode, :string],
+      [:app_version, :string],
+      [:operating_system, :string],
+      [:generator, :string],
+      [:schema_migrations, :string],
+      [:initial_file, :boolean]
+    ].each do |attr_name, attr_type|
+      columns << ActiveRecord::ConnectionAdapters::Column.new(attr_name.to_s, nil, attr_type.to_s, true)
+      validates_presence_of attr_name unless attr_type == :boolean
+    end
+    
+    def self.safe_to_load_from_cargo_stream?
+      true
+    end
+    
+    def self.new_from_group(group, initial_file = false)
+      mode = Offroad::app_online? ? "online" : "offline"
+      migration_query = "SELECT version FROM schema_migrations ORDER BY version"
+      migrations = Offroad::group_base_model.connection.select_all(migration_query).map{ |r| r["version"] }
+      return MirrorInfo.new(
+        :created_at => Time.now.to_s,
+        :online_site => Offroad::online_url,
+        :app => Offroad::app_name,
+        :app_mode => mode.titleize,
+        :app_version => Offroad::app_version,
+        :operating_system => RUBY_PLATFORM,
+        :generator => "Offroad " + Offroad::VERSION,
+        :schema_migrations => migrations.join(","),
+        :initial_file => initial_file
+      )
+    end
+    
+    def save
+      raise DataError.new("Cannot save MirrorInfo records")
+    end
+  end
+end

data/lib/app/models/offroad/model_state.rb

@@ -0,0 +1,36 @@
+module Offroad
+  private
+  
+  class ModelState < ActiveRecord::Base
+    set_table_name "offroad_model_states"
+    
+    validates_presence_of :app_model_name
+    
+    def validate
+      model = nil
+      begin
+        model = app_model
+      rescue NameError
+        errors.add_to_base "Given model name does not correspond to a constant"
+      end
+      
+      if model
+        errors.add_to_base "Constant is not a mirrored model" unless self.class.valid_model?(model)
+      end
+    end
+    
+    named_scope :for_model, lambda { |model| { :conditions => {
+      :app_model_name => valid_model?(model) ? model.name : nil
+    } } }
+    
+    def app_model
+      app_model_name.constantize
+    end
+    
+    private
+    
+    def self.valid_model?(model)
+      model.respond_to?(:acts_as_offroadable?) && model.acts_as_offroadable?
+    end
+  end
+end

data/lib/app/models/offroad/received_record_state.rb

@@ -0,0 +1,109 @@
+module Offroad
+  private
+  
+  class ReceivedRecordState < ActiveRecord::Base
+    set_table_name "offroad_received_record_states"
+    
+    belongs_to :model_state, :class_name => "::Offroad::ModelState"
+    
+    belongs_to :group_state, :class_name => "::Offroad::GroupState"
+    
+    def validate
+      unless model_state
+        errors.add_to_base "Cannot find associated model state"
+        return
+      end
+      model = model_state.app_model
+
+      if Offroad::app_offline?
+        if model.offroad_group_data?
+          errors.add_to_base "Cannot allow received record state for group data in offline app"
+        end
+      elsif Offroad::app_online?
+        if model.offroad_global_data?
+          errors.add_to_base "Cannot allow received record state for global records in online app"
+        elsif group_state.nil?
+          errors.add_to_base "Cannot allow received record state for online group records in online app"
+        end
+      end
+
+      if model.offroad_global_data? && group_state
+        errors.add_to_base "Cannot allow received record state for global records to also be assoc with a group"
+      end
+
+      begin
+        app_record
+      rescue ActiveRecord::RecordNotFound
+        errors.add_to_base "Cannot find associated app record"
+      end
+    end
+    
+    named_scope :for_model, lambda { |model| { :conditions => {
+      :model_state_id => model.offroad_model_state.id
+    } } }
+
+    named_scope :for_model_and_group_if_apropos, lambda { |model, group| { :conditions => {
+      :model_state_id => model.offroad_model_state.id,
+      :group_state_id => (group && model.offroad_group_data? && group.group_state) ? group.group_state.id : 0
+    } } }
+    
+    named_scope :for_record, lambda { |rec| { :conditions => {
+      :model_state_id => rec.class.offroad_model_state.id,
+      :group_state_id => (rec.class.offroad_group_data? && rec.group_state) ? rec.group_state.id : 0,
+      :local_record_id => rec.id
+    } } }
+
+    def app_record
+      model_state.app_model.find(local_record_id)
+    end
+
+    def self.redirect_to_local_ids(records, column, model, group)
+      column = column.to_sym
+      source = self.for_model_and_group_if_apropos(model, group)
+      already_allocated = source.all(:conditions => { :remote_record_id => records.map{|r| r[column]} }).index_by(&:remote_record_id)
+
+      remaining = {} # Maps newly discovered remote id to list of records in batch that reference that id
+      records.each do |r|
+        remote_id = r[column]
+        next unless remote_id && remote_id > 0
+        # TODO Check for illegal references here (i.e. group model referencing global model)
+        if already_allocated.has_key?(remote_id)
+          r[column] = already_allocated[remote_id].local_record_id
+        else
+          # Target doesn't exist yet, we'll figure out what its local id will be later
+          if remaining.has_key?(remote_id)
+            remaining[remote_id] << r
+          else
+            remaining[remote_id] = [r]
+          end
+        end
+      end
+
+      return unless remaining.size > 0
+
+      # Reserve access to a block of local ids by creating temporary records to advance the autoincrement counter
+      # TODO I'm pretty sure this is safe because it'll always be used in a transaction, but I should check
+      model.import([model.primary_key.to_sym], [[nil]]*remaining.size, :validate => false, :timestamps => false)
+      last_id = model.last(:select => model.primary_key, :order => model.primary_key).id
+      local_ids = (last_id+1-remaining.size)..last_id
+      model.delete(local_ids)
+
+      # Create the corresponding RRSes
+      model_state_id = model.offroad_model_state.id
+      group_state = model.offroad_group_data? && group ? group.group_state : nil
+      group_state_id = group_state ? group_state.id : 0
+      self.import(
+        [:model_state_id, :group_state_id, :local_record_id, :remote_record_id],
+        local_ids.zip(remaining.keys).map{|here, there| [model_state_id, group_state_id, here, there]},
+        :validate => false, :timestamps => false
+      )
+      
+      # Finally do the redirection to the new ids
+      remaining.each_key.each_with_index do |remote_id, i|
+        remaining[remote_id].each do |r|
+          r[column] = local_ids.first+i
+        end
+      end
+    end
+  end
+end

data/lib/app/models/offroad/sendable_record_state.rb

@@ -0,0 +1,91 @@
+module Offroad
+  private
+  
+  class SendableRecordState < ActiveRecord::Base
+    set_table_name "offroad_sendable_record_states"
+    
+    belongs_to :model_state, :class_name => "::Offroad::ModelState"
+    
+    def validate
+      unless model_state
+        errors.add_to_base "Cannot find associated model state"
+        return
+      end
+      
+      rec = nil
+      unless deleted
+        begin
+          rec = app_record
+        rescue ActiveRecord::RecordNotFound
+          errors.add_to_base "Cannot find associated app record"
+        end
+      end
+      
+      if rec
+        if Offroad::app_offline? && app_record.class.offroad_global_data?
+          errors.add_to_base "Cannot create sendable record state for global data in offline app"
+        elsif Offroad::app_online? && app_record.class.offroad_group_data?
+          errors.add_to_base "Cannot create sendable record state for group data in online app"
+        end
+      end
+    end
+    
+    named_scope :for_model, lambda { |model| { :conditions => {
+      :model_state_id => model.offroad_model_state.id
+    } } }
+    
+    named_scope :for_deleted_records, :conditions => { :deleted => true }
+    named_scope :for_non_deleted_records, :conditions => { :deleted => false }
+    
+    named_scope :with_version_greater_than, lambda { |v| { :conditions => ["mirror_version > ?", v] } }
+    
+    named_scope :for_record, lambda { |rec| { :conditions => {
+      :model_state_id => rec.class.offroad_model_state.id,
+      :local_record_id => rec.id
+    } } }
+    
+    def app_record
+      model_state.app_model.find(local_record_id)
+    end
+    
+    # We put SRS records in mirror files to represent deleted records
+    def self.safe_to_load_from_cargo_stream?
+      true
+    end
+    
+    def self.note_record_destroyed(record)
+      mark_record_changes(record) do |rec|
+        rec.deleted = true
+      end
+    end
+    
+    def self.note_record_created_or_updated(record)
+      mark_record_changes(record)
+    end
+
+    def self.setup_imported(model, batch)
+      model_state_id = model.offroad_model_state.id
+      mirror_version = SystemState::current_mirror_version
+      self.import(
+        [:model_state_id, :local_record_id, :mirror_version],
+        batch.map{|r| [model_state_id, r.id, mirror_version]},
+        :validate => false,
+        :timestamps => false
+      )
+    end
+
+    private
+    
+    def self.mark_record_changes(record)
+      transaction do
+        scope = for_record(record)
+        rec_state = scope.first || scope.create
+        rec_state.lock!
+        rec_state.mirror_version = SystemState::current_mirror_version
+        yield(rec_state) if block_given?
+        rec_state.save!
+      end
+    end
+    
+  end
+end

data/lib/app/models/offroad/system_state.rb

@@ -0,0 +1,33 @@
+require 'forwardable'
+
+module Offroad
+  private
+  
+  # State of the Offroad-managed app as a whole; there should only be one record in this table
+  # Attributes of that record can be read via the class methods of this class
+  class SystemState < ActiveRecord::Base
+    set_table_name "offroad_system_state"
+    
+    # Create validators and class-level attribute getters for the columns that contain system settings
+    extend SingleForwardable
+    for column in columns
+      sym = column.name.to_sym
+      next if sym == :id
+      def_delegator :instance_record, sym
+    end
+    
+    def self.increment_mirror_version
+      self.increment_counter(:current_mirror_version, instance_record.id)
+    end
+    
+    # Returns the singleton record, first creating it if necessary
+    def self.instance_record
+      sys_state = first
+      if sys_state
+        return sys_state
+      else
+        return create(:current_mirror_version => 1)
+      end
+    end
+  end
+end