nsync 0.0.2

data/LICENSE

@@ -0,0 +1,21 @@
+Copyright (c) 2010 Ben Hughes
+Copyright (c) 2010 NabeWise Media, Inc.
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.
+

data/README.md

@@ -0,0 +1,204 @@
+Nsync: Git based database synchronization
+=========================================
+
+Nsync allows you to keep disparate data sources synchronized for core data.
+The use case this is designed to solve is when you have a data processing
+system and one or many consumer facing services that depend on a canonical,
+processed version of the data.  All of this is based on the power of Git.
+Nsync makes no assumptions about your data stores, ORMs, or storage practices
+other than that the data from the producer has something that can serve as a
+unique primary key and that the consumer can be queried by a key indicating
+its source.  That said, Nsync comes with extensions for ActiveRecord 2.3.x
+that handle the simple case.
+
+A NabeWise Story
+----------------
+
+Nsync was born out of our needs at NabeWise (http://nabewise.com).  We deal
+with neighborhoods and cities.  Our source data consists of about 70,000
+neighborhoods across the US.  We carefully curate neighborhoods for each of
+our cities. Oftentimes this involves editing boundaries, making changes to
+neighborhood names, or other slight adjustments to the underlying data. We
+also process and refine the boundaries for display through a number of
+automated processes to get the data to where we want.  This occurs in a Rails
+app based on top of PostgreSQL with PostGIS.  We have to get this data to our
+website, which runs on top of MySQL and Redis.  We have enough data that full
+reloads from files are impractical, and we also want to handle events like
+neighborhood deletion intelligently.  Nsync solves these issues.
+
+Installation
+------------
+
+    gem install nsync
+
+Nsync depends on two gems that I've forked, schleyfox-grit and
+schleyfox-lockfile. I'm sorry, but this is how it has to be.  Nsync also
+currently depends on ActiveSupport ~> 2.3.5, but I am working to remove this
+dependency.
+
+Terminology
+-----------
+
+In Nsync lingo, a producer is an object/class that creates data that will go
+into a repository and propagate to consumers.  It adheres to the Producer
+interface. A consumer is an object/class that takes data from the repo and
+updates itself accordingly.  It adheres to the Consumer interface.  A producer
+is also a consumer of itself.
+
+Producer Usage
+--------------
+
+To start off with, you have to configure your shiny new producer app. This
+configuration should happen before the producer is ever used.
+
+    Nsync::Config.run do |c|
+      # The producer uses a standard repository
+      # This will automatically be created if it does not exist
+      c.repo_path = "/local/path/to/hold/data"
+      # The remote repository url will get data pushed to it
+      c.repo_push_url = "git@examplegithost:username/data.git"
+    
+      # This must be Nsync::GitVersionManager if you want things like
+      # rollback to work.
+      c.version_manager = Nsync::GitVersionManager.new
+    
+      # A lock file path to use for this app
+      c.lock_file = "/tmp/app_name_nsync.lock"
+    end
+
+Now you need to let your objects know the joy of Nsync. This is not strictly
+necessary, but can help out. If you are using ActiveRecord, do this:
+
+    ActiveRecord::Base.send(:extend, Nsync::ActiveRecord::ClassMethods)
+
+If you are using something else, do this:
+
+    YourBaseObject.send(:extend, Nsync::ClassMethods)
+
+Now, set your data classes up as producers
+
+    class Post < ActiveRecord::Base
+      nsync_producer
+    end
+
+By default, this will write out the json-ified contents of its attributes (if
+its ActiveRecord, you have to define its representation otherwise) to
+"CLASS_NAME/ID.json" in the repo.
+
+If not all of your data should be exported, you can specify an :if function
+
+    class Post
+      nsync_producer :if => lambda {|o| o.should_be_exported }
+    end
+
+After you make some data changes, you can commit and push them by doing
+
+    producer = Nsync::Producer.new
+    producer.commit("Short Message Describing Changes")
+
+See Nsync::ClassMethods, Nsync::ActiveRecord::ClassMethods,
+Nsync::Producer::InstanceMethods, and
+Nsync::ActiveRecord::Producer::InstanceMethods for more information
+
+Consumer Usage
+--------------
+
+Every good producer needs one (or many) good consumers. Again, the first step
+is configuration.
+
+The Consumer is a little less straight forward.  It requires that the classes
+from the Producer side be mapped to classes on the Consumer side.  This
+happens using Nsync::Config#map_class, which maps from a Producer class name
+to one or many Consumer classes.
+
+It also requires that Nsync::Config#version_manager is set to a class or
+instance that conforms to the VersionManager interface. This is probably a
+class on top of a database that stores versions (by commit id) as they are
+loaded into the system, such that the current version and all previous
+versions can be easily accessed. The ActiveRecord integration tests
+demonstrate this.
+
+    Nsync::Config.run do |c|
+      # The consumer uses a read-only, bare repository (one ending in .git)
+      # This will automatically be created if it does not exist
+      c.repo_path = "/local/path/to/hold/data.git"
+      # The remote repository url from which to pull data
+      c.repo_url = "git@examplegithost:username/data.git"
+    
+      # An object that implements the VersionManager interface 
+      # (see Nsync::GitVersionManager) for an example
+      c.version_manager = MyCustomVersionManager.new
+    
+      # A lock file path to use for this app
+      c.lock_file = "/tmp/app_name_nsync.lock"
+    
+      # The class mapping maps from the class names of the producer classes to
+      # the class names of their associated consuming classes. A producer can
+      # map to one or many consumers, and a consumer can be mapped to one or many
+      # producers. Consumer classes should implement the Consumer interface.
+      c.map_class "RawDataPostClass", "Post"
+      c.map_class "RawDataInfo", "Info"
+    end
+
+Now you should let your classes know about the Nsync way.If you are using
+ActiveRecord, do this:
+
+    ActiveRecord::Base.send(:extend, Nsync::ActiveRecord::ClassMethods)
+
+If you are using something else, do this:
+
+    YourBaseObject.send(:extend, Nsync::ClassMethods)
+
+Now it's time to let your objects know that they are consumers
+
+    class Post < ActiveRecord::Base
+      nsync_consumer
+    end
+
+You can (and probably should) override all or some of the default methods in
+the Consumer interface.  By default, it basically just attempts to copy hash
+from the file into the consuming database.  If your object has any relations,
+this will probably fail tragically. A better Post class would be
+
+    class Post < ActiveRecord::Base
+
+      nsync_consumer
+
+      def self.nsync_add_data(consumer, event_type, filename, data)
+        post = new
+        post.source_id = data['id']
+        post.nsync_update(consumer, event_type, filename, data)
+      end
+
+      def nsync_update(consumer, event_type, filename, data)
+        if event_type == :deleted
+          destroy
+        else
+          self.author = Author.nsync_find(data['author_id']).first
+          self.content = data['content']
+          
+          related_post_source_ids = data['related_post_ids']
+          post = self
+
+          consumer.after_current_class_finished(lambda {
+            post.related_posts = Post.all(:conditions => {:source_id =>
+              related_post_source_ids})
+          })
+
+          self.save
+        end
+      end
+    end
+
+This also demonstrates how to add callbacks to queues.
+
+You can update from the repo like so:
+
+    consumer = Nsync::Consumer.new
+    consumer.update
+
+
+
+
+
+

data/Rakefile

@@ -0,0 +1,47 @@
+require 'rubygems'
+require 'rake'
+
+gem "schleyfox-grit", ">= 2.3.0.1"
+require 'grit'
+
+require 'jeweler'
+require 'jeweler_monkey_patch'
+      
+    
+Jeweler::Tasks.new do |gem|
+  gem.name = "nsync"
+  gem.summary = %Q{Keep your data processors and apps in sync}
+  gem.description = %Q{Nsync is designed to allow you to have a separate data
+  processing app with its own data processing optimized database and a consumer
+  app with its own database, while keeping the data as in sync as you want it.}
+  gem.email = "ben@pixelmachine.org"
+  gem.homepage = "http://github.com/schleyfox/nsync"
+  gem.authors = ["Ben Hughes"]
+
+  gem.add_dependency "json"
+  gem.add_dependency "activesupport", "~> 2.3.5"
+  gem.add_dependency "schleyfox-grit", ">= 2.3.0.1"
+  gem.add_dependency "schleyfox-lockfile", ">= 1.0.0"
+
+  gem.add_development_dependency "shoulda"
+  gem.add_development_dependency "mocha"
+end
+
+
+require 'rake/testtask'
+Rake::TestTask.new(:test) do |test|
+  test.libs << 'lib' << 'test'
+  test.pattern = 'test/**/*_test.rb'
+  test.verbose = true
+end
+
+desc "Generate RCov test coverage and open in your browser"
+task :coverage do
+  require 'rcov'
+  sh "rm -fr coverage"
+  sh "rcov -Itest -Ilib -x \"rubygems/*,/Library/Ruby/Site/*,gems/*,rcov*\" --html test/*_test.rb"
+  sh "open coverage/index.html"
+end
+
+require 'yard'
+YARD::Rake::YardocTask.new

data/VERSION

@@ -0,0 +1 @@
+0.0.2

data/jeweler_monkey_patch.rb

@@ -0,0 +1,59 @@
+# because the outdated git-ruby version is not interface compatible with grit,
+# we have to prevent it from being used
+module Jeweler::Specification
+  def set_jeweler_defaults(base_dir, git_base_dir = nil)
+    base_dir = File.expand_path(base_dir)
+    git_base_dir = if git_base_dir
+                     File.expand_path(git_base_dir)
+                   else
+                     base_dir
+                   end
+    can_git = git_base_dir && base_dir.include?(git_base_dir) && File.directory?(File.join(git_base_dir, '.git'))
+
+    Dir.chdir(git_base_dir) do
+      all_files = `git ls-files`.split("\n").reject{|file| file =~ /^\./ }
+
+      if blank?(files)
+        base_dir_with_trailing_separator = File.join(base_dir, "")
+
+        self.files = all_files.reject{|file| file =~ /^(doc|pkg|test|spec|examples)/ }.compact.map do |file|
+          File.expand_path(file).sub(base_dir_with_trailing_separator, "")
+        end
+      end
+
+      if blank?(test_files)
+        self.test_files = all_files.select{|file| file =~ /^(test|spec|examples)/ }.compact.map do |file|
+          File.expand_path(file).sub(base_dir_with_trailing_separator, "")
+        end
+      end
+
+      if blank?(executables)
+        self.executables = all_files.select{|file| file =~ /^bin/}.map do |file|
+          File.basename(file)
+        end
+      end
+
+      if blank?(extensions)
+        self.extensions = FileList['ext/**/{extconf,mkrf_conf}.rb']
+      end
+
+      self.has_rdoc = true
+
+      if blank?(extra_rdoc_files)
+        self.extra_rdoc_files = FileList['README*', 'ChangeLog*', 'LICENSE*', 'TODO']
+      end
+
+      if File.exist?('Gemfile')
+        require 'bundler'
+        bundler = Bundler.load
+        bundler.dependencies_for(:default, :runtime).each do |dependency|
+          self.add_dependency dependency.name, *dependency.requirement.as_list
+        end
+        bundler.dependencies_for(:development).each do |dependency|
+          self.add_development_dependency dependency.name, *dependency.requirement.as_list
+        end
+      end
+      
+    end
+  end
+end

data/lib/nsync/active_record/consumer/methods.rb

@@ -0,0 +1,47 @@
+module Nsync
+  module ActiveRecord
+    module Consumer
+      module ClassMethods
+        def nsync_find(ids)
+          nsync_opts = read_inheritable_attribute(:nsync_opts)
+          all(:conditions => {nsync_opts[:id_key] => ids})
+        end
+
+        def nsync_add_data(consumer, event_type, filename, data)
+          data = data.dup
+          nsync_opts = read_inheritable_attribute(:nsync_opts)
+          if nsync_opts[:id_key].to_s != "id"
+            data[nsync_opts[:id_key].to_s] = data.delete("id")
+            create(data)
+          else
+            id = data.delete("id")
+            obj = new(data)
+            obj.id = id
+            obj.save
+          end
+        end
+      end
+
+      module InstanceMethods
+        def self.included(base)
+          base.send(:extend, ClassMethods)
+        end
+        def nsync_update(consumer, event_type, filename, data)
+          data = data.dup
+          if event_type == :deleted
+            destroy
+          else
+            nsync_opts = self.class.read_inheritable_attribute(:nsync_opts)
+            if nsync_opts[:id_key].to_s != "id"
+              data[nsync_opts[:id_key].to_s] = data.delete("id")
+            else
+              self.id = data.delete("id")
+              update_attributes(data)
+            end
+          end
+        end
+      end
+    end
+  end
+end
+          

data/lib/nsync/active_record/methods.rb

@@ -0,0 +1,23 @@
+require File.join(File.dirname(__FILE__), 'consumer/methods')
+require File.join(File.dirname(__FILE__), 'producer/methods')
+
+module Nsync
+  module ActiveRecord
+    module ClassMethods
+      # Makes this class an Nsync Consumer
+      def nsync_consumer(opts={})
+        nsync_opts = {:id_key => :source_id}.merge(opts)
+        write_inheritable_attribute(:nsync_opts, nsync_opts)
+        include Nsync::ActiveRecord::Consumer::InstanceMethods
+      end
+  
+      def nsync_producer(opts={})
+        nsync_opts = {:id_key => :id}.merge(opts)
+        write_inheritable_attribute(:nsync_opts, nsync_opts)
+        include Nsync::ActiveRecord::Consumer::InstanceMethods
+        include Nsync::Producer::InstanceMethods
+        include Nsync::ActiveRecord::Producer::InstanceMethods
+      end
+    end
+  end
+end

data/lib/nsync/active_record/producer/methods.rb

@@ -0,0 +1,21 @@
+module Nsync
+  module ActiveRecord
+    module Producer
+      module InstanceMethods
+        def self.included(base)
+          puts "foo"
+          base.class_eval do
+            p base
+            after_save :nsync_write
+            before_destroy :nsync_destroy
+          end
+        end
+
+        def to_nsync_hash
+          attributes
+        end
+      end
+    end
+  end
+end
+      

data/lib/nsync/class_methods.rb

@@ -0,0 +1,17 @@
+require File.join(File.dirname(__FILE__), 'producer/methods')
+module Nsync
+  module ClassMethods
+    # Makes this class an Nsync Consumer
+    def nsync_consumer(opts={})
+      nsync_opts = {:id_key => :source_id}.merge(opts)
+      write_inheritable_attribute(:nsync_opts, nsync_opts)
+    end
+
+    def nsync_producer(opts={})
+      nsync_opts = {:id_key => :id}.merge(opts)
+      write_inheritable_attribute(:nsync_opts, nsync_opts)
+      include Nsync::Producer::InstanceMethods
+    end
+  end
+end
+

data/lib/nsync/config.rb

@@ -0,0 +1,96 @@
+module Nsync
+  def self.config
+    @config ||= Config.new
+  end
+
+  def self.reset_config
+    @config = nil
+  end
+
+  class Config
+    #required to be user specified
+    attr_accessor :version_manager, :repo_path
+
+    #optional
+    attr_accessor :ordering, :repo_url, :repo_push_url, :log, :lock_file, 
+      :producer_instance
+
+    include Lockfile
+
+    def initialize
+      clear_class_mappings
+      self.log = ::Logger.new(STDERR)
+      self.lock_file = "/tmp/nsync.lock"
+    end
+
+    def lock
+      ret = nil
+      success = with_lock_file(lock_file) do
+        ret = yield
+      end
+      if success != false
+        return ret
+      else
+        log.error("[NSYNC] Could not obtain lock!; exiting")
+        return false
+      end
+    end
+
+    def cd
+      old_pwd = FileUtils.pwd
+      begin
+        FileUtils.cd repo_path
+        yield
+      ensure
+        FileUtils.cd old_pwd
+      end
+    end
+
+
+    def map_class(producer_class, *consumer_classes)
+      @class_mappings[producer_class] ||= []
+      @class_mappings[producer_class] += consumer_classes
+    end
+
+    def clear_class_mappings
+      @class_mappings = {}
+    end
+
+    def consumer_classes_for(producer_class)
+      Array(@class_mappings[producer_class]).map do |klass|
+        begin
+          klass.constantize
+        rescue NameError => e
+          log.error(e.inspect)
+          log.warn("[NSYNC] Could not find class '#{klass}'; skipping")
+          nil
+        end
+      end.compact
+    end
+
+    def version_manager
+      return @version_manager if @version_manager
+      raise "Must define config.version_manager"
+    end
+
+    def producer_instance
+      @producer_instance ||= Nsync::Producer.new
+    end
+
+    def self.run
+      yield Nsync.config
+    end
+
+    def local?
+      !repo_url
+    end
+
+    def remote?
+      !!repo_url
+    end
+
+    def remote_push?
+      !!repo_push_url
+    end
+  end
+end