andyjeffries-rubyrep 1.2.1

data/History.txt

@@ -0,0 +1,83 @@
+== 1.2.0 2011-03-07
+
+* Feature: compatibility with Rails 3
+* Feature: do not replicate record updates that didn't change any fields (props to daudo)
+* Bug fix: reducing deadlock problems (props to gtanzillo)
+* Bug fix: adding missing schema prefix in PostgreSQL triggers
+* Bug fix: scans / syncs fail due to incorrect handling of case sensitivity of string primary key columns
+* Bug fix: reducing risk of foreign key conflicts during replication (props for root cause analysis to TonyB)
+
+== 1.1.2 2009-05-10
+
+* Bug fix: escape primary keys in replication triggers
+
+== 1.1.1 2009-02-02
+
+* Feature: optional use of an SQL logger
+* Bug fix: under postgresql, rubyrep triggers now finds the pending_changes table also if not included in current search path
+
+== 1.1.0 2009-12-15
+
+* Feature: filtering of sync / replication events with custom specified functionality
+* Feature: compatibility with JRuby 1.4.0 and ActiveRecord 2.3.5
+
+== 1.0.9 2009-10-14
+
+* Bug fix: ReplicationRun#run should load only one change log record to figure out if there are pending changes
+
+== 1.0.8 2009-10-01
+
+* Feature: replication more robust against connection failures
+* Feature: initial syncs before replication can be disabled with :initial_sync option
+* Bug fix: better detection of failed update and delete replications
+* Bug fix: fix scenario where replication could be logged as successful even when failed
+* Bug fix: make proxied replication work under JRuby
+
+== 1.0.7 2009-07-26
+
+* Bug fix: buffered LoggedChangeLoader#update to avoid timeouts with large replication backlogs.
+
+== 1.0.6 2009-07-25
+
+* Bug fix: do not assume anymore that replication log events appear in sequential order
+* Feature: increased resilience of rubyrep to network timeouts
+* Feature: introduced :after_infrastructure_setup handler
+
+== 1.0.5 2009-07-03
+
+* Bug fix: rubyrep replication runs should survive update or delete attempts rejected by the database.
+* Bug fix: support for multiple PostgreSQL schemas holding tables of the same name.
+
+== 1.0.4 2009-06-20
+
+* Bug fix: added missing file (log_helper.rb) to gem manifest
+
+== 1.0.3 2009-06-20
+
+* Minor enhancements:
+  * Added support (and according tests) for non-standard table / column names
+  * Verified compatibility with jruby 1.3.0
+* Bug fixes:
+  * Also big exception messages should be accommodated in event log table.
+
+== 1.0.2 2009-06-13
+
+* 3 minor enhancements:
+  * Simplified low-level functions to read records from the database
+    (easier integration of additional databases)
+  * Better replication efficiency with large replication backlogs
+  * Reduced costs of an empty replication run
+  * Ignore tables that only exist in one database
+    (tables created for testing / backup don't block scan / sync / replication)
+
+== 1.0.1 2009-03-25
+
+* 3 minor enhancements:
+  * Verified compatibility with active_record 2.3.2 and jruby 1.2.0
+  * Released jruby bunde with the updated activerecord and jruby
+  * Catch ctrl+c and execute graceful shutdown
+
+== 1.0.0 2009-03-07
+
+* 1 major enhancement:
+  * Initial release

data/License.txt

@@ -0,0 +1,20 @@
+Copyright (c) 2007 Arndt Lehmann
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/Manifest.txt

@@ -0,0 +1,151 @@
+History.txt
+License.txt
+Manifest.txt
+README.txt
+Rakefile
+bin/rubyrep
+config/hoe.rb
+config/mysql_config.rb
+config/postgres_config.rb
+config/proxied_test_config.rb
+config/redmine_config.rb
+config/rep_config.rb
+config/requirements.rb
+config/test_config.rb
+lib/rubyrep.rb
+lib/rubyrep/base_runner.rb
+lib/rubyrep/command_runner.rb
+lib/rubyrep/committers/buffered_committer.rb
+lib/rubyrep/committers/committers.rb
+lib/rubyrep/configuration.rb
+lib/rubyrep/connection_extenders/connection_extenders.rb
+lib/rubyrep/connection_extenders/jdbc_extender.rb
+lib/rubyrep/connection_extenders/mysql_extender.rb
+lib/rubyrep/connection_extenders/postgresql_extender.rb
+lib/rubyrep/database_proxy.rb
+lib/rubyrep/direct_table_scan.rb
+lib/rubyrep/generate_runner.rb
+lib/rubyrep/initializer.rb
+lib/rubyrep/log_helper.rb
+lib/rubyrep/logged_change.rb
+lib/rubyrep/logged_change_loader.rb
+lib/rubyrep/noisy_connection.rb
+lib/rubyrep/proxied_table_scan.rb
+lib/rubyrep/proxy_block_cursor.rb
+lib/rubyrep/proxy_connection.rb
+lib/rubyrep/proxy_cursor.rb
+lib/rubyrep/proxy_row_cursor.rb
+lib/rubyrep/proxy_runner.rb
+lib/rubyrep/replication_difference.rb
+lib/rubyrep/replication_extenders/mysql_replication.rb
+lib/rubyrep/replication_extenders/postgresql_replication.rb
+lib/rubyrep/replication_extenders/replication_extenders.rb
+lib/rubyrep/replication_helper.rb
+lib/rubyrep/replication_initializer.rb
+lib/rubyrep/replication_run.rb
+lib/rubyrep/replication_runner.rb
+lib/rubyrep/replicators/replicators.rb
+lib/rubyrep/replicators/two_way_replicator.rb
+lib/rubyrep/scan_progress_printers/progress_bar.rb
+lib/rubyrep/scan_progress_printers/scan_progress_printers.rb
+lib/rubyrep/scan_report_printers/scan_detail_reporter.rb
+lib/rubyrep/scan_report_printers/scan_report_printers.rb
+lib/rubyrep/scan_report_printers/scan_summary_reporter.rb
+lib/rubyrep/scan_runner.rb
+lib/rubyrep/session.rb
+lib/rubyrep/sync_helper.rb
+lib/rubyrep/sync_runner.rb
+lib/rubyrep/syncers/syncers.rb
+lib/rubyrep/syncers/two_way_syncer.rb
+lib/rubyrep/table_scan.rb
+lib/rubyrep/table_scan_helper.rb
+lib/rubyrep/table_sorter.rb
+lib/rubyrep/table_spec_resolver.rb
+lib/rubyrep/table_sync.rb
+lib/rubyrep/task_sweeper.rb
+lib/rubyrep/trigger_mode_switcher.rb
+lib/rubyrep/type_casting_cursor.rb
+lib/rubyrep/uninstall_runner.rb
+lib/rubyrep/version.rb
+rubyrep
+rubyrep.bat
+script/destroy
+script/generate
+script/txt2html
+setup.rb
+sims/performance/big_rep_spec.rb
+sims/performance/big_scan_spec.rb
+sims/performance/big_sync_spec.rb
+sims/performance/performance.rake
+sims/sim_helper.rb
+spec/base_runner_spec.rb
+spec/buffered_committer_spec.rb
+spec/command_runner_spec.rb
+spec/committers_spec.rb
+spec/configuration_spec.rb
+spec/connection_extender_interface_spec.rb
+spec/connection_extenders_registration_spec.rb
+spec/database_proxy_spec.rb
+spec/database_rake_spec.rb
+spec/db_specific_connection_extenders_spec.rb
+spec/db_specific_replication_extenders_spec.rb
+spec/direct_table_scan_spec.rb
+spec/dolphins.jpg
+spec/generate_runner_spec.rb
+spec/initializer_spec.rb
+spec/log_helper_spec.rb
+spec/logged_change_loader_spec.rb
+spec/logged_change_spec.rb
+spec/noisy_connection_spec.rb
+spec/postgresql_replication_spec.rb
+spec/postgresql_schema_support_spec.rb
+spec/postgresql_support_spec.rb
+spec/progress_bar_spec.rb
+spec/proxied_table_scan_spec.rb
+spec/proxy_block_cursor_spec.rb
+spec/proxy_connection_spec.rb
+spec/proxy_cursor_spec.rb
+spec/proxy_row_cursor_spec.rb
+spec/proxy_runner_spec.rb
+spec/replication_difference_spec.rb
+spec/replication_extender_interface_spec.rb
+spec/replication_extenders_spec.rb
+spec/replication_helper_spec.rb
+spec/replication_initializer_spec.rb
+spec/replication_run_spec.rb
+spec/replication_runner_spec.rb
+spec/replicators_spec.rb
+spec/rubyrep_spec.rb
+spec/scan_detail_reporter_spec.rb
+spec/scan_progress_printers_spec.rb
+spec/scan_report_printers_spec.rb
+spec/scan_runner_spec.rb
+spec/scan_summary_reporter_spec.rb
+spec/session_spec.rb
+spec/spec.opts
+spec/spec_helper.rb
+spec/strange_name_support_spec.rb
+spec/sync_helper_spec.rb
+spec/sync_runner_spec.rb
+spec/syncers_spec.rb
+spec/table_scan_helper_spec.rb
+spec/table_scan_spec.rb
+spec/table_sorter_spec.rb
+spec/table_spec_resolver_spec.rb
+spec/table_sync_spec.rb
+spec/task_sweeper_spec.rb
+spec/trigger_mode_switcher_spec.rb
+spec/two_way_replicator_spec.rb
+spec/two_way_syncer_spec.rb
+spec/type_casting_cursor_spec.rb
+spec/uninstall_runner_spec.rb
+tasks/database.rake
+tasks/deployment.rake
+tasks/environment.rake
+tasks/java.rake
+tasks/redmine_test.rake
+tasks/rspec.rake
+tasks/rubyrep.tailor
+tasks/stats.rake
+tasks/task_helper.rb
+tasks/website.rake

data/README.txt

@@ -0,0 +1,37 @@
+= rubyrep
+
+== MISSION:
+
+Development of an open-source solution for asynchronous, master-master replication of relational databases that is
+* ridiculously easy to use
+* database independent
+
+
+== MORE INFORMATION:
+
+Refer to the project website at http://www.rubyrep.org
+
+== LICENSE:
+
+(The MIT License)
+
+Copyright (c) 2009 Arndt Lehmann
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+'Software'), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
+CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
+TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
+SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/bin/rubyrep

@@ -0,0 +1,8 @@
+#!/usr/bin/ruby
+
+$LOAD_PATH.unshift File.dirname(__FILE__) + '/../lib'
+
+require 'rubyrep'
+
+exit RR::CommandRunner.run(ARGV)
+

data/lib/rubyrep.rb

@@ -0,0 +1,72 @@
+$LOAD_PATH.unshift File.dirname(__FILE__)
+$LOAD_PATH.unshift File.dirname(__FILE__) + "/rubyrep"
+
+require 'rubygems'
+require 'yaml'
+
+gem 'activerecord', '>= 3.0.5'
+require 'active_record'
+
+require 'version'
+require 'configuration'
+require 'initializer'
+require 'session'
+require 'connection_extenders/connection_extenders'
+require 'table_scan_helper'
+require 'table_scan'
+require 'type_casting_cursor'
+require 'proxy_cursor'
+require 'proxy_block_cursor'
+require 'proxy_row_cursor'
+require 'direct_table_scan'
+require 'proxied_table_scan'
+require 'database_proxy'
+require 'command_runner'
+require 'proxy_runner'
+require 'proxy_connection'
+require 'table_spec_resolver'
+require 'scan_report_printers/scan_report_printers'
+require 'scan_report_printers/scan_summary_reporter'
+require 'scan_report_printers/scan_detail_reporter'
+require 'scan_progress_printers/scan_progress_printers'
+require 'scan_progress_printers/progress_bar'
+require 'base_runner'
+require 'scan_runner'
+require 'committers/committers'
+require 'committers/buffered_committer'
+require 'log_helper'
+require 'sync_helper'
+require 'table_sorter'
+require 'table_sync'
+require 'syncers/syncers'
+require 'syncers/two_way_syncer'
+require 'sync_runner'
+require 'trigger_mode_switcher'
+require 'logged_change_loader'
+require 'logged_change'
+require 'replication_difference'
+require 'replication_helper'
+require 'replicators/replicators'
+require 'replicators/two_way_replicator'
+require 'task_sweeper'
+require 'replication_run'
+require 'replication_runner'
+require 'uninstall_runner'
+require 'generate_runner'
+require 'noisy_connection'
+
+Dir["#{File.dirname(__FILE__)}/rubyrep/connection_extenders/*.rb"].each do |extender|
+  # jdbc_extender.rb is only loaded if we are running on jruby
+  require extender unless extender =~ /jdbc/ and not RUBY_PLATFORM =~ /java/
+end
+
+require 'replication_initializer'
+require 'replication_extenders/replication_extenders'
+
+Dir["#{File.dirname(__FILE__)}/rubyrep/replication_extenders/*.rb"].each do |extender|
+  require extender
+end
+
+module RR
+  
+end

data/lib/rubyrep/base_runner.rb

@@ -0,0 +1,195 @@
+$LOAD_PATH.unshift File.dirname(__FILE__) + '/../lib'
+
+require 'optparse'
+
+module RR
+  # This class implements the base functionality for runners that process table
+  # specs.
+  class BaseRunner
+
+    # Default options if not overriden in command line
+    DEFAULT_OPTIONS = {
+      :table_specs => []
+    }
+
+    # Provided options. Possible values:
+    # * +:config_file+: path to config file
+    # * +:table_specs+: array of table specification strings
+    attr_accessor :options
+
+    # The class for the selected report printer
+    attr_accessor :report_printer_class
+
+    # The specified option parameter for the report printer
+    attr_accessor :report_printer_arg
+
+    # Returns the active ScanReportPrinters instance (as selected through the
+    # command line options OR if none was selected, the default one).
+    def report_printer
+      unless @report_printer
+        printer_class = report_printer_class || ScanReportPrinters::ScanSummaryReporter
+        @report_printer ||= printer_class.new(session, report_printer_arg)
+      end
+      @report_printer
+    end
+
+    # Returns the command line selected ScanProgressPrinters class
+    attr_accessor :selected_progress_printer
+
+    # Returns the active ScanProgressPrinter class (as selected through the
+    # command line options OR if none was selected, the default one).
+    def progress_printer
+      if selected_progress_printer
+        selected_progress_printer
+      else
+        printer_key = session.configuration.options[:scan_progress_printer]
+        ScanProgressPrinters.printers[printer_key][:printer_class]
+      end
+    end
+
+    # Returns the default command summary description (nothing).
+    # Should be overwritten by child classes.
+    def summary_description; ""; end
+
+    # Parses the given command line parameter array.
+    # Returns the status (as per UNIX conventions: 1 if parameters were invalid,
+    # 0 otherwise)
+    def process_options(args)
+      status = 0
+      self.options = DEFAULT_OPTIONS.clone
+
+      parser = OptionParser.new do |opts|
+        opts.banner = <<EOS
+Usage: #{$0} #{self.class.name.sub(/^.*::(.*)Runner$/, '\\1').downcase} [options] [table_spec] [table_spec] ...
+
+  #{summary_description}
+
+  table_spec can be either:
+    * a specific table name (e. g. 'users') or
+    * a pair of (specific) table names (e. g.: 'users,users_backup')
+        (In this case the first table in the 'left' database is compared
+         with the second table in the 'right' database.)
+    * a regular expression (e. g. '/^user/') [case insensitive match]
+  If no table_specs are provided via command line, the ones from the
+  configuration file are used.
+EOS
+        opts.separator ""
+        opts.separator "  Specific options:"
+
+        ScanReportPrinters.on_printer_selection(opts) do |printer_class, arg|
+          self.report_printer_class = printer_class
+          self.report_printer_arg = arg
+        end
+
+        ScanProgressPrinters.on_printer_selection(opts) do |printer|
+          self.selected_progress_printer = printer
+        end
+
+        opts.on("-c", "--config", "=CONFIG_FILE",
+          "Mandatory. Path to configuration file.") do |arg|
+          options[:config_file] = arg
+        end
+        
+        add_specific_options(opts)
+
+        opts.on_tail("--help", "Show this message") do
+          $stderr.puts opts
+          self.options = nil
+        end
+      end
+
+      begin
+        unprocessed_args = parser.parse!(args)
+        if options # this will be +nil+ if the --help option is specified
+          options[:table_specs] = unprocessed_args
+          raise("Please specify configuration file") unless options.include?(:config_file)
+        end
+      rescue Exception => e
+        $stderr.puts "Command line parsing failed: #{e}"
+        $stderr.puts parser.help
+        self.options = nil
+        status = 1
+      end
+
+      return status
+    end
+
+    # Signals scan completion to the (active) scan report printer if it supports
+    # that method.
+    def signal_scanning_completion
+      if report_printer.respond_to? :scanning_finished
+        report_printer.scanning_finished
+      end
+    end
+
+    # Creates a processor that does something with the given table.
+    # A processor needs to implement a +run+ method that yields for progress
+    # reporting purposes pairs of diff_type and row as defined under
+    # DirectTableScan#run.
+    def create_processor(left_table, right_table)
+      # not implemented in the base class
+    end
+
+    # Intended to be overwritten by derived classes that need to add additional
+    # options to the provided +OptionParser+ object.
+    def add_specific_options(opts)
+    end
+
+    # Intended to be overwritten by derived classes that need to modify the
+    # table_pairs.
+    # * table_pairs: array of table pairs as returned by TableSpecResolver#resolve
+    # Returns the new table pairs array.
+    def prepare_table_pairs(table_pairs)
+      table_pairs
+    end
+    
+    # Returns the active +Session+. 
+    # Loads config file and creates session if necessary.
+    def session
+      unless @session
+        load options[:config_file]
+        @session = Session.new Initializer.configuration
+      end
+      @session
+    end
+
+    attr_writer :session
+
+    # Returns the table pairs that should be processed.
+    # Refer to TableSpecRsolver#resolve for format of return value.
+    def table_pairs
+      prepare_table_pairs(session.configured_table_pairs(options[:table_specs]))
+    end
+
+    # Executes a run based on the established options.
+    def execute
+      session.configuration.exclude_rubyrep_tables
+      table_pairs.each do |table_pair|
+        report_printer.scan table_pair[:left], table_pair[:right] do
+          processor = create_processor \
+            table_pair[:left], table_pair[:right]
+          processor.progress_printer = progress_printer
+          processor.run do |diff_type, row|
+            report_printer.report_difference diff_type, row
+          end
+        end
+      end
+      signal_scanning_completion
+    end
+
+    # Entry points for executing a processing run.
+    # args: the array of command line options that were provided by the user.
+    def self.run(args)
+      runner = new
+
+      status = runner.process_options(args)
+      if runner.options
+        runner.execute
+      end
+      status
+    end
+
+  end
+end
+
+