slubydoo 0.0.4.1

data/README

@@ -0,0 +1,111 @@
+Copyright (c) August 2008 EFishman, released under the MIT license
+
+
+-----
+Notes
+-----
+SLUBYDOO is a feature set that allows you to work with slony - ruby - postgresql.  Slubydoo is not just an API.  With Slubydoo you can use migrations to manipulate your database and still use slony for replication.  You may be able to get by using Slubydoo with a minimal knowledge of slony.  However, I advise you to read up on Slony to see if it truly is a fit for your database schema.  Slony is trigger based, so Slubydoo is intended to work with slonys strengths while insulating from finer details.  I currently use Slubydoo with Ruby On Rails ActiveRecord, however it is easy to change a couple lines and you are off and running with another ORM.   
+
+
+-------------------
+Yaml Configurations
+-------------------
+The principal to recognize first of all is that slony is highly configurable.  I have attempted to create a highly configurable feature set that is adaptable to your configurations and choices.  There are two configuration files slony_nodes.yml and slony_paths.yml.  Here is where you put your database connection information, remote machine connection information and file system path information.  Slubydoo will run off of your configurations. 
+
+The application_configuration gem is required and allows you to set these configs to your specific systems.
+
+In your application_configuration.yml file add a param db_slony_replication_status to true or false.  Then in your app you can refer to this variable as app_config.db_slony_replication_status.  This allows you to switch replication on or off which can be useful in development and test modes.
+
+Example app_config/slubydoo/slony_nodes.yml file:
+
+---
+nodes:
+  -
+    node_name: master1
+    node_id: 1
+    node_schema: public
+    node_cluster_name: slony_cluster
+    node_host: 127.0.0.1
+    node_database_name: master
+    node_username: foo
+    node_password: password
+    node_port: 5432
+    node_adapter: postgresql
+    node_encoding: unicode
+    node_type: master
+    node_system_username: foo
+    node_system_password: password
+    node_slonik_cmd_path: "/usr/local/pgsql/bin/slonik"
+    node_slon_cmd_path: "/usr/local/pgsql/bin/slon"
+  -
+    node_name: slave2
+    node_id: 2
+    node_schema: public
+    node_cluster_name: slony_cluster
+    node_host: 127.0.0.0
+    node_database_name: slave
+    node_username: foo
+    node_password: password
+    node_port: 5432
+    node_adapter: postgresql
+    node_encoding: unicode
+    node_type: slave
+    node_system_username: foo
+    node_system_password: password
+    node_slonik_cmd_path: "/usr/bin/slonik"
+    node_slon_cmd_path: "/usr/bin/slon"
+
+
+Example app_config/slubydoo/slony_paths.yml file:
+
+slubydoo_initialize_yaml_dir_path: "/lib/slubydoo/sluby_snacks/"
+slubydoo_config_dir_path: "/config/app_config/slubydoo/"
+slubydoo_node_file_name: "slony_nodes.yml"
+slubydoo_tables_file_name: "slon_tables.yml"
+slubydoo_sequences_file_name: "slon_sequences.yml"
+slubydoo_sets_file_name: "slon_sets.yml"
+slubydoo_execute_dir_path: "/tmp/"
+slubydoo_log_file_path: "/tmp/foo.log"
+slubydoo_slonik_dir_path: "/tmp/"
+slubydoo_delete_files_on_remote_hosts: false
+slubydoo_refine_table_search_terms: ""
+slubydoo_sleep_time: 40
+slubydoo_master_slave_process_conditions: "-d 1 -s 20000 -g 50 -t 60000"
+
+Make sure all these paths exist and in the same path locations on both your master and slave databases.
+
+
+------------------
+Other Useful Intel
+------------------
+For legacy tables, you have several choices with slubydoo.  You can create yaml files according to my specifications (See SlonySet) and slubydoo will create slony sets and tables based on these yamls of your design.  This will allow you to design your slony sets according to your databases foreign keys and relationships.  The other choice is to automaticaly create one set for each table that has a primary key (default).  Slubydoo will search your database catalog and determine the tables with primary keys and which ones have sequences.  It's your choice.  There is nothing stopping you from altering it further if you like a different approach.  If you want the magic push one button approach where you don't have to configure anything, slony is NOT for you and neither is slubydoo.
+
+Make sure that you can ssh from the application servers directly to the database machines with just a username, password and hostaddress.  Also that you can ssh from the master db to the slave db and vice versa.
+
+I have put sql and connection specific information in the SlonyPreset class.  That way you only have to change this file if you want to switch from postgresql specific sql or from ActiveRecord methods for example.
+
+
+----------
+Rake Tasks
+----------
+rake slubydoo:replication_start
+Where to begin.  This will create slony sets based on either your yaml configs or your current schema (default).  It will include tables with primary keys only.  If you want to include tables without primary keys you can't use slony.  I put these rake tasks in my app, but you could uncomment them and use them here if you want.
+
+
+----------
+Migrations
+----------
+See the RDocumentation on how to use slubydoo with migrations.  This works nicely and is pretty much the entire reason of why I built this functionality.  
+#replicate_nodes(rep_action, options = {}) {|if block_given?| ...}
+#basically runs the same migration code on more than one database you wrap your migrations like this: 
+#self.up
+#  replicate_nodes(:alter_table, :rep_table => :table1) do
+#    add_column :table1, :column2, :integer
+#  end
+#end
+
+
+---------------
+Recommendations
+---------------
+Browse through the Slubydoo RDocumentation after reading this.

data/lib/slubydoo.rb

@@ -0,0 +1,4 @@
+require File.join(File.dirname(__FILE__),"slubydoo", "slony_set")
+require File.join(File.dirname(__FILE__),"slubydoo", "slony_cluster")
+require File.join(File.dirname(__FILE__),"slubydoo", "slony_node")
+require File.join(File.dirname(__FILE__),"slubydoo", "slony_preset")

data/lib/slubydoo/slony_cluster.rb

@@ -0,0 +1,206 @@
+require 'net/ssh'
+require 'net/sftp'
+require 'stringio'
+
+class SlonyCluster
+
+  class << self
+
+    # Kick it ALL off!
+    # Make sure your configs are all set first.
+    def slubydoo_start
+      bresult = false
+      begin
+        puts "\nBeginning creation of slony nodes, processing of sets, and starting processes.\n"
+        mn = SlonyNode.master_node
+        sn = SlonyNode.slave_nodes
+        bresult = initialize_nodes(mn, sn)
+        bresult = master_and_slave_processes("start", mn, sn)  if bresult == true
+        bresult = SlonySet.subscribe_sets(mn, sn)  if bresult == true
+      rescue Exception => ex
+        puts "Error while running slubydoo_start: #{ex}\n" if bresult != true
+      end # begin
+      bresult
+    end
+     
+    def initialize_nodes(mn, sn)
+      bresult = false
+      nodes = []
+      nodes << mn
+      nodes << sn
+      nodes.flatten.each do |node| 
+        begin
+          conf_string = connection_namespace(mn, sn) 
+          conf_string << init_cluster if node["node_type"] == "master"
+          conf_string << create_cluster_store(mn, node) if node["node_type"] == "slave"
+          sfname = "#{app_config.slubydoo_slonik_dir_path}#{node["node_name"]}_init_" << String.create_random << ".sh"
+          bresult = create_file_on_system(node, sfname, conf_string)
+          bresult = execute_command_on_system(node, "#{node["node_slonik_cmd_path"]} #{sfname}") if bresult == true
+          bresult = delete_file_on_system(node, sfname) if bresult == true and app_config.slubydoo_delete_files_on_remote_hosts
+          sleep app_config.slubydoo_sleep_time
+        rescue Exception => ex
+          puts "Error while initializing nodes: #{ex}\n" if bresult != true
+        end
+      end
+      bresult
+    end
+
+    # slon is the daemon application that "runs" Slony-I replication. A slon instance must be run for each node in a Slony-I cluster. 
+    def master_and_slave_processes(start_stop, mn = SlonyNode.master_node, sn = SlonyNode.slave_nodes, log_path = app_config.slubydoo_log_file_path, conditions = app_config.slubydoo_master_slave_process_conditions )
+      bresult = false
+      sn = [mn] << sn
+      begin
+        sn.flatten.each do |node|
+          puts %{#{node["node_slon_cmd_path"]} will #{start_stop} after #{Time.now} on db=#{node["node_database_name"]} host=#{node["node_host"]}\n}
+          filestr = %{#{node["node_slon_cmd_path"]} #{conditions} #{node["node_cluster_name"]} #{conninfo(node)}} if start_stop == "start"
+          filestr = "killall -15 slon" if start_stop == "stop"
+          #sfname = "#{app_config.slubydoo_slonik_dir_path}#{node["node_name"]}_#{start_stop}_process_" << String.create_random << ".sh"
+          #bresult = create_file_on_system(node, sfname, filestr)
+          #bresult = execute_command_on_system(node, "sh #{sfname} &") if bresult == true
+          bresult = execute_command_on_system(node, "#{filestr}")
+          bresult = true if start_stop == "stop"
+        end
+      rescue Exception => ex
+        puts "Error with master_and_slave_processes: #{ex}\n"
+        bresult = false
+      end # begin
+      bresult
+    end
+             
+    # This is used to create all sets, tables, sequences, and subscribe those sets, 
+    # depending on what arguments are in the yamls that you feed it.
+    # For ease of use to create a file and run it with the same method.
+    def run_slonik_on_system(mn = SlonyNode.master_node, sn = SlonyNode.slave_nodes, in_text = "", description = "")
+      bresult = false
+      if in_text == ""
+        puts "You did not supply any text for run_slonik_on_system. Are your yamls configured correctly?\n"
+      else
+        begin
+          new_file_str = "#!/bin/sh\n"
+          new_file_str = connection_namespace(mn, sn) << in_text
+          sfname = "#{app_config.slubydoo_slonik_dir_path}slony_#{description.downcase.gsub(/[\s]/, '_')}_" << String.create_random << ".sh"
+          bresult = create_file_on_system(mn, sfname, new_file_str)
+          bresult = execute_command_on_system(mn, "#{mn["node_slonik_cmd_path"]} #{sfname}") if bresult == true
+          bresult = delete_file_on_system(mn, sfname) if bresult == true and app_config.slubydoo_delete_files_on_remote_hosts
+          sleep app_config.slubydoo_sleep_time # give things a few seconds to catch up
+        rescue Exception => ex 
+          puts "Error message: #{ex}\nFailed to #{description}  \nThis was the attempted text: \n\n#{new_file_str}\n"
+        end # begin
+      end # if in_text == ""
+      bresult
+    end
+    
+    # allows you to take a command and execute it remotely using the gem net-ssh
+    def execute_command_on_system(event_node, in_command)
+      bresult = false
+      output = ""
+      begin
+        Net::SSH.start(event_node["node_host"], event_node["node_system_username"], :password => event_node["node_system_password"]) do |ssh|
+          output = ssh.exec!(in_command << " > #{app_config.slubydoo_log_file_path} 2>&1 &")
+        end # Net::SSH.start
+        puts "This command was excecuted: #{in_command}\nThe host was #{event_node["node_host"]} #{output}\n"
+        bresult = true
+      rescue Exception => ex
+        puts "Failed to execute command: #{in_command}. Error message: #{ex}\n"
+      end # begin
+      bresult
+    end
+    
+    # allows you to create a file remotely using the gem net-sftp
+    def create_file_on_system(event_node, sfname_remote, text_in_file)
+      bresult = false
+      output = ""
+      begin
+        Net::SFTP.start(event_node["node_host"], event_node["node_system_username"], :password => event_node["node_system_password"]) do |sftp|
+          io = StringIO.new(text_in_file)
+          sftp.upload! io, sfname_remote
+        end
+        Net::SSH.start(event_node["node_host"], event_node["node_system_username"], :password => event_node["node_system_password"]) do |ssh|
+          output = ssh.exec!("chmod u+x #{sfname_remote}" << " > #{app_config.slubydoo_log_file_path} 2>&1 &") # exec prints the bresults to $stdout, catch into output
+        end # Net::SSH.start
+        puts "This file was created: #{sfname_remote}\nThe host was #{event_node["node_host"]}\nThe text in the file is: #{text_in_file}\n #{output}\n"
+        bresult = true
+      rescue Exception => ex 
+        puts "Failed to create file: #{sfname_remote}\nError message: #{ex}\n"
+        puts "\nThe host was #{event_node["node_host"]}\n" rescue ""
+        bresult =  false
+      end # begin
+      bresult
+    end
+    
+    # allows you to delete a file remotely using the gem net-ssh
+    def delete_file_on_system(event_node, sfname_remote)
+      bresult = false
+      output = ""
+      begin
+        Net::SSH.start(event_node["node_host"], event_node["node_system_username"], :password => event_node["node_system_password"]) do |ssh|
+          output = ssh.exec!("rm #{sfname_remote}" << " > #{app_config.slubydoo_log_file_path} 2>&1 &")
+        end # Net::SSH.start
+        puts "This file was deleted: #{sfname_remote}\nThe host was #{event_node["node_host"]}\n #{output}\n"
+        bresult = true
+      rescue Exception => ex 
+        puts "Failed to delete file: #{sfname_remote}\nError message: #{ex}\n"
+        puts "\nThe host was #{event_node["node_host"]}\n" rescue ""
+      end # begin
+      bresult
+    end
+
+    def store_paths(master_node = SlonyNode.master_node, slave_nodes = SlonyNode.slave_nodes)
+      str = ""
+      master_and_slave_processes("stop")
+      slave_nodes.each do |slave_node|
+      str << store_path( master_node["node_id"], slave_node["node_id"], conninfo(master_node)) # master node
+        str << store_path( slave_node["node_id"], master_node["node_id"], conninfo(slave_node)) # slave nodes
+      end
+      str
+    end
+  
+  
+    private
+    
+    # used by methods that need connection information
+    def conninfo(node)
+      %{'dbname=#{node["node_database_name"]} host=#{node["node_host"]} user=#{node["node_username"]} password=#{node["node_password"]} port=#{node["node_port"].to_s}'}
+    end
+    
+    # init the first node.  Its id MUST be 1.  This creates the schema
+    # _$CLUSTERNAME containing all replication system specific database objects.
+    def init_cluster( id = 1, comment = "Master Node")
+      %{INIT CLUSTER ( ID = #{id}, COMMENT = '#{comment} #{id}');\n}
+    end
+
+    # custom slony store node adds each subsequent node.
+    def store_node(id, comment = "Slave Node", ev = 1)
+      %{STORE NODE ( ID = #{id}, COMMENT = '#{comment} #{id}', EVENT NODE = #{ev});\n}
+    end
+
+    # custom slony he path is defined by designating one node as a server and the other as a client for messaging.
+    def store_path(server, client, conninfo)
+      %{STORE PATH (SERVER = #{server}, CLIENT = #{client}, CONNINFO = #{conninfo});\n}
+    end
+    
+    def drop_path(server, client)
+      %{DROP PATH (SERVER = #{server}, CLIENT = #{client});\n}
+    end
+
+    # Defines which namespace the replication system uses
+    # Admin conninfo's are used by the slonik program to connect to the node databases.
+    def connection_namespace(mn, sn)
+      conf_string = ""
+      conf_string << "CLUSTER NAME = #{mn["node_cluster_name"]};\n"
+      conf_string << %{NODE #{mn["node_id"]} ADMIN CONNINFO = #{conninfo(mn)};\n}
+      sn.each {|s| conf_string << %{NODE #{s["node_id"]} ADMIN CONNINFO = #{conninfo(s)};\n} }
+      conf_string
+    end
+
+    # Create the slave node and tell the two nodes how to connect to
+    # each other and how they should listen for events.
+    def create_cluster_store(mn, slave_node, str = "\n")
+      str << store_node(slave_node["node_id"])  # slave nodes
+      str << store_path( mn["node_id"], slave_node["node_id"], conninfo(mn)) # master node
+      str << store_path( slave_node["node_id"], mn["node_id"], conninfo(slave_node)) # slave nodes
+      str
+    end
+
+  end # class << self
+end # class SlonyCluster

data/lib/slubydoo/slony_node.rb

@@ -0,0 +1,60 @@
+class SlonyNode
+
+
+  class << self
+    
+    def slave_nodes
+      if @slave_nodes.nil? || @slave_nodes.empty?
+        x = YAML.load File.open(SlonyPreset.app_slubydoo_config_dir_path << app_config.slubydoo_node_file_name)
+        node_info = x["nodes"]
+        @slave_nodes = []
+        node_info.each do |n|
+          @slave_nodes << {}.merge(n) if n["node_type"] == "slave"
+        end
+      end
+      @slave_nodes
+    end
+  
+    def master_node
+      if @master_node.nil? || @master_node.empty?
+        x = YAML.load File.open(SlonyPreset.app_slubydoo_config_dir_path << app_config.slubydoo_node_file_name)
+        node_info = x["nodes"]
+        @master_node = {}
+        node_info.each do |n| 
+          if n["node_type"] == "master"
+            @master_node = {}.merge(n) 
+            break
+          end # if n["node_type"] == "master"
+        end # node_info.each do |n| 
+      end
+      @master_node
+    end
+    
+    # basically runs the same migration code on more than one database
+    # you wrap your migrations like this:
+    # self.up
+    #   replicate_nodes(:alter_table, :rep_table => :table1) do
+    #     add_column :table1, :column2, :integer
+    #   end 
+    # end
+    def replicate_nodes(rep_action, options = {})
+      yield if block_given? # run the migration on the master node
+      if app_config.db_slony_replication_status == true
+        begin
+          nodes = slave_nodes
+          nodes.each do |node|
+            SlonyPreset.establish_node_connection(node)
+            yield if block_given?
+            SlonyPreset.restore_node_connection(node)
+          end
+          SlonySet.execute_default_set_script(master_node, nodes, {:rep_action => rep_action}.merge(options)) if rep_action.to_s != "schema_only"
+        rescue Exception => ex
+          SlonyPreset.restore_node_connection(master_node)
+          puts "Error: #{ex}"
+        end
+      end
+    end
+        
+  end # class << self
+  
+end # class SlonyNode

data/lib/slubydoo/slony_preset.rb

@@ -0,0 +1,116 @@
+class SlonyPreset
+  # put any configurations here that might change depending on the database/adapter/framework
+  
+  class << self
+    
+    def default_root_path
+      RAILS_ROOT
+    end
+  
+    def default_logger
+      RAILS_DEFAULT_LOGGER
+    end
+        
+    def app_slubydoo_config_dir_path
+      "#{default_root_path}#{app_config.slubydoo_config_dir_path}"
+    end
+
+    def app_slubydoo_yaml_dir_path
+      "#{default_root_path}#{app_config.slubydoo_initialize_yaml_dir_path}"
+    end
+
+    def establish_node_connection(node)
+      ActiveRecord::Base.establish_connection(
+        :adapter  => node["node_adapter"],
+        :host     => node["node_host"],
+        :username => node["node_username"],
+        :password => node["node_password"],
+        :database => node["node_database_name"],
+        :port => node["node_port"],
+        :encoding => node["node_encoding"],
+        :schema_search_path => node["node_schema"]  
+      )
+    end
+
+    def restore_node_connection(node)
+      ActiveRecord::Base.establish_connection
+    end
+
+    def execute_sql(sqlscript)
+      res = ActiveRecord::Base.connection.execute(sqlscript).result.flatten
+      res
+    end
+    
+    def find_all_tables_with_primary_keys(further_refine_search = "")
+      execute_sql %{ 
+        SELECT distinct(r.relname)
+        FROM pg_class r, pg_constraint c
+        WHERE r.oid = c.conrelid AND c.contype = 'p'
+        AND r.relname not like 'sl_%' #{further_refine_search}
+        ORDER BY r.relname desc;
+      }
+    end
+    
+    def find_all_sequences
+      execute_sql %{ 
+        SELECT distinct(relname) FROM pg_class
+        WHERE relkind = 'S'
+        ORDER BY relname DESC;
+      }
+    end
+    
+    def find_all_columns_in_table(tabname = "")
+      execute_sql %{ 
+        SELECT distinct(attname) FROM pg_attribute
+        WHERE attrelid = '#{tabname}'::regclass
+          AND attisdropped IS FALSE
+          AND attnum >= 1
+        ORDER BY attnum;
+        }
+    end
+    
+    # this should also check for a sequence in the catalog
+    def has_id_seq_column?(tabname = "")
+      col_count = execute_sql %{
+        SELECT count(*)::integer FROM pg_class 
+        WHERE relkind = 'S' 
+        AND relname = '#{default_sequence_name_structure(tabname)}';
+        }
+      return true if (col_count.first.to_i > 0)
+      false
+    end
+    
+    def find_tables_set(node_cluster_name,tablname)
+      execute_sql("SELECT tab_set FROM _#{node_cluster_name}.sl_table WHERE tab_relname = '#{tablname}';").first.to_s
+    end
+    
+    def find_sequences_set(node_cluster_name,seqname)
+      execute_sql("SELECT seq_set FROM _#{node_cluster_name}.sl_sequence WHERE seq_relname = '#{seqname}';").first.to_s
+    end
+    
+    def find_max_sequence_id(node_cluster_name)
+      execute_sql("SELECT (max(seq_id))::integer as foo_id FROM _#{node_cluster_name}.sl_sequence;").first.to_i
+    end
+    
+    def find_max_table_id(node_cluster_name)
+      execute_sql("SELECT (max(tab_id))::integer as foo_id FROM _#{node_cluster_name}.sl_table;").first.to_i
+    end
+    
+    def find_max_set_id(node_cluster_name)
+      execute_sql("SELECT (max(set_id))::integer as foo_id FROM _#{node_cluster_name}.sl_set;").first.to_i
+    end
+    
+    def bounce_sql_statement
+      "SELECT 1;"
+    end
+    
+    def default_sequence_name_structure(tablename, columname = "id")
+      "#{tablename}_#{columname}_seq"
+    end
+    
+    def drop_rep_schema(schemaname)
+      execute_sql("DROP SCHEMA IF EXISTS #{schemaname} CASCADE;")
+    end
+    
+  end # class << self
+end # class SlonyPreset

data/lib/slubydoo/slony_set.rb

@@ -0,0 +1,328 @@
+class SlonySet
+  
+  class << self
+     
+      
+    ########  sets  ########
+    # The smallest unit one node can subscribe for replication from another node is a set. A set always has an origin. 
+    # ID of the set to be created, ORIGIN = Initial origin node of the set, COMMENT = descriptive text
+    def create_set(id, origin, comment)
+      %{CREATE SET ( ID = #{id}, ORIGIN = #{origin}, COMMENT = '#{comment}');\n}
+    end
+    
+    # All sets need to be created if they are new or they have been dropped
+    # this gets run before any create table/sequence runs
+    # after sets have been created, then they can get subscribed.
+    # Creates a set for each table that has a primary key automatically.
+    # You can use the alternative create_all_sets_from_yaml if you want to specify your sets manually.
+    def create_all_sets(master_node = SlonyNode.master_node, set_str = "")
+      prepared_sets = one_set_for_each_table
+      prepared_sets.each_with_index {|r,i| set_str << create_set((i+1).to_s, master_node["node_id"], "set_#{r}")}
+      set_str
+    end
+    
+    # This is what uses your configured yaml if you've chosen to use a yaml to configure on your own
+    # otherwise you can use create_all_sets which will automatically create sets based on your db tables.
+    def create_all_sets_from_yaml(set_str = "")
+      prepared_sets = read_slon_set_yaml
+      prepared_sets.each {|c| set_str << create_set(c["setid"], c["set_origin"], c["set_comment"])}
+      set_str
+    end
+    
+    # Drop a set of tables from the Slony-I  configuration. This automatically unsubscribes all nodes from the set 
+    # and restores the original triggers and rules on all subscribers.
+    # ID = ID of the set to be dropped, ORIGIN = Current origin node of the set.
+    def drop_set(id, origin)
+      "DROP SET ( ID = #{id}, ORIGIN = #{origin} );\n"
+    end
+
+    # If you do this, you'll need to run create sets, tables and sequences again
+    # then you'll have to re subscribe those sets again.
+    # This is pretty much a reset button that undoes everything, but you'll need to 
+    # use the other resources to put it back together.
+    def drop_sets(master_node = SlonyNode.master_node, set_str = "")
+      prepared_sets = one_set_for_each_table
+      prepared_sets.each_with_index {|r,i| set_str << drop_set((i+1).to_s, master_node["node_id"])}
+      set_str
+    end
+    
+    def drop_all_sets_from_yaml(set_str = "")
+      prepared_sets = read_slon_set_yaml
+      prepared_sets.each {|c| set_str << drop_set(c["setid"], c["set_origin"])}
+      set_str
+    end
+    
+    # Initiates replication for a replication set OR Revising subscription information for already-subscribed nodes.
+    # ID = ID of the set to subscribe, PROVIDER = Node ID of the data provider from which this node draws data, 
+    # RECEIVER = Node ID of the new subscriber, FORWARD = boolean Flag whether or not the new subscriber should store the log information.
+    def subscribe_set( id, receiver, provider, forward = "NO")
+      %{SUBSCRIBE SET ( ID = #{id}, PROVIDER = #{provider}, RECEIVER = #{receiver}, FORWARD = #{forward} );\n}
+    end
+
+    # To get this to work, you need to first shut down the master and slave daemons
+    # then unsubscribe and subscribe, then restart the daemons
+    # otherwise it will not truncate the tables.
+    def subscribe_all_sets(mn = SlonyNode.master_node, sn = SlonyNode.slave_nodes, set_str = "")
+      prepared_sets = one_set_for_each_table
+      sn.each {|node| prepared_sets.each_with_index {|c,i| set_str << subscribe_set((i+1).to_s, node["node_id"], mn["node_id"])}}
+      set_str
+    end
+
+    def subscribe_all_sets_from_yaml(mn, sn, set_str = "")
+      prepared_sets = read_slon_set_yaml
+      sn.each {|node| prepared_sets.each {|c| set_str << subscribe_set(c["setid"], node["node_id"], mn["node_id"])}}
+      set_str
+    end
+  
+    # Stops the subscriber from replicating the set.
+    # ID = ID of the set to unsubscribe, RECEIVER = Node ID of the subscriber
+    def unsubscribe_set(id, receiver)
+      %{UNSUBSCRIBE SET ( ID = #{id}, RECEIVER = #{receiver} );\n}
+    end
+  
+    # To get this to work, you need to first shut down the master and slave daemons
+    # then unsubscribe and subscribe, then restart the daemons
+    # otherwise it will not truncate the tables.
+    def unsubscribe_all_sets(mn = SlonyNode.master_node, sn = SlonyNode.slave_nodes, set_str = "")
+      prepared_sets = one_set_for_each_table
+      sn.each {|node| prepared_sets.each_with_index {|c,i| set_str << unsubscribe_set((i+1).to_s, node["node_id"])}}
+      set_str
+    end
+    
+    def unsubscribe_all_sets_from_yaml(mn, sn, set_str = "")
+      prepared_sets = read_slon_set_yaml
+      sn.each {|node| prepared_sets.each {|c| set_str << unsubscribe_set(c["setid"], node["node_id"])}}
+      set_str
+    end
+    
+    ########  end of sets  ########
+    
+    ########  tables  ########
+    # SET ID of the set to which the table is to be added, ORIGIN = Origin node for the set, ID = Unique ID of the table,
+    # FULLY QUALIFIED NAME = The full table name, COMMENT = descriptive text
+    def set_add_table( set_id, set_tablename, set_tableid, set_comment, set_schema, set_origin)
+      %{SET ADD TABLE (SET ID = #{set_id}, ORIGIN = #{set_origin}, ID = #{set_tableid}, FULLY QUALIFIED NAME = '#{set_schema}.#{set_tablename}', COMMENT = '#{set_schema}.#{set_tablename}');\n}
+    end    
+
+    # after a set gets created you can add table/s, sequences to it
+    # Todo: redo the sequence area a bit in next version
+    def add_all_tables_with_sequences(master_node, set_str = "")
+      prepared_sets = one_set_for_each_table
+      prepared_sets.each_with_index do |tabl,i| 
+        i += 1
+        set_str << set_add_table(i.to_s, tabl.to_s, i.to_s, tabl.to_s, master_node["node_schema"], master_node["node_id"])
+        set_str << set_add_sequence(i.to_s, SlonyPreset.default_sequence_name_structure(tabl.to_s), i.to_s, SlonyPreset.default_sequence_name_structure(tabl.to_s), master_node["node_schema"], master_node["node_id"]) if SlonyPreset.has_id_seq_column?(tabl)
+      end
+      set_str
+    end
+    
+    # Use prepared tables with associated sets you prepare beforehand in a yaml file
+    # it's your choice to do this or just build with add_all_tables.
+    def add_all_tables_from_yaml(set_str = "")
+      prepared_sets = read_table_set_yaml
+      prepared_sets.each {|c| set_str << set_add_table(c["tab_set"], c["tab_relname"], c["tab_id"], c["tab_comment"], c["node_schema"], c["node_id"])}
+      set_str
+    end
+    ########  end of tables  ########
+    
+    ########  sequences  ########
+    # SET ID of the set to which the sequence is to be added, ORIGIN = Origin node for the set, ID = Unique ID of the sequence,
+    # FULLY QUALIFIED NAME = The full sequence name, COMMENT = descriptive text.
+    # (happier are you if you don't use sequences at all)
+    def set_add_sequence( set_id, set_sequence, set_sequenceid, set_comment, set_schema, set_origin)
+      %{SET ADD SEQUENCE (SET ID = #{set_id}, ORIGIN = #{set_origin}, ID = #{set_sequenceid}, FULLY QUALIFIED NAME = '#{set_schema}.#{set_sequence}', COMMENT = '#{set_schema}.#{set_sequence}');\n}
+    end
+    
+    # After a set gets created you can add table/s, sequences to it
+    # but don't add sequences only sets.
+    def add_all_sequences_from_yaml(set_str = "")
+      prepared_sets = read_sequence_set_yaml
+      prepared_sets.each {|c| set_str << set_add_sequence(c["seq_set"], c["seq_relname"], c["seq_id"], c["seq_comment"], c["node_schema"], c["node_id"])}
+      set_str
+    end
+    ########  end of sequences  ########
+        
+    # Install all table, set and sequence information through command line slonik
+    # for everything on first time install.
+    def subscribe_sets(mn = SlonyNode.master_node, sn = SlonyNode.slave_nodes)
+      result = false
+      result = SlonyCluster.run_slonik_on_system(mn, sn, create_all_sets(mn), description = "create_all_sets") 
+      result = SlonyCluster.run_slonik_on_system(mn, sn, add_all_tables_with_sequences(mn), description = "add_all_tables") if result == true
+      prepared_sets = one_set_for_each_table
+      sn.each {|node| prepared_sets.each_with_index {|c,i| SlonyCluster.run_slonik_on_system(mn, sn, subscribe_set((i+1).to_s, node["node_id"], mn["node_id"]), description = "subscribe_sets")}} if result == true
+      result
+    end
+    
+    # This is useful for more freedom in migrations.  That way you can run your migration and run an sql file.
+    # options are {:rep_table => [:tablename], :rep_sequence => :columnname, :rep_description => string, :rep_filename => "/path", :rep_set_id => "1", :rep_set_origin => "1"}
+    # slony execute script needs to run a file, I optionally create one if you want to alter something in a migration
+    # There are basically three table manipulation statements here in slubydoo, create, drop and alter.
+    # Therefore, I have included ways of handling these three.
+    # Note that :rep_table should be an array of table names. 
+    # Todo: need to be able to combine table names with create_table
+    def execute_default_set_script(mn = SlonyNode.master_node, sn = SlonyNode.slave_nodes, options = {})
+      result = false
+      if options[:rep_table].nil? or options[:rep_action].nil? or options[:rep_table].empty?
+        puts "Incorrect input, you must supply both :rep_table and :rep_action options."
+      else
+        options[:rep_table].each do |rep_table_name|
+          options[:rep_set_origin] = "#{mn["node_id"].to_s}" if options[:rep_set_origin].nil?
+          options[:rep_description] = "#{rep_table_name.to_s}"
+          case options[:rep_action].to_s
+          when "create_table"
+            result = create_slonik_table(rep_table_name.to_s, mn, sn, options)
+          when "drop_table"
+            result = drop_slonik_table_with_set(rep_table_name.to_s, mn, sn, options)
+          else #alter_table
+            result = alter_slonik_table(rep_table_name.to_s, mn, sn, options)
+          end # case options[:rep_action]
+        end
+      end # options[:rep_table].each do |rep_table_name|
+      result  
+    end
+        
+    
+    private
+
+    # Basically executes a script against a particular set derived from the input table
+    # or alternatively you can pass in a set id with :rep_set_id.
+    def alter_slonik_table(rep_table_name, mn = SlonyNode.master_node, sn = SlonyNode.slave_nodes, options = {})
+      result = false
+      new_str = ""
+      begin
+        rep_set_id = options[:rep_set_id].to_s if !options[:rep_set_id].nil?
+        rep_set_id = SlonyPreset.find_tables_set(mn["node_cluster_name"],rep_table_name) if rep_set_id.nil?
+        rep_set_id = SlonyPreset.find_sequences_set(mn["node_cluster_name"], SlonyPreset.default_sequence_name_structure(rep_table_name, options[:rep_sequence].to_s))  if rep_set_id.nil? and !options[:rep_sequence].nil?
+        if options[:rep_filename].nil?
+          new_rep_filename = "#{app_config.slubydoo_execute_dir_path}#{options[:rep_description].to_s.downcase.gsub(/[\s]/, '_')}_#{String.create_random}.sql"
+          result = SlonyCluster.create_file_on_system(mn, new_rep_filename, SlonyPreset.bounce_sql_statement)
+        end
+        set_str = %{EXECUTE SCRIPT ( SET ID = #{rep_set_id}, FILENAME = '#{new_rep_filename}', EVENT NODE = #{mn["node_id"]} );}
+        result = SlonyCluster.run_slonik_on_system(mn, sn, set_str, options[:rep_description].to_s)
+        result = SlonyCluster.delete_file_on_system(mn, new_rep_filename) if result == true and app_config.slubydoo_delete_files_on_remote_hosts
+      rescue Exception => ex
+        puts "Error: #{ex}\n"
+        result = false
+      end
+      result  
+    end        
+      
+    # Since I have one set for one table I have altered this method to drop set instead of altering a set.
+    # If you want this altered you can simply change this to be the same as the alter table.
+    # TODO: expand this in a later release to use optional multi table set drop of table.
+    def drop_slonik_table_with_set(rep_table_name, mn = SlonyNode.master_node, sn = SlonyNode.slave_nodes, options = {})
+      result = false
+      new_str = ""
+      begin
+        tab_set = SlonyPreset.find_tables_set(mn["node_cluster_name"],rep_table_name)
+        sn.each {|slave_node| new_str << unsubscribe_set(tab_set, slave_node["node_id"]) }
+        new_str << drop_set(tab_set, options[:rep_set_origin].to_s)
+        result = SlonyCluster.run_slonik_on_system(mn, sn, new_str, options[:rep_description].to_s) 
+      rescue Exception => ex
+        puts "Error: #{ex}\n"
+        result = false
+      end
+      result  
+    end        
+        
+    # If you don't pass in a sequence name, it will not create a sequence.
+    # opts = options from migrations or other application scripts
+    # adds a new set for this table.
+    # Creates a table in a supplied set if you give options a :rep_set_id.
+    def create_slonik_table(rep_table_name, mn = SlonyNode.master_node, sn = SlonyNode.slave_nodes, options = {})
+      begin
+        new_str = ""
+        rep_set_id = options[:rep_set_id].to_s if !options[:rep_set_id].nil?
+        rep_set_id = SlonyPreset.find_max_set_id(mn["node_cluster_name"]) + 1 if rep_set_id.nil?
+        tableid = SlonyPreset.find_max_table_id(mn["node_cluster_name"]) + 1 
+        seqid = SlonyPreset.find_max_sequence_id(mn["node_cluster_name"]) + 1  if !options[:rep_sequence].nil?
+        new_str << create_set(rep_set_id, options[:rep_set_origin].to_s, options[:rep_description].to_s)  if options[:rep_set_id].nil?
+        new_str << set_add_table(rep_set_id, rep_table_name, tableid, rep_table_name, mn["node_schema"], mn["node_id"])
+        new_str << set_add_sequence(rep_set_id, SlonyPreset.default_sequence_name_structure(rep_table_name, options[:rep_sequence].to_s), seqid, SlonyPreset.default_sequence_name_structure(rep_table_name, options[:rep_sequence].to_s), mn["node_schema"], mn["node_id"])  if !options[:rep_sequence].nil?
+        sn.each {|node| new_str << subscribe_set(rep_set_id, node["node_id"], mn["node_id"])}
+        result = SlonyCluster.run_slonik_on_system(mn, sn, new_str, options[:rep_description].to_s) 
+      rescue Exception => ex
+        puts "Error: #{ex}\n"
+        result = false
+      end
+      result
+    end
+        
+    ########  yaml reader methods  ########
+    # This is how you can start with existing tables.
+    # Put them in yamls, they will get added all at once.
+    # From then on you can use other tools above to add sets/tables/sequences.
+    # What I did was dump my current tables and sequences out into yamls.
+    # Then I created a few methods to create the other set yamls
+    # This is completely different for everyone, depending on your current schema, 
+    # so you need to do this part yourself.  Decide how you want to make your sets, 
+    # which tables/sequences you want in your sets.
+    # Set up your yaml files following the pattern below:
+    
+    # ---
+    # sets:
+    #   - 
+    #     setid: 1
+    #     set_origin: 1
+    #     set_comment: create set 1
+    #   - 
+    #     setid: 2
+    #     set_origin: 1
+    #     set_comment: create set 2
+    def read_slon_set_yaml(prepared_sets = [])
+       configs = YAML.load File.open(SlonyPreset.app_slubydoo_yaml_dir_path << app_config.slubydoo_sets_file_name)
+       sets = configs["sets"]
+       sets.each {|c| prepared_sets << {}.merge(c) }
+       prepared_sets
+     end
+    
+     # ---
+     # tables:
+     #   - 
+     #     tab_set: "1"
+     #     tab_relname: table1
+     #     tab_id: "1"
+     #     tab_comment: table1 comment
+     #   - 
+     #     tab_set: "1"
+     #     tab_relname: table2
+     #     tab_id: "2"
+     #     tab_comment: table2 comment
+     def read_table_set_yaml(prepared_sets = [])
+       configs = YAML.load File.open(SlonyPreset.app_slubydoo_yaml_dir_path << app_config.slubydoo_tables_file_name)
+       sets = configs["tables"]
+       x = SlonyNode.master_node
+       sets.each {|c| prepared_sets << {}.merge(c).merge(x) }
+       prepared_sets
+     end
+     
+     # ---
+     # sequences:
+     #   - 
+     #     seq_set: "1"
+     #     seq_id: "1"
+     #     seq_relname: table1_id_seq
+     #     seq_comment: table1 sequence comment
+     #   - 
+     #     seq_set: "1"
+     #     seq_id: "2"
+     #     seq_relname: table2_id_seq
+     #     seq_comment: table2 sequence comment
+     def read_sequence_set_yaml(prepared_sets = [])
+       configs = YAML.load File.open(SlonyPreset.app_slubydoo_yaml_dir_path << app_config.slubydoo_sequences_file_name)
+       sets = configs["sequences"]
+       x = SlonyNode.master_node
+       sets.each {|c| prepared_sets << {}.merge(c).merge(x) }
+       prepared_sets
+     end
+     
+     # Default way to make new slony sets in slubydoo
+     # instead of the set-table-sequence yamls.
+     def one_set_for_each_table(search_terms = app_config.slubydoo_refine_table_search_terms)
+       SlonyPreset.find_all_tables_with_primary_keys(search_terms)
+     end
+     
+     ########  end of yaml reader methods  ########
+    
+  end # class << self
+end # class SlonySet

data/lib/slubydoo/slony_tasks.rake

@@ -0,0 +1,57 @@
+# require 'rake'
+# 
+# namespace :slubydoo do
+#   
+#   # ====================================================
+#   # task :replication_start
+#   # ====================================================
+#   desc "Initialize and start replicating all in one command"
+#   task :replication_start => :environment do |t|
+#     value = SlonyCluster.slubydoo_start  if app_config.db_slony_replication_status == true
+#     puts "Slony PostgreSQL Replication configuration parameter is not set"  if app_config.db_slony_replication_status != true
+#   end # task :replication_start 
+# 
+#   # ====================================================
+#   # task :master_and_slave_processes
+#   # ====================================================
+#   desc <<-DESC
+#     Start or Stop individual processes on the master and on the slave \
+#     on already initialized nodes.
+#   DESC
+#   task :master_and_slave_processes => :environment do |t|
+#     start_stop = ENV["PROCESS_ACTION"] ||= "start"
+#     start_stop = "start" if (start_stop != "start") and (start_stop != "stop")
+#     value = SlonyCluster.master_and_slave_processes(start_stop)  if app_config.db_slony_replication_status == true
+#     puts "Slony PostgreSQL Replication configuration parameter is not set"  if app_config.db_slony_replication_status != true
+#   end # task :master_and_slave_processes 
+# 
+#   # ====================================================
+#   # task :drop_sets
+#   # ====================================================
+#   desc "drop_sets"
+#   task :drop_sets => :environment do |t|
+#     value = SlonyCluster.run_slonik_on_system(SlonyNode.master_node, SlonyNode.slave_nodes, SlonySet.drop_sets, description = "drop_sets")  if app_config.db_slony_replication_status == true
+#     puts "Slony PostgreSQL Replication configuration parameter is not set" if app_config.db_slony_replication_status != true
+#   end # task :drop_sets 
+# 
+#   # ====================================================
+#   # task :drop_set
+#   # ====================================================
+#   desc "drop_set"
+#   task :drop_set => :environment do |t|
+#     setid = ENV["SETID"]
+#     value = SlonyCluster.run_slonik_on_system(SlonyNode.master_node, SlonyNode.slave_nodes, SlonySet.drop_set(setid, SlonyNode.master_node["node_id"]), description = "drop_set_#{setid}")  if app_config.db_slony_replication_status == true
+#     puts "Slony PostgreSQL Replication configuration parameter is not set" if app_config.db_slony_replication_status != true
+#   end # task :drop_set 
+# 
+#   # ====================================================
+#   # task :store_paths
+#   # ====================================================
+#   desc "If your ip address changes on either your master or slave run store_paths to get slony up-to-date."
+#   task :store_paths => :environment do |t|
+#     value = SlonyCluster.run_slonik_on_system(SlonyNode.master_node, SlonyNode.slave_nodes, SlonyCluster.store_paths, description = "store_paths")  if app_config.db_slony_replication_status == true
+#     puts "Slony PostgreSQL Replication configuration parameter is not set" if app_config.db_slony_replication_status != true
+#   end # task :store_paths 
+# 
+# 
+# end # namespace :slubydoo do

data/lib/slubydoo/tasks.rb

@@ -0,0 +1 @@
+load File.join(File.dirname(__FILE__), "slony_tasks.rake")

data/test/slubydoo_spec.rb

@@ -0,0 +1,5 @@
+require File.dirname(__FILE__) + '/spec_helper.rb'
+
+describe "Slubydoo" do
+
+end

data/test/spec.opts

@@ -0,0 +1,2 @@
+--format specdoc
+--colour

data/test/spec_helper.rb

@@ -0,0 +1,3 @@
+require 'rubygems'
+require 'spec'
+require File.join(File.dirname(__FILE__),"..", "lib", "slubydoo")

metadata

@@ -0,0 +1,89 @@
+--- !ruby/object:Gem::Specification 
+name: slubydoo
+version: !ruby/object:Gem::Version 
+  version: 0.0.4.1
+platform: ruby
+authors: 
+- efishman
+autorequire: 
+bindir: bin
+cert_chain: []
+
+date: 2009-06-12 00:00:00 -04:00
+default_executable: 
+dependencies: 
+- !ruby/object:Gem::Dependency 
+  name: application_configuration
+  version_requirement: 
+  version_requirements: !ruby/object:Gem::Requirement 
+    requirements: 
+    - - "="
+      - !ruby/object:Gem::Version 
+        version: 1.5.3.1
+    version: 
+- !ruby/object:Gem::Dependency 
+  name: net-ssh
+  version_requirement: 
+  version_requirements: !ruby/object:Gem::Requirement 
+    requirements: 
+    - - "="
+      - !ruby/object:Gem::Version 
+        version: 2.0.3
+    version: 
+- !ruby/object:Gem::Dependency 
+  name: net-sftp
+  version_requirement: 
+  version_requirements: !ruby/object:Gem::Requirement 
+    requirements: 
+    - - "="
+      - !ruby/object:Gem::Version 
+        version: 2.0.1
+    version: 
+description: "slubydoo was developed by: efishman"
+email: 
+executables: []
+
+extensions: []
+
+extra_rdoc_files: []
+
+files: 
+- lib/slubydoo/slony_cluster.rb
+- lib/slubydoo/slony_node.rb
+- lib/slubydoo/slony_preset.rb
+- lib/slubydoo/slony_set.rb
+- lib/slubydoo/slony_tasks.rake
+- lib/slubydoo/tasks.rb
+- lib/slubydoo.rb
+- README
+has_rdoc: true
+homepage: 
+post_install_message: 
+rdoc_options: []
+
+require_paths: 
+- lib
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement 
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      version: "0"
+  version: 
+required_rubygems_version: !ruby/object:Gem::Requirement 
+  requirements: 
+  - - ">="
+    - !ruby/object:Gem::Version 
+      version: "0"
+  version: 
+requirements: []
+
+rubyforge_project: slubydoo
+rubygems_version: 1.0.1
+signing_key: 
+specification_version: 2
+summary: slubydoo
+test_files: 
+- test/slubydoo_spec.rb
+- test/spec.opts
+- test/spec_helper.rb