db_leftovers 0.9.2 → 1.0.0

data/Gemfile

@@ -17,3 +17,7 @@ group :development do
   gem "rcov", ">= 0"
 end
 
+group :test do
+  gem 'activerecord-postgresql-adapter'
+  gem 'activerecord-mysql2-adapter'
+end

data/Gemfile.lock

@@ -24,6 +24,10 @@ GEM
       activesupport (= 3.1.3)
       arel (~> 2.2.1)
       tzinfo (~> 0.3.29)
+    activerecord-mysql2-adapter (0.0.3)
+      mysql2
+    activerecord-postgresql-adapter (0.0.1)
+      pg
     activeresource (3.1.3)
       activemodel (= 3.1.3)
       activesupport (= 3.1.3)
@@ -47,6 +51,8 @@ GEM
       treetop (~> 1.4.8)
     mime-types (1.17.2)
     multi_json (1.0.4)
+    mysql2 (0.3.11)
+    pg (0.14.1)
     polyglot (0.3.3)
     rack (1.3.6)
     rack-cache (1.1)
@@ -99,6 +105,8 @@ PLATFORMS
   ruby
 
 DEPENDENCIES
+  activerecord-mysql2-adapter
+  activerecord-postgresql-adapter
   bundler
   jeweler (~> 1.6.4)
   rails (>= 3.0.0)

data/README.html

@@ -3,12 +3,19 @@
 <p>db_leftovers lets you define indexes, foreign keys, and CHECK constraints for your Rails app
 in one place using an easy-to-read DSL,
 then run a rake task to bring your database up-to-date.
-I wrote this because I didn't want indexes and constraints scattered throughout my migrations or buried in my <code>schema.rb</code>, and I wanted a command I could run to ensure that they matched across my development, test, staging, and production databases.
-This was particularly a problem for Heroku projects, because <code>db:push</code> and <code>db:pull</code> do not transfer your foreign keys or other constraints.
-But now that it's written, I'm finding it useful on non-Heroku projects as well.</p>
+Whenever you edit the DSL, you can re-run the rake task and db_leftovers will alter your database accorindgly.
+This is useful because of the following limitations in vanilla Rails:</p>
 
-<p>At present db_leftovers works only on PostgreSQL databases,
-but it could easily be extended to cover other RDBMSes.</p>
+<ul>
+<li>There are no built-in migration methods to create foreign keys or CHECK constraints.</li>
+<li>Even if created, foreign keys and CHECK constraints won't appear in your schema.rb.</li>
+<li>If you're using Heroku, <code>db:push</code> and <code>db:pull</code> won't transfer your foreign keys and CHECK constraints.</li>
+<li>Creating indexes in your migrations makes it hard to manage them.</li>
+</ul>
+
+<p>That last point deserves some elaboration. Using <code>create_index</code> in your migrations is bug-prone because without rare developer discipline (My rule is "never change a migration after a <code>git push</code>, but I haven't seen this followed elsewhere."), you wind up missing indexes in some environments. It also means you don't have a central place to see all your indexes so you can analyze which are needed. With db_leftovers, you can rest assured that each environment conforms to a definition that is easy to read and checked into version control.</p>
+
+<p>At present db_leftovers supports PostgreSQL and MySQL, although since MySQL doesn't support index WHERE clauses or CHECK constraints, using that functionality will raise errors. (If you need to share the same definitions across Postgres and MySQL, you can run arbitrary Ruby code inside the DSL to avoid defining unsupported objects when run against MySQL.)</p>
 
 <h2>Configuration File</h2>
 
@@ -70,7 +77,7 @@ All parameters are strings or symbols.</p>
 
 <h3>table(table_name, &amp;block)</h3>
 
-<p>The <code>table</code> call is just a convenience so you can group all a table's indexes etcetera together and not keep repeating the table name. You use it like this:</p>
+<p>The <code>table</code> call is just a convenience so you can group all a table's indexes et cetera together and not keep repeating the table name. You use it like this:</p>
 
 <pre><code>table :books do
   index :author_id
@@ -104,7 +111,8 @@ The second time you run db_leftovers, it will read the expression from Postgres
 and so it will drop and re-create the constraint.
 It will drop and re-create it every time you run the rake task.
 To get around this, make sure your config file uses the same expression as printed by db_leftovers in the rake output.
-This can also happen for index WHERE clauses, fixable by a similar workaround.</p>
+This can also happen for index WHERE clauses, fixable by a similar workaround.
+MySQL doesn't have this problem because it doesn't support CHECK constraints or index WHERE clauses.</p>
 
 <p>To print messages even about indexes/foreign keys/constraints that haven't changed, you can say:</p>
 
@@ -119,8 +127,7 @@ This can also happen for index WHERE clauses, fixable by a similar workaround.</
 <h2>Known Issues</h2>
 
 <ul>
-<li>db_leftovers only supports PostgreSQL databases.
-If you want to add support for something else, just send me a pull request!</li>
+<li>When db_leftovers interrogates your database for the currently-defined indexes et cetera, it doesn't filter things by the current database name. So if you have mutliple Rails projects all accessible to the same user, you'll wind up changing more than you like (probably by DROPing things).</li>
 </ul>
 
 <h2>Contributing to db_leftovers</h2>

data/README.md

@@ -4,12 +4,17 @@ db\_leftovers
 db\_leftovers lets you define indexes, foreign keys, and CHECK constraints for your Rails app
 in one place using an easy-to-read DSL,
 then run a rake task to bring your database up-to-date.
-I wrote this because I didn't want indexes and constraints scattered throughout my migrations or buried in my `schema.rb`, and I wanted a command I could run to ensure that they matched across my development, test, staging, and production databases.
-This was particularly a problem for Heroku projects, because `db:push` and `db:pull` do not transfer your foreign keys or other constraints.
-But now that it's written, I'm finding it useful on non-Heroku projects as well.
+Whenever you edit the DSL, you can re-run the rake task and db\_leftovers will alter your database accorindgly.
+This is useful because of the following limitations in vanilla Rails:
 
-At present db\_leftovers works only on PostgreSQL databases,
-but it could easily be extended to cover other RDBMSes.
+  * There are no built-in migration methods to create foreign keys or CHECK constraints.
+  * Even if created, foreign keys and CHECK constraints won't appear in your schema.rb.
+  * If you're using Heroku, `db:push` and `db:pull` won't transfer your foreign keys and CHECK constraints.
+  * Creating indexes in your migrations makes it hard to manage them.
+  
+That last point deserves some elaboration. Using `create_index` in your migrations is bug-prone because without rare developer discipline (My rule is "never change a migration after a `git push`, but I haven't seen this followed elsewhere."), you wind up missing indexes in some environments. It also means you don't have a central place to see all your indexes so you can analyze which are needed. With db\_leftovers, you can rest assured that each environment conforms to a definition that is easy to read and checked into version control.
+
+At present db\_leftovers supports PostgreSQL and MySQL, although since MySQL doesn't support index WHERE clauses or CHECK constraints, using that functionality will raise errors. (If you need to share the same definitions across Postgres and MySQL, you can run arbitrary Ruby code inside the DSL to avoid defining unsupported objects when run against MySQL.)
 
 Configuration File
 ------------------
@@ -66,7 +71,7 @@ All parameters are strings or symbols.
 
 ### table(table\_name, &block)
 
-The `table` call is just a convenience so you can group all a table's indexes etcetera together and not keep repeating the table name. You use it like this:
+The `table` call is just a convenience so you can group all a table's indexes et cetera together and not keep repeating the table name. You use it like this:
 
     table :books do
       index :author_id
@@ -100,6 +105,7 @@ and so it will drop and re-create the constraint.
 It will drop and re-create it every time you run the rake task.
 To get around this, make sure your config file uses the same expression as printed by db\_leftovers in the rake output.
 This can also happen for index WHERE clauses, fixable by a similar workaround.
+MySQL doesn't have this problem because it doesn't support CHECK constraints or index WHERE clauses.
 
 To print messages even about indexes/foreign keys/constraints that haven't changed, you can say:
 
@@ -115,8 +121,7 @@ or
 Known Issues
 ------------
 
-* db\_leftovers only supports PostgreSQL databases.
-  If you want to add support for something else, just send me a pull request!
+* When db\_leftovers interrogates your database for the currently-defined indexes et cetera, it doesn't filter things by the current database name. So if you have mutliple Rails projects all accessible to the same user, you'll wind up changing more than you like (probably by DROPing things).
 
   
 

data/TODO

@@ -1,9 +1,2 @@
-- Refactor DatabaseInterface so all methods are instance methods and Definition/DSL uses an instance, set according to the Rails database adapter and also settable to a test subclass, so we can change the definition from test to test.
-
-- Add optional tests that use a real Postgres/MySQL/SQLite if available.
-  - Run all the same tests, but against a real DatabaseInterface instance.
-
 - Make sure everything works if the current db user has access to multiple databases: only run the code on the db given in the Rails database.yml.
 
-- Support MySQL
-

data/VERSION

@@ -1 +1 @@
-0.9.2
+1.0.0

data/db_leftovers.gemspec

@@ -5,11 +5,11 @@
 
 Gem::Specification.new do |s|
   s.name = "db_leftovers"
-  s.version = "0.9.2"
+  s.version = "1.0.0"
 
   s.required_rubygems_version = Gem::Requirement.new(">= 0") if s.respond_to? :required_rubygems_version=
   s.authors = ["Paul A. Jungwirth"]
-  s.date = "2012-09-30"
+  s.date = "2012-10-02"
   s.description = "        Define indexes and foreign keys for your Rails app\n        in one place using an easy-to-read DSL,\n        then run a rake task to bring your database up-to-date.\n"
   s.email = "pj@illuminatedcomputing.com"
   s.extra_rdoc_files = [
@@ -30,15 +30,23 @@ Gem::Specification.new do |s|
     "db_leftovers.gemspec",
     "lib/db_leftovers.rb",
     "lib/db_leftovers/constraint.rb",
-    "lib/db_leftovers/database_interface.rb",
     "lib/db_leftovers/definition.rb",
     "lib/db_leftovers/dsl.rb",
     "lib/db_leftovers/foreign_key.rb",
+    "lib/db_leftovers/generic_database_interface.rb",
     "lib/db_leftovers/index.rb",
+    "lib/db_leftovers/mysql_database_interface.rb",
+    "lib/db_leftovers/postgres_database_interface.rb",
     "lib/db_leftovers/table_dsl.rb",
     "lib/tasks/leftovers.rake",
+    "spec/config/database.yml.sample",
     "spec/db_leftovers_spec.rb",
-    "spec/spec_helper.rb"
+    "spec/mysql_spec.rb",
+    "spec/postgres_spec.rb",
+    "spec/spec_helper.rb",
+    "spec/support/mock_database_interface.rb",
+    "spec/support/shared_db_tests.rb",
+    "spec/support/sql_matcher.rb"
   ]
   s.homepage = "http://github.com/pjungwir/db_leftovers"
   s.licenses = ["MIT"]

data/lib/db_leftovers.rb

@@ -1,4 +1,6 @@
-require 'db_leftovers/database_interface.rb'
+require 'db_leftovers/generic_database_interface.rb'
+require 'db_leftovers/postgres_database_interface.rb'
+require 'db_leftovers/mysql_database_interface.rb'
 require 'db_leftovers/index.rb'
 require 'db_leftovers/foreign_key.rb'
 require 'db_leftovers/constraint.rb'

data/lib/db_leftovers/definition.rb

@@ -5,9 +5,12 @@ module DBLeftovers
       opts = {
         :do_indexes => true,
         :do_foreign_keys => true,
-        :do_constraints => true
+        :do_constraints => true,
+        :db_interface => nil
       }.merge(opts)
-      dsl = DSL.new(:verbose => ENV['DB_LEFTOVERS_VERBOSE'] || false)
+      dsl = DSL.new(
+        :verbose => ENV['DB_LEFTOVERS_VERBOSE'] || false,
+        :db_interface => opts[:db_interface])
       dsl.define(&block)
       dsl.record_indexes        if opts[:do_indexes]
       dsl.record_foreign_keys   if opts[:do_foreign_keys]

data/lib/db_leftovers/dsl.rb

@@ -8,7 +8,7 @@ module DBLeftovers
 
     def initialize(opts={})
       @verbose = !!opts[:verbose]
-      @db = DatabaseInterface.new
+      @db = opts[:db_interface] || get_database_interface
 
       @indexes_by_table = {}      # Set from the DSL
       @old_indexes = @db.lookup_all_indexes
@@ -226,6 +226,18 @@ module DBLeftovers
       "fk_#{from_table}_#{from_column}"
     end
 
+    def get_database_interface
+      db = ActiveRecord::Base.configurations[Rails.env]['adapter']
+      case db
+      when 'postgresql'
+        DBLeftovers::PostgresDatabaseInterface.new
+      when 'mysql2'
+        DBLeftovers::MySQLInterface.new
+      else
+        raise "Unsupported database: #{db}"
+      end
+    end
+
   end
 
 end

data/lib/db_leftovers/generic_database_interface.rb

@@ -0,0 +1,67 @@
+module DBLeftovers
+
+  class GenericDatabaseInterface
+
+    def lookup_all_indexes
+      raise "Should be overriden by a database-specific interface"
+    end
+
+    def lookup_all_foreign_keys
+      raise "Should be overriden by a database-specific interface"
+    end
+
+    def lookup_all_constraints
+      raise "Should be overriden by a database-specific interface"
+    end
+
+    def execute_add_index(idx)
+      unique = idx.unique? ? 'UNIQUE' : ''
+      where = idx.where_clause.present? ? "WHERE #{idx.where_clause}" : ''
+
+      sql = <<-EOQ
+        CREATE #{unique} INDEX #{idx.index_name}
+        ON #{idx.table_name}
+        (#{idx.column_names.join(', ')})
+        #{where}
+      EOQ
+      execute_sql(sql)
+    end
+
+    def execute_drop_index(table_name, index_name)
+      sql = <<-EOQ
+          DROP INDEX #{index_name}
+      EOQ
+      execute_sql(sql)
+    end
+
+    def execute_add_foreign_key(fk)
+      on_delete = "ON DELETE CASCADE" if fk.cascade
+      on_delete = "ON DELETE SET NULL" if fk.set_null
+      execute_sql %{ALTER TABLE #{fk.from_table}
+                ADD CONSTRAINT #{fk.constraint_name}
+                FOREIGN KEY (#{fk.from_column})
+                REFERENCES #{fk.to_table} (#{fk.to_column})
+                #{on_delete}}
+    end
+
+    def execute_drop_foreign_key(constraint_name, from_table, from_column)
+      execute_sql %{ALTER TABLE #{from_table} DROP CONSTRAINT #{constraint_name}}
+    end
+
+    def execute_add_constraint(chk)
+      sql = <<-EOQ
+          ALTER TABLE #{chk.on_table} ADD CONSTRAINT #{chk.constraint_name} CHECK (#{chk.check})
+      EOQ
+      execute_sql sql
+    end
+
+    def execute_drop_constraint(constraint_name, on_table)
+      execute_sql %{ALTER TABLE #{on_table} DROP CONSTRAINT #{constraint_name}}
+    end
+
+    def execute_sql(sql)
+      @conn.execute(sql)
+    end
+
+  end
+end

data/lib/db_leftovers/mysql_database_interface.rb

@@ -0,0 +1,83 @@
+module DBLeftovers
+
+  class MysqlDatabaseInterface < GenericDatabaseInterface
+
+    def initialize(conn=nil)
+      @conn = conn || ActiveRecord::Base.connection
+    end
+
+    def lookup_all_indexes
+      # TODO: Constrain it to the database for the current Rails project:
+      ret = {}
+      @conn.select_values("SHOW TABLES").each do |table_name|
+        indexes = {}
+        # Careful, MySQL automatically creates indexes whenever you define a foreign key.
+        # Use our foreign key naming convention to ignore these:
+        @conn.select_rows("SHOW INDEXES FROM #{table_name} WHERE key_name NOT LIKE 'fk_%'").each do |_, non_unique, key_name, seq_in_index, column_name, collation, cardinality, sub_part, packed, has_nulls, index_type, comment|
+          unless key_name == 'PRIMARY'
+            # Combine rows for multi-column indexes
+            h = (indexes[key_name] ||= { unique: non_unique == 0, name: key_name, columns: {} })
+            h[:columns][seq_in_index.to_i] = column_name
+          end
+        end
+
+        indexes.each do |index_name, h|
+          ret[index_name] = Index.new(
+            table_name,
+            h[:columns].sort.map{|k, v| v},
+            unique: h[:unique],
+            name: h[:name]
+          )
+        end
+      end
+
+      return ret
+    end
+
+    def lookup_all_foreign_keys
+      # TODO: Support multi-column foreign keys:
+      # TODO: Constrain it to the database for the current Rails project:
+      ret = {}
+      sql = <<-EOQ
+          SELECT  c.constraint_name,
+                  c.table_name,
+                  k.column_name,
+                  c.referenced_table_name,
+                  k.referenced_column_name,
+                  c.delete_rule
+          FROM    information_schema.referential_constraints c,
+                  information_schema.key_column_usage k
+          WHERE   c.constraint_schema = k.constraint_schema
+          AND     c.constraint_name = k.constraint_name
+      EOQ
+      @conn.select_rows(sql).each do |constr_name, from_table, from_column, to_table, to_column, del_type|
+        del_type = case del_type
+                   when 'RESTRICT'; nil
+                   when 'CASCADE'; :cascade
+                   when 'SET NULL'; :set_null
+                   else; raise "Unknown del type: #{del_type}"
+                   end
+        ret[constr_name] = ForeignKey.new(constr_name, from_table, from_column, to_table, to_column, :on_delete => del_type)
+      end
+      return ret
+    end
+
+    def lookup_all_constraints
+      # TODO: Constrain it to the database for the current Rails project:
+      # MySQL doesn't support CHECK constraints:
+      return []
+    end
+
+    def execute_drop_index(table_name, index_name)
+      sql = <<-EOQ
+          DROP INDEX #{index_name} ON #{table_name}
+      EOQ
+      execute_sql(sql)
+    end
+
+    def execute_drop_foreign_key(constraint_name, from_table, from_column)
+      execute_sql %{ALTER TABLE #{from_table} DROP FOREIGN KEY #{constraint_name}}
+    end
+
+  end
+end

data/lib/db_leftovers/{database_interface.rb → postgres_database_interface.rb}

@@ -1,9 +1,14 @@
 module DBLeftovers
 
-  class DatabaseInterface
+  class PostgresDatabaseInterface < GenericDatabaseInterface
+
+    def initialize(conn=nil)
+      @conn = conn || ActiveRecord::Base.connection
+    end
 
     def lookup_all_indexes
-      # TODO: Constraint it to the database for the current Rails project:
+      # TODO: Constrain it to the database for the current Rails project:
+      #       (current_database(), current_schema())
       ret = {}
       sql = <<-EOQ
           SELECT  ix.indexrelid,
@@ -34,7 +39,7 @@ module DBLeftovers
                     ix.indpred
           ORDER BY t.relname, i.relname
       EOQ
-      ActiveRecord::Base.connection.select_rows(sql).each do |indexrelid, indrelid, table_name, index_name, is_unique, column_numbers, where_clause|
+      @conn.select_rows(sql).each do |indexrelid, indrelid, table_name, index_name, is_unique, column_numbers, where_clause|
         where_clause = remove_outer_parens(where_clause) if where_clause
         ret[index_name] = Index.new(
           table_name,
@@ -53,7 +58,8 @@ module DBLeftovers
       # confdeltype: a=nil, c=cascade, n=null
       ret = {}
       # TODO: Support multi-column foreign keys:
-      # TODO: Constraint it to the database for the current Rails project:
+      # TODO: Constrain it to the database for the current Rails project:
+      #       (current_database(), current_schema())
       sql = <<-EOQ
           SELECT  c.conname,
                   t1.relname,
@@ -84,7 +90,7 @@ module DBLeftovers
           AND     pg_catalog.pg_table_is_visible(t1.oid)
           AND     pg_catalog.pg_table_is_visible(t2.oid)
       EOQ
-      ActiveRecord::Base.connection.select_rows(sql).each do |constr_name, from_table, from_column, to_table, to_column, del_type|
+      @conn.select_rows(sql).each do |constr_name, from_table, from_column, to_table, to_column, del_type|
         del_type = case del_type
                    when 'a'; nil
                    when 'c'; :cascade
@@ -97,7 +103,8 @@ module DBLeftovers
     end
 
     def lookup_all_constraints
-      # TODO: Constraint it to the database for the current Rails project:
+      # TODO: Constrain it to the database for the current Rails project:
+      #       (current_database(), current_schema())
       ret = {}
       sql = <<-EOQ
           SELECT  c.conname,
@@ -113,61 +120,12 @@ module DBLeftovers
           AND     n.nspname NOT IN ('pg_catalog', 'pg_toast')
           AND     pg_catalog.pg_table_is_visible(t.oid)
       EOQ
-      ActiveRecord::Base.connection.select_rows(sql).each do |constr_name, on_table, check_expr|
+      @conn.select_rows(sql).each do |constr_name, on_table, check_expr|
         ret[constr_name] = Constraint.new(constr_name, on_table, remove_outer_parens(check_expr))
       end
       return ret
     end
 
-    def execute_add_index(idx)
-      unique = idx.unique? ? 'UNIQUE' : ''
-      where = idx.where_clause.present? ? "WHERE #{idx.where_clause}" : ''
-
-      sql = <<-EOQ
-        CREATE #{unique} INDEX #{idx.index_name}
-        ON #{idx.table_name}
-        (#{idx.column_names.join(', ')})
-        #{where}
-      EOQ
-      execute_sql(sql)
-    end
-
-    def execute_drop_index(table_name, index_name)
-      sql = <<-EOQ
-          DROP INDEX #{index_name}
-      EOQ
-      execute_sql(sql)
-    end
-
-    def execute_add_foreign_key(fk)
-      on_delete = "ON DELETE CASCADE" if fk.cascade
-      on_delete = "ON DELETE SET NULL" if fk.set_null
-      execute_sql %{ALTER TABLE #{fk.from_table}
-                ADD CONSTRAINT #{fk.constraint_name}
-                FOREIGN KEY (#{fk.from_column})
-                REFERENCES #{fk.to_table} (#{fk.to_column})
-                #{on_delete}}
-    end
-
-    def execute_drop_foreign_key(constraint_name, from_table, from_column)
-      execute_sql %{ALTER TABLE #{from_table} DROP CONSTRAINT #{constraint_name}}
-    end
-
-    def execute_add_constraint(chk)
-      sql = <<-EOQ
-          ALTER TABLE #{chk.on_table} ADD CONSTRAINT #{chk.constraint_name} CHECK (#{chk.check})
-      EOQ
-      execute_sql sql
-    end
-
-    def execute_drop_constraint(constraint_name, on_table)
-      execute_sql %{ALTER TABLE #{on_table} DROP CONSTRAINT #{constraint_name}}
-    end
-
-    def execute_sql(sql)
-      ActiveRecord::Base.connection.execute(sql)
-    end
-
     private
 
     def column_names_for_index(table_id, column_numbers)
@@ -178,7 +136,7 @@ module DBLeftovers
             WHERE   attrelid = #{table_id}
             AND     attnum = #{c}
         EOQ
-        ActiveRecord::Base.connection.select_value(sql)
+        @conn.select_value(sql)
       end
     end
 
@@ -186,6 +144,5 @@ module DBLeftovers
       str ? str.gsub(/^\((.*)\)$/, '\1') : nil
     end
 
-
   end
 end