attribute-stats 0.1.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 02a8733b9a9fee7f8391d8855ee3d1a9106f1196
+  data.tar.gz: 2b07627c92d58fa5ac459ed1fbb8ab27493d1390
+SHA512:
+  metadata.gz: bcdfba6645eea57014decabf2e8ba5ee935cf6fa3285c545622a59b8e2c13c87512b628824d551b0be8bfc7f95e3fa113043facc720025084371f2a9676bdac0
+  data.tar.gz: f6f439d0aeac841c04846708bdd90a40550afeda98710a1d497f8f5a0ce9968f27a3f304c8ef24e8d15dd39647a9b45a2c51251322bee2459c7290a28ac280cb

data/LICENSE

@@ -0,0 +1,21 @@
+The MIT License (MIT)
+
+Copyright (c) 2015 Steve Hodges
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,122 @@
+Attribute Stats gives you insight into which persisted attributes are actually used in your Rails models. Whether you're joining an existing project or have been using it for years, get a quick look at the landscape of the database.
+
+It helps you find smells in your project:
+- Attributes which have never had data set (indicates a potentially forgotten attribute)
+- Tables which haven't been updated for X years (indicates a potentially unused or legacy model)
+- Attributes used by very few objects in your table (is this being used? Should it be an attribute?)
+- Attributes which are all set to the default value (if they're all default, is this really being used?)
+
+## Installation
+
+[![Gem Version](https://badge.fury.io/rb/attribute-stats.svg)](https://badge.fury.io/rb/attribute-stats)
+
+Add `gem 'attribute-stats'` to your Gemfile, bundle, and follow the Usage instructions below.
+
+View [attribute-stats on RubyGems](https://rubygems.org/gems/attribute-stats) for more info, or just read the [gemspec](attribute-stats.gemspec).
+
+## Usage
+
+### Programmatic
+
+You can use `AttributeStats` from within Rails (or a Rails console):
+
+    irb:1> stats = AttributeStats::StatsGenerator.new()
+    irb:2> stats.attribute_usage
+    => # a list of your attributes and what % of records have a value set
+    irb:3> stats.unused_attributes
+    => # a list of your attributes which have no value set in the database
+    irb:4> stats.dormant_tables
+    => # a list of your tables which have not been updated in awhile
+    irb:5> stats.migration
+    => # sample migration up/down to remove your unused attributes
+    irb:6> stats.set_formatter :json
+    irb:7> stats.migration # returns json instead of a hash
+
+### Rake Tasks
+
+Rake tasks are available once you've installed the gem in your Gemfile and bundled:
+
+* rake db:stats:dormant_tables
+* rake db:stats:attribute_usage
+* rake db:stats:unused_attributes
+* rake attribute-stats:generate_migration
+
+Each allows you to change the output to JSON if you'd like to pipe it into another application.
+
+---
+
+#### rake db:stats:unused_attributes
+Lists all attributes which are unused (have a nil or empty value).
+
+**Argument Options:**
+1. consider_defaults_unused: true or false (default: false). This option considers attributes set to the databse default value to be unused.
+2. format: tabular, json  (default: tabular)
+3. verbose: true, false (default: true)
+
+i.e. `rake db:stats:unused_attributes[true,json,false]`
+
+---
+
+#### rake db:stats:attribute_usage
+Lists usage statistics for all attributes (count and percent which are unused). Note: this does a full scan of your tables, so it will be slow for tables with a lot of data.
+
+**Argument Options:**
+1. consider_defaults_unused: true or false (default: false). This option considers attributes set to the databse default value to be unused.
+2. format: tabular, json  (default: tabular)
+3. verbose: true, false (default: true)
+
+i.e. `rake db:stats:attribute_usage[true,json,false]`
+
+---
+
+#### rake db:stats:dormant_tables
+Lists tables which have not been updated in the past X months (default is 3.months.ago)
+
+**Argument Options:**
+1. date_expression: A Rails date expression, i.e. '3.months.ago'. Tables with updated_at values after that date are not considered dormant.
+2. format: tabular, json  (default: tabular)
+3. verbose: true, false (default: true)
+
+i.e. `rake db:stats:dormant_tables['1.year.ago',json,false]`
+
+---
+
+#### rake attribute-stats:migration
+Generates a sample migration syntax to remove all unused attributes. (This is just output, not saved to disk. See the TODO below.)
+
+**Argument Options:**
+1. consider_defaults_unused: true or false (default: false). This option considers attributes set to the databse default value to be unused.
+
+*TODO: actually save that generated file to the db/migrate path of the host Rails app.*
+
+## Caveats
+
+The gem does not support:
+
+1. Detection of unset encrypted attributes. `attribute-stats` works by searching for empty values using SQL. If you are encrypting data before storing it in the database, a value of `nil` might have a value in the database which, decrypted, equals `nil`.
+1. Custom table names. If you are defining a model called House, and configure the model to use the table 'domiciles' instead of 'houses', `attribute-stats` skips analysis of the table. *(TODO: fix)*
+1. Tables not associated with a model. If your database has tables which do not correspond with a model (see point #2), `attribute-stats` skips analysis of that table. *(TODO: fix)*
+
+## Compatability
+
+The gem is tested and works with Rails version 4.2 through 5.2, with `mysql2`, `sqlite3`, and `postgresql` database adapters. You can view all tested dependency sets in [Appraisals](Appraisals)
+
+*Due to changes in ActiveRecord between 4.1 and 4.2, the gem breaks in versions below 4.2. In a future version of `attribute-stats`, I intend to add support to previous versions of Rails (at least back to 4.0).*
+
+## Testing the gem
+
+To test the gem against the current version of Rails (in [Gemfile.lock](Gemfile.lock)) and sqlite:
+
+1. `bundle install`
+2. `bundle exec rspec`
+
+Or, you can run tests for all supported Rails versions and supported databases:
+
+1. Add your database config at `spec/database.yml` pointing to empty local mysql and postgres test databases (see [database.yml.sample](spec/database.yml.sample))
+1. `gem install appraisal`
+1. `bundle exec appraisal install` *(this Generates gemfiles for all permutations of our dependencies, so you'll see lots of bundler output))*
+1. `bundle exec appraisal rspec`. *(This runs rspec for each dependency permutation. If one fails, appraisal exits immediately and does not test permutations it hasn't gotten to yet. Tests are not considered passing until all 12 permutations are passing)*
+
+If you only want to test a certain dependency set, such as Rails 5.2 for MySQL: `bundle exec appraisals rails-5-2-mysql`. In this case, you would *not* need to configure postgresql in your database.yml, nor have postgres running on your machine.
+
+You can view all available dependency sets in [Appraisals](Appraisals)

data/lib/attribute-stats.rb

@@ -0,0 +1 @@
+Dir[File.join(__dir__, '**', '*.rb')].each{|f| require f }

data/lib/attribute-stats/railtie.rb

@@ -0,0 +1,9 @@
+require 'rails'
+
+module AttributeStats
+  class Railtie < Rails::Railtie
+    rake_tasks do
+      Dir[File.join(__dir__, 'tasks/*.rake')].each { |f| load f }
+    end
+  end
+end

data/lib/attribute-stats/tasks/attribute_usage.rake

@@ -0,0 +1,15 @@
+require 'attribute-stats'
+namespace :db do
+	namespace :stats do
+		desc "Usage statistics for all attributes [options CONSIDER_DEFAULTS_UNUSED: false, FORMAT: json, tabular, VERBOSE: false]"
+		task :attribute_usage, [:consider_defaults_unused,:format,:verbose] => :environment do |task, args|
+			args.with_defaults(consider_defaults_unused: 'false', format: 'tabular', verbose: 'true')
+      options = {
+        consider_defaults_unused: args[:consider_defaults_unused].downcase != 'false',
+                       formatter: args[:format],
+                         verbose: args[:verbose].downcase != 'false',
+                          source: :cli }
+      AttributeStats::StatsGenerator.new(options).attribute_usage
+		end
+	end
+end

data/lib/attribute-stats/tasks/dormant.rake

@@ -0,0 +1,15 @@
+require 'attribute-stats'
+namespace :db do
+	namespace :stats do
+		desc "View tables not updated in the past months [options DATE_EXPRESSION: X.months.ago, FORMAT: json, tabular, VERBOSE: false]"
+		task :dormant_tables, [:date_expression,:format,:verbose] => :environment do |task, args|
+			args.with_defaults(date_expression: '3.months.ago', format: 'tabular', verbose: 'true')
+      options = {
+        dormant_table_age: args[:date_expression],
+	              formatter: args[:format],
+	                verbose: args[:verbose] != 'false',
+	                 source: :cli }
+      AttributeStats::StatsGenerator.new(options).dormant_tables
+		end
+	end
+end

data/lib/attribute-stats/tasks/migration.rake

@@ -0,0 +1,13 @@
+require 'attribute-stats'
+namespace :'attribute-stats' do
+	desc "Generate sample migration to remove unused attributes (and optionally those using default values) [options CONSIDER_DEFAULTS_UNUSED: false, FORMAT: json, tabular, VERBOSE: false]"
+	task :migration, [:consider_defaults_unused,:format,:verbose] => :environment do |task, args|
+		args.with_defaults(consider_defaults_unused: 'false', format: 'tabular', verbose: 'true')
+    options = {
+      consider_defaults_unused: args[:consider_defaults_unused].downcase != 'false',
+                     formatter: args[:format],
+                       verbose: args[:verbose].downcase != 'false',
+                        source: :cli }
+    AttributeStats::StatsGenerator.new(options).migration
+	end
+end

data/lib/attribute-stats/tasks/unused_attributes.rake

@@ -0,0 +1,15 @@
+require 'attribute-stats'
+namespace :db do
+  namespace :stats do
+    desc "View attributes with no data in the database (and optionally those using default values) [options CONSIDER_DEFAULTS_UNUSED: false, FORMAT: json, tabular, VERBOSE: false]"
+    task :unused_attributes, [:consider_defaults_unused,:format,:verbose] => :environment do |task, args|
+      args.with_defaults(consider_defaults_unused: 'false', format: 'tabular', verbose: 'true')
+      options = {
+        consider_defaults_unused: args[:consider_defaults_unused].downcase != 'false',
+                       formatter: args[:format],
+                         verbose: args[:verbose].downcase != 'false',
+                          source: :cli }
+      AttributeStats::StatsGenerator.new(options).unused_attributes
+    end
+  end
+end

data/lib/attribute-stats/version.rb

@@ -0,0 +1,3 @@
+module AttributeStats
+  VERSION = "0.1.0"
+end

data/lib/entities/attribute_info.rb

@@ -0,0 +1,23 @@
+module AttributeStats
+  class AttributeInfo
+    attr_reader :count, :usage_percent, :name, :empty
+
+    def initialize(attribute_name)
+      @name = attribute_name
+    end
+
+    def set_usage(record_count, table_total_record_count)
+      @count = record_count
+      @usage_percent = (record_count / table_total_record_count.to_f).round(5)
+      @empty = record_count == 0
+    end
+
+    def set_emptyness(is_empty)
+      @empty = is_empty
+    end
+
+    def empty?
+      @empty
+    end
+  end
+end

data/lib/entities/table_data.rb

@@ -0,0 +1,50 @@
+module AttributeStats
+  class TableData
+    attr_reader :count, :table_name, :attributes, :name, :last_updated, :model
+    def initialize(model)
+      @model        = model
+      @name         = model.name
+      @table_name   = model.table_name
+      @attributes   = []
+      @dormant      = false
+      @count        = 0
+      @column_names = model.columns.map(&:name)
+    end
+
+    def column_names
+      @column_names
+    end
+
+    def make_dormant(last_updated)
+      @dormant = true
+      last_updated  = last_updated.to_datetime unless last_updated.nil?
+      @last_updated = last_updated
+    end
+
+    def dormant?
+      @dormant
+    end
+
+    def set_count(total_record_count)
+      @count = total_record_count
+    end
+
+    def attribute_for(attribute_name)
+      unless attribute = attributes.detect{|a| a.name == attribute_name }
+        attribute = AttributeInfo.new(attribute_name)
+        @attributes << attribute
+      end
+      attribute
+    end
+
+    def unused_attributes
+      @unused_attributes ||= begin
+        attrs = []
+        @attributes.each do |attribute|
+          attrs << attribute.name if attribute.empty?
+        end
+        attrs
+      end
+    end
+  end
+end

data/lib/formatters/hash_formatter.rb

@@ -0,0 +1,49 @@
+module AttributeStats
+  class HashFormatter
+
+    def initialize(options: {}, table_info: {}, migration: [])
+      @options, @table_info, @migration = options, table_info, migration
+    end
+
+    def output_all_attributes
+      output = []
+      @table_info.each do |table_info|
+        table_info.attributes.each do |attribute|
+          output << {
+                    model: table_info.name,
+                attribute: attribute.name,
+                    count: attribute.count,
+            usage_percent: attribute.usage_percent
+          }
+        end
+      end
+      output
+    end
+
+    def output_dormant_tables
+      output = []
+      @table_info.each do |table_info|
+        output << table_info.table_name if table_info.dormant?
+      end
+      output
+    end
+
+    def output_unused_attributes
+      output = []
+      @table_info.each do |table_info|
+        table_info.attributes.sort_by(&:name).each do |attribute|
+          next unless attribute.empty?
+          output << {
+                model: table_info.name,
+            attribute: attribute.name
+          } 
+        end
+      end
+      output
+    end
+
+    def output_migration
+      @migration.data
+    end
+  end
+end

data/lib/formatters/json_formatter.rb

@@ -0,0 +1,20 @@
+module AttributeStats
+  class JSONFormatter < HashFormatter
+
+    def output_all_attributes
+      super.to_json
+    end
+
+    def output_dormant_tables
+      super.to_json
+    end
+
+    def output_unused_attributes
+      super.to_json
+    end
+
+    def output_migration
+      super.to_json
+    end
+  end
+end

data/lib/formatters/tabular_formatter.rb

@@ -0,0 +1,124 @@
+require 'hirb'
+
+module AttributeStats
+  class TabularFormatter
+
+    def initialize(options: {}, table_info: {}, migration: nil)
+      @options, @table_info, @migration = options, table_info, migration
+      @buffer = ''
+    end
+
+    def output_all_attributes
+      attribute_results = []
+      @table_info.each do |table_info|
+        table_info.attributes.sort_by(&:usage_percent).each do |attribute|
+          attribute_results << {
+                    model: table_info.name,
+                set_count: attribute.count,
+              set_percent: (attribute.usage_percent * 100).round(1).to_s.rjust(5),
+           attribute_name: attribute.name
+          }
+        end
+      end
+      if attribute_results.empty?
+        puts "No attributes found"
+      else
+        print_table attribute_results, title: 'Attributes Utilization'
+      end
+      @buffer
+    end
+
+    def output_dormant_tables
+      output = []
+      @table_info.each do |table_info|
+        date = table_info.last_updated
+        output << {
+                 model: table_info.name,
+          last_updated: date.nil? ? 'Never Updated' : table_info.last_updated.to_date.to_s(:long)
+        } if table_info.dormant?
+      end
+      if output.empty?
+        puts "No dormant tables"
+      else
+        print_table output, title: ['Dormant Tables', "No updated_ats after #{@options[:dormant_table_age].to_date.to_s(:long)}"]
+      end
+      @buffer
+    end
+
+    def output_unused_attributes
+      output = []
+      @table_info.each do |table_info|
+        table_info.attributes.sort_by(&:name).each do |attribute|
+          output << {
+                     model: table_info.name,
+            attribute_name: attribute.name
+          } if attribute.empty?
+        end
+      end
+      unused_values = ['Nil', 'Empty']
+      unused_values << 'Default Values' if @options[:consider_defaults_unused]
+      if output.empty?
+        puts "No unused attributes (good for you!)"
+      else
+        print_table output, title: ['Unused Attributes', unused_values.join(', ')]
+      end
+      @buffer
+    end
+
+    def output_migration
+      print_section_header "Migration Syntax to delete unused attributes"
+      @migration.to_migration_file
+    end
+
+    private
+
+    def puts(*values)
+      @buffer << values.join("\n")
+      @buffer << "\n"
+    end
+
+    def print_table_header(table_info)
+      line = '-'*(table_info.name.length+4)
+      puts '', line, "  #{table_info.name}  ", line
+    end
+
+    def print_section_header(text)
+      line = '#'*(text.length+4)
+      puts '', line, "  #{text}  ", line
+    end
+
+    def print_table(data, title: nil)
+      column_names = data.first.keys
+      output = Hirb::Helpers::AutoTable.render(data,
+         fields: header_order(column_names),
+        headers: formatted_headers(column_names))
+      output_table_title(title, output) unless title.blank?
+      puts output
+    end
+
+    def output_table_title(title_rows, table_output)
+      header_line = table_output.split(/\n/).first
+      puts '', header_line
+
+      Array(title_rows).each do |title|
+        if header_line.length > title.length+4
+          puts "| #{title.center(header_line.length - 4)} |"
+        else
+          puts "| #{title}"
+        end
+      end
+    end
+
+    def header_order(column_names)
+      [:model, :table_name, :attribute_name,
+        :last_updated, :set_count, :set_percent] & column_names
+    end
+
+    def formatted_headers(column_names)
+      column_names.inject({}) do |hash, name|
+        hash[name] = name.to_s.titleize
+        hash
+      end
+    end
+  end
+end

data/lib/stats_generation/generate_migration.rb

@@ -0,0 +1,151 @@
+module AttributeStats
+  class GenerateMigration
+
+    def initialize(table_info: [], options: {})
+      @table_info, @options = table_info, options
+      @up_buffer   = []
+      @down_buffer = []
+      @table_info.each do |table_info|
+        add_migrations_for_table_to_buffer(table_info)
+      end
+    end
+
+    def data
+      {
+          up_commands: @up_buffer,
+        down_commands: @down_buffer,
+              warning: warning
+      }
+    end
+
+    def to_migration_file
+      output = "class RemoveUnusedAttributes < ActiveRecord::Migration"
+      output << "[#{Rails::VERSION::STRING}]" if Rails::VERSION::MAJOR >= 5
+      output << "\n#{warning_to_width}"
+      output << <<-OUTPUT
+  def up
+    #  #{@up_buffer.join("\n    #  ")}
+  end
+
+  def down
+    #  #{@down_buffer.join("\n    #  ")}
+  end
+end
+      OUTPUT
+      output
+    end
+
+    def to_json
+      {
+          up_commands: @up_buffer,
+        down_commands: @down_buffer,
+              warning: warning
+      }
+    end
+
+    private
+
+    # This is a modified version of the ActiveRecord::SchemaDumper#table method.
+    # Unfortunately, Rails generators cannot accept multiple tables in a single RemoveXXXFromXXX
+    # migration (maybe I should submit a PR to Rails?)
+    # ActiveRecord::SchemaDumper cannot be reused for this case, so I had to extract it.
+    def add_migrations_for_table_to_buffer(table_info)
+      @connection ||= ActiveRecord::Base.connection
+      @types ||= @connection.native_database_types
+
+      column_specs = []
+      @connection.columns(table_info.table_name).each do |column|
+        next unless column.name.to_s.in? table_info.unused_attributes
+        column_specs << column_spec_for(column)
+      end
+
+      quoted_table_name  = @connection.quote_table_name(table_info.table_name)
+      column_specs.each do |colspec|
+        down_syntax    = [quoted_table_name, colspec[:name], ":#{colspec[:type]}", colspec[:options].presence].compact.join(', ')
+        @down_buffer  << "add_column #{down_syntax}"
+        @up_buffer    << "remove_column #{quoted_table_name}, #{colspec[:name]}"
+      end
+    end
+
+    def warning_to_width(target_width=78)
+      line_start = "  # "
+      output = ""
+      warning.split("\n").each do |line|
+        line_buffer = "\n#{line_start}"
+        line.split(/ +/).each do |word|
+          if word.length + line_buffer.length > target_width
+            output << line_buffer
+            line_buffer = "\n#{line_start}"
+          end
+          line_buffer << "#{word} "
+        end
+        output << line_buffer+"\n#{line_start}"
+      end
+      output + "\n"
+    end
+
+    def column_spec_for(column)
+      # Rails 5.2
+      # SchemaDumper           returns an array like: [:text, {:limit=>"255"}]
+      # Rails 5.1
+      # connection.column_spec returns an array like: [:text, {:limit=>"255"}]
+      # prior to Rails 5.1
+      # connection.column_spec returns a  hash  like: {:name=>"\"line_2\"", :type=>"string", :limit=>"limit: 255"}
+      if    Rails::VERSION::MAJOR >= 5 && Rails::VERSION::MINOR == 2
+        column_spec_for_5_2(column)
+      elsif Rails::VERSION::MAJOR >= 5 && Rails::VERSION::MINOR >= 1
+        column_spec_for_5_1(column)
+      elsif Rails::VERSION::MAJOR >= 5 && Rails::VERSION::MINOR == 0
+        column_spec_for_5_0(column)
+      else
+        column_spec_for_4(column)
+      end
+    end
+
+    def column_spec_for_5_2(column)
+      # Unfortunately Rails has moved this logic from Connection to a private method on SchemaDumper...
+      # Rails really doesn't want people using this stuff!
+      colspec = ActiveRecord::ConnectionAdapters::SchemaDumper
+                  .send(:new, @connection, {})
+                  .send(:column_spec, column)
+      spec = {
+        name: @connection.quote_column_name(column.name),
+        type: colspec[0]
+      }
+      spec[:options] = colspec[1].map{|k,v| "#{k}: #{v}"}.join(', ') if colspec[1].is_a?(Hash)
+      spec
+    end
+
+
+    def column_spec_for_5_1(column)
+      colspec = @connection.column_spec(column)
+      spec = {
+        name: @connection.quote_column_name(column.name),
+        type: colspec[0],
+      }
+      spec[:options] = colspec[1].map{|k,v| "#{k}: #{v}"}.join(', ') if colspec[1].is_a?(Hash)
+      spec
+    end
+
+    def column_spec_for_5_0(column)
+      colspec = @connection.column_spec(column)
+      colspec[:options] = colspec.except(:type, :name).values.join(', ')
+      colspec
+    end
+
+    def column_spec_for_4(column)
+      colspec = @connection.column_spec(column, @types)
+      colspec[:options] = colspec.except(:type, :name).values.join(', ')
+      colspec
+    end
+
+    def warning
+      <<-WARNING
+BUYER BEWARE/CAVEAT EMPTOR. Review this migration carefully!
+This migration code was autogenerated by analyzing database columns which are empty at the time the script was executed.
+IT IS YOUR RESPONSIBILITY to verify that these attributes are indeed unused in your application before running this migration. For your protection, the generated commands are COMMENTED OUT.
+If you were to uncomment the below commands and migrate your database, it is extremely likely that your application will break due to references to the removed attributes in your application code.
+WARNING
+    end
+  end
+end

data/lib/stats_generation/set_attribute_stats.rb

@@ -0,0 +1,93 @@
+require_relative 'terminal'
+module AttributeStats
+  class SetAttributeStats
+    include Terminal
+
+    def initialize(table_info: [], options: {})
+      @table_info, @options = table_info, options
+    end
+
+    def set_counts
+      @query_method = :count
+      set_stats
+    end
+
+    def set_empty
+      @query_method = :exists?
+      set_stats
+    end
+
+    def set_stats
+      @table_info.each_with_index do |t,index|
+        @table_count = index
+        fetch_table_stats(t)
+      end
+      erase_line if @options[:verbose]
+      true
+    end
+
+    private
+
+    def fetch_table_stats(table)
+      @current_table = table
+      @current_model = table.model
+      set_table_count
+      print("Scanning #{in_color(table.name,@table_count)} ") if @options[:verbose]
+      return if @current_table.count == 0
+      set_attribute_counts
+      erase_line if @options[:verbose]
+    end
+
+    def set_table_count
+      @current_table.set_count(@current_model.all.count)
+    end
+
+    def set_attribute_counts
+      attribs = @current_model.columns.dup
+      attribs.reject!{|c| ["id","created_at","updated_at"].include? c.name } unless @options[:defaults]
+
+      attribs.each do |column_specification|
+        attribute_info = @current_table.attribute_for(column_specification.name)
+        if @query_method == :count
+          next if attribute_info.count
+          record_count = attribute_query(column_specification).count
+          empty = record_count == 0
+          attribute_info.set_usage record_count, @current_table.count
+        else
+          next unless attribute_info.empty.nil?
+          empty = !attribute_query(column_specification).exists?
+          attribute_info.set_emptyness empty
+        end
+        next unless @options[:verbose]
+        print empty ? red('E') : green('.')
+      end
+    end
+
+    def attribute_query(column_specification)
+      query = @current_model.where.not(column_specification.name => nil)
+      query = safe_add_not_empty_query(query, column_specification)
+      query = add_condition_for_default_value(query, column_specification) if @options[:consider_defaults_unused]
+      query
+    end
+
+    def add_condition_for_default_value(query, attrib)
+      default_value = @current_model.columns_hash[attrib.name].default
+      return query if default_value.blank?
+      query.where.not(attrib.name => default_value)
+    end
+
+    # In addition to normal string attributes, this allows us to query serialized columns.
+    # If you serialize an attribute as an array or hash or whatever, Rails still stores empty
+    # array/hashes as an empty string in the db. Using the more rails-y
+    #   where.not(serialized_attribute_name => '')
+    # results in a SerializationTypeMismatch error. So we construct the subquery by hand.
+    def safe_add_not_empty_query(query, column_specification)
+      return query unless ([:text, :string].include?(column_specification.type))
+      query.where.not("#{quoted_column_name(column_specification.name)} = ''") 
+    end
+
+    def quoted_column_name(column_name)
+      ActiveRecord::Base.connection.quote_column_name column_name
+    end
+  end
+end

data/lib/stats_generation/set_dormant_tables.rb

@@ -0,0 +1,39 @@
+require_relative 'terminal'
+
+module AttributeStats
+  class SetDormantTables
+    include Terminal
+
+    def initialize(table_info: [], options: {})
+      @table_info, @options = table_info, options
+    end
+
+    def call
+      @table_info.each_with_index do |table_data,index|
+        @table_count = index
+        set_dormant_table(table_data)
+      end
+      erase_line if @options[:verbose]
+      true
+    end
+
+    private
+
+    def set_dormant_table(table_data)
+      query_column = (['updated_at', 'created_at'] & table_data.column_names)[0]
+      return if query_column.nil?
+      print("Scanning #{in_color(table_data.name,@table_count)} ") if @options[:verbose]
+      updated_at = table_data.model.maximum(query_column)
+      table_data.make_dormant(updated_at) if updated_at.nil? || updated_at <= dormant_table_age
+      erase_line if @options[:verbose]
+    end
+
+    def dormant_table_age
+      return @options[:dormant_table_age] if @options[:dormant_table_age].respond_to?(:strftime)
+      # Safely generate Rails duration (instead of risker eval)
+      parts = @options[:dormant_table_age].split('.')
+      duration_expression = parts[0].to_i.send(parts[1]).send(parts[2])
+      @options[:dormant_table_age] = duration_expression
+    end
+  end
+end

data/lib/stats_generation/stats_generator.rb

@@ -0,0 +1,127 @@
+module AttributeStats
+  class StatsGenerator
+    attr_reader :options, :table_info, :migration
+
+    DEFAULT_OPTIONS = {
+      consider_defaults_unused: true,
+                      defaults: false,
+                     formatter: :hash,
+                        source: :code,
+             dormant_table_age: '3.months.ago',
+                       verbose: false }
+
+    def initialize(opts={})
+      @options = DEFAULT_OPTIONS.merge(opts)
+    end
+
+    def attribute_usage
+      fetch_attribute_usage
+      output formatter.output_all_attributes
+    end
+
+    def dormant_tables
+      fetch_dormant_tables
+      output formatter.output_dormant_tables
+    end
+
+    def unused_attributes
+      fetch_empty_attributes
+      output formatter.output_unused_attributes
+    end
+
+    def migration
+      generate_migration
+      output formatter.output_migration
+    end
+
+    def set_formatter(formatter_type)
+      formatter_was = options[:formatter]
+      case formatter_type.to_s.downcase
+      when 'json'
+        options[:formatter] = :json
+      when 'tabular'
+        options[:formatter] = :tabular
+      when 'hash'
+        options[:formatter] = :hash
+      end
+      @formatter = nil unless options[:formatter] == formatter_was
+      options[:formatter]
+    end
+
+    def inspect
+      "StatsGenerator(results: #{table_info.to_s[0..200]}, options: #{options})"
+    end
+
+    private
+
+    def formatter
+      @formatter ||= begin
+        output = case options[:formatter].to_s.downcase
+        when 'json'
+          JSONFormatter
+        when 'hash'
+          HashFormatter
+        else
+          TabularFormatter
+        end.new(options: options, table_info: table_info, migration: @migration)
+      end
+    end
+
+    def output(value)
+      return if value.nil?
+      if options[:source] == :cli || value.is_a?(String)
+        puts value
+      else
+        value
+      end
+    end
+
+    def fetch_attribute_usage
+      @fetch_attribute_usage ||= begin
+        initialize_tables
+        attribute_stats_setter.set_counts
+        true
+      end
+    end
+
+    def fetch_empty_attributes
+      @fetch_empty_attributes ||= begin
+        initialize_tables
+        attribute_stats_setter.set_empty
+        true
+      end
+    end
+
+    def attribute_stats_setter
+      @attribute_stats_setter ||= SetAttributeStats.new(table_info: table_info, options: options)
+    end
+
+    def fetch_dormant_tables
+      @fetched_dormant_tables ||= begin
+        initialize_tables
+        SetDormantTables.new(table_info: table_info, options: options).call
+        true
+      end
+    end
+
+    def generate_migration
+      fetch_empty_attributes
+      @migration ||= GenerateMigration.new(table_info: table_info, options: options)
+    end
+
+    def initialize_tables
+      return unless table_info.nil?
+      @table_info = []
+      tables.sort.each{|table| setup_table_and_model(table) }
+    end
+
+    def setup_table_and_model(table)
+      @table_info << TableData.new(table.classify.constantize)
+    rescue NameError => ex
+    end
+
+    def tables
+      ActiveRecord::Base.connection.data_sources
+    end
+  end
+end

data/lib/stats_generation/terminal.rb

@@ -0,0 +1,29 @@
+module AttributeStats
+  module Terminal
+    # https://en.wikipedia.org/wiki/ANSI_escape_code#Escape_sequences
+    START_OF_NEXT_LINE = "\e[1E"
+    LINE_ABOVE = "\e[1A"
+    CLEAR_LINE_TO_RIGHT = "\e[K"
+    RED = "\e[31m"
+    GREEN = "\e[32m"
+
+    def erase_line
+      print START_OF_NEXT_LINE, LINE_ABOVE, CLEAR_LINE_TO_RIGHT
+    end
+
+    RESET_COLOR = "\e[0m"
+
+    def in_color(text, index=0)
+      code = (31..37).to_a[index % 7]
+      "\e[#{code}m#{text}#{RESET_COLOR}"
+    end
+
+    def red(text)
+      print RED, text, RESET_COLOR
+    end
+
+    def green(text)
+      print GREEN, text, RESET_COLOR
+    end
+  end
+end

data/rakefile.rb

@@ -0,0 +1,4 @@
+require "rubygems"
+require "bundler/setup"
+require "bundler/gem_tasks"
+require 'rake/testtask'

metadata

@@ -0,0 +1,176 @@
+--- !ruby/object:Gem::Specification
+name: attribute-stats
+version: !ruby/object:Gem::Version
+  version: 0.1.0
+platform: ruby
+authors:
+- Steve Hodges
+autorequire: 
+bindir: exe
+cert_chain: []
+date: 2018-09-08 00:00:00.000000000 Z
+dependencies:
+- !ruby/object:Gem::Dependency
+  name: rails
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '4.2'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '6'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - ">="
+      - !ruby/object:Gem::Version
+        version: '4.2'
+    - - "<"
+      - !ruby/object:Gem::Version
+        version: '6'
+- !ruby/object:Gem::Dependency
+  name: hirb
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.7'
+  type: :runtime
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '0.7'
+- !ruby/object:Gem::Dependency
+  name: appraisal
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.2'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '2.2'
+- !ruby/object:Gem::Dependency
+  name: bundler
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.9'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.9'
+- !ruby/object:Gem::Dependency
+  name: rake
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '10.0'
+- !ruby/object:Gem::Dependency
+  name: rspec
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '3'
+- !ruby/object:Gem::Dependency
+  name: sqlite3
+  requirement: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.0'
+  type: :development
+  prerelease: false
+  version_requirements: !ruby/object:Gem::Requirement
+    requirements:
+    - - "~>"
+      - !ruby/object:Gem::Version
+        version: '1.0'
+description: |
+  Attribute Stats gives you insight into which attributes are actually used in your Rails models. Whether you're joining an existing project or have been using it for years, get quick info.
+
+  The stats help you find smells in your project:
+
+  Attributes which have never had data set (indicates a potentially forgotten attribute); Tables which haven't been updated for X years (indicates a potentially unused or legacy model); Attributes used by very few objects in your table (is this being used? Should it be an attribute?)
+
+  It also generates sample Rails database migration code to allow you to drop the database columns.
+email:
+- sjhodges@gmail.com
+executables: []
+extensions: []
+extra_rdoc_files: []
+files:
+- LICENSE
+- README.md
+- lib/attribute-stats.rb
+- lib/attribute-stats/railtie.rb
+- lib/attribute-stats/tasks/attribute_usage.rake
+- lib/attribute-stats/tasks/dormant.rake
+- lib/attribute-stats/tasks/migration.rake
+- lib/attribute-stats/tasks/unused_attributes.rake
+- lib/attribute-stats/version.rb
+- lib/entities/attribute_info.rb
+- lib/entities/table_data.rb
+- lib/formatters/hash_formatter.rb
+- lib/formatters/json_formatter.rb
+- lib/formatters/tabular_formatter.rb
+- lib/stats_generation/generate_migration.rb
+- lib/stats_generation/set_attribute_stats.rb
+- lib/stats_generation/set_dormant_tables.rb
+- lib/stats_generation/stats_generator.rb
+- lib/stats_generation/terminal.rb
+- rakefile.rb
+homepage: https://github.com/stevehodges/attribute_stats
+licenses:
+- MIT
+metadata: {}
+post_install_message: 
+rdoc_options: []
+require_paths:
+- lib
+required_ruby_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: 2.2.0
+required_rubygems_version: !ruby/object:Gem::Requirement
+  requirements:
+  - - ">="
+    - !ruby/object:Gem::Version
+      version: '0'
+requirements: []
+rubyforge_project: 
+rubygems_version: 2.6.14.1
+signing_key: 
+specification_version: 4
+summary: Statistics about attribute usage, unused attributes, and dormant tables in
+  Rails projects
+test_files: []