sql_partitioner 0.6.0

data/.gitignore

@@ -0,0 +1,61 @@
+# standard ignores
+log/*
+tmp/*
+TAGS
+.idea/*
+
+# Apparently this file should not be checked in, for GEM projects: http://yehudakatz.com/2010/12/16/clarifying-the-roles-of-the-gemspec-and-gemfile/
+Gemfile.lock
+
+# Rbenv
+.ruby-version
+
+# rcov generated
+coverage
+coverage.data
+
+# rdoc generated
+rdoc
+
+# yard generated
+doc
+.yardoc
+
+# bundler
+.bundle
+
+# jeweler generated
+pkg
+
+# Have editor/IDE/OS specific files you need to ignore? Consider using a global gitignore: 
+#
+# * Create a file at ~/.gitignore
+# * Include files you want ignored
+# * Run: git config --global core.excludesfile ~/.gitignore
+#
+# After doing this, these files will be ignored in all your git projects,
+# saving you from having to 'pollute' every project you touch with them
+#
+# Not sure what to needs to be ignored for particular editors/OSes? Here's some ideas to get you started. (Remember, remove the leading # of the line)
+#
+# For MacOS:
+#
+.DS_Store
+
+# For TextMate
+#*.tmproj
+#tmtags
+
+# For emacs:
+*~
+\#*
+.\#*
+
+# For vim:
+*.swp
+
+# For redcar:
+.redcar
+
+# For rubinius:
+*.rbc

data/.rspec

@@ -0,0 +1,3 @@
+--colour
+--format=documentation
+--backtrace

data/.travis.yml

@@ -0,0 +1,10 @@
+language: ruby
+rvm:
+  - ree-1.8.7-2012.02
+  - 2.1.2
+sudo: false     # makes build run in docker container, which starts faster than a VM
+cache: bundler  # speed up
+before_script:
+  - mysql -e 'create database sql_partitioner_test;'
+script:
+  - bundle exec rspec

data/CHANGELOG.md

@@ -0,0 +1,30 @@
+## 0.6.0 (2014-11-03)
+
+Features:
+
+  - Filled out README
+  - Added and improved YARD docs
+  - Improved specs
+
+## 0.5.0 (2014-10-14)
+
+Features:
+
+  - Added support for running specs in Travis CI
+  - Improved specs to exercise both database adapters (ActiveRecord, DataMapper)
+  
+Bugfixes:
+
+  - in Ruby 1.8.7 fixed `SqlPartitioner::Partition#to_log` output 
+
+## 0.4.0 (2014-10-02)
+
+Features:
+
+  - Added development dependency: SimpleCov
+  - Improved test coverage
+  
+Bugfixes:
+  
+  - Fixed return value for `_execute_and_display_partition_info` when SQL is executed.
+    Before, it returned whatever `logger.info` happened to return. Now it returns true.

data/Gemfile

@@ -0,0 +1,4 @@
+source 'http://rubygems.org'
+
+# Specify your gem's dependencies in sql_partitioner.gemspec
+gemspec

data/LICENSE

@@ -0,0 +1,19 @@
+The MIT License (MIT)
+ 
+Copyright (c) 2014 RightScale, Inc, All Rights Reserved Worldwide.
+ 
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.md

@@ -0,0 +1,119 @@
+# SqlPartitioner
+[![Build Status](https://travis-ci.org/rightscale/sql_partitioner.png)](https://travis-ci.org/rightscale/sql_partitioner)
+
+SqlPartitioner provides a `PartitionManager` class to help maintain partitioned tables in MySQL.
+If you have a table that is partitioned based on a timestamp, you will likely need to regularly add new partitions 
+into the future as well as remove older partitions to free up space. This gem will help.
+
+## Supported Features
+SqlPartitioner works with MySQL partitioned tables that are partitioned by a `timestamp` column, expressed as an integer 
+representing a Unix epoch timestamp in either seconds or micro-seconds.
+
+You can use ActiveRecord or DataMapper.
+
+Supported functionality:
+
+- initializing partitioning on a table
+- adding new partitions of a given size (expressed in months or days)
+- removing partitions older than a given timestamp or number of days
+
+You can run the above operations directly or pass a flag to only do a dry-run.
+
+## Unsupported Features
+
+Does not yet support databases other than MySQL. Target table can only be partitioned by its `timestamp` column representing seconds or micro-seconds.
+
+## Getting Started
+You'll need to `require 'sql_partitioner'`.
+
+Here's an example for initializing a `PartitionManager` instance, using `DataMapper`:
+
+```ruby
+partition_manager = SqlPartitioner::PartitionsManager.new(
+    :table_name        => 'my_partitioned_table', # target table for partitioning operations
+    :time_unit         => :micro_seconds, # or :seconds, as appropriate for the table's `timestamp` column
+    :lock_wait_timeout => 1, #(seconds)
+    :adapter           => SqlPartitioner::DMAdapter.new(DataMapper.repository.adapter),
+    :logger            => Logger.new(STDOUT)
+)
+```
+
+If you are using `ActiveRecord`, you can instead supply the following for `:adapter`:
+```ruby
+SqlPartitioner::ARAdapter.new(ActiveRecord::Base.connection)
+```
+
+Regarding the `:lock_wait_timeout` option: any partitioning statement must acquire a table lock on the partitioned table, 
+and while it is waiting to acquire this lock, any subsequent queries on that table will be blocked and have to wait. 
+It may take a long time to acquire a table lock if there were already long-running queries in progress. 
+Therefore, setting a short timeout (e.g. 1 second) ensures the partitioning statement will timeout quickly,
+ so any other SQL operations on that table will not be delayed. 
+If the partitioning command times-out, it will have to be retried later. 
+MySQL's default value for [lock_wait_timeout](http://dev.mysql.com/doc/refman/5.5/en/server-system-variables.html#sysvar_lock_wait_timeout) is 1 year.
+
+### Initialize partitioning
+Here's an example for initializing partitioning on the table. It will create partitions of size 30 days, as needed, to cover 90 days into the future:
+
+```ruby
+days_into_future = 90
+partition_size = 30
+partition_size_unit = :days
+dry_run = false
+partition_manager.initialize_partitioning_in_intervals(days_into_future, partition_size_unit, partition_size, dry_run)
+```
+
+### Adding partitions
+Here's an example for appending partitions to cover time periods into the future. It will create partitions of size 30 days, as needed, to cover 180 days into the future:
+
+```ruby
+days_into_future = 180
+partition_size = 30
+partition_size_unit = :days
+dry_run = false
+partition_manager.append_partition_intervals(partition_size_unit, partition_size, days_into_future, dry_run)
+```
+
+Here's an example for appending a single partition with the given name and "until" timestamp (using microseconds in this case):
+
+```ruby
+partition_data = {'until_2014_11_01' => 1414870869000000}
+dry_run = false
+partition_manager.reorg_future_partition(partition_data, dry_run)
+```
+
+### Dropping partitions
+Here's an example for dropping partitions as needed to only cover 360 days of the past:
+
+```ruby
+days_into_past = 360
+dry_run = false
+partition_manager.drop_partitions_older_than_in_days(days_into_past, dry_run)
+```
+
+Here's an example for dropping a single partition, `until_2014_11_01`, by name: 
+
+```ruby
+partition_names = ['until_2014_11_01']
+dry_run = false
+partition_manager.drop_partitions(partition_names, dry_run)
+```
+
+### Suggested use:
+The above operations can be helpful when creating a rake task that can initialize partitioning for a given table, 
+and gets called periodically to add and remove partitions as needed.
+
+## Compatibility
+Tested with Ruby 1.8.7 and 2.1.2, and MySQL 5.5.
+
+## Contributing
+Pull requests welcome.
+
+## Maintained by
+
+- [Dominic Metzger](https://github.com/dominicm)
+- [Sumner McCarty](https://github.com/sumner-mccarty)
+- [Prakash Selvaraj](https://github.com/PrakashSelvaraj)
+- [Jim Slattery](https://github.com/jim-slattery-rs)
+
+## License
+MIT License, see [LICENSE](LICENSE)

data/Rakefile

@@ -0,0 +1,6 @@
+require "bundler/gem_tasks"
+require 'rspec/core/rake_task'
+
+RSpec::Core::RakeTask.new(:spec)
+
+task :default => :spec

data/lib/sql_partitioner/adapters/ar_adapter.rb

@@ -0,0 +1,39 @@
+require File.expand_path("./base_adapter", File.dirname(__FILE__))
+require "active_record"
+
+module SqlPartitioner
+
+  # Adapter wrapping an Active Record Connection
+  class ARAdapter < BaseAdapter
+    def initialize(connection)
+      @connection = connection
+    end
+
+    def select(*args)
+      result = []
+      strukt = nil
+      
+      sanitized_sql = ActiveRecord::Base.send(:sanitize_sql_array, args)
+      conn_result = @connection.send(:select, sanitized_sql)
+      conn_result.each do |h|
+        if h.keys.size == 1
+          result << h.values.first
+        else
+          strukt ||= Struct.new(*h.keys.map{ |k| k.downcase.to_sym})
+          result << strukt.new(*h.values)
+        end
+      end
+      result
+    end
+
+    def execute(*args)
+      sanitized_sql = ActiveRecord::Base.send(:sanitize_sql_array, args)
+      @connection.execute(sanitized_sql)
+    end
+
+    def schema_name
+      @connection.current_database 
+    end
+  end
+
+end

data/lib/sql_partitioner/adapters/base_adapter.rb

@@ -0,0 +1,18 @@
+module SqlPartitioner
+
+  class BaseAdapter
+    # -- needs to return an array of structs or an array of values if columns selected == 1
+    def select(*args)
+      raise "select(*args) MUST BE IMPLEMENTED!"
+    end
+
+    def execute(*args)
+      raise "execute(*args) MUST BE IMPLEMENTED!"
+    end
+
+    def schema_name
+      raise "schema_name MUST BE IMPLEMENTED!"
+    end
+  end
+
+end

data/lib/sql_partitioner/adapters/dm_adapter.rb

@@ -0,0 +1,24 @@
+require File.expand_path("./base_adapter", File.dirname(__FILE__))
+
+module SqlPartitioner
+
+  # Adapter wrapping an Active Record Connection
+  class DMAdapter < BaseAdapter
+    def initialize(dm_adapter)
+      @dm_adapter = dm_adapter
+    end
+
+    def select(*args)
+      @dm_adapter.select(*args)
+    end
+
+    def execute(*args)
+      @dm_adapter.execute(*args)
+    end
+
+    def schema_name
+      @dm_adapter.schema_name
+    end
+  end
+
+end

data/lib/sql_partitioner/base_partitions_manager.rb

@@ -0,0 +1,231 @@
+module SqlPartitioner
+  class BasePartitionsManager
+
+    attr_accessor :table_name, :adapter, :logger, :current_timestamp
+
+    FUTURE_PARTITION_NAME = 'future'
+
+    FUTURE_PARTITION_VALUE = 'MAXVALUE'
+
+    # @param [{Symbol=>Object}] options
+    # @options options [SqlPartitioner::BaseAdapter] :adapter for DB communication
+    # @options options [String] :table_name target table for the partition management operations
+    # @options options [Symbol] :time_unit to use for the table's `timestamp` column, defaults to :seconds
+    # @options options [Fixnum] :current_time unix epoch in seconds
+    # @options options [Logger] :logger
+    # @options options [Fixnum] :lock_wait_timeout (in seconds) Each SQL statement will be executed with `@@local.lock_wait_timeout`
+    #  having been temporarily set to this value.
+    #  Background: Any partitioning statement must acquire a table lock on the partitioned table,
+    #  and while it is waiting to acquire this lock, any subsequent queries on that table will be blocked and have to wait.
+    #  It may take a long time to acquire a table lock if there were already long-running queries in progress.
+    #  Therefore, setting a short timeout (e.g. 1 second) ensures the partitioning statement will timeout quickly,
+    #  so any other SQL operations on that table will not be delayed.
+    #  If the partitioning command times-out, it will have to be retried later.
+    def initialize(options = {})
+      @adapter            = options[:adapter]
+      @tuc                = TimeUnitConverter.new(options[:time_unit] || :seconds)
+
+      @current_timestamp  = @tuc.from_seconds((options[:current_time] || Time.now).to_i)
+      @table_name         = options[:table_name]
+      @logger             = options[:logger]
+      @lock_wait_timeout  = options[:lock_wait_timeout]
+    end
+
+    # initialize partitioning on the given table based on partition_data
+    # provided.
+    # partition data should be of form
+    #   {partition_name1 => partition_timestamp_1 ,
+    #    partition_name2 => partition_timestamp_2...}
+    # For example:
+    #   {'until_2014_03_17' => 1395077901193149,
+    #    'until_2014_04_01' => 1396373901193398}
+    #
+    # @param [Hash<String,Fixnum>] partition_data of form { partition_name1 => timestamp1..}
+    # @param [Boolean] dry_run Defaults to false. If true, query wont be executed.
+    # @raise [ArgumentError] if partition data is not hash or if one of name id
+    #                    is not a String or if one of the value is not
+    #                    Integer
+    def initialize_partitioning(partition_data, dry_run = false)
+      partition_data = partition_data.merge(FUTURE_PARTITION_NAME => FUTURE_PARTITION_VALUE)
+
+      _validate_partition_data(partition_data)
+
+      init_sql = SqlPartitioner::SQL.initialize_partitioning(table_name, partition_data)
+      _execute_and_display_partition_info(init_sql, dry_run)
+    end
+
+    # Drop partitions by name
+    # @param [Array<String>] partition_names array of String partition_names
+    # @param [Boolean] dry_run Defaults to false. If true, query wont be executed.
+    # @return [String] drop sql if dry run is true
+    # @raise [ArgumentError] if input is not an Array or if partition name is
+    #                        not a string
+    def drop_partitions(partition_names, dry_run = false)
+      _validate_drop_partitions_names(partition_names)
+
+      drop_sql = SqlPartitioner::SQL.drop_partitions(table_name, partition_names)
+      _execute_and_display_partition_info(drop_sql, dry_run)
+    end
+
+    # Reorgs future partition into partitions provided as input.
+    #
+    # @param [Hash<String,Fixnum>] partition_data of form { partition_name1 => timestamp1..}
+    # @param [Boolean] dry_run Defaults to false. If true, query wont be executed.
+    # @return [Boolean] true if not dry run and query is executed else false
+    # @return [String] sql if dry_run is true
+    def reorg_future_partition(partition_data, dry_run = false)
+      partition_data = partition_data.dup
+
+      if partition_data.any?
+        partition_data[FUTURE_PARTITION_NAME] = FUTURE_PARTITION_VALUE
+      end
+
+      _validate_partition_data(partition_data)
+
+      reorg_sql = SqlPartitioner::SQL.reorg_partitions(table_name, partition_data, FUTURE_PARTITION_NAME)
+      _execute_and_display_partition_info(reorg_sql, dry_run)
+    end
+
+    def log(message, prefix = true)
+      message = "[#{self.class.name}]#{message}" if prefix
+      @logger.info "#{message}"
+    end
+
+    # generates name of for "until_yyyy_mm_dd" from the given timestamp.
+    # returns future partition name if value is FUTURE_PARTITION_VALUE
+    #
+    # @param [Fixnum] timestamp  timestamp for which the name has to be
+    #                            generated.
+    #
+    # @return [String] partition_name
+    def name_from_timestamp(timestamp)
+      if timestamp == FUTURE_PARTITION_VALUE
+         FUTURE_PARTITION_NAME
+      else
+        seconds = @tuc.to_seconds(timestamp)
+        "until_#{Time.at(seconds).utc.strftime("%Y_%m_%d")}"
+      end
+    end
+
+
+    #----------------
+    private # methods
+    #----------------
+
+    #----------- Validation Helpers ---------------
+
+    def _validate_positive_fixnum(parameter_name, parameter)
+      _validate_class(parameter_name, parameter, Fixnum)
+
+      if parameter <= 0
+        _raise_arg_err "#{parameter_name} should be > 0"
+      end
+      true
+    end
+
+    def _validate_class(parameter_name, parameter, expected_class)
+      if !parameter.kind_of?(expected_class)
+        _raise_arg_err("class of #{parameter_name} expected to be #{expected_class} but instead was #{parameter.class}")
+      end
+      true
+    end
+
+    def _validate_timestamp(timestamp)
+      return true if timestamp == FUTURE_PARTITION_VALUE
+
+      _validate_positive_fixnum(:timestamp, timestamp)
+
+      true
+    end
+
+    def _validate_partition_name(partition_name)
+      _validate_class('partition_name', partition_name, String)
+    end
+
+    def _validate_partition_names(partition_names)
+      _validate_class('partition_names', partition_names, Array)
+
+      partition_names.each do |name|
+        _validate_partition_name(name)
+      end
+
+      true
+    end
+
+    def _validate_partition_names_allowed_to_drop(partition_names)
+      black_listed_partitions = [FUTURE_PARTITION_NAME]
+
+      if active_partition = Partition.all(adapter, table_name).current_partition(self.current_timestamp)
+        black_listed_partitions << active_partition.name
+      end
+
+      if (partition_names & black_listed_partitions).any?
+        _raise_arg_err "current and future partition can never be dropped"
+      end
+
+      true
+    end
+
+    def _validate_drop_partitions_names(partition_names)
+      _validate_partition_names(partition_names)
+      _validate_partition_names_allowed_to_drop(partition_names)
+
+      true
+    end
+
+    def _validate_partition_data(partition_data)
+      _validate_class('partition_data', partition_data, Hash)
+
+      partition_data.each_pair do |key, value|
+        _validate_partition_name(key)
+        _validate_timestamp(value)
+
+        if key == FUTURE_PARTITION_NAME && value != FUTURE_PARTITION_VALUE ||
+            key != FUTURE_PARTITION_NAME && value == FUTURE_PARTITION_VALUE
+          _raise_arg_err "future partition name '#{FUTURE_PARTITION_NAME}' must use timestamp '#{FUTURE_PARTITION_VALUE}',"\
+                         "but got name #{key} and timestamp #{value}"
+        end
+      end
+
+      true
+    end
+
+    # executes the sql
+    # @param [String] sql to be executed
+    # @return [Boolean] true
+    def _execute(sql)
+      if @lock_wait_timeout
+        SqlPartitioner::LockWaitTimeoutHandler.with_lock_wait_timeout(@adapter, @lock_wait_timeout) do
+          adapter.execute(sql)
+        end
+      else
+        adapter.execute(sql)
+      end
+    end
+
+    # executes the sql and then displays the partition info
+    # @param [String] sql to be executed
+    # @param [Boolean] dry_run Defaults to true. If true, query wont be executed.
+    # @return [String/Boolean] returns SQL to be executed if dry_run=true
+    def _execute_and_display_partition_info(sql, dry_run=true)
+      if sql
+        if dry_run
+          sql
+        else
+          _execute(sql)
+       
+          log "\n#{Partition.to_log(Partition.all(adapter, table_name))}", false
+
+          true
+        end
+      else
+        false
+      end
+    end
+
+    def _raise_arg_err(err_message)
+      raise ArgumentError.new err_message
+    end
+
+  end
+end

data/lib/sql_partitioner/loader.rb

@@ -0,0 +1,16 @@
+module SqlPartitioner
+  class Loader
+
+    def self.require_or_skip(path, required_constant)
+      if Object.const_defined?(required_constant)
+        require path
+
+        true
+      else
+        # "No need to `require '#{path}'` since #{required_constant} is not defined at this point."
+        false
+      end
+    end
+
+  end
+end

data/lib/sql_partitioner/lock_wait_timeout_handler.rb

@@ -0,0 +1,17 @@
+module SqlPartitioner
+  class LockWaitTimeoutHandler
+    
+    # Temporarily sets the `@@local.lock_wait_timeout` to the given value,
+    # executes the `block`, and restores `@@local.lock_wait_timeout` to its original value.
+    def self.with_lock_wait_timeout(adapter, timeout, &block)
+      lock_wait_timeout_before = adapter.select("SELECT @@local.lock_wait_timeout").first
+      adapter.execute("SET @@local.lock_wait_timeout = ?", timeout)
+      begin
+        return block.call
+      ensure
+        adapter.execute("SET @@local.lock_wait_timeout = ?", lock_wait_timeout_before.to_i)
+      end
+    end
+
+  end
+end