cleansweep 1.0.0

checksums.yaml

@@ -0,0 +1,7 @@
+---
+SHA1:
+  metadata.gz: 03e179a7416960bd102d09f4dd50f1c7f204fbc4
+  data.tar.gz: 559252c905e8b3302bf2eafe2d9774c574d322aa
+SHA512:
+  metadata.gz: 7f9019b2cf6b6fb339a5ac4ed32eb8146cb8b7742282da6cb7924e4c319414fb4307c0ca093b2bfcd8a26f58c768221183fd5cb85a576b40ab022b560802d28a
+  data.tar.gz: ae47587361ffe0ac2cfdca18f99095c259c0838442b22a6dccbaacc3224cccafe37f07ed99ed1bd3eb461f46261b187d52bdedd2bf03b3f0b3d4816e12b4a321

data/.gitignore

@@ -0,0 +1,20 @@
+/.bundle/
+.ruby-version
+/.yardoc
+/Gemfile.lock
+/_yardoc/
+/coverage/
+/doc/
+/pkg/
+/spec/reports/
+/tmp/
+*.bundle
+*.so
+*.o
+*.a
+*~
+mkmf.log
+.idea/
+log/
+*#
+.#*

data/Gemfile

@@ -0,0 +1,4 @@
+source 'https://rubygems.org'
+
+# Specify your gem's dependencies in cleansweep.gemspec
+gemspec

data/LICENSE.txt

@@ -0,0 +1,22 @@
+Copyright (c) 2014 Bill Kayser
+
+MIT License
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+"Software"), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND
+NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE
+LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION
+OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION
+WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.md

@@ -0,0 +1,201 @@
+Cleansweep is a utility for scripting purges using ruby in an efficient, low-impact manner on
+mysql innodb tables.  Based on the Percona `pt-archive` utility.
+
+## Installation
+
+Add this line to your application's Gemfile:
+
+```ruby
+gem 'cleansweep'
+```
+
+And then execute:
+
+    $ bundle
+
+Or install it yourself as:
+
+    $ gem install cleansweep
+
+## How it works
+
+Consider the table:
+```sql
+    create table comments (
+       `id` int(11) primary key auto_increment,
+       `timestamp` datetime,
+       `account` int(11),
+       `liked` boolean,
+       key comments_on_account_timestamp(account, timestamp)
+    )
+```
+Assume there is an active record model for it:
+
+    class Comment < ActiveRecord::Base ; end
+
+### Purging by traversing an index
+
+The most efficient way to work through a table is by scanning through an index one chunk
+at a time.
+
+Let's assume we want to purge Comments older than 1 month.  We can
+scan the primary key index or the `account`,`timestamp` index.  In this case the latter will
+probably work better since we are evaluating the timestamp for the purge.
+
+```ruby
+    r = CleanSweep::PurgeRunner.new model: Comment,
+                                    index: 'comments_on_account_timestamp' do | scope |
+        scope.where('timestamp < ?', 1.month.ago)
+    end
+```
+
+To execute the purge, do:
+
+```ruby
+    count = r.execute_in_batches
+    puts "Deleted #{count} rows"
+```
+
+Check what it will do:
+
+```ruby
+    r.print_queries($stdout)
+```
+
+This will show you what it will do by printing out the three different statements used:
+
+```sql
+    Initial Query:
+        SELECT  `id`,`account`,`timestamp`
+        FROM `comments` FORCE INDEX(comments_on_account_timestamp)
+        WHERE (timestamp < '2014-11-25 21:47:43')
+        ORDER BY `account` ASC,`timestamp` ASC
+        LIMIT 500
+    Chunk Query:
+        SELECT  `id`,`account`,`timestamp`
+        FROM `comments` FORCE INDEX(comments_on_account_timestamp)
+        WHERE (timestamp < '2014-11-25 21:47:43') AND (`account` > 0 OR (`account` = 0 AND `timestamp` > '2014-11-18 21:47:43'))\n    ORDER BY `account` ASC,`timestamp` ASC
+        LIMIT 500
+    Delete Statement:
+        DELETE
+        FROM `comments`
+        WHERE (`id` = 2)
+```
+
+It does the initial statement once to get the first chunk of rows.  Then it does subsequent queries
+starting at the index where the last chunk left off, thereby avoiding a complete index scan.  This works
+fine as long as you don't have rows with duplicate account id and timestamps.  If you do, you'll possibly
+miss rows between chunks.
+
+To avoid missing duplicates, you can traverse the index using only the first column with an inclusive comparator
+like `>=` instead of `>`.  Here's what that would look like:
+
+```ruby
+    r = CleanSweep::PurgeRunner.new model:Comment,
+                                    index: 'comments_on_account_timestamp',
+                                    first_only: true do | scope |
+        scope.where('timestamp < ?', 1.month.ago)
+    end
+```
+
+The chunk query looks like:
+
+```sql
+    SELECT  `id`,`account`,`timestamp`
+    FROM `comments` FORCE INDEX(comments_on_account_timestamp)
+    WHERE (timestamp < '2014-11-25 21:47:43') AND (`account` >= 0)
+    LIMIT 500
+```
+
+You can scan the index in either direction.  To specify descending order, use the `reverse: true` option.
+
+### Copying rows from one table to another
+
+You can use the same technique to copy rows from one table to another.  Support in CleanSweep is pretty
+minimal.  It won't _move_ rows, only copy them, although it would be easy to fix this.
+I used this to copy ids into a temporary table which I then
+used to delete later.
+
+Here's an example that copies rows from the `Comment` model to the `ExpiredComment` model (`expired_comments`).
+Comments older than one week are copied.
+
+```ruby
+      copier = CleanSweep::PurgeRunner.new model: Comment,
+                                           index: 'comments_on_account_timestamp',
+                                           dest_model: ExpiredComment,
+                                           copy_columns: %w[liked] do do | model |
+        model.where('last_used_at < ?', 1.week.ago)
+      end
+```
+
+The `copy_columns` option specifies additional columns to be inserted into the `expired_comments` table.
+
+### Watching the history list and replication lag
+
+You can enter thresholds for the history list size and replication lag that will be used to pause the
+purge if either of those values get into an unsafe territory.  The script will pause for 5 minutes and
+only start once the corresponding metric goes back down to 90% of the specified threshold.
+
+### Logging and monitoring progress
+
+You pass in a standard log instance to capture all running output.  By default it will log to your
+`ActiveRecord::Base` logger, or stdout if that's not set up.
+
+If you specify a reporting interval
+with the `report` option it will print the status of the purge at that interval.  This is useful to track
+progress and assess the rate of deletion.
+
+### Joins and subqueries
+
+You can add subqueries and joins to your query in the scope block, but be careful.  The index and order
+clause may work against you if the table you are joining with doesn't have good parity with the indexes
+in your target table.
+
+### Limitations
+
+* Only works for mysql (as far as I know).  I have only used it against 5.5.
+* Should work with ActiveRecord 3.* - 4.*.
+* Using a non-unique index risks missing duplicate rows unless you use the `first_only` option.
+* Using the `first_only` option risks rescanning many rows if you have many more duplicates than your
+  chunk size
+* An index is required but you should be able to run a purge without one.  It just means you're not
+  scanning the index in chunks.  This might be okay if you are deleting everything as you go along because
+  then you're not rescanning the rows.  It wouldn't require much to modify CleanSweep to support this
+  mode.
+
+### Other options
+
+There are a number of other options you can use to tune the script.  For details look at the
+[API on the `PurgeRunner` class](http://bkayser.github.io/cleansweep/rdoc/CleanSweep/PurgeRunner.html)
+
+### NewRelic integration
+
+The script requires the [New Relic](http://github.com/newrelic/rpm) gem.  It won't impact anyting if you
+don't have a New Relic account to report to, but if you do use New Relic it is configured to show you
+detailed metrics.  I recommend turning off transaction traces for long purge jobs to reduce your memory
+footprint.
+
+## Testing
+
+To run the specs, start a local mysql instance.  The default user is root with an empty password.
+Override the user/password with environment variables `DB_USER` and `DB_PASSWORD`.  The test
+creates a db called 'cstest'.
+
+## Contributing
+
+1. Fork it ( https://github.com/bkayser/cleansweep/fork )
+2. Create your feature branch (`git checkout -b my-new-feature`)
+3. Commit your changes (`git commit -am 'Add some feature'`)
+4. Push to the branch (`git push origin my-new-feature`)
+5. Create a new Pull Request
+
+## License and Copyright
+
+Copyright 2014 New Relic, Inc., and Bill Kayser
+
+Covered by the MIT [LICENSE](LICENSE.txt).
+
+### Credits
+
+This was all inspired and informed by [Percona's `pt-archiver` script](http://www.percona.com/doc/percona-toolkit/2.1/pt-archiver.html)
+written by Baron Schwartz.

data/Rakefile

@@ -0,0 +1,3 @@
+require "bundler/gem_tasks"
+require 'rspec/core/rake_task'
+RSpec::Core::RakeTask.new(:spec)

data/cleansweep.gemspec

@@ -0,0 +1,34 @@
+# coding: utf-8
+lib = File.expand_path('../lib', __FILE__)
+$LOAD_PATH.unshift(lib) unless $LOAD_PATH.include?(lib)
+require 'clean_sweep/version'
+
+Gem::Specification.new do |spec|
+  spec.name          = "cleansweep"
+  spec.version       = CleanSweep::VERSION
+  spec.authors       = ["Bill Kayser"]
+  spec.email         = ["bkayser@newrelic.com"]
+  spec.summary       = %q{Utility to purge or archive rows in mysql tables}
+  spec.description   = <<-EOF
+     Purge data from mysql innodb tables efficiently with low overhead and impact.
+     Based on the Percona pt-archive utility.
+  EOF
+  spec.homepage      = "http://github.com/bkayser/cleansweep"
+  spec.license       = "MIT"
+
+  spec.files         = `git ls-files -z`.split("\x0")
+  spec.executables   = spec.files.grep(%r{^bin/}) { |f| File.basename(f) }
+  spec.test_files    = spec.files.grep(%r{^spec/})
+  spec.require_paths = ["lib"]
+
+  spec.add_runtime_dependency 'activerecord', '~>3'
+  spec.add_runtime_dependency 'newrelic_rpm', '~>3'
+  spec.add_runtime_dependency 'mysql2', '~> 0.3.17'
+
+  spec.add_development_dependency 'pry', '~>0'
+  spec.add_development_dependency 'bundler', '~> 1.7'
+  spec.add_development_dependency 'rake', '~> 10.0'
+  spec.add_development_dependency 'rspec', '~> 3.1'
+  spec.add_development_dependency 'factory_girl', '~> 4.4'
+  spec.add_development_dependency 'awesome_print', '~>1.2'
+end

data/lib/clean_sweep/purge_runner/logging.rb

@@ -0,0 +1,38 @@
+module CleanSweep::PurgeRunner::Logging
+
+  def report(force=false)
+    report_duration = Time.now - @report_interval_start
+    if (force || report_duration >= @report_interval)
+      while (@report_interval_start < Time.now - @report_interval) do
+        @report_interval_start += @report_interval
+      end
+      report  = []
+      elapsed = [1, (Time.now - @start).to_i].max
+      rate    = (@total_deleted / elapsed).to_i
+      rate    = "#{rate > 0 ? '%12i' % rate : ('%12s' %'< 1')} records/second"
+      report << "report:"
+      if copy_mode?
+        report << "  #{@dry_run ? 'queried' : 'copied'}: #{'%12i' % @total_deleted} #{@model.table_name} records"
+      else
+        report << "  #{@dry_run ? 'queried' : 'deleted'}: #{'%12i' % @total_deleted} #{@model.table_name} records"
+      end
+      report << "  elapsed: #{'%12s' % format(elapsed)}"
+      report << "  rate:    #{rate}"
+      log :info,  report.join("\n")
+    end
+  end
+
+  def log level, msg
+    prefix = level == :debug ? " *** " : " ** "
+    out = msg.split("\n").map {|line| prefix + line}.join("\n")
+    @logger.send level, out
+  end
+
+  def format(time)
+    format_string = "%H:%M:%S"
+    if (time.to_i > (24 * 60 * 60))
+      format_string = "%d days, %H:%M"
+    end
+    Time.at(time).strftime(format_string)
+  end
+end

data/lib/clean_sweep/purge_runner/mysql_status.rb

@@ -0,0 +1,82 @@
+require 'newrelic_rpm'
+
+class CleanSweep::PurgeRunner::MysqlStatus
+
+  # Options:
+  # logger, model, max_history, max_repl_lag, check_period
+  def initialize(options={})
+    @logger = options[:logger] || ActiveRecord::Base.logger
+    @model = options[:model]
+    @max_history = options[:max_history]
+    @max_replication_lag = options[:max_repl_lag]
+    @check_period = options[:check_period] || 2.minutes
+    @last_check = @check_period.ago
+  end
+
+  def check!
+    return if Time.now - @check_period < @last_check
+    while (v = get_violations).any? do
+      @logger.warn("pausing 5 minutes (#{v.to_a.map{ |key, value| "#{key} = #{value}"}.join(", ")})") if !paused?
+      @paused = true
+      pause 5.minutes
+    end
+    @logger.info("violations clear") if paused?
+    all_clear!
+  end
+
+  def get_violations
+    violations = {}
+    if @max_history
+      current = get_history_length
+      violations["history length"] = current if threshold(@max_history) < current
+    end
+    if @max_replication_lag
+      current = get_replication_lag
+      violations["replication lag"] = current if threshold(@max_replication_lag) < current
+    end
+    return violations
+  end
+
+  # Return the threshold to use in the check.  If we are already failing, don't
+  # start up again until we've recovered at least 10%
+  def threshold(value)
+    if paused?
+      value = 0.90 * value
+    else
+      value
+    end
+  end
+
+  def pause time
+    Kernel.sleep time
+  end
+  add_method_tracer :pause
+
+  def paused?
+    @paused
+  end
+
+  def all_clear!
+    @last_check = Time.now
+    @paused = nil
+  end
+
+  def get_replication_lag
+    rows = @model.connection.select_rows 'SHOW SLAVE STATUS'
+    if rows.nil? || rows.empty?
+      return 0
+    else
+      return rows[0][32]
+    end
+  end
+
+
+  def get_history_length
+    rows = @model.connection.select_rows <<-EOF
+        show engine innodb status
+    EOF
+    status_string = rows.first[2]
+    return /History list length ([0-9]+)/.match(status_string)[1].to_i
+  end
+
+end

data/lib/clean_sweep/purge_runner.rb

@@ -0,0 +1,211 @@
+require 'stringio'
+# This is a utility built to mimic some of the features of the pt_archive script
+# to make really big purges go faster with no production impact.
+#
+# It uses a strategy of descending an index, querying for purgeable ids and then
+# deleting them in batches.
+#
+# === Required Options
+#
+# [:model]
+#    Required: the Active Record model for the table being purged or copied from
+#
+# === Optional Options
+#
+# [:chunk_size]
+#    The number of rows to copy in each block.  Defaults to 500.
+# [:index]
+#    The index to traverse in ascending order doing the purge.  Rows are read in the order of
+#    the index, which must be a btree index.  If not specified, <tt>PRIMARY</tt> is assumed.
+# [:reverse]
+#    Traverse the index in reverse order.  For example, if your index is on <tt>account_id</tt>,
+#    <tt>timestamp</tt>, this option will move through the rows starting at the highest account
+#    number, then move through timestamps starting with the most recent.
+# [:first_only]
+#    Traverse only the first column of the index, and do so inclusively using the <tt>&gt;=</tt> operator
+#    instead of the strict <tt>&gt;</tt> operator.  This is important if the index is not unique and there
+#    are a lot of duplicates.  Otherwise the delete could miss rows.  Not allowed in copy mode because you'd
+#    be inserting duplicate rows.
+# [:dry_run]
+#    Print out the queries that are going to be used.  You should run explain on these.
+# [:stop_after]
+#    The operation will end after copying this many rows.
+# [:report]
+#    Specify an interval in seconds between status messages being printed out.
+# [:logger]
+#    The log instance to use.  Defaults to the <tt>ActiveRecord::Base.logger</tt>
+#    if not nil, otherwise it uses _$stdout_
+# [:dest_model]
+#    When this option is present nothing is deleted, and instead rows are copied to
+#    the table for this model.  This model must
+#    have identically named columns as the source model.  By default, only columns in the
+#    named index and primary key are copied but these can be augmented with columns in the
+#    <tt>copy_columns</tt> option.
+# [:copy_columns]
+#    Extra columns to add when copying to a dest model.
+#
+# === Safety thresholds
+# [:sleep]
+#    Time in seconds to sleep between each chunk.
+# [:max_history]
+#    The history list size (if available) is checked every 5 minutes and if it exceeds this size
+#    the purge will pause until the history list is below 90% of this value.
+# [:max_repl_lag]
+#    The maximum length of the replication lag.  Checked every 5 minutes and if exceeded the purge
+#    pauses until the replication lag is below 90% of this value.
+
+class CleanSweep::PurgeRunner
+
+  require 'clean_sweep/purge_runner/logging'
+  require 'clean_sweep/purge_runner/mysql_status'
+
+  include CleanSweep::PurgeRunner::Logging
+
+  # This helps us track the state of replication and history list and pause
+  # if necessary
+  attr_reader :mysql_status
+  
+  def initialize(options={})
+    @model     = options[:model] or raise "source model class required"
+    @limit            = options[:chunk_size] || 500
+
+    @target_model     = options[:dest_model]
+    @stop_after       = options[:stop_after]
+    @report_interval  = options[:report] || 10.seconds
+    @logger           = options[:logger] || ActiveRecord::Base.logger || Logger.new($stdout)
+    @dry_run          = options[:dry_run]
+    @sleep            = options[:sleep]
+
+    @max_history      = options[:max_history]
+    @max_repl_lag     = options[:max_repl_lag]
+
+    @table_schema     = CleanSweep::TableSchema.new @model,
+                                                    key_name: options[:index],
+                                                    ascending: !options[:reverse],
+                                                    extra_columns: options[:copy_columns],
+                                                    first_only: options[:first_only]
+
+    if (@max_history || @max_repl_lag)
+      @mysql_status = CleanSweep::PurgeRunner::MysqlStatus.new model: @model,
+                                                               max_history: @max_history,
+                                                               max_repl_lag: @max_repl_lag,
+                                                               check_period: options[:check_period],
+	                                                             logger: @logger
+    end
+
+    raise "You can't copy rows from a table into itself" if copy_mode? && @model == @target_model
+    raise "An index is required in copy mode" if copy_mode? && @table_schema.traversing_key.nil?
+    raise "first_only option not allowed in copy mode" if copy_mode? && @table_schema.first_only?
+
+    @report_interval_start = Time.now
+
+    @query                 = @table_schema.initial_scope.limit(@limit)
+
+    @query = yield(@query) if block_given?
+  end
+
+
+  def copy_mode?
+    @target_model.present?    
+  end
+
+  # Execute the purge in chunks according to the parameters given on instance creation.
+  # Will raise <tt>CleanSweep::PurgeStopped</tt> if a <tt>stop_after</tt> option was provided and
+  # that limit is hit.
+  #
+  # Returns the number of rows copied or deleted.
+  #
+  def execute_in_batches
+
+    print_queries($stdout) and return 0 if @dry_run
+
+    @start = Time.now
+    verb = copy_mode? ? "copying" : "purging"
+
+    msg = "starting: #{verb} #{@table_schema.name} records in batches of #@limit"
+    msg << " to #{@target_model.table_name}" if copy_mode?
+
+
+    log :info,  "sleeping #{@sleep} seconds between purging" if @sleep && !copy_mode?
+    @total_deleted = 0
+
+    # Iterate through the rows in limit chunks
+    log :debug, "find rows: #{@query.to_sql}" if @logger.level == Logger::DEBUG
+
+    @mysql_status.check! if @mysql_status
+
+    rows = NewRelic::Agent.with_database_metric_name(@model.name, 'SELECT') do
+      @model.connection.select_rows @query.to_sql
+    end
+    while rows.any? && (!@stop_after || @total_deleted < @stop_after) do
+#      index_entrypoint_args = Hash[*@source_keys.zip(rows.last).flatten]
+      log :debug, "#{verb} #{rows.size} records between #{rows.first.inspect} and #{rows.last.inspect}" if @logger.level == Logger::DEBUG
+      stopped = @stop_after && rows.size + @total_deleted > @stop_after
+
+      rows = rows.first(@stop_after - @total_deleted) if stopped
+      last_row = rows.last
+      if copy_mode?
+        metric_op_name = 'INSERT'
+        statement = @table_schema.insert_statement(@target_model, rows)
+      else
+        metric_op_name = 'DELETE'
+        statement = @table_schema.delete_statement(rows)
+      end
+      log :debug, statement if @logger.level == Logger::DEBUG
+      chunk_deleted = NewRelic::Agent.with_database_metric_name(@target_model, metric_op_name) do
+        @model.connection.update statement
+      end
+
+      @total_deleted += chunk_deleted
+      raise CleanSweep::PurgeStopped.new("stopped after #{verb} #{@total_deleted} #{@model} records", @total_deleted) if stopped
+      q = @table_schema.scope_to_next_chunk(@query, last_row).to_sql
+      log :debug, "find rows: #{q}" if @logger.level == Logger::DEBUG
+
+      sleep @sleep if @sleep && !copy_mode?
+      @mysql_status.check! if @mysql_status
+
+      rows = NewRelic::Agent.with_database_metric_name(@model, 'SELECT') do
+        @model.connection.select_rows(q)
+      end
+      report
+    end
+    report(true)
+    if copy_mode?
+      log :info,  "completed after #{verb} #{@total_deleted} #{@table_schema.name} records to #{@target_model.table_name}"
+    else
+      log :info,  "completed after #{verb} #{@total_deleted} #{@table_schema.name} records"
+    end
+
+    return @total_deleted
+  end
+
+  def sleep duration
+    Kernel.sleep duration
+  end
+
+  add_method_tracer :sleep
+  add_method_tracer :execute_in_batches
+
+  def print_queries(io)
+    io.puts 'Initial Query:'
+    io.puts format_query('    ', @query.to_sql)
+    rows = @model.connection.select_rows @query.limit(1).to_sql
+    io.puts "Chunk Query:"
+    io.puts format_query('    ', @table_schema.scope_to_next_chunk(@query, rows.first).to_sql)
+    if copy_mode?
+      io.puts "Insert Statement:"
+      io.puts format_query('    ', @table_schema.insert_statement(@target_model, rows))
+    else
+      io.puts "Delete Statement:"
+      io.puts format_query('    ', @table_schema.delete_statement(rows))
+    end
+  end
+
+  private
+
+  def format_query indentation, query
+    lines = query.split(/ (?=values|from|where|order|limit)/i)
+    lines.map {|line| indentation + line.strip }.join("\n")
+  end
+end
+

data/lib/clean_sweep/purge_stopped.rb

@@ -0,0 +1,9 @@
+
+class CleanSweep::PurgeStopped < RuntimeError
+
+  attr_reader :stopped_at
+
+  def initialize(message, limit)
+    @stopped_at = limit
+  end
+end

data/lib/clean_sweep/table_schema/column_schema.rb

@@ -0,0 +1,23 @@
+class CleanSweep::TableSchema::ColumnSchema
+
+  attr_reader :name
+  attr_accessor :select_position
+
+  def initialize(name, model)
+    @name = name.to_sym
+    col_num = model.column_names.index(name.to_s) or raise "Can't find #{name} in #{model.name}"
+    @model = model
+    @column = model.columns[col_num]
+  end
+
+  def quoted_name
+    "`#{name}`"
+  end
+  def value(row)
+    row[select_position]
+  end
+  def quoted_value(row)
+    @model.quote_value(value(row), @column)
+  end
+end
+