db_sucker 3.0.1 → 3.2.1

checksums.yaml

@@ -1,7 +1,7 @@
 ---
 SHA256:
-  metadata.gz: 401d1701ef72b4acc718017a1bd234328e52efc2563439f2dac10fe566098b9d
-  data.tar.gz: 0e1d5e4456ae52412fd56b6417d9fb66cf60424a3ea0e922a73db3705a9450c4
+  metadata.gz: 41ee20b2f776fdf1308842c417e64fff4b5d51178bb50ee91db5215ee6e6d660
+  data.tar.gz: 50999e195fa36291660360afee4e78abd4761a5acabb9ea38bb004e9abb937e0
 SHA512:
-  metadata.gz: 89202e7556c6f89156e12e67c4c3899fd12caf542e76bb75548b888bb588945031c5628061672e1a8805cd02338ee5d1e1104b069867aac2bf19a3dd8d0251f6
-  data.tar.gz: ba11c360337e2ba2952f2e87069b83541d6111ba131e6cd81118026fb26a966b955b75c450bb7caed05aa9923fed18f18fa0baa01c4d359d94b77575d589391f
+  metadata.gz: a743447ba674c4213736395e1a6a7d7f0aed5f2305bef00e4cd1ee7903eb4e458a63a55fad1576de818079d9ea3f13f693f7af4f3519b4d7871b7f9f80301d09
+  data.tar.gz: 6d9c944d4f1b0420d785882b0c60b2c79c8803d9bdec83b801756997dec8db68a01c747a5b37a8a90533071358697ce63b3748675dc796c285c88a1d9449e8d9

data/CHANGELOG.md

@@ -1,3 +1,70 @@
+## 3.2.1
+
+### Updates
+
+* Added means to signal all threads in order to wake them up as they occasionally get stuck for some reason. Press `S` or use `:signal-threads` to execute. Have yet to find the culprint :)
+* Slight UX improvements, silent commands briefly flash the screen, invalid commands ring the bell
+
+### Fixes
+
+* don't warn about orphaned concurrent-ruby threads, debug log trace of remaining threads
+* join killed threads to ensure they are dead
+* minor sshdiag fixes/enhancements
+
+-------------------
+
+## 3.2.0
+
+### Updates
+
+* Added support for native sftp command line utility (see application option `file_transport`) but it
+  only works with non-interactive key authentication.
+
+### Fixes
+
+* Prevent application from crashing when eval produces non-StandardErrors (e.g. SyntaxErrors)
+
+-------------------
+
+## 3.1.1
+
+### Fixes
+
+* Prevent a single task (running in main thread) to summon workers (and fail) when getting deferred
+
+-------------------
+
+## 3.1.0
+
+### Fixes
+
+* Prevent app to run out of consumers when tasks are waiting for defer ready
+* Prevent IO errors on Ruby < 2.3 in uncompress
+* Prevent racing conditions in SSH diagnose task
+* Minor fixes
+
+-------------------
+
+## 3.0.3
+
+* no changes, fixed my tag screwup
+
+-------------------
+
+## 3.0.2
+
+* no changes, github tags can't be altered, I screwed up
+
+-------------------
+
+## 3.0.1
+
+### Fixes
+
+* hotfix for 3.0.0 release
+
+-------------------
+
 ## 3.0.0
 
 ### Updates

data/README.md

@@ -7,7 +7,8 @@ You configure your hosts via an YAML configuration in which you can define multi
 
 This tool is meant for pulling live data into your development environment. **It is not designed for backups!** but you might get away with it.
 
-![screenshot](https://i.imgur.com/EAjWrEd.png)
+[![screenshot](https://i.imgur.com/EAjWrEd.png)](https://www.youtube.com/watch?v=jdhyQzJOIkE)
+[▶ see it in action](https://www.youtube.com/watch?v=jdhyQzJOIkE)
 
 ---
 ## Alpha product (v3 is a rewrite), use at your own risk, always have a backup!
@@ -18,7 +19,7 @@ This tool is meant for pulling live data into your development environment. **It
 * independent parallel dump / download / import cycle for each table
 * verifies file integrity via SHA
 * flashy and colorful curses based interface with keyboard shortcuts
-* more status indications than you would ever want (even more if the remote has a somewhat recent `pv` (pipeviewer) installed)
+* more status indications than you would ever want (even more if the remote has `pv` (pipeviewer) >= 1.3.8 installed)
 * limit concurrency of certain type of tasks (e.g. limit downloads, imports, etc.)
 * uses more threads than any application should ever use (seriously it's a nightmare)
 
@@ -31,7 +32,7 @@ Currently `db_sucker` only handles the following data-flow constellation:
 
 On the local side you will need:
   - unixoid OS
-  - Ruby (>= 2.0, != 2.3.1 see gotchas)
+  - Ruby (>= 2.0, != 2.3.0 see [Caveats](#caveats---bugs))
   - mysql2 gem
   - MySQL client (`mysql` command will be used for importing)
 
@@ -41,6 +42,7 @@ On the remote side you will need:
   - any folder with write permissions (for the temporary dumps)
   - mysqldump executable
   - MySQL credentials :)
+  - Optional: Install "pv" aka pipeviewer with a version >= 1.3.8 for progress displays on remote tasks
 
 
 ## Installation
@@ -49,9 +51,13 @@ Simple as:
 
     $ gem install db_sucker
 
-At the moment you are advised to adjust the MaxSessions limit on your remote SSH server if you run into issues, see Caveats.
+You will need mysql2 as well (it's not a dependency as we might support other DBMS in the future):
 
-You will also need at least one configuration, see Configuration.
+    $ gem install mysql2
+
+At the moment you are advised to adjust the MaxSessions limit on your remote SSH server if you run into issues, see [Caveats](#caveats---bugs).
+
+You will also need at least one configuration, see [Configuration](#configuration-for-sucking---yaml-format).
 
 
 ## Usage
@@ -124,12 +130,12 @@ To get a list of available interface options and shortcuts press `?` or type `:h
 
 ## Configuration (application) - Ruby format
 
-DbSucker has a lot of settings and other mechanisms which you can tweak and utilize by creating a `~/.db_sucker/config.rb` file. You can change settings, add hooks or define own actions. For more information please take a look at the [documented example config](https://github.com/2called-chaos/db_sucker/blob/master/doc/config_example.rb) and/or [complete list of all settings](https://github.com/2called-chaos/db_sucker/blob/master/lib/db_sucker/application.rb#L58-L129).
+DbSucker has a lot of settings and other mechanisms which you can tweak and utilize by creating a `~/.db_sucker/config.rb` file. You can change settings, add hooks or define own actions. For more information please take a look at the [documented example config](https://github.com/2called-chaos/db_sucker/blob/master/doc/config_example.rb) and/or [complete list of all settings](https://github.com/2called-chaos/db_sucker/blob/master/lib/db_sucker/application.rb#L60-L134).
 
 
 ## Deferred import
 
-Tables with an uncompressed filesize of over 50MB will be queued up for import. Files smaller than 50MB will be imported concurrently with other tables. When all those have finished the large ones will import one after another. You can skip this behaviour with the `-n` resp. `--no-deffer` option. The threshold is changeable in your `config.rb`, see Configuration.
+Tables with an uncompressed filesize of over 50MB will be queued up for import. Files smaller than 50MB will be imported concurrently with other tables. When all those have finished the large ones will import one after another. You can skip this behaviour with the `-n` resp. `--no-deffer` option. The threshold is changeable in your `config.rb`, see [Configuration](#configuration-application---ruby-format).
 
 
 ## Importer
@@ -159,7 +165,6 @@ Currently there is only the "binary" importer which will use the mysql client bi
 ### General
 
 * Ruby 2.3.0 has a bug that might segfault your ruby if some exceptions occur, this is fixed since 2.3.1 and later
-* Consumers that are waiting (e.g. deferred or slot pool) won't release their tasks, if you have to few consumers you might softlock
 
 ### SSH errors / MaxSessions
 
@@ -170,7 +175,7 @@ Under certain conditions the program might softlock when the remote unexpectedly
 If you get warnings that SSH errors occured (and most likely tasks fail), please do any of the following to prevent the issue:
 
   * Raise the MaxSession setting on the remote SSHd server if you can (recommended)
-  * Lower the amount of slots for concurrent downloads (see Configuration)
+  * Lower the amount of slots for concurrent downloads (see [Configuration](#configuration-application---ruby-format))
   * Lower the amount of consumers (not recommended, use slots instead)
 
 You can run basic SSH diagnosis tests with `db_sucker <config_identifier> -a sshdiag`.

data/VERSION

@@ -1 +1 @@
-3.0.1
+3.2.1

data/doc/config_example.rb

@@ -14,6 +14,10 @@ opts[:deferred_threshold] = 50_000_000  # 50 MB
 opts[:status_format] = :full            # used for IO operations, can be one of: none, minimal, full
 opts[:pv_enabled] = true                # disable pv utility autodiscovery (force non-usage)
 
+# Use native SFTP command for way faster transfer rates, only use with non-interactive key authentication!
+opts[:file_transport] = :ruby
+#opts[:file_transport] = :native
+
 # used to open core dumps (should be a blocking call, e.g. `subl -w' or `mate -w')
 # MUST be windowed! vim, nano, etc. will not work!
 opts[:core_dump_editor] = "subl -w"

data/lib/db_sucker.rb

@@ -4,7 +4,6 @@ end
 
 # stdlib
 require "benchmark"
-require "optparse"
 require "fileutils"
 require "thread"
 require "monitor"
@@ -33,6 +32,7 @@ require "db_sucker/application/sklaven_treiber/log_spool"
 require "db_sucker/application/sklaven_treiber/worker/io/base"
 require "db_sucker/application/sklaven_treiber/worker/io/throughput"
 require "db_sucker/application/sklaven_treiber/worker/io/sftp_download"
+require "db_sucker/application/sklaven_treiber/worker/io/sftp_native_download"
 require "db_sucker/application/sklaven_treiber/worker/io/file_copy"
 require "db_sucker/application/sklaven_treiber/worker/io/file_gunzip"
 require "db_sucker/application/sklaven_treiber/worker/io/file_shasum"

data/lib/db_sucker/application.rb

@@ -37,10 +37,17 @@ module DbSucker
             else "Unhandled exception terminated application!"
           end
         ensure
+          $core_runtime_exiting = 1
           app.fire(:core_shutdown)
-          remain = Thread.list.length
-          if remain > 1
-            app.warning "#{remain} threads remain (should be 1)..."
+          remain = app.filtered_threads
+          if remain.length > 1
+            app.error "[WARN] #{remain.length} threads remain (should be 1)..."
+            remain.each do |thr|
+              app.debug "[THR] #{Thread.main == thr ? "MAIN" : "THREAD"}\t#{thr.alive? ? "ALIVE" : "DEAD"}\t#{thr.inspect}", 10
+              thr.backtrace.each do |l|
+                app.debug "[THR]\t#{l}", 20
+              end
+            end
           else
             app.debug "1 thread remains..."
           end
@@ -77,6 +84,12 @@ module DbSucker
         status_format: :full,            # used for IO operations, can be one of: none, minimal, full
         pv_enabled: true,                # disable pv utility autodiscovery (force non-usage)
 
+        # file transport: how to copy files from the remote
+        #   ruby       Use ruby sftp-library (why slow?)
+        #   native     Shell out to native sftp command (we try our best to build the command according to your SSH settings)
+        #              NOTE: only use with non-interactive key authentication
+        file_transport: :ruby,
+
         # sklaven treiber
         window_enabled: true,            # if disabled effectively disables any status progress or window drawing
         window_draw: true,               # wether to refresh screen or not

data/lib/db_sucker/application/core.rb

@@ -1,6 +1,12 @@
 module DbSucker
   class Application
     module Core
+      def filtered_threads
+        Thread.list.reject do |thr|
+          thr.backtrace[0]["gems/concurrent-ruby"] rescue false
+        end
+      end
+
       def sandboxed &block
         block.call
       rescue StandardError => ex
@@ -49,20 +55,22 @@ module DbSucker
           end
 
           # worker info
-          f.puts "\n\n#{sklaventreiber.workers.length} workers:\n"
-          sklaventreiber.workers.each do |w|
-            f.puts "#{"[SSH] " if w.sshing} #{w.descriptive} #{w.state}".strip
-          end
-
-          # slot pool
-          f.puts "\n\n#{sklaventreiber.slot_pools.length} slot pools:\n"
-          sklaventreiber.slot_pools.each do |name, pool|
-            f.puts "#{name}: #{pool.slots}"
-            pool.active.each do |thr|
-              f.puts "\tactive\t #{thr} [#{thr[:itype]}]"
+          if sklaventreiber
+            f.puts "\n\n#{sklaventreiber.workers.length} workers:\n"
+            sklaventreiber.workers.each do |w|
+              f.puts "#{"[SSH] " if w.sshing} #{w.descriptive} #{w.state}".strip
             end
-            pool.waiting.each do |wthr, tthr|
-              f.puts "\twaiting\t #{tthr} [#{tthr[:itype]}]"
+
+            # slot pool
+            f.puts "\n\n#{sklaventreiber.slot_pools.length} slot pools:\n"
+            sklaventreiber.slot_pools.each do |name, pool|
+              f.puts "#{name}: #{pool.slots}"
+              pool.active.each do |thr|
+                f.puts "\tactive\t #{thr} [#{thr[:itype]}]"
+              end
+              pool.waiting.each do |wthr, tthr|
+                f.puts "\twaiting\t #{tthr} [#{tthr[:itype]}]"
+              end
             end
           end
         end
@@ -94,7 +102,7 @@ module DbSucker
           end.tap do |thr|
             # set type and priority
             thr[:itype] = type
-            thr.priority = @opts[:"tp_#{type}"]
+            thr.priority = @opts[:"tp_#{type}"] || 0
             thr.abort_on_exception = true
 
             # add signal methods

data/lib/db_sucker/application/dispatch.rb

@@ -101,55 +101,73 @@ module DbSucker
             i += 1
           end
         end
-        t.kill
+        t.kill.join
         puts "Thread priority: -#{m[1].abs}..+#{m[2].abs}"
       end
 
       # via -a/--action sshdiag
       def dispatch_sshdiag
-        log c("\nPlease wait while we run some tests...\n", :blue)
         _identifier, _ctn = false, false, false
         idstr = ARGV.shift
         varstr = ARGV.shift
 
         configful_dispatch(idstr, varstr) do |identifier, ctn, variation, var|
+          unless ctn
+            abort "This test requires a config identifier with an SSH connection!"
+          end
+
+          log c("\nPlease wait while we run some tests...\n", :blue)
           _identifier = identifier
           _ctn = ctn
           channels = []
+          monitor = Monitor.new
           stop = false
           maxsessions = :unknown
           begin
             t = Thread.new {
               begin
                 loop do
-                  ctn.loop_ssh(0.1)
+                  ctn.loop_ssh(0.1) { monitor.synchronize { channels.any? } }
                 end
               rescue DbSucker::Application::Container::SSH::ChannelOpenFailedError
-                maxsessions = channels.length - channels.select{|c| c[:open_failed] }.length
-                stop = true
+                monitor.synchronize do
+                  maxsessions = channels.length - channels.select{|c| c[:open_failed] }.length
+                  stop = true
+                  print "!"
+                end
                 retry
               end
             }
             250.times do
-              break if stop
-              c, r = ctn.blocking_channel_result("sleep 60", blocking: false, channel: true, use_sh: true)
-              channels << c
+              break if monitor.synchronize { stop }
+              c, r = ctn.blocking_channel_result("sleep 900", blocking: false, channel: true, use_sh: true)
+              monitor.synchronize do
+                channels << c
+                print "+"
+              end
               sleep 0.1
             end
           ensure
             debug "Stopping sessions (#{channels.length})..."
-            channels.each_with_index do |c, i|
-              debug "Channel ##{i+1} #{c[:pid] ? "with PID #{c[:pid]}" : "has no PID"}"
+            puts # newline for style
+            i = 1
+            loop do
+              break if monitor.synchronize { channels.empty? }
+              c = monitor.synchronize { channels.shift }
+              debug "Channel ##{i} #{c[:pid] ? "with PID #{c[:pid]}" : "has no PID"}"
               ctn.kill_remote_process(c[:pid]) if c[:pid]
+              print "-"
+              i += 1
             end
-            log c("\nSSH MaxSessions: #{c maxsessions, :magenta}", :cyan)
+            maxsessions = "#{maxsessions}+" if maxsessions.to_i >= 250
+            log c("\n\nSSH MaxSessions: #{c maxsessions, :magenta}", :cyan)
             log "This value determines how many sessions we can multiplex over a single TCP connection."
             log "Currently, DbSucker can only utilize one connection, thus this value defines the maxmium concurrency."
             log "If you get errors you can either"
             log "  * increase the SSHd `MaxSessions' setting on the remote (if you can)"
             log "  * reduce the amount of workers and/or remote slots"
             log "  * fix the mess that is this tool, visit #{c "https://github.com/2called-chaos/db_sucker", :blue}"
-            t.kill
+            t.kill.join
           end
         end
       rescue Net::SSH::AuthenticationFailed => ex

data/lib/db_sucker/application/sklaven_treiber.rb

@@ -242,8 +242,24 @@ module DbSucker
         @threads.each{|t| t[:start] = true; t.signal }
 
         # master thread (control)
+        additionals = 0
+        Thread.current[:summon_workers] = 0
         while @threads.any?(&:alive?)
           _control_thread
+          Thread.current.sync do
+            Thread.current[:summon_workers].times do
+              app.debug "Spawned additional worker due to deferred import to prevent softlocks"
+              @threads << app.spawn_thread(:sklaventreiber_worker) {|thr|
+                begin
+                  additionals += 1
+                  thr[:managed_worker] = cnum + additionals
+                  _queueoff
+                rescue Interrupt
+                end
+              }
+            end
+            Thread.current[:summon_workers] = 0
+          end
           Thread.current.wait(0.1)
         end
         @threads.each(&:join)

data/lib/db_sucker/application/sklaven_treiber/worker.rb

@@ -4,6 +4,7 @@ module DbSucker
       class Worker
         SlotPoolNotInitializedError = Class.new(::RuntimeError)
         ChannelFailRetryError = Class.new(::RuntimeError)
+        UnknownFileTransportError = Class.new(::RuntimeError)
 
         include Core
         include Accessors

data/lib/db_sucker/application/sklaven_treiber/worker/helpers.rb

@@ -15,6 +15,12 @@ module DbSucker
             end
           end
 
+          def sftp_native_download *args, &block
+            IO::SftpNativeDownload.new(self, *args).tap do |op|
+              block.try(:call, op)
+            end
+          end
+
           def file_copy *args, &block
             IO::FileCopy.new(self, *args).tap do |op|
               block.try(:call, op)

data/lib/db_sucker/application/sklaven_treiber/worker/io/file_gunzip.rb

@@ -42,7 +42,6 @@ module DbSucker
                 ensure
                   @state = :finishing
                   gz.close
-                  @in_file.close
                   @out_file.close
                 end
 

data/lib/db_sucker/application/sklaven_treiber/worker/io/pv_wrapper.rb

@@ -63,7 +63,7 @@ module DbSucker
               end
               killer.signal.join
             ensure
-              killer.kill
+              killer.kill.join
             end
           end
         end

data/lib/db_sucker/application/sklaven_treiber/worker/io/sftp_native_download.rb

@@ -0,0 +1,79 @@
+module DbSucker
+  class Application
+    class SklavenTreiber
+      class Worker
+        module IO
+          class SftpNativeDownload < Base
+            UnknownEventError = Class.new(::RuntimeError)
+            attr_reader :downloader
+
+            def init
+              @label = "downloading"
+              @entity = "download"
+              @throughput.categories << :inet << :inet_down
+            end
+
+            def reset_state
+              super
+              @downloader = nil
+            end
+
+            def build_sftp_command src, dst
+              [].tap{|cmd|
+                cmd << %{sftp}
+                cmd << %{-P #{@ctn.source["ssh"]["port"]}} if @ctn.source["ssh"]["port"]
+                @ctn.ssh_key_files.each {|f| cmd << %{-i "#{f}"} }
+                cmd << %{"#{@ctn.source["ssh"]["username"]}@#{@ctn.source["ssh"]["hostname"]}:#{src}"}
+                cmd << %{"#{dst}"}
+              }.join(" ").strip
+            end
+
+            def download! opts = {}
+              opts = opts.reverse_merge(tries: 3, read_size: @read_size, force_new_connection: true)
+              cmd = build_sftp_command(@remote, @local)
+              prepare_local_destination
+
+              execute(opts.slice(:tries).merge(sleep_error: 3)) do
+                begin
+                  @state = :init
+                  @ctn.sftp_start(opts[:force_new_connection]) do |sftp|
+                    @filesize = sftp.lstat!(@remote).size
+                  end
+
+                  # status thread
+                  status_thread = @worker.app.spawn_thread(:sklaventreiber_worker_ctrl) do |thr|
+                    loop do
+                      @offset = File.size(@local) if File.exist?(@local)
+                      break if thr[:stop]
+                      thr.wait(0.25)
+                    end
+                  end
+
+                  @state = :downloading
+                  debug "Opening process `#{cmd}'"
+                  Open3.popen2e(cmd, pgroup: true) do |_stdin, _stdouterr, _thread|
+                    # close & exit status
+                    _stdin.close_write
+                    exit_status = _thread.value
+                    if exit_status == 0
+                      debug "Process exited (#{exit_status}) `#{cmd}'"
+                    else
+                      warning "Process exited (#{exit_status}) `#{cmd}'"
+                    end
+                  end
+
+                  status_thread[:stop] = true
+                  status_thread.signal
+                  status_thread.join
+                ensure
+                  @state = :finishing
+                end
+              end
+              @state = :done
+            end
+          end
+        end
+      end
+    end
+  end
+end

data/lib/db_sucker/application/sklaven_treiber/worker/io/throughput.rb

@@ -66,12 +66,12 @@ module DbSucker
 
             def register target
               sync do
-                if @instances[target]
+                if @instances[target.object_id]
                   raise InstanceAlreadyRegisteredError, "throughput manager cannot register more than once on the same target: `#{target}'"
                 else
                   raise NotImplementedError, "throughput manager requires the target to respond_to?(:filesize)" unless target.respond_to?(:filesize)
                   raise NotImplementedError, "throughput manager requires the target to respond_to?(:offset)" unless target.respond_to?(:offset)
-                  @instances[target] = Instance.new(self, target)
+                  @instances[target.object_id] = Instance.new(self, target)
                 end
               end
             end

data/lib/db_sucker/application/sklaven_treiber/worker/routines.rb

@@ -142,11 +142,23 @@ module DbSucker
             @local_file_compressed = local_tmp_file(File.basename(@remote_file_compressed))
             @local_files_to_remove << @local_file_compressed
 
-            sftp_download(@ctn, @remote_file_compressed => @local_file_compressed) do |dl|
-              dl.status_format = app.opts[:status_format]
-              @status = [dl, "yellow"]
-              dl.abort_if { @should_cancel }
-              dl.download!
+            case app.opts[:file_transport]
+            when :ruby
+              sftp_download(@ctn, @remote_file_compressed => @local_file_compressed) do |dl|
+                dl.status_format = app.opts[:status_format]
+                @status = [dl, "yellow"]
+                dl.abort_if { @should_cancel }
+                dl.download!
+              end
+            when :native
+              sftp_native_download(@ctn, @remote_file_compressed => @local_file_compressed) do |dl|
+                dl.status_format = app.opts[:status_format]
+                @status = [dl, "yellow"]
+                dl.abort_if { @should_cancel }
+                dl.download!
+              end
+            else
+              raise UnknownFileTransportError, "Unknown file transport `#{app.opts[:file_transport]}' configured, valid are `ruby' and `native'!"
             end
           end
 
@@ -247,6 +259,9 @@ module DbSucker
 
           def _l_wait_for_workers
             @perform << "l_import_file_deferred"
+            unless Thread.current[:managed_worker] == :main
+              Thread.main.sync { Thread.main[:summon_workers] += 1 }
+            end
             wait_defer_ready
           end
 
@@ -302,7 +317,7 @@ module DbSucker
                 end
               }
             else
-              raise ImporterNotFoundError, "variation `#{cfg.name}/#{name}' defines unknown importer `#{imp}' (in `#{cfg.src}')"
+              raise Container::Variation::ImporterNotFoundError, "variation `#{@var.cfg.name}/#{@var.name}' defines unknown importer `#{imp}' (in `#{@var.cfg.src}')"
             end
             t.join
           end

data/lib/db_sucker/application/window.rb

@@ -4,9 +4,6 @@ module DbSucker
       include Core
       include Curses
       COLOR_GRAY = 8
-      COL1 = 20
-      COL2 = 25
-      COL3 = 20
       OutputHelper.hook(self)
 
       attr_reader :app, :sklaventreiber, :keypad, :tick, :spinner_frames
@@ -193,7 +190,7 @@ module DbSucker
         if opts[:threads]
           next_line
           yellow "       Threads: "
-          blue "#{Thread.list.length} ".ljust(COL1, " ")
+          blue "#{Thread.list.length}".ljust(3, " ")
         end
 
         if opts[:started]
@@ -292,7 +289,7 @@ module DbSucker
 
         puts
         puts c("        Status: ") << c(sklaventreiber.status[0], sklaventreiber.status[1].presence || "red")
-        puts c("       Threads: ") << c("#{Thread.list.length} ".ljust(COL1, " "), :blue)
+        puts c("       Threads: ") << c("#{Thread.list.length} ", :blue)
         puts c("       Started: ") << c("#{@app.boot}", :blue) << c(" (") << c(human_seconds(Time.current - app.boot), :blue) << c(")")
         puts c("Transaction ID: ") << c("#{sklaventreiber.trxid}", :cyan)
         puts c("      Database: ") << c(t_db || "?", :magenta) << c(" (transferred ") << c(t_total || "?", :blue) << c(" of ") << c(t_done || "?", :blue) << c(" tables)")

data/lib/db_sucker/application/window/core.rb

@@ -83,6 +83,10 @@ module DbSucker
           init_pair(Window::COLOR_GRAY, 0, -1)
         end
 
+        def flashbang
+          flash
+        end
+
         def set_cursor visibility
           curs_set(visibility)
         end

data/lib/db_sucker/application/window/keypad.rb

@@ -20,12 +20,14 @@ module DbSucker
             ["T", "create core dump and open in editor"],
             ["q", "quit prompt"],
             ["Q", "same as ctrl-c"],
+            ["S", "signal/wakeup all threads"],
             [":", "main prompt"],
           ],
           main_commands: [
             [["?", %w[h elp]], [], "shows this help"],
             [[%w[q uit]], [], "quit prompt"],
             [["q!", "quit!"], [], "same as ctrl-c"],
+            [["signal-threads"], [], "signal/wakeup all threads"],
             [["kill"], [], "(dirty) interrupts all workers"],
             [["kill!"], [], "(dirty) essentially SIGKILL (no cleanup)"],
             [["dump"], [], "create and open coredump"],
@@ -50,6 +52,7 @@ module DbSucker
               when "?" then show_help
               when "q" then quit_dialog
               when "Q" then $core_runtime_exiting = 1
+              when "S" then signal_threads
               end
             end
           end
@@ -71,6 +74,8 @@ module DbSucker
               when "eval" then args.any? ? _eval(args.join(" ")) : eval_prompt
               when "p", "pause" then pause_workers(args)
               when "r", "resume" then resume_workers(args)
+              when "signal-threads" then signal_threads
+              else app.print("\a")
             end
           end
         end
@@ -125,6 +130,7 @@ module DbSucker
 
         def pause_workers args
           if args[0].is_a?(String)
+            window.flashbang
             _detect_worker(args.join(" "), &:pause)
           else
             prompt!("Usage: :p(ause) <table_name|--all>", color: :yellow)
@@ -133,6 +139,7 @@ module DbSucker
 
         def resume_workers args
           if args[0].is_a?(String)
+            window.flashbang
             _detect_worker(args.join(" "), &:unpause)
           else
             prompt!("Usage: :r(esume) <table_name|--all>", color: :yellow)
@@ -141,6 +148,7 @@ module DbSucker
 
         def cancel_workers args
           if args[0].is_a?(String)
+            window.flashbang
             _detect_worker(args.join(" ")) do |wrk|
               wrk.cancel! "canceled by user"
             end
@@ -150,16 +158,29 @@ module DbSucker
         end
 
         def kill_ssh_poll
-          return if sklaventreiber.workers.select{|w| !w.done? || w.sshing }.any?
-          sklaventreiber.poll.try(:kill)
+          if sklaventreiber.workers.select{|w| !w.done? || w.sshing }.any?
+            app.print("\a")
+            prompt!("Error: cannot kill SSH poll whilst in use", color: :red)
+          else
+            window.flashbang
+            sklaventreiber.poll.try(:kill)
+          end
         end
 
         def kill_workers
+          window.flashbang
           Thread.list.each do |thr|
             thr.raise(Interrupt) if thr[:managed_worker]
           end
         end
 
+        def signal_threads
+          window.flashbang
+          Thread.list.each do |thr|
+            thr.signal if thr.respond_to?(:signal)
+          end
+        end
+
         def kill_app
           exit!
         end
@@ -171,4 +192,3 @@ module DbSucker
     end
   end
 end
-

data/lib/db_sucker/application/window/keypad/core.rb

@@ -72,7 +72,7 @@ module DbSucker
               begin
                 f.puts("#{evil}\n\n")
                 f.puts(app.sync{ app.instance_eval(evil) })
-              rescue StandardError => ex
+              rescue Exception => ex
                 f.puts("#{ex.class}: #{ex.message}")
                 ex.backtrace.each {|l| f.puts("  #{l}") }
               end

data/lib/db_sucker/patches/developer.rb

@@ -12,7 +12,7 @@ module DbSucker
             end
             app.debug "IO-stats: {#{iostats * ", "}}"
           end
-          app.dump_core if Thread.list.length > 1
+          app.dump_core if app.filtered_threads.length > 1
           app.debug "RSS: #{app.human_bytes(`ps h -p #{Process.pid} -o rss`.strip.split("\n").last.to_i * 1024)}"
         end
 

data/lib/db_sucker/version.rb

@@ -1,4 +1,4 @@
 module DbSucker
-  VERSION = "3.0.1"
+  VERSION = "3.2.1"
   UPDATE_URL = "https://raw.githubusercontent.com/2called-chaos/db_sucker/master/VERSION"
 end

metadata

@@ -1,14 +1,14 @@
 --- !ruby/object:Gem::Specification
 name: db_sucker
 version: !ruby/object:Gem::Version
-  version: 3.0.1
+  version: 3.2.1
 platform: ruby
 authors:
 - Sven Pachnit
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2018-02-05 00:00:00.000000000 Z
+date: 2021-02-13 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: curses
@@ -175,6 +175,7 @@ files:
 - lib/db_sucker/application/sklaven_treiber/worker/io/file_shasum.rb
 - lib/db_sucker/application/sklaven_treiber/worker/io/pv_wrapper.rb
 - lib/db_sucker/application/sklaven_treiber/worker/io/sftp_download.rb
+- lib/db_sucker/application/sklaven_treiber/worker/io/sftp_native_download.rb
 - lib/db_sucker/application/sklaven_treiber/worker/io/throughput.rb
 - lib/db_sucker/application/sklaven_treiber/worker/routines.rb
 - lib/db_sucker/application/slot_pool.rb
@@ -209,8 +210,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubyforge_project: 
-rubygems_version: 2.7.3
+rubygems_version: 3.1.2
 signing_key: 
 specification_version: 4
 summary: Sucks your remote databases via SSH for local tampering.