qik-loops 2.1.0

data/.gitignore

@@ -0,0 +1,8 @@
+.DS_Store
+doc
+pkg
+.yardoc
+.idea
+spec/rails/logs
+spec/rails/tmp
+*.pid

data/LICENSE

@@ -0,0 +1,21 @@
+The MIT License
+
+Copyright (c) 2008-2009 Alexey Kovyrin
+
+Permission is hereby granted, free of charge, to any person obtaining a copy
+of this software and associated documentation files (the "Software"), to deal
+in the Software without restriction, including without limitation the rights
+to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
+copies of the Software, and to permit persons to whom the Software is
+furnished to do so, subject to the following conditions:
+
+The above copyright notice and this permission notice shall be included in
+all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
+IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
+FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
+AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
+LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
+OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN
+THE SOFTWARE.

data/README.rdoc

@@ -0,0 +1,259 @@
+= Simple background loops framework
+
+<tt>loops</tt> is a small and lightweight framework for Ruby on Rails, Merb and other ruby
+frameworks created to support simple background loops in your application which are usually
+used to do some background data processing on your servers (queue workers, batch tasks
+processors, etc).
+
+_Warning_: If you use some pre-2.0 version of this plugin, read a dedicated paragraph below.
+
+
+== What would you use it for?
+
+Originally loops plugin was created to make our own loops code a bit more organized. We used
+to have dozens of different modules with methods that were called with script/runner and then
+used with nohup and other painful backgrounding techniques. When you have such a number of
+loops/workers to run in background it becomes a nightmare to manage them on a regular basis
+(restarts, code upgrades, status/health checking, etc).
+
+After a few takes on writing our scripts in a more organized way we were able to generalize most
+of the code so now our loops started looking like a classes with a single mandatory public method
+called *run*. Everything else (spawning many workers, managing them, logging, backgrounding,
+pid-files management, etc) is handled by the plugin itself.
+
+
+== But there are dozens of libraries like this! Why do we need yet another one?
+
+The major idea behind this small project was to create a deadly simple and yet robust framework to
+be able to run some tasks in background and do not think about spawning many workers, restarting
+them when they die, etc. So, if you need to be able to run either one or many copies of your worker
+and you do not want to think about re-spawning your scripts when they die and you do not want to
+spend megabytes of RAM on separate copies of Ruby interpreter (when you run each copy of your
+loop as a separate process controlled by monit/god/etc), then you should try this framework --
+you're going to like it.
+
+
+== How to install?
+
+There are two options when approaching db-charmer installation:
+
+* using the gem (recommended)
+* install as a Rails plugin
+
+To install as a gem, add this to your environment.rb:
+
+    config.gem 'loops'
+
+And then run the command:
+
+    sudo rake gems:install
+
+To install loops as a Rails plugin you need to do rhw following:
+
+    ./script/plugin install git://github.com/kovyrin/loops.git
+
+This will install the whole package in your vendor/plugins directory.
+For merb applications, just check out the code and place it to the vendor/plugins directory.
+
+
+After you are done with the installation, you need to generate binary and configuration
+files by running:
+
+    ./script/generate loops
+
+This will create the following list of files:
+* <tt>./script/loops</tt> - binary file that will be used to manage your loops
+* <tt>./config/loops.yml</tt> - example configuration file
+* <tt>./app/loops/simple.rb</tt> - REALLY simple loop example
+* <tt>./app/loops/queue_loop.rb</tt> - simple ActiveMQ queue worker
+
+
+== How to use?
+
+Here is a simple loop scaffold for you to start from (put this file to
+<tt>app/loops/hello_world_loop.rb</tt>):
+
+    class HelloWorldLoop < Loops::Base
+      def run
+        with_period_of(1) do # period is in seconds
+          debug("Hello, debug log!")
+          sleep(config['sleep_period']) # Do something "useful" and make it configurable
+          debug("Hello, debug log (yes, once again)!")
+        end
+      end
+    end
+
+When you have your loop ready to use, add the following lines to your (maybe empty yet)
+<tt>config/loops.yml</tt> file:
+
+    hello_world:
+      type: simple
+      sleep_period: 10
+
+This is it! To manage your loop, just run one of the following commands:
+
+* To list all configured loops:
+
+    $ ./script/loops list
+
+* To run all enabled (actually non-disabled) loops in foreground:
+
+    $ ./script/loops start
+
+* To run all enabled loops in background:
+
+    $ ./script/loops start -d
+
+* To run specific loop in background:
+
+    $ ./script/loops start hello_world -d
+
+* To see all possible options:
+
+    $ ./script/loops help
+
+
+_Notice:_ If you use loops as a gem, you will get a system binary called <tt>loops</tt> which you
+could use instead of your <tt>./script/loops<tt> binary:
+
+    $ loops help
+
+
+== How to run more than one worker?
+
+If you want to have more than one copy of your worker running, that is as simple as adding one
+option to your loop configuration:
+
+    hello_world:
+      type: simple
+      sleep_period: 10
+      workers_number: 1
+
+This <tt>workers_number</tt> option would tell loops manager to spawn more than one copy of
+your loop and run them in parallel. The only thing you'd need to do is to remember about
+concurrent work of your loops. For example, if you have some kind of database table with elements
+you need to process, you can create a simple database-based locks system or use any
+memcache-based locks.
+
+
+== How to run more than one loop using the same class?
+
+You can run the same loop class with different configuration parameters by explicitly identifying
+the loop class to execute:
+
+    hello:
+      type: simple
+      loop_name: some_module/my_worker
+      language: English
+
+    salut:
+      type: simple
+      loop_name: some_module/my_worker
+      language: French
+
+Now two independent sets of loops are using the same class <tt>SomeModule::MyWorkerLoop</tt>
+customized by the language parameter.
+
+
+== I want to keep my loop running on machine reboots. How to do it?
+
+We use monit to keep loop monitors runnings. You could use something like this in your configs:
+
+    check process loop-slow_logs with pidfile /your/project/current/tmp/pids/loop-slow_logs.pid
+      group loops
+      start program "/your/project/current/script/loops start slow_logs -e loops -p tmp/pids/loop-slow_logs.pid -d"
+      stop  program "/your/project/current/script/loops stop  slow_logs -e loops -p tmp/pids/loop-slow_logs.pid"
+
+
+== ActiveMQ-based workers? What's that?
+
+In some of our worker loops we poll ActiveMQ queue and process its items to perform some
+asynchronous operations. So, to make it simpler for us to create such a workers, we've created
+really simple loops class extension that wraps your code with basic queue polling/acknowledging
+code and as the result, you can create a loops like this:
+
+    class MyQueueLoop < Loops::Queue
+      def process_message(message)
+        debug "Received a message: #{message.body}"
+        debug "sleeping..."
+        sleep(0.5 + rand(10) / 10.0) # do something "useful" with the message :-)
+        debug "done..."
+      end
+    end
+
+With configs like this:
+
+    # An example of a STOMP queue-based loop
+    my_queue:
+      type: queue
+      host: 127.0.0.1
+      port: 61613
+      queue_name: blah
+
+Of course, this solution scales perfectly and to make your queue processing faster you just
+need to add more workers (by adding <tt>workers_number: N</tt> option).
+
+_Warning_: This type of loops requires you to have the <tt>stomp</tt> gem installed in your
+system.
+
+
+== There is this <tt>workers_engine</tt> option in the config file. What do you use it for?
+
+There are two so called "workers engines" in loops: <tt>fork</tt> and <tt>thread</tt>.
+They're used to control the way process manager would spawn new loops workers:
+with the <tt>fork</tt> engine we'll load all the loops classes and then fork ruby interpreter
+as many times as many workers we need. With the <tt>thread</tt> engine we'd do Thread.new
+instead of forking. Thread engine could be useful if you are sure your loop won't lock
+ruby interpreter (it does not do native calls, etc) or if you use some interpreter that does
+not support forks (like jruby).
+
+The default engine is <tt>fork</tt>.
+
+
+== What Ruby implementations does it work for?
+
+We've tested and used the plugin on MRI 1.8.6/1.8.7 and on JRuby 1.4.0. At this point we do
+not support demonization in JRuby and never tested the code on Ruby 1.9. Obviously because
+of JVM limitations you won't be able to use <tt>fork</tt> workers engine in JRuby, but
+threaded workers do pretty well.
+
+Recommended version of ruby ro run loops is Ruby Enterprise Edition. This is because we
+have a support for their Copy-On-Write friendly garbage collector that makes your loops
+much smaller in memory (since they share the most of the code). Even with one loop process
+you'd save some memory because your loop monitor process would share most of the memory with
+the loop itself. When you run on RubyEE, you could use <tt>loops stats</tt> command to get
+detailed loops memory stats:
+
+        [root@analyics current]# ./script/loops stats
+
+        --------- Loops processes ----------
+        PID   PPID  VMSize    Private  Name
+        ------------------------------------
+        9062  1     199.3 MB  32.4 MB  loops monitor: activemq
+        9234  9062  211.9 MB  37.5 MB  loop worker: activemq
+        9251  9062  213.3 MB  38.4 MB  loop worker: activemq
+        9267  9062  211.9 MB  37.1 MB  loop worker: activemq
+        9268  9062  211.9 MB  38.0 MB  loop worker: activemq
+        ### Processes: 5
+        ### Total private dirty RSS: 183.33 MB
+
+
+
+== Migrating from pre-2.0 releases
+
+Before version 2.0 has been released, this code was developed as a Rails plugin only and
+did not have any versions numbering system in place. So we call all those old versions
+a pre-2.0 releases. If you use one of those relases (if your loops plugin does not have
+the <tt>VERSION.yml</tt> file in the root directory), be careful when upgrading because
+there are a few incompatible changes we have made in the loops command: <tt>-h</tt>,
+<tt>-a</tt>, <tt>-s</tt>, <tt>-L</tt> and <tt>-l</tt> options were deprecated
+and replaced with a friendlier word commands.
+Use <tt>loops help</tt> to get help.
+
+
+== Who are the authors?
+
+This plugin has been created in Scribd.com for our internal use and then the sources were opened
+for other people to use. All the code in this package has been developed by Alexey Kovyrin,
+Dmytro Shteflyuk and Alexey Verkhovsky for Scribd.com and is released under the MIT license.
+For more details, see LICENSE file.

data/Rakefile

@@ -0,0 +1,48 @@
+require 'rake'
+
+begin
+  require 'jeweler'
+  Jeweler::Tasks.new do |gemspec|
+    gemspec.name        = 'qik-loops'
+    gemspec.summary     = 'Simple background loops framework for ruby (qik.com)'
+    gemspec.description = 'Loops is a small and lightweight framework for Ruby on Rails, Merb and other ruby frameworks created to support simple background loops in your application which are usually used to do some background data processing on your servers (queue workers, batch tasks processors, etc).'
+    gemspec.email       = 'gleb.pomykalov@qik.com'
+    gemspec.homepage    = 'http://github.com/glebpom/loops'
+    gemspec.authors     = ['Alexey Kovyrin', 'Dmytro Shteflyuk', 'Gleb Pomykalov']
+    gemspec.files.include ['lib/**/*']
+  end
+  Jeweler::GemcutterTasks.new
+rescue LoadError
+  puts 'Jeweler not available. Install it with: sudo gem install jeweler'
+end
+
+begin
+  require 'spec/rake/spectask'
+
+  desc 'Default: run unit tests.'
+  task :default => :spec
+
+  desc 'Test the loops plugin.'
+  Spec::Rake::SpecTask.new(:spec) do |t|
+    t.libs << 'lib'
+    t.pattern = 'spec/**/*_spec.rb'
+    t.verbose = true
+    t.spec_opts = ['-cfs']
+  end
+rescue LoadError
+  puts 'RSpec not available. Install it with: sudo gem install rspec'
+end
+
+begin
+  require 'yard'
+  YARD::Rake::YardocTask.new(:yard) do |t|
+    t.options = ['--title', 'Loops Documentation']
+    if ENV['PRIVATE']
+      t.options.concat ['--protected', '--private']
+    else
+      t.options.concat ['--protected', '--no-private']
+    end
+  end
+rescue LoadError
+  puts 'Yard not available. Install it with: sudo gem install yard'
+end

data/VERSION.yml

@@ -0,0 +1,5 @@
+--- 
+:minor: 1
+:patch: 0
+:build: 
+:major: 2

data/bin/loops

@@ -0,0 +1,16 @@
+#!/usr/bin/env ruby
+$:.unshift(File.dirname(__FILE__) + '/../lib') unless $:.include?(File.dirname(__FILE__) + '/../lib')
+
+require File.dirname(__FILE__) + '/../lib/loops'
+
+begin
+  # The dup is to keep ARGV intact, so that tools like ruby-debug can respawn.
+  failure = Loops::CLI.execute
+  Kernel.exit(failure ? 1 : 0)
+rescue SystemExit => e
+  Kernel.exit(e.status)
+rescue Exception => e
+  STDERR.puts("Unhandled exception: #{e.message} (#{e.class})")
+  STDERR.puts(e.backtrace.join("\n"))
+  Kernel.exit(1)
+end

data/bin/loops-memory-stats

@@ -0,0 +1,259 @@
+#!/usr/bin/env ruby
+
+#
+# This script is based on Phusion's passenger-memory-stats
+# Almost all of the code is Copyright (c) 2008, 2009 Phusion
+# For more details go to http://www.modrails.com/
+#
+
+# ANSI color codes
+RESET   = "\e[0m"
+BOLD    = "\e[1m"
+WHITE   = "\e[37m"
+YELLOW  = "\e[33m"
+BLUE_BG = "\e[44m"
+
+# Container for tabular data.
+class Table
+	def initialize(column_names)
+		@column_names = column_names
+		@rows = []
+	end
+	
+	def add_row(values)
+		@rows << values.to_a
+	end
+	
+	def add_rows(list_of_rows)
+		list_of_rows.each do |row|
+			add_row(row)
+		end
+	end
+	
+	def remove_column(name)
+		i = @column_names.index(name)
+		@column_names.delete_at(i)
+		@rows.each do |row|
+			row.delete_at(i)
+		end
+	end
+	
+	def to_s(title = nil)
+		max_column_widths = [1] * @column_names.size
+		(@rows + [@column_names]).each do |row|
+			row.each_with_index do |value, i|
+				max_column_widths[i] = [value.to_s.size, max_column_widths[i]].max
+			end
+		end
+		
+		format_string = max_column_widths.map{ |i| "%#{-i}s" }.join("  ")
+		header = sprintf(format_string, *@column_names).rstrip << "\n"
+		if title
+			free_space = header.size - title.size - 2
+			if free_space <= 0
+				left_bar_size = 3
+				right_bar_size = 3
+			else
+				left_bar_size = free_space / 2
+				right_bar_size = free_space - left_bar_size
+			end
+			result = "#{BLUE_BG}#{BOLD}#{YELLOW}\n"
+			result << "#{"-" * left_bar_size} #{title} #{"-" * right_bar_size}\n"
+			if !@rows.empty?
+				result << WHITE
+				result << header
+			end
+		else
+			result = header.dup
+		end
+		if @rows.empty?
+			result << RESET
+		else
+			result << ("-" * header.size) << "#{RESET}\n"
+			@rows.each do |row|
+				result << sprintf(format_string, *row).rstrip << "\n"
+			end
+		end
+		result
+	end
+end
+
+class MemoryStats
+	class Process
+		attr_accessor :pid
+		attr_accessor :ppid
+		attr_accessor :threads
+		attr_accessor :vm_size              # in KB
+		attr_accessor :rss                  # in KB
+		attr_accessor :name
+		attr_accessor :private_dirty_rss    # in KB
+		
+		def vm_size_in_mb
+			return sprintf("%.1f MB", vm_size / 1024.0)
+		end
+
+		def rss_in_mb
+			return sprintf("%.1f MB", rss / 1024.0)
+		end
+		
+		def private_dirty_rss_in_mb
+			if private_dirty_rss.is_a?(Numeric)
+				return sprintf("%.1f MB", private_dirty_rss / 1024.0)
+			else
+				return "?"
+			end
+		end
+		
+		def to_a
+			return [pid, ppid, vm_size_in_mb, private_dirty_rss_in_mb, rss_in_mb, name]
+		end
+	end
+	
+	def start
+		loops_processes = list_processes(:match =>
+			/^loops* (monitor|worker)/)
+		print_process_list("Loops processes", loops_processes, :show_ppid => true)
+		
+		if RUBY_PLATFORM !~ /linux/
+			puts
+			puts "*** WARNING: The private dirty RSS can only be displayed " <<
+				"on Linux. You're currently using '#{RUBY_PLATFORM}'."
+		elsif ::Process.uid != 0 && loops_processes.any?{ |p| p.private_dirty_rss.nil? }
+			puts
+			puts "*** WARNING: Please run this tool as root. Otherwise the " <<
+				"private dirty RSS of processes cannot be determined."
+		end
+	end
+	
+	# Returns a list of Process objects that match the given search criteria.
+	#
+	#  # Search by executable path.
+	#  list_processes(:exe => '/usr/sbin/apache2')
+	#  
+	#  # Search by executable name.
+	#  list_processes(:name => 'ruby1.8')
+	#  
+	#  # Search by process name.
+	#  list_processes(:match => 'Passenger FrameworkSpawner')
+	def list_processes(options)
+		if options[:exe]
+			name = options[:exe].sub(/.*\/(.*)/, '\1')
+			if RUBY_PLATFORM =~ /linux/
+				ps = "ps -C '#{name}'"
+			else
+				ps = "ps -A"
+				options[:match] = Regexp.new(Regexp.escape(name))
+			end
+		elsif options[:name]
+			if RUBY_PLATFORM =~ /linux/
+				ps = "ps -C '#{options[:name]}'"
+			else
+				ps = "ps -A"
+				options[:match] = Regexp.new(" #{Regexp.escape(options[:name])}")
+			end
+		elsif options[:match]
+			ps = "ps -A"
+		else
+			raise ArgumentError, "Invalid options."
+		end
+		
+		processes = []
+		case RUBY_PLATFORM
+		when /solaris/
+			list = `#{ps} -o pid,ppid,nlwp,vsz,rss,comm`.split("\n")
+			threads_known = true
+		when /darwin/
+			list = `#{ps} -w -o pid,ppid,vsz,rss,command`.split("\n")
+			threads_known = false
+		else
+			list = `#{ps} -w -o pid,ppid,nlwp,vsz,rss,command`.split("\n")
+			threads_known = true
+		end
+		list.shift
+		list.each do |line|
+			line.gsub!(/^ */, '')
+			line.gsub!(/ *$/, '')
+			
+			p = Process.new
+			if threads_known
+				p.pid, p.ppid, p.threads, p.vm_size, p.rss, p.name = line.split(/ +/, 6)
+			else
+				p.pid, p.ppid, p.vm_size, p.rss, p.name = line.split(/ +/, 5)
+				p.threads = "?"
+			end
+			p.name.sub!(/\Aruby: /, '')
+			p.name.sub!(/ \(ruby\)\Z/, '')
+			if p.name !~ /^ps/ && (!options[:match] || p.name.match(options[:match]))
+				# Convert some values to integer.
+				[:pid, :ppid, :vm_size, :rss].each do |attr|
+					p.send("#{attr}=", p.send(attr).to_i)
+				end
+				p.threads = p.threads.to_i if threads_known
+
+				if platform_provides_private_dirty_rss_information?
+					p.private_dirty_rss = determine_private_dirty_rss(p.pid)
+				end
+				processes << p
+			end
+		end
+		return processes
+	end
+
+private
+	def platform_provides_private_dirty_rss_information?
+		return RUBY_PLATFORM =~ /linux/
+	end
+
+	# Returns the private dirty RSS for the given process, in KB.
+	def determine_private_dirty_rss(pid)
+		total = 0
+		File.read("/proc/#{pid}/smaps").split("\n").each do |line|
+			line =~ /^(Private)_Dirty: +(\d+)/
+			if $2
+				total += $2.to_i
+			end
+		end
+		if total == 0
+			return nil
+		else
+			return total
+		end
+	rescue Errno::EACCES, Errno::ENOENT
+		return nil
+	end
+	
+	def print_process_list(title, processes, options = {})
+		table = Table.new(%w{PID PPID VMSize Private Resident Name})
+		table.add_rows(processes)
+		if options.has_key?(:show_ppid) && !options[:show_ppid]
+			table.remove_column('PPID')
+		end
+		if platform_provides_private_dirty_rss_information?
+			table.remove_column('Resident')
+		else
+			table.remove_column('Private')
+		end
+		puts table.to_s(title)
+		
+		if platform_provides_private_dirty_rss_information?
+			total_private_dirty_rss = 0
+			some_private_dirty_rss_cannot_be_determined = false
+			processes.each do |p|
+				if p.private_dirty_rss.is_a?(Numeric)
+					total_private_dirty_rss += p.private_dirty_rss
+				else
+					some_private_dirty_rss_cannot_be_determined = true
+				end
+			end
+			puts   "### Processes: #{processes.size}"
+			printf "### Total private dirty RSS: %.2f MB", total_private_dirty_rss / 1024.0
+			if some_private_dirty_rss_cannot_be_determined
+				puts " (?)"
+			else
+				puts
+			end
+		end
+	end
+end
+
+MemoryStats.new.start