webapp_worker 0.0.2 → 0.0.3

data/README.markdown

@@ -1,6 +1,6 @@
 ## Information
 
-Provides a way to have workers on your webapp servers, espeically useful for webapps that tend to scale up and down with X amount of servers in the load balancer.  Also good way to not use another dependent resource like a job scheduler/queue.  Keeps your application all packaged up nicely, no cron jobs to set, nothing else to think about setting up and nothing else to maintain.
+Provides a way to have workers on your webapp servers, especially useful for webapps that tend to scale up and down with X amount of servers in the load balancer.  Also good way to not use another dependent resource like a job scheduler/queue.  Keeps your application all packaged up nicely, no cron jobs to set, nothing else to think about setting up and nothing else to maintain.
 
 ## Installation
 
@@ -14,8 +14,10 @@ or use it in your Gemfile
 
 	waw -e development -f jobs.yml (parses data)
 	waw -e development -f jobs.yml  -j (parses and shows jobs)
-	waw -e development -f jobs.yml  -n 1 (parses and shows next X number of jobs that will run)
+	waw -e development -f jobs.yml  -n 4 (parses and shows the next N run times, by job)
 	waw -e development -f jobs.yml -r (parses and starts to run)
+	waw -e development -f jobs.yml -d (parses and turns on debugging)
+	waw -e development -f jobs.yml -v (parses and turns on verbose logging)
 
 ## Using in your webapp
 
@@ -45,39 +47,51 @@ You don't have to use a jobs file, you can specify the yaml or hash in the code
 
 ## Example Output of waw
 
+See if Webapp Worker can parse the jobs file and show you all the jobs it can run from the file
+
 	$ waw -e local -f config/jobs.yml -j
 	Job File: config/jobs.yml
 
 	Host: localhost
 	Mailto:
 	Environment: development
-	Amount of Jobs: 9
-
-	Command to Run: rake job:run
-	   Next Run: [2012-01-03 22:00:00 -0700]
-	Command to Run: rake job:run
-	   Next Run: [2012-01-03 22:02:00 -0700]
-	Command to Run: rake job:run
-	   Next Run: [2012-01-03 21:12:00 -0700]
-	Command to Run: rake job:run
-	   Next Run: [2012-01-03 21:14:00 -0700]
-	Command to Run: rake job:run
-	   Next Run: [2012-01-03 21:16:00 -0700]
-	Command to Run: rake job:run
-	   Next Run: [2012-01-03 21:18:00 -0700]
-	Command to Run: rake job:run
-	   Next Run: [2012-01-03 21:22:00 -0700]
-	Command to Run: rake job:run
-	   Next Run: [2012-01-03 21:24:00 -0700]
-	Command to Run: rake job:run
-	   Next Run: [2012-01-03 21:30:00 -0700]
+	Amount of Jobs: 3
+
+	Job: rake job:run
+	Job: rake job:run
+	Job: rake job:run
+
+See when the jobs are supposed to run next, next two times, next three times, etc...
+
+	$ waw -e local -f config/jobs.yml -n 4
+	Job: rake job:run
+	   Next Run Time(s): [2012-01-03 22:00:00 -0700, 2012-01-03 22:05:00 -0700, 2012-01-03 22:10:00 -0700, 2012-01-03 22:15:00 -0700]
+	Job: rake job:run
+	   Next Run Time(s): [2012-01-03 22:00:00 -0700, 2012-01-03 22:05:00 -0700, 2012-01-03 22:10:00 -0700, 2012-01-03 22:15:00 -0700]
+	Job: rake job:run
+
+	$ waw -e local -f config/jobs.yml -r (optional -d and -v for debug and verbose)
+	Running Jobs
+
+## Other Options
+
+You may want to know what version of the gem Webapp Worker is using, so just send a USR1 signal to it
+
+	$ kill -s USR1 11682
+	Webapp Worker Version: 0.0.3
+
+You may need to turn on debugging for the Webapp Worker while its running, so just send a USR2 signal to it
+
+	$ kill -s USR2 11682
+	Changed logger level to Debug
 
 ## Roadmap
 
-- Process also needs to understand when to die and to start back up. (new version being used in the web app server)
+- Use the mailto attribute in application to actually do something
 - Start having the webapp worker registering to a central point or do UDP mutlicasting to find each other.
 - Once self registering is enabled, webapp_workers need to communicate effectively.
-- Once communication is esatablished webapp_workers need to do the scheduling for themselves.
+- Once communication is established webapp_workers need to do the scheduling for themselves.
+- Do logging for each type of job in a jobs directory under tmp/webapp_worker, allowing for troubleshooting.
 - Spit out reports of the different jobs and how fast they run.
 
 ## Contributing

data/bin/waw

@@ -15,7 +15,7 @@ where [options] are:
   opt :jobfile, "A YAML config file", :type => String, :short => "-f"
   opt :run, "Run the jobs", :default => false, :short => "-r"
   opt :jobs, "Show the jobs", :default => false, :short => "-j"
-  opt :nextrun, "Find the next possible command(s) to run (i.e. 1,2...)", :type => Integer, :short => "-n"
+  opt :nextrun, "Find the next N run times, by job (use an integer)", :type => Integer, :short => "-n"
   opt :debug, "Local Debug", :short => "-d"
   opt :verbose, "Verbose Output", :short => "-v"
 end
@@ -30,8 +30,8 @@ a.parse_yaml(job_file)
 
 if opts[:nextrun] != nil
 	a.next_command_run?(opts[:nextrun]).each do |command,time|
-		puts "Next Command Run: #{command}"
-		puts "   Next Run: #{time}"
+		puts "Job: #{command}"
+		puts "   Next Run Time(s): #{time}"
 	end
 elsif opts[:run] == false && opts[:jobs] == false
 	puts
@@ -49,8 +49,7 @@ elsif opts[:run] == false && opts[:jobs] == true
 	puts
 	a.jobs.each do |job|
 		j = WebappWorker::Job.new(job)
-		puts "Command to Run: #{j.command}"
-		puts "   Next Run: #{j.next_run?}"
+		puts "Job: #{j.command}"
 	end
 else
 	puts "Running Jobs"

data/lib/webapp_worker.rb

@@ -2,6 +2,7 @@ require "webapp_worker/version"
 
 require "webapp_worker/application"
 require "webapp_worker/job"
+require "webapp_worker/system"
 
 module WebappWorker
 end

data/lib/webapp_worker/application.rb

@@ -1,20 +1,6 @@
 require 'socket'
 require 'timeout'
 require 'open4'
-require 'logger'
-
-module Process
-  class << self
-    def alive?(pid)
-      begin
-        Process.kill(0, pid.to_i)
-        true
-      rescue Errno::ESRCH
-        false
-      end
-    end
-  end
-end
 
 module WebappWorker
 	class Application
@@ -44,6 +30,44 @@ module WebappWorker
 			return Socket.gethostname.downcase
 		end
 
+		def check_file_modification_time
+			mtime = File.mtime(@file)
+
+			if mtime != @file_mtime
+				@file_mtime = mtime
+				self.parse_yaml(@file)
+			end
+		end
+
+		def next_command_run?(til)
+			commands = {}
+			c = {}
+			next_commands = {}
+			new_jobs = []
+
+			(0..til).each do |i|
+				@jobs.each do |j|
+					new_jobs << j
+				end
+			end
+
+			new_jobs.flatten.each do |job|
+				j = WebappWorker::Job.new(job)
+				commands.store(j.command,j.next_runs?(til))
+			end
+
+			(commands.sort_by { |key,value| value }).collect { |key,value| c.store(key,value) }
+
+			counter = 0
+			c.each do |key,value|
+				next_commands.store(key,value)
+				counter = counter + 1
+				break if counter >= @jobs.length
+			end
+
+			return next_commands
+		end
+
 		def next_command_run_time?
 			commands = {}
 			c = {}
@@ -79,162 +103,26 @@ module WebappWorker
 			return next_commands
 		end
 
-		def check_file_modification_time
-			mtime = File.mtime(@file)
-
-			if mtime != @file_mtime
-				@file_mtime = mtime
-				self.parse_yaml(@file)
-			end
-		end
-
-		def check_for_directory
-			dir = "/tmp/webapp_worker"
-
-			if Dir.exists?(dir)
-			else
-				Dir.mkdir(dir, 0700)
-			end
-		end
-
-		def create_pid(logger)
-			logger.info "Creating Pid File at /tmp/webapp_worker/waw.pid"
-
-			File.open("/tmp/webapp_worker/waw.pid", 'w') { |f| f.write(Process.pid) }
-			$0="Web App Worker - Job File: #{@file}"
-
-			logger.info "Pid File created: #{Process.pid} at /tmp/webapp_worker/waw.pid"
-		end
-
-		def check_for_process(logger)
-			file = "/tmp/webapp_worker/waw.pid"
-
-			if File.exists?(file)
-				possible_pid = ""
-				pid_file = File.open(file, 'r').each { |f| possible_pid+= f }
-				pid_file.close
-
-				if Process.alive?(possible_pid)
-					puts "Already found webapp_worker running, pid is: #{possible_pid}, exiting..."
-					logger.fatal "Found webapp_worker already running with pid: #{possible_pid}, Pid File: #{file} exiting..."
-					exit 1
-				else
-					logger.warn "Found pid file, but no process running, recreating pid file with my pid: #{Process.pid}"
-					File.delete(file)
-					self.create_pid(logger)
-				end
-			else
-				self.create_pid(logger)
-			end
-
-			logger.info "Starting Webapp Worker"
-		end
-
-		def graceful_termination(logger)
-			stop_loop = true
-
-			begin
-				puts
-				puts "Graceful Termination started, waiting 60 seconds before KILL signal send"
-				logger.info "Graceful Termination started, waiting 60 seconds before KILL signal send"
-
-				Timeout::timeout(60) do
-					@command_processes.each do |pid,command|
-						logger.debug "Sending INT Signal to #{command} Process with PID: #{pid}"
-						begin
-							Process.kill("INT",pid.to_i)
-						rescue => error
-						end
-					end
-
-					@threads.each do |thread,command|
-						thread.join
-					end
-				end
-			rescue Timeout::Error
-				puts "Graceful Termination bypassed, killing processes and threads"
-				logger.info "Graceful Termination bypassed, killing processes and threads"
-
-				@command_processes.each do |pid,command|
-					logger.debug "Killing #{command} Process with PID: #{pid}"
-					begin
-						Process.kill("KILL",pid.to_i)
-					rescue => error
-					end
-				end
-
-				@threads.each do |thread,command|
-					logger.debug "Killing Command Thread: #{command}"
-					Thread.kill(thread)
-				end
-			end
-
-			puts "Stopping Webapp Worker"
-			logger.info "Stopping Webapp Worker"
-			file = "/tmp/webapp_worker/waw.pid"
-			File.delete(file)
-			exit 0
-		end
-
 		def run(debug=nil,verbose=nil)
-			self.check_for_directory
-
-			logger = Logger.new("/tmp/webapp_worker/#{@environment}.log", 5, 5242880)
-
-			if debug
-				logger.level = Logger::DEBUG
-			elsif verbose
-				logger.level = Logger::INFO
-			else
-				logger.level = Logger::WARN
-			end
-
 			p = Process.fork do
-				begin
-					self.check_for_process(logger)
-				rescue => error
-					puts error.inspect
-					logger.fatal error.inspect
-				end
-
-				@command_processes = {}
-				@threads = {}
-				stop_loop = false
-
-				Signal.trap('HUP', 'IGNORE')
+				#Some Setup Work
+				$0="WebApp Worker - Job File: #{@file}"
+				waw_system = WebappWorker::System.new
+				waw_system.setup(debug,verbose)
+				logger = waw_system.logger
 
 				%w(INT QUIT TERM TSTP).each do |sig|
 					Signal.trap(sig) do
+						stop_loop = true
 						logger.warn "Received a #{sig} signal, stopping current commands."
-						self.graceful_termination(logger)
+						waw_system.graceful_termination(@threads,@command_processes)
 					end
 				end
 
-				Signal.trap('USR1') do
-					version = WebappWorker::VERSION
-					puts
-					puts "Webapp Worker Version: #{version}"
-					logger.info "Received USR1 signal, sent version: #{version}"
-				end
-
-				Signal.trap('USR2') do
-					logger.level = Logger::DEBUG
-					puts
-					puts "Changed logger level to Debug"
-					logger.info "Changed logger level to Debug"
-				end
-
-				#Signal.trap('STOP') do |s|
-				#	#Stop Looping until
-				#	stop_loop = true
-				#	logger.warn "Received signal #{s}, pausing current loop."
-				#end
-				#
-				#Signal.trap('CONT') do
-				#	#Start Looping again (catch throw?)
-				#	stop_loop = false
-				#	logger.warn "Received signal #{s}, starting current loop."
-				#end
+				#WebApp Worker is setup now do the real work
+				@command_processes = {}
+				@threads = {}
+				stop_loop = false
 
 				logger.debug "Going into Loop"
 				until stop_loop
@@ -263,6 +151,8 @@ module WebappWorker
 								pid, stdin, stdout, stderr = Open4::popen4 command
 								@command_processes.store(pid,command)
 
+								#make logger log to a specific job file log file
+
 								ignored, status = Process::waitpid2 pid
 
 								if status.to_i == 0
@@ -299,6 +189,5 @@ module WebappWorker
 
 			Process.detach(p)
 		end
-
 	end
 end

data/lib/webapp_worker/system.rb

@@ -0,0 +1,205 @@
+require 'drb'
+require 'etc'
+require 'logger'
+
+module WebappWorker
+	class System
+
+		attr_accessor :user, :tmp_dir, :pid_file, :ipc_file, :logger
+
+		def initialize(user_supplied_hash={})
+			standard_hash = { user:"nobody", tmp_dir:"/tmp/webapp_worker/", pid_file:"/tmp/webapp_worker/waw.pid", ipc_file:"/tmp/webapp_worker/waw", logger:"" }
+
+			user_supplied_hash = {} unless user_supplied_hash
+			user_supplied_hash = standard_hash.merge(user_supplied_hash)
+
+			user_supplied_hash.each do |key,value|
+				self.instance_variable_set("@#{key}", value)
+				self.class.send(:define_method, key, proc{self.instance_variable_get("@#{key}")})
+				self.class.send(:define_method, "#{key}=", proc{|x| self.instance_variable_set("@#{key}", x)})
+			end
+		end
+
+    def self.process_alive?(pid)
+      begin
+        Process.kill(0, pid.to_i)
+        true
+      rescue Errno::ESRCH
+        false
+      end
+    end
+
+    def setup(debug=nil,verbose=nil)
+      self.check_for_directory
+      self.create_logger(debug,verbose)
+      self.check_for_process
+      self.start_listening
+
+      Signal.trap('HUP', 'IGNORE')
+
+      Signal.trap('USR1') do
+        version = WebappWorker::VERSION
+        puts
+        puts "Webapp Worker Version: #{version}"
+        logger.info "Received USR1 signal, sent version: #{version}"
+      end
+
+      Signal.trap('USR2') do
+        logger.level = Logger::DEBUG
+        puts
+        puts "Changed logger level to Debug"
+        logger.info "Changed logger level to Debug"
+      end
+    end
+
+		def check_for_directory
+			if Dir.exists?(@tmp_dir)
+			else
+				Dir.mkdir(@tmp_dir, 0700)
+			end
+		end
+
+    def create_logger(debug=nil,verbose=nil)
+      @logger = Logger.new("#{@tmp_dir}waw.log", 5, 5242880)
+
+      if debug
+        @logger.level = Logger::DEBUG
+      elsif verbose
+        @logger.level = Logger::INFO
+      else
+        @logger.level = Logger::WARN
+      end
+    end
+
+    def delete_files
+      @logger.fatal "Deleting both PID and IPC files"
+
+      begin
+				File.delete(@pid_file)
+        @logger.fatal "Deleted PID File: #{@pid_file}"
+      rescue => error
+        @logger.fatal "Error at Deleting PID File: #{@pid_file}: #{error}"
+      end
+
+      begin
+				File.delete(@ipc_file)
+        @logger.fatal "Deleted IPC File: #{@ipc_file}"
+      rescue => error
+        @logger.fatal "Error at Deleting IPC File: #{@ipc_file}: #{error}"
+      end
+    end
+
+		def create_pid
+      self.delete_files
+
+			@logger.info "Creating Pid File at #{@pid_file}"
+			File.open(@pid_file, 'w') { |f| f.write(Process.pid) }
+			@logger.info "Pid File created: #{Process.pid} at #{@pid_file}"
+		end
+
+		def check_for_process
+			if File.exists?(@pid_file)
+				possible_pid = ""
+				pf = File.open(@pid_file, 'r').each { |f| possible_pid+= f }
+				pf.close
+
+				if WebappWorker::System.process_alive?(possible_pid)
+          version = self.check_process_version
+          my_version = WebappWorker::VERSION
+
+					if version.to_s == my_version.to_s
+						puts "Already found webapp_worker running, pid is: #{possible_pid}, exiting..."
+						@logger.fatal "Found webapp_worker already running with pid: #{possible_pid}, Pid File: #{@pid_file} exiting..."
+						exit 1
+					else
+						puts "Found old version of webapp_worker #{version}, running with pid: #{possible_pid}, asking it to terminate since my version is: #{my_version}"
+						@logger.fatal "Found old version of webapp_worker #{version}, running with pid: #{possible_pid}, asking it to terminate since my version is: #{my_version}"
+
+						Process.kill("INT",possible_pid.to_i)
+
+						while WebappWorker::System.process_alive?(possible_pid)
+							sleep 120
+
+							begin
+                Process.kill("KILL",possible_pid.to_i)
+							rescue => error
+							end
+
+              self.create_pid
+						end
+					end
+				else
+					@logger.warn "Found pid file, but no process running"
+					self.create_pid
+				end
+			else
+				@logger.info "Did not find a pid file"
+				self.create_pid
+			end
+
+			@logger.info "Starting Webapp Worker"
+		end
+
+		def start_listening
+			@logger.info "Starting to listen on IPC: #{@ipc_file}"
+
+			DRb.start_service("drbunix:#{@ipc_file}", WebappWorker::VERSION)
+
+			@logger.info "Now listenting on IPC: #{@ipc_file}"
+		end
+
+		def check_process_version
+			@logger.info "Asking the processes version from IPC: #{@ipc_file}"
+
+			DRb.start_service
+			version = DRbObject.new_with("drbunix:#{@ipc_file}", nil)
+
+			@logger.info "The process' version is: #{version}"
+			return version
+		end
+
+		def graceful_termination(threads,command_processes)
+			begin
+				puts
+				puts "Graceful Termination started, waiting 60 seconds before KILL signal send"
+				@logger.info "Graceful Termination started, waiting 60 seconds before KILL signal send"
+
+				Timeout::timeout(60) do
+					command_processes.each do |pid,command|
+						@logger.debug "Sending INT Signal to #{command} Process with PID: #{pid}"
+
+						if WebappWorker::System.process_alive?(pid)
+							Process.kill("INT",pid.to_i)
+						end
+					end
+
+					threads.each do |thread,command|
+						thread.join
+					end
+				end
+			rescue Timeout::Error
+				puts "Graceful Termination bypassed, killing processes and threads"
+				@logger.info "Graceful Termination bypassed, killing processes and threads"
+
+				command_processes.each do |pid,command|
+					@logger.debug "Killing #{command} Process with PID: #{pid}"
+
+          if WebappWorker::System.process_alive?(pid)
+            Process.kill("KILL",pid.to_i)
+          end
+				end
+
+				threads.each do |thread,command|
+					@logger.debug "Killing Command Thread: #{command}"
+					Thread.kill(thread)
+				end
+			end
+
+			puts "Stopping Webapp Worker"
+			@logger.info "Stopping Webapp Worker"
+
+      self.delete_files
+			exit 0
+		end
+	end
+end

data/lib/webapp_worker/version.rb

@@ -1,3 +1,3 @@
 module WebappWorker
-  VERSION = "0.0.2"
+  VERSION = "0.0.3"
 end

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification
 name: webapp_worker
 version: !ruby/object:Gem::Version
-  version: 0.0.2
+  version: 0.0.3
   prerelease: 
 platform: ruby
 authors:
@@ -9,11 +9,11 @@ authors:
 autorequire: 
 bindir: bin
 cert_chain: []
-date: 2012-01-16 00:00:00.000000000Z
+date: 2012-01-18 00:00:00.000000000Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: trollop
-  requirement: &83276430 !ruby/object:Gem::Requirement
+  requirement: &80202920 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - =
@@ -21,10 +21,10 @@ dependencies:
         version: 1.16.2
   type: :runtime
   prerelease: false
-  version_requirements: *83276430
+  version_requirements: *80202920
 - !ruby/object:Gem::Dependency
   name: open4
-  requirement: &83272260 !ruby/object:Gem::Requirement
+  requirement: &80201930 !ruby/object:Gem::Requirement
     none: false
     requirements:
     - - =
@@ -32,7 +32,7 @@ dependencies:
         version: 1.3.0
   type: :runtime
   prerelease: false
-  version_requirements: *83272260
+  version_requirements: *80201930
 description: Allow the webapp to handle your workers, no need to use a job scheduler
 email:
 - nickwillever@gmail.com
@@ -49,6 +49,7 @@ files:
 - lib/webapp_worker.rb
 - lib/webapp_worker/application.rb
 - lib/webapp_worker/job.rb
+- lib/webapp_worker/system.rb
 - lib/webapp_worker/version.rb
 - webapp_worker.gemspec
 homepage: https://nictrix.github.com/webapp_worker