wcc 0.0.3 → 0.0.4

data/README.md

@@ -8,11 +8,21 @@ from old content to new content so minor changes produce only few lines of text
 Note: wcc relies on native `diff` command to produce the unified diff shown in mails -
 plans are to remove this dependency by using [something like this](https://github.com/samg/diffy) later...
 
+Setup
+-----
+
+You need Ruby (preferably version 1.8.7) and Rubygems installed
+(consider using [rvm](http://beginrescueend.com/)). Install wcc:
+
+	gem install wcc
+
+(If you *don't* use [rvm](http://beginrescueend.com/) you should add a 'sudo'.)
+
 Usage
 -----
 
-wcc is packaged as a gem named 'wcc' and provides it's main script as ´wcc´ via the
-command line. It can invoked by hand or automatically via *cron* on a server environment.
+The installed 'wcc' gem provides a ´wcc´ binary on the command line.
+It can invoked by hand or automatically via *cron* on a server environment.
 
 For using wcc you need to specify some options:
 
@@ -20,7 +30,7 @@ For using wcc you need to specify some options:
 * or in a configuration file in [YAML](https://secure.wikimedia.org/wikipedia/en/wiki/YAML) format
 
 The location of the configuration file (usually called 'conf.yml' or something like this)
-can itself be given on command line as last argument. Each option has an hard-coded default
+can itself be given on command line as last argument. Each option has a hard-coded default
 (e.g. the configuration file name is assumed to be './conf.yml'). Command line options
 overwrite configuration file entries.
 
@@ -32,16 +42,6 @@ An example crontab entry that runs wcc every 10 minutes might look like this:
 
 	*/10 *  * * *   root    cd /path/to/dir/with/conf;./wcc
 
-By default wcc only outputs ERROR messages to avoid your cron daemon spammin' around.
-It is recommended to place 'conf.yml' (and optionally the 'filter.d') within an separate 
-directory and use `cd` in cron entry.
-
-Setup
------
-
-You need Ruby (preferably version 1.8.7) and Rubygems installed
-(consider using [rvm](http://beginrescueend.com/)). Install wcc:
-
-	gem install wcc
-
-(If you *don't* use [rvm](http://beginrescueend.com/) you should add a 'sudo'.)
+By default wcc only outputs ERROR and FATAL messages to avoid your cron daemon spammin' around.
+It is recommended to place 'conf.yml' (and optionally the 'filter.d' and 'template.d') within
+a separate directory and use `cd` in cron entry.

data/doc/Filters.md

@@ -0,0 +1,37 @@
+Filter-API
+==========
+
+Filters formally are Ruby Proc blocks (mostly contained in small ruby
+scripts) referenced by an ID that decide upon a given website diff whether
+to notify the user (by email) or not.
+
+These Proc blocks get registered using `Filter.add` with a given ID (a
+string, not a :symbol to prevent some problems).
+
+wcc provides an autoloading mechanism that loads all ruby files contained in
+a specific directory - the *filter.d* - via `require`.  The filter file
+should contain some bootstrap code that may rely only on the *Filter* class
+out of wcc:
+
+	Filter.add 'my_custom_id' do |data,arguments|
+	    # do your filter magic here
+		# and return a boolean
+	end
+
+The format of the `data` may change over time, currently it's a string
+containing all lines separated by newline of the unified diff (raw from unix
+diff).  The arguments are a - possibly empty - hash directly passed in from
+the configuration file to parameterize the filter (e.g. ignoring all diffs
+shorter than let's say 10 lines).
+
+The output of this filter should be boolean true or false, indicating that
+an email should be sent or not.
+
+In the configuration file a filter is referenced like this:
+
+	sites:
+	  - url: ...
+	    :
+	    filters:
+	      - test
+	      - paramtest: { minLines: 10 }

data/lib/wcc/filter.rb

@@ -0,0 +1,40 @@
+
+module WCC
+	class FilterRef
+		attr_reader :id, :arguments
+
+		def initialize(id, arguments = {})
+			@id = id
+			@arguments = arguments
+		end
+		
+		def to_s; @id end
+	end
+
+	class Filter
+		@@filters = {}
+		
+		def self.add(id, &block)
+			WCC.logger.info "Adding filter '#{id}'"
+			@@filters[id] = block
+		end
+		
+		def self.accept(data, filters)
+			return true if filters.nil?
+			
+			WCC.logger.info "Testing with filters: #{filters.join(', ')}"
+			filters.each do |fref|
+				block = @@filters[fref.id]
+				if block.nil?
+					WCC.logger.error "Requested filter '#{fref.id}' not found, skipping it."
+					next
+				end
+				if not block.call(data, fref.arguments)
+					WCC.logger.info "Filter '#{fref.id}' failed!"
+					return false
+				end
+			end
+			true
+		end
+	end
+end

data/lib/wcc/mail.rb

@@ -0,0 +1,60 @@
+
+module WCC
+	# An email address container with internal conversion
+	# routines.
+	class MailAddress
+		def initialize(email)
+			email = email.to_s if email.is_a?(MailAddress)
+			@email = email.strip
+		end
+		
+		# Extract the 'name' out of an mail address:
+		#   "Me <me@example.org>" -> "Me"
+		#   "me2@example.org" -> "me2"
+		#
+		# @return [String] name
+		def name
+			if @email =~ /^[\w\s]+<.+@[^@]+>$/
+				@email.gsub(/<.+?>/, '').strip
+			else
+				@email.split("@")[0...-1].join("@")
+			end
+		end
+
+		# Return the real mail address:
+		#   "Me <me@example.org>" -> "me@example.org"
+		#   "me2@example.org" -> "me2@example.org"
+		#
+		# @return [String] mail address
+		def address
+			if @email =~ /^[\w\s]+<.+@[^@]+>$/
+				@email.match(/<([^>]+@[^@>]+)>/)[1]
+			else
+				@email
+			end
+		end
+		
+		def to_s; @email end
+	end
+
+	# SmtpMailer is a specific implementation of an mail deliverer that
+	# does plain SMTP to host:port using [Net::SMTP].
+	class SmtpMailer
+		def initialize(host, port)
+			@host = host
+			@port = port
+		end
+		
+		def send(data, template, from, tos = [])
+			Net::SMTP.start(@host, @port) do |smtp|
+				tos.each do |to|
+					# eval ERB
+					msg = template.result(binding)
+					smtp.send_message(msg, from.address, to.address)
+				end
+			end
+		rescue
+			WCC.logger.fatal "Cannot send mails via SMTP to #{@host}:#{@port} : #{$!.to_s}"
+		end
+	end
+end

data/lib/wcc/site.rb

@@ -0,0 +1,75 @@
+
+module WCC
+	class Site
+		attr_reader :uri, :emails, :filters, :auth, :cookie, :id
+
+		def initialize(url, strip_html, emails, filters, auth, cookie)
+			@uri = URI.parse(url)
+			@strip_html = strip_html
+			@emails = emails.is_a?(Array) ? emails : [emails]
+			@filters = filters.is_a?(Array) ? filters : [filters]
+			@auth = auth
+			@cookie = cookie
+			@id = Digest::MD5.hexdigest(url.to_s)[0...8]
+			# invalid hashes are ""
+			load_hash
+		end
+
+		def strip_html?; @strip_html end
+
+		def new?
+			hash.empty?
+		end
+		
+		def load_hash
+			file = Conf.file(@id + '.md5')
+			if File.exists?(file)
+				WCC.logger.debug "Load hash from file '#{file}'"
+				File.open(file, 'r') { |f| @hash = f.gets; break }
+			else
+				WCC.logger.info "Site #{uri.host} was never checked before."
+				@hash = ""
+			end
+		end
+		
+		def load_content
+			file = Conf.file(@id + '.site')
+			if File.exists?(file)
+				File.open(file, 'r') { |f| @content = f.read }
+			end
+		end
+		
+		def hash; @hash end
+		
+		def hash=(hash)
+			@hash = hash
+			File.open(Conf.file(@id + '.md5'), 'w') { |f| f.write(@hash) } unless Conf.simulate?
+		end
+		
+		def content; load_content if @content.nil?; @content end
+		
+		def content=(content)
+			@content = content
+			File.open(Conf.file(@id + '.site'), 'w') { |f| f.write(@content) } unless Conf.simulate?
+		end
+
+		def fetch
+			http = Net::HTTP.new(@uri.host, @uri.port)
+			if @uri.is_a?(URI::HTTPS)
+				http.use_ssl = true
+				http.verify_mode = OpenSSL::SSL::VERIFY_NONE
+			end
+			http.start do |http|
+				req = Net::HTTP::Get.new(@uri.request_uri)
+				if @auth['type'] == 'basic'
+					WCC.logger.debug "Doing basic auth"
+					req.basic_auth(@auth['username'], @auth['password'])
+				end
+				if not @cookie.nil?
+					req.add_field("Cookie", @cookie)
+				end
+				http.request(req)
+			end
+		end
+	end
+end

data/lib/wcc.rb

@@ -1,12 +1,14 @@
 
 require 'base64'
 require 'digest/md5'
+require 'erb'
 require 'iconv'
 require 'logger'
 require 'net/http'
 require 'net/https'
 require 'net/smtp'
 require 'optparse'
+require 'ostruct'
 require 'pathname'
 require 'singleton'
 require 'tempfile'
@@ -16,8 +18,13 @@ require 'yaml'
 # ruby gem dependencies
 require 'htmlentities'
 
+# wcc
+require 'wcc/filter'
+require 'wcc/mail'
+require 'wcc/site'
+
 class String
-	# Remove all HTML <tags> with at least one character name and
+	# Remove all HTML tags with at least one character name and
 	# decode all HTML entities into utf-8 characters.
 	#
 	# @return [String] stripped string
@@ -31,6 +38,7 @@ module WCC
 	DIFF_TIME_FMT = '%Y-%m-%d %H:%M:%S %Z'
 	
 	# logging via WCC.logger.blub
+	
 	def self.logger
 		@logger
 	end
@@ -57,10 +65,12 @@ module WCC
 				:simulate => false,
 				:clean => false,
 				:nomails => false,
-				:dir => '/var/tmp/wcc',
+				# when you want to use ./tmp it must be writeable
+				:cache_dir => '/var/tmp/wcc',
 				:tag => 'wcc',
 				:syslog => false,
-				:filterd => './filter.d',
+				:filter_dir => './filter.d',
+				:template_dir => './template.d',
 				:mailer => 'smtp',
 				:smtp_host => 'localhost',
 				:smtp_port => 25
@@ -75,14 +85,15 @@ module WCC
 				opts.banner += "\nOptions:\n"
 				opts.on('-v', '--verbose', 'Output more information') do self[:verbose] = true end
 				opts.on('-d', '--debug', 'Enable debug mode') do self[:debug] = true end
-				opts.on('-o', '--dir DIR', 'Save hash and diff files to DIR') do |dir| self[:dir] = dir end
+				opts.on('--cache-dir DIR', 'Save hash and diff files to DIR') do |dir| self[:cache_dir] = dir end
 				opts.on('-s', '--simulate', 'Check for update but do not save hash or diff files') do self[:simulate] = true end
-				opts.on('-c', '--clean', 'Remove all saved hash and diff files') do self[:clean] = true end
+				opts.on('--clean', 'Remove all saved hash and diff files') do self[:clean] = true end
 				opts.on('-t', '--tag TAG', 'Set TAG used in output') do |t| self[:tag] = t end
 				opts.on('-n', '--no-mails', 'Do not send any emails') do self[:nomails] = true end
 				opts.on('-f', '--from MAIL', 'Set From: mail address') do |m| self[:from_mail] = m end
 				opts.on('--host HOST', 'Set SMTP host') do |h| self[:host] = h end
 				opts.on('--port PORT', 'Set SMTP port') do |p| self[:port] = p end
+				#opts.on('--init', '--initialize')
 				opts.on('--show-config', 'Show config after loading config file (debug purposes)') do self[:show_config] = true end
 				opts.on('-h', '-?', '--help', 'Display this screen') do
 					puts opts
@@ -115,10 +126,11 @@ module WCC
 			yaml = YAML.load_file(self[:conf])
 			if yaml.is_a?(Hash) and (yaml = yaml['conf']).is_a?(Hash)
 				@options[:from_mail] ||= yaml['from_addr']
-				@options[:dir] ||= yaml['cache_dir']
+				@options[:cache_dir] ||= yaml['cache_dir']
 				@options[:tag] ||= yaml['tag']
 				@options[:syslog] ||= yaml['use_syslog']
-				@options[:filterd] ||= yaml['filterd']
+				@options[:filter_dir] ||= yaml['filterd']
+				@options[:template_dir] ||= yaml['templated']
 				
 				if yaml['email'].is_a?(Hash)
 					if yaml['email']['smtp'].is_a?(Hash)
@@ -142,18 +154,23 @@ module WCC
 				exit 0
 			end
 			
-			# create dir for hash files
-			Dir.mkdir(self[:dir]) unless File.directory?(self[:dir])
+			# create cache dir for hash and diff files
+			Dir.mkdir(self[:cache_dir]) unless File.directory?(self[:cache_dir])
 			
 			if(self[:clean])
 				WCC.logger.warn "Cleanup hash and diff files"
-				Dir.foreach(self[:dir]) do |f|
+				Dir.foreach(self[:cache_dir]) do |f|
 					File.delete(self.file(f)) if f =~ /^.*\.(md5|site)$/
 				end
 			end
 			
 			# read filter.d
-			Dir[File.join(self[:filterd], '*.rb')].each { |file| require file }
+			Dir[File.join(self[:filter_dir], '*.rb')].each { |file| require file }
+			
+			# attach --no-mails filter
+			WCC::Filter.add '--no-mails' do |data|
+				!self[:nomails]
+			end
 		end
 		
 		def self.sites
@@ -163,19 +180,25 @@ module WCC
 			
 			WCC.logger.debug "Load sites from '#{Conf[:conf]}'"
 			
-			# may be false if file is empty
+			# may be *false* if file is empty
 			yaml = YAML.load_file(Conf[:conf])
 			
+			if not yaml
+				WCC.logger.info "No sites loaded"
+				return @sites
+			end
+			
 			yaml['sites'].to_a.each do |yaml_site|
-				filterrefs = []
+				# query --no-mails filter for every site
+				frefs = [FilterRef.new('--no-mails')]
 				(yaml_site['filters'] || []).each do |entry|
 					if entry.is_a?(Hash)
 						# hash containing only one key (filter id),
 						# the value is the argument hash
 						id = entry.keys[0]
-						filterrefs << FilterRef.new(id, entry[id])
+						frefs << FilterRef.new(id, entry[id])
 					else entry.is_a?(String)
-						filterrefs << FilterRef.new(entry, {})
+						frefs << FilterRef.new(entry)
 					end
 				end
 				
@@ -187,10 +210,10 @@ module WCC
 					yaml_site['url'], 
 					yaml_site['strip_html'] || true,
 					yaml_site['emails'].map { |m| MailAddress.new(m) } || [],
-					filterrefs,
+					frefs,
 					yaml_site['auth'] || {},
 					cookie)
-			end if not yaml.nil?
+			end
 			
 			WCC.logger.debug @sites.length.to_s + (@sites.length == 1 ? ' site' : ' sites') + " loaded\n" +
 				@sites.map { |s| "  #{s.uri.host.to_s}\n    url: #{s.uri.to_s}\n    id: #{s.id}" }.join("\n")
@@ -199,197 +222,21 @@ module WCC
 		end
 		
 		def self.mailer
-			if @mailer.nil?
-				# smtp mailer
-				if Conf[:mailer] == 'smtp'
-					@mailer = SmtpMailer.new(Conf[:smtp_host], Conf[:smtp_port])
-				end
+			return @mailer unless @mailer.nil?
+
+			# smtp mailer
+			if Conf[:mailer] == 'smtp'
+				@mailer = SmtpMailer.new(Conf[:smtp_host], Conf[:smtp_port])
 			end
+
 			@mailer
 		end
 		
-		def self.file(path = nil) File.join(self[:dir], path) end
+		def self.file(path = nil) File.join(self[:cache_dir], path) end
 		def self.simulate?; self[:simulate] end
-		def self.send_mails?; !self[:nomails] end
 		def self.[](key); Conf.instance[key] end
 	end
 
-	class FilterRef
-		attr_reader :id, :arguments
-
-		def initialize(id, arguments)
-			@id = id
-			@arguments = arguments
-		end
-		
-		def to_s; @id end
-	end
-
-	class Site
-		attr_reader :uri, :emails, :filters, :auth, :cookie, :id
-
-		def initialize(url, strip_html, emails, filters, auth, cookie)
-			@uri = URI.parse(url)
-			@strip_html = strip_html
-			@emails = emails.is_a?(Array) ? emails : [emails]
-			@filters = filters.is_a?(Array) ? filters : [filters]
-			@auth = auth
-			@cookie = cookie
-			@id = Digest::MD5.hexdigest(url.to_s)[0...8]
-			# invalid hashes are ""
-			load_hash
-		end
-
-		def strip_html?; @strip_html end
-
-		def new?
-			hash.empty?
-		end
-		
-		def load_hash
-			file = Conf.file(@id + '.md5')
-			if File.exists?(file)
-				WCC.logger.debug "Load hash from file '#{file}'"
-				File.open(file, 'r') { |f| @hash = f.gets; break }
-			else
-				WCC.logger.info "Site #{uri.host} was never checked before."
-				@hash = ""
-			end
-		end
-		
-		def load_content
-			file = Conf.file(@id + '.site')
-			if File.exists?(file)
-				File.open(file, 'r') { |f| @content = f.read }
-			end
-		end
-		
-		def hash; @hash end
-		
-		def hash=(hash)
-			@hash = hash
-			File.open(Conf.file(@id + '.md5'), 'w') { |f| f.write(@hash) } unless Conf.simulate?
-		end
-		
-		def content; load_content if @content.nil?; @content end
-		
-		def content=(content)
-			@content = content
-			File.open(Conf.file(@id + '.site'), 'w') { |f| f.write(@content) } unless Conf.simulate?
-		end
-
-		def fetch
-			http = Net::HTTP.new(@uri.host, @uri.port)
-			if @uri.is_a?(URI::HTTPS)
-				http.use_ssl = true
-				http.verify_mode = OpenSSL::SSL::VERIFY_NONE
-			end
-			http.start do |http|
-				req = Net::HTTP::Get.new(@uri.request_uri)
-				if @auth['type'] == 'basic'
-					WCC.logger.debug "Doing basic auth"
-					req.basic_auth(@auth['username'], @auth['password'])
-				end
-				if not @cookie.nil?
-					req.add_field("Cookie", @cookie)
-				end
-				http.request(req)
-			end
-		end
-	end
-
-	class MailAddress
-		def initialize(email)
-			email = email.to_s if email.is_a?(MailAddress)
-			@email = email.strip
-		end
-		
-		def name
-			if @email =~ /^[\w\s]+<.+@[^@]+>$/
-				@email.gsub(/<.+?>/, '').strip
-			else
-				@email.split("@")[0...-1].join("@")
-			end
-		end
-
-		def address
-			if @email =~ /^[\w\s]+<.+@[^@]+>$/
-				@email.match(/<([^>]+@[^@>]+)>/)[1]
-			else
-				@email
-			end
-		end
-		
-		def to_s; @email end
-	end
-
-	class Mail
-		attr_reader :title, :message
-		
-		def initialize(title, message, options = {})
-			@title = title
-			@message = message
-			@options = {:from => MailAddress.new(Conf[:from_mail])}
-			@options[:from] = MailAddress.new(options[:from]) unless options[:from].nil?
-		end
-		
-		def send(tos = [])
-			Conf.mailer.send(self, @options[:from], tos)
-		end
-	end
-
-	class SmtpMailer
-		def initialize(host, port)
-			@host = host
-			@port = port
-		end
-		
-		def send(mail, from, to = [])
-			Net::SMTP.start(@host, @port) do |smtp|
-				to.each do |toaddr|
-					msg  = "From: #{from.name} <#{from.address}>\n"
-					msg += "To: #{toaddr}\n"
-					msg += "Subject: #{mail.title.gsub(/\s+/, ' ')}\n"
-					msg += "Content-Type: text/plain; charset=\"utf-8\"\n"
-					msg += "Content-Transfer-Encoding: base64\n"
-					msg += "\n"
-					msg += Base64.encode64(mail.message)
-					
-					smtp.send_message(msg, from.address, toaddr.address)
-				end
-			end
-		rescue
-			WCC.logger.fatal "Cannot send mails at #{@host}:#{@port} : #{$!.to_s}"
-		end
-	end
-
-	class Filter
-		@@filters = {}
-		
-		def self.add(id, &block)
-			WCC.logger.info "Adding filter '#{id}'"
-			@@filters[id] = block
-		end
-		
-		def self.accept(data, filters)
-			return true if filters.nil?
-			
-			WCC.logger.info "Testing with filters: #{filters.join(', ')}"
-			filters.each do |fref|
-				block = @@filters[fref.id]
-				if block.nil?
-					WCC.logger.error "Requested filter '#{fref.id}' not found, skipping it."
-					next
-				end
-				if not block.call(data, fref.arguments)
-					WCC.logger.info "Filter #{fref.id} failed!"
-					return false
-				end
-			end
-			true
-		end
-	end
-
 	class LogFormatter
 		def initialize(use_color = true)
 			@color = use_color
@@ -450,7 +297,7 @@ module WCC
 				return false
 			end
 			
-			# strip html _before_ diffing
+			# strip html
 			new_content = new_content.strip_html if site.strip_html?
 			new_hash = Digest::MD5.hexdigest(new_content)
 			
@@ -459,7 +306,6 @@ module WCC
 			
 			# do not try diff or anything if site was never checked before
 			if site.new?
-				# update content
 				site.hash, site.content = new_hash, new_content
 				
 				# set custom diff message
@@ -474,19 +320,20 @@ module WCC
 				old_label = "OLD (%s)" % File.mtime(Conf.file(site.id + ".md5")).strftime(DIFF_TIME_FMT)
 				new_label = "NEW (%s)" % Time.now.strftime(DIFF_TIME_FMT)
 			
-				# do update
 				site.hash, site.content = new_hash, new_content
 				
 				# diff between OLD and NEW
 				diff = %x[diff -U 1 --label "#{old_label}" --label "#{new_label}" #{old_site_file.path} #{Conf.file(site.id + '.site')}]
 			end
 			
+			# HACK: there *was* an update but no notification is required
 			return false if not Filter.accept(diff, site.filters)
 			
-			Mail.new(
-				"[#{Conf[:tag]}] #{site.uri.host} changed",
-				"Change at #{site.uri.to_s} - diff follows:\n\n#{diff}"
-				).send(site.emails) if Conf.send_mails?
+			data = OpenStruct.new
+			data.title = "[#{Conf[:tag]}] #{site.uri.host} changed"
+			data.message = "Change at #{site.uri.to_s} - diff follows:\n\n#{diff}"
+			
+			Conf.mailer.send(data, @@mail_plain, MailAddress.new(Conf[:from_mail]), site.emails)
 			
 			system("logger -t '#{Conf[:tag]}' 'Change at #{site.uri.to_s} (tag #{site.id}) detected'") if Conf[:syslog]
 			
@@ -495,8 +342,13 @@ module WCC
 
 		# main
 		def self.run!
+			# first use of Conf initializes it
 			WCC.logger = Logger.new(STDOUT)
 			
+			mp_path = File.join(Conf[:template_dir], 'mail-plain.erb')
+			mp = File.open(mp_path, 'r') { |f| f.read }
+			@@mail_plain = ERB.new(mp)
+			
 			Conf.sites.each do |site|
 				if checkForUpdate(site)
 					WCC.logger.warn "#{site.uri.host.to_s} has an update!"

metadata

@@ -1,13 +1,13 @@
 --- !ruby/object:Gem::Specification 
 name: wcc
 version: !ruby/object:Gem::Version 
-  hash: 25
+  hash: 23
   prerelease: 
   segments: 
   - 0
   - 0
-  - 3
-  version: 0.0.3
+  - 4
+  version: 0.0.4
 platform: ruby
 authors: 
 - Christian Nicolai
@@ -15,7 +15,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2011-10-01 00:00:00 Z
+date: 2011-10-02 00:00:00 Z
 dependencies: 
 - !ruby/object:Gem::Dependency 
   name: htmlentities
@@ -41,6 +41,10 @@ extra_rdoc_files: []
 
 files: 
 - bin/wcc
+- doc/Filters.md
+- lib/wcc/filter.rb
+- lib/wcc/mail.rb
+- lib/wcc/site.rb
 - lib/wcc.rb
 - README.md
 homepage: https://github.com/cmur2/wcc