websitary 0.4 → 0.5

data/History.txt

@@ -1,3 +1,18 @@
+= 0.5
+
+* mailto: and javascript: hrefs are now handled via the exclude option
+* rewrite absolute URLs sans host correctly
+* strip href and image src tags in order to prevent parser errors
+* some scaffolding for mechanize
+* global proxy option (currently only used for mechanize)
+* use -nolist for lynx
+* catch errors in Websitary::App#execute_downdiff
+* :rss_find_enclosure => LAMBDA: Extract the enclosure URL from the item 
+  description
+* :rss_format_local_copy => STRING|BLOCK/2: Format the display of the 
+  local copy.
+
+
 = 0.4
 
 * Sources may have a :timeout option.

data/README.txt

@@ -212,6 +212,12 @@ Known global options:
 <tt>:toggle_body => BOOLEAN</tt>::
   If true, make a news body collabsable on mouse-clicks (sort of).
 
+<tt>:proxy => STRING</tt>, <tt>:proxy => ARRAY</tt>::
+  The proxy. (currently only supported by mechanize)
+
+<tt>:user_agent => STRING</tt>::
+  Set the user agent (only for certain queries).
+
 
 ==== output_format FORMAT, output_format [FORMAT1, FORMAT2, ...]
 Set the output format.
@@ -318,11 +324,29 @@ Options
 <tt>:rss_enclosure => true|"DIRECTORY"</tt>::
   If true, save rss feed enclosures in 
   "~/.websitary/attachments/RSS_FEED_NAME/". If a string, use this as 
-  destination directory.
+  destination directory. Only enclosures of new items will be saved -- 
+  i.e. when downloading a feed for the first time, no enclosures will be 
+  saved.
+
+<tt>:rss_find_enclosure => BLOCK</tt>::
+  Certain RSS-feeds embed enclosures in the description. Use this option 
+  to scan the description (a Hpricot document) for an URL that is then saved 
+  as enclosure if the :rss_enclosure option is set.
+  Example:
+      source 'http://www.example.com/rss',
+        :title => 'Example',
+        :use => :rss, :rss_enclosure => true,
+        :rss_find_enclosure => lambda {|item, doc| (doc / 'img').map {|e| e['src']}[0]}
 
 <tt>:rss_format (default: "plain_text")</tt>::
     When output format is :rss, create rss item descriptios as plain text.
 
+<tt>:rss_format_local_copy => FORMAT_STRING | BLOCK</tt>::
+    By default a hypertext reference to the local copy of an RSS 
+    enclosure is added to entry. Sometimes you may want to display 
+    something inline (e.g. an image). You can then use this option to 
+    define a format string (one field = the local copy's file url).
+
 <tt>:show_initial => true</tt>::
     Include initial copies in the report (may not always work properly). 
     This can also be set as a global option.
@@ -388,6 +412,13 @@ Example:
   Use open-uri for downloading the source. Use webdiff for generating 
   diffs. This doesn't handle cookies and the like.
 
+<tt>:mechanize</tt>::
+  Use mechanize (must be installed) for downloading the source. Use 
+  webdiff for generating diffs. This calls the URL's :mechanize property 
+  (a lambda that takes 3 arguments: URL, agent, page => HTML as string) 
+  to post-process the page (or if not available, use the page body's 
+  HTML).
+
 <tt>:text</tt>::
   This requires hpricot to be installed. Use open-uri for downloading 
   and hpricot for converting HTML to plain text. This still requires 
@@ -730,7 +761,7 @@ Now check out the configuration commands in the Synopsis section.
 
 == LICENSE:
 websitary Webpage Monitor
-Copyright (C) 2007 Thomas Link
+Copyright (C) 2007-2008 Thomas Link
 
 This program is free software; you can redistribute it and/or modify
 it under the terms of the GNU General Public License as published by

data/Rakefile

@@ -21,11 +21,11 @@ require 'rtagstask'
 RTagsTask.new
 
 task :ctags do
-    `ctags --extra=+q --fields=+i+S -R bin lib`
+    puts `ctags --extra=+q --fields=+i+S -R bin lib`
 end
 
 task :files do
-    `find bin lib -name "*.rb" > files.lst`
+    puts `find bin lib -name "*.rb" > files.lst`
 end
 
 # vim: syntax=Ruby

data/bin/websitary

@@ -1,6 +1,6 @@
 #! /usr/bin/env ruby
 # websitary.rb -- The website news, rss feed, podcast catching monitor
-# @Last Change: 2007-12-26.
+# @Last Change: 2008-02-12.
 # Author::      Thomas Link (micathom at gmail com)
 # License::     GPL (see http://www.gnu.org/licenses/gpl.txt)
 # Created::     2007-06-09.
@@ -11,7 +11,7 @@ require 'websitary'
 
 if __FILE__ == $0
     w = Websitary::App.new(ARGV)
-    t = w.configuration.get_optionvalue(:global, :timer)
+    t = w.configuration.optval_get(:global, :timer)
     if t
         exit_code = 0
         while exit_code <= 1

data/lib/websitary.rb

@@ -1,5 +1,5 @@
 # websitary.rb
-# @Last Change: 2008-01-13.
+# @Last Change: 2008-03-11.
 # Author::      Thomas Link (micathom AT gmail com)
 # License::     GPL (see http://www.gnu.org/licenses/gpl.txt)
 # Created::     2007-09-08.
@@ -7,7 +7,8 @@
 
 require 'cgi'
 require 'digest/md5'
-require 'ftools'
+# require 'ftools'
+require 'fileutils'
 require 'net/ftp'
 require 'optparse'
 require 'pathname'
@@ -33,8 +34,8 @@ end
 
 module Websitary
     APPNAME     = 'websitary'
-    VERSION     = '0.4'
-    REVISION    = '2464'
+    VERSION     = '0.5'
+    REVISION    = '2476'
 end
 
 require 'websitary/applog'
@@ -72,7 +73,7 @@ class Websitary::App
         unless File.exists?(css)
             $logger.info "Copying default css file: #{css}"
             @configuration.write_file(css, 'w') do |io|
-                io.puts @configuration.get_option(:page, :css)
+                io.puts @configuration.opt_get(:page, :css)
             end
         end
     end
@@ -99,7 +100,7 @@ class Websitary::App
     def execute_configuration
         keys = @configuration.options.keys
         urls = @configuration.todo
-        # urls = @configuration.todo..sort {|a,b| @configuration.get(a, :title, a) <=> @configuration.get(b, :title, b)}
+        # urls = @configuration.todo..sort {|a,b| @configuration.url_get(a, :title, a) <=> @configuration.url_get(b, :title, b)}
         urls.each_with_index do |url, i|
             data = @configuration.urls[url]
             text = [
@@ -107,7 +108,7 @@ class Websitary::App
                 "<b>current</b><br/>#{CGI.escapeHTML(@configuration.latestname(url, true))}<br/>",
                 "<b>backup</b><br/>#{CGI.escapeHTML(@configuration.oldname(url, true))}<br/>",
                 *((data.keys | keys).map do |k|
-                        v = @configuration.get(url, k).inspect
+                        v = @configuration.url_get(url, k).inspect
                   "<b>:#{k}</b><br/>#{CGI.escapeHTML(v)}<br/>"
                 end)
             ]
@@ -205,7 +206,7 @@ class Websitary::App
         rv = 0
         @configuration.todo.each do |url|
             opts = @configuration.urls[url]
-            name = @configuration.get(url, :title, url)
+            name = @configuration.url_get(url, :title, url)
             $logger.debug "Source: #{name}"
             aggrbase  = @configuration.encoded_filename('aggregate', url, true, 'md5')
             aggrfiles = Dir["#{aggrbase}_*"]
@@ -223,7 +224,7 @@ class Websitary::App
     def execute_show
         @configuration.todo.each do |url|
             opts = @configuration.urls[url]
-            $logger.debug "Source: #{@configuration.get(url, :title, url)}"
+            $logger.debug "Source: #{@configuration.url_get(url, :title, url)}"
             aggrbase  = @configuration.encoded_filename('aggregate', url, true, 'md5')
             difftext  = []
             aggrfiles = Dir["#{aggrbase}_*"]
@@ -233,7 +234,7 @@ class Websitary::App
             difftext.compact!
             difftext.delete('')
             unless difftext.empty?
-                joindiffs = @configuration.get(url, :joindiffs, lambda {|t| t.join("\n")})
+                joindiffs = @configuration.url_get(url, :joindiffs, lambda {|t| t.join("\n")})
                 difftext  = @configuration.call_cmd(joindiffs, [difftext], :url => url) if joindiffs
                 accumulate(url, difftext, opts)
             end
@@ -255,13 +256,13 @@ class Websitary::App
         end
         @configuration.todo.each do |url|
             opts = @configuration.urls[url]
-            $logger.debug "Source: #{@configuration.get(url, :title, url)}"
+            $logger.debug "Source: #{@configuration.url_get(url, :title, url)}"
 
             diffed = @configuration.diffname(url, true)
             $logger.debug "diffname: #{diffed}"
 
             if File.exists?(diffed)
-                $logger.warn "Reuse old diff: #{@configuration.get(url, :title, url)} => #{diffed}"
+                $logger.warn "Reuse old diff: #{@configuration.url_get(url, :title, url)} => #{diffed}"
                 difftext = File.read(diffed)
                 accumulate(url, difftext, opts)
             else
@@ -272,17 +273,22 @@ class Websitary::App
                 older = @configuration.oldname(url, true)
                 $logger.debug "older: #{older}"
 
-                if rebuild or download(url, opts, latest, older)
-                    difftext = diff(url, opts, latest, older)
-                    if difftext
-                        @configuration.write_file(diffed, 'wb') {|io| io.puts difftext}
-                        # $logger.debug "difftext: #{difftext}" #DBG#
-                        if accumulator
-                            accumulator.call(url, difftext, opts)
-                        else
-                            accumulate(url, difftext, opts)
+                begin
+                    if rebuild or download(url, opts, latest, older)
+                        difftext = diff(url, opts, latest, older)
+                        if difftext
+                            @configuration.write_file(diffed, 'wb') {|io| io.puts difftext}
+                            # $logger.debug "difftext: #{difftext}" #DBG#
+                            if accumulator
+                                accumulator.call(url, difftext, opts)
+                            else
+                                accumulate(url, difftext, opts)
+                            end
                         end
                     end
+                rescue Exception => e
+                    $logger.error e.to_s
+                    $logger.info e.backtrace.join("\n")
                 end
             end
         end
@@ -291,20 +297,22 @@ class Websitary::App
 
 
     def move(from, to)
-        copy_move(:rename, from, to)
+        # copy_move(:rename, from, to) # ftools
+        copy_move(:mv, from, to) # FileUtils
     end
 
 
     def copy(from, to)
-        copy_move(:copy, from, to)
+        # copy_move(:copy, from, to)
+        copy_move(:cp, from, to)
     end
 
 
     def copy_move(method, from, to)
         if File.exists?(from)
-            $logger.debug "Overwriting: #{from} -> #{to}" if File.exists?(to)
+            $logger.debug "Overwrite: #{from} -> #{to}" if File.exists?(to)
             lst = File.lstat(from)
-            File.send(method, from, to)
+            FileUtils.send(method, from, to)
             File.utime(lst.atime, lst.mtime, to)
             @configuration.mtimes.set(from, lst.mtime)
             @configuration.mtimes.set(to, lst.mtime)
@@ -347,16 +355,16 @@ class Websitary::App
 
     def download(url, opts, latest, older=nil)
         if @configuration.done.include?(url)
-            $logger.info "Already downloaded: #{@configuration.get(url, :title, url).inspect}"
+            $logger.info "Already downloaded: #{@configuration.url_get(url, :title, url).inspect}"
             return false
         end
 
-        $logger.warn "Download: #{@configuration.get(url, :title, url).inspect}"
+        $logger.warn "Download: #{@configuration.url_get(url, :title, url).inspect}"
         @configuration.done << url
-        text = @configuration.call_cmd(@configuration.get(url, :download), [url], :url => url)
+        text = @configuration.call_cmd(@configuration.url_get(url, :download), [url], :url => url)
         # $logger.debug text #DBG#
         unless text
-            $logger.warn "no contents: #{@configuration.get(url, :title, url)}"
+            $logger.warn "no contents: #{@configuration.url_get(url, :title, url)}"
             return false
         end
 
@@ -390,7 +398,7 @@ class Websitary::App
             text = text.join("\n")
         end
 
-        pprc = @configuration.get(url, :downloadprocess)
+        pprc = @configuration.url_get(url, :downloadprocess)
         if pprc
             $logger.debug "download process: #{pprc}"
             text = @configuration.call_cmd(pprc, [text], :url => url)
@@ -416,25 +424,25 @@ class Websitary::App
     def diff(url, opts, new, old)
         if File.exists?(old)
             $logger.debug "diff: #{old} <-> #{new}"
-            difftext = @configuration.call_cmd(@configuration.get(url, :diff), [old, new], :url => url)
+            difftext = @configuration.call_cmd(@configuration.url_get(url, :diff), [old, new], :url => url)
             # $logger.debug "diff: #{difftext}" #DBG#
 
             if difftext =~ /\S/
-                if (pprc = @configuration.get(url, :diffprocess))
+                if (pprc = @configuration.url_get(url, :diffprocess))
                     $logger.debug "diff process: #{pprc}"
                     difftext = @configuration.call_cmd(pprc, [difftext], :url => url)
                 end
                 # $logger.debug "difftext: #{difftext}" #DBG#
                 if difftext =~ /\S/
-                    $logger.warn "Changed: #{@configuration.get(url, :title, url).inspect}"
+                    $logger.warn "Changed: #{@configuration.url_get(url, :title, url).inspect}"
                     return difftext
                 end
             end
 
-            $logger.debug "Unchanged: #{@configuration.get(url, :title, url).inspect}"
+            $logger.debug "Unchanged: #{@configuration.url_get(url, :title, url).inspect}"
 
         elsif File.exist?(new) and
-            (@configuration.get(url, :show_initial) or @configuration.get_optionvalue(:global, :show_initial))
+            (@configuration.url_get(url, :show_initial) or @configuration.optval_get(:global, :show_initial))
 
             return File.read(new)
 
@@ -451,16 +459,16 @@ class Websitary::App
             tdiff = tdiff_with(opts, tn, tl)
             case tdiff
             when nil, false
-                $logger.debug "Age requirement fulfilled: #{@configuration.get(url, :title, url).inspect}: #{format_tdiff(td)} old"
+                $logger.debug "Age requirement fulfilled: #{@configuration.url_get(url, :title, url).inspect}: #{format_tdiff(td)} old"
                 return false
             when :skip, true
-                $logger.info "Skip #{@configuration.get(url, :title, url).inspect}: Only #{format_tdiff(td)} old"
+                $logger.info "Skip #{@configuration.url_get(url, :title, url).inspect}: Only #{format_tdiff(td)} old"
                 return true
             when Numeric
                 if td < tdiff
                     tdd = tdiff - td
                     @tdiff_min = tdd if @tdiff_min.nil? or tdd < @tdiff_min
-                    $logger.info "Skip #{@configuration.get(url, :title, url).inspect}: Only #{format_tdiff(td)} old (#{format_tdiff(tdiff)})"
+                    $logger.info "Skip #{@configuration.url_get(url, :title, url).inspect}: Only #{format_tdiff(td)} old (#{format_tdiff(tdiff)})"
                     return true
                 end
             else
@@ -509,7 +517,7 @@ class Websitary::App
             when Integer
                 return eligible != now
             else
-                $logger.error "#{@configuration.get(url, :title, url)}: Wrong type for :days_of_week=#{dweek.inspect}"
+                $logger.error "#{@configuration.url_get(url, :title, url)}: Wrong type for :days_of_week=#{dweek.inspect}"
                 return :skip
             end
         end

data/lib/websitary/configuration.rb

@@ -1,5 +1,5 @@
 # configuration.rb
-# @Last Change: 2008-01-09.
+# @Last Change: 2008-05-23.
 # Author::      Thomas Link (micathom AT gmail com)
 # License::     GPL (see http://www.gnu.org/licenses/gpl.txt)
 # Created::     2007-09-08.
@@ -47,7 +47,6 @@ class Websitary::Configuration
         @cmd_edit          = 'vi "%s"'
         @execute           = 'downdiff'
         @quicklist_profile = 'quicklist'
-        @user_agent        = "websitary/#{Websitary::VERSION}"
         @view              = 'w3m "%s"'
 
         @allow             = {}
@@ -60,7 +59,7 @@ class Websitary::Configuration
         @profiles          = []
         @robots            = {}
         @todo              = []
-        @exclude           = []
+        @exclude           = [/^\s*(javascript|mailto):/]
         @urlencmap         = {}
         @urls              = {}
 
@@ -190,10 +189,16 @@ class Websitary::Configuration
     end
 
 
+    def url_set(url, items)
+        opts = @urls[url] ||= {}
+        opts.merge!(items)
+    end
+
+
     # Retrieve an option for an url
     # url:: String
     # opt:: Symbol
-    def get(url, opt, default=nil)
+    def url_get(url, opt, default=nil)
         opts = @urls[url]
         unless opts
             $logger.debug "Non-registered URL: #{url}"
@@ -221,7 +226,7 @@ class Websitary::Configuration
         when nil
         when Symbol
             $logger.debug "get: val=#{val}"
-            success, rv = get_option(opt, val)
+            success, rv = opt_get(opt, val)
             $logger.debug "get: #{success}, #{rv}"
             if success
                 return rv
@@ -231,7 +236,7 @@ class Websitary::Configuration
             return val
         end
         unless default
-            success, default1 = get_option(opt, :default)
+            success, default1 = opt_get(opt, :default)
             default = default1 if success
         end
 
@@ -240,10 +245,10 @@ class Websitary::Configuration
     end
 
 
-    def get_optionvalue(opt, val, default=nil)
+    def optval_get(opt, val, default=nil)
         case val
         when Symbol
-            ok, val = get_option(opt, val)
+            ok, val = opt_get(opt, val)
             if ok
                 val
             else
@@ -255,22 +260,22 @@ class Websitary::Configuration
     end
 
 
-    def get_option(opt, val)
+    def opt_get(opt, val)
         vals = @options[opt]
         $logger.debug "val=#{val} vals=#{vals.inspect}"
         if vals and vals.has_key?(val)
             rv = vals[val]
-            $logger.debug "get_option ok: #{opt} => #{rv.inspect}"
+            $logger.debug "opt_get ok: #{opt} => #{rv.inspect}"
             case rv
             when Symbol
-                $logger.debug "get_option re: #{rv}"
-                return get_option(opt, rv)
+                $logger.debug "opt_get re: #{rv}"
+                return opt_get(opt, rv)
             else
-                $logger.debug "get_option true, #{rv}"
+                $logger.debug "opt_get true, #{rv}"
                 return [true, rv]
             end
         else
-            $logger.debug "get_option no: #{opt} => #{val.inspect}"
+            $logger.debug "opt_get no: #{opt} => #{val.inspect}"
             return [false, val]
         end
     end
@@ -409,7 +414,7 @@ class Websitary::Configuration
     # urls:: String
     def source(urls, opts={})
         urls.split("\n").flatten.compact.each do |url|
-            @urls[url] = @default_options.dup.update(opts)
+            url_set(url, @default_options.dup.update(opts))
             to_do url
         end
     end
@@ -477,9 +482,9 @@ class Websitary::Configuration
 
 
     def format_text(url, text)
-        enc = get(url, :iconv)
+        enc = url_get(url, :iconv)
         if enc
-            denc = get_optionvalue(:global, :encoding)
+            denc = optval_get(:global, :encoding)
             begin
                 require 'iconv'
                 text = Iconv.conv(denc, enc, text)
@@ -493,7 +498,7 @@ class Websitary::Configuration
 
     # Format a diff according to URL's source options.
     def format(url, difftext)
-        fmt  = get(url, :format)
+        fmt  = url_get(url, :format)
         text = format_text(url, difftext)
         eval_arg(fmt, [text], text)
     end
@@ -527,7 +532,7 @@ class Websitary::Configuration
     def call_cmd(cmd, cmdargs, args={})
         default = args[:default]
         url     = args[:url]
-        timeout = url ? get(url, :timeout) : nil
+        timeout = url ? url_get(url, :timeout) : nil
         if timeout
             begin
                 Timeout::timeout(timeout) do |timeout_length|
@@ -583,7 +588,7 @@ class Websitary::Configuration
             if difftext
                 difftext = html_to_text(difftext) if is_html?(difftext)
                 !difftext.empty? && [
-                    eval_arg(get(url, :rewrite_link, '%s'), [url]), 
+                    eval_arg(url_get(url, :rewrite_link, '%s'), [url]), 
                     difftext_annotation(url), 
                     nil, 
                     difftext
@@ -594,32 +599,32 @@ class Websitary::Configuration
 
 
     def get_output_rss(difftext)
-        success, rss_url = get_option(:rss, :url)
+        success, rss_url = opt_get(:rss, :url)
         if success
-            success, rss_version = get_option(:rss, :version)
+            success, rss_version = opt_get(:rss, :version)
             # require "rss/#{rss_version}"
 
             rss         = RSS::Rss.new(rss_version)
             chan        = RSS::Rss::Channel.new
             chan.title  = @output_title
             [:description, :copyright, :category, :language, :image, :webMaster, :pubDate].each do |field|
-                ok, val = get_option(:rss, field)
+                ok, val = opt_get(:rss, field)
                 item.send(format_symbol(field, '%s='), val) if ok
             end
             chan.link   = rss_url
             rss.channel = chan
 
             cnt = difftext.map do |url, text|
-                rss_format = get(url, :rss_format, 'plain_text')
+                rss_format = url_get(url, :rss_format, 'plain_text')
                 text = strip_tags(text, :format => rss_format)
                 next if text.empty?
 
                 item = RSS::Rss::Channel::Item.new
                 item.date  = Time.now
-                item.title = get(url, :title, File.basename(url))
-                item.link  = eval_arg(get(url, :rewrite_link, '%s'), [url])
+                item.title = url_get(url, :title, File.basename(url))
+                item.link  = eval_arg(url_get(url, :rewrite_link, '%s'), [url])
                 [:author, :date, :enclosure, :category, :pubDate].each do |field|
-                    val = get(url, format_symbol(field, 'rss_%s'))
+                    val = url_get(url, format_symbol(field, 'rss_%s'))
                     item.send(format_symbol(field, '%s='), val) if val
                 end
 
@@ -647,7 +652,7 @@ class Websitary::Configuration
 
     def get_output_html(difftext)
         difftext = difftext.map do |url, text|
-            tags = get(url, :strip_tags)
+            tags = url_get(url, :strip_tags)
             text = strip_tags(text, :tags => tags) if tags
             text.empty? ? nil : [url, text]
         end
@@ -655,7 +660,7 @@ class Websitary::Configuration
         sort_difftext!(difftext)
 
         toc = difftext.map do |url, text|
-            ti  = get(url, :title, File.basename(url))
+            ti  = url_get(url, :title, File.basename(url))
             tid = html_toc_id(url)
             bid = html_body_id(url)
             %{<li id="#{tid}" class="toc"><a class="toc" href="\##{bid}">#{ti}</a></li>}
@@ -664,9 +669,9 @@ class Websitary::Configuration
         idx = 0
         cnt = difftext.map do |url, text|
             idx += 1
-            ti   = get(url, :title, File.basename(url))
+            ti   = url_get(url, :title, File.basename(url))
             bid  = html_body_id(url)
-            if (rewrite = get(url, :rewrite_link))
+            if (rewrite = url_get(url, :rewrite_link))
                 urlr = eval_arg(rewrite, [url])
                 ext  = ''
             else
@@ -676,7 +681,7 @@ class Websitary::Configuration
                 urlr = url
             end
             note    = difftext_annotation(url)
-            onclick = get_optionvalue(:global, :toggle_body) ? 'onclick="ToggleBody(this)"' : ''
+            onclick = optval_get(:global, :toggle_body) ? 'onclick="ToggleBody(this)"' : ''
             <<HTML
 <div id="#{bid}" class="webpage" #{onclick}>
 <div class="count">
@@ -697,9 +702,9 @@ class Websitary::Configuration
 HTML
         end.join(('<hr class="separator"/>') + "\n")
 
-        success, template = get_option(:page, :format)
+        success, template = opt_get(:page, :format)
         unless success
-            success, template = get_option(:page, :simple)
+            success, template = opt_get(:page, :simple)
         end
         return eval_arg(template, [@output_title, toc, cnt])
     end
@@ -735,12 +740,12 @@ HTML
 
 
     def encoded_filename(dir, url, ensure_dir=false, type=nil)
-        type ||= get(url, :cachetype, 'tree')
+        type ||= url_get(url, :cachetype, 'tree')
         $logger.debug "encoded_filename: type=#{type} url=#{url}"
         rv = File.join(@cfgdir, dir, encoded_basename(url, type))
         rd = File.dirname(rv)
         $logger.debug "encoded_filename: rv0=#{rv}"
-        fm = get_optionvalue(:global, :filename_size, 255)
+        fm = optval_get(:global, :filename_size, 255)
         rdok = !ensure_dir || @app.ensure_dir(rd, false)
         if !rdok or rv.size > fm or File.directory?(rv)
             # $logger.debug "Filename too long (:global=>:filename_size = #{fm}), try md5 encoded filename instead: #{url}"
@@ -796,6 +801,24 @@ HTML
     end
 
 
+    def save_dir(url, dir, title=nil)
+        case dir
+        when true
+            title ||= url_get(url, :title)
+            dir = File.join(@cfgdir, 'attachments', encode(title))
+        when Proc
+            dir = dir.call(url)
+        end
+        @app.ensure_dir(dir) if dir
+        return dir
+    end
+
+
+    def clean_url(url)
+        url && url.strip
+    end
+
+
     # Strip the url's last part (after #).
     def canonic_url(url)
         url.sub(/#.*$/, '')
@@ -803,7 +826,7 @@ HTML
 
 
     def strip_tags_default
-        success, tags = get_option(:strip_tags, :default)
+        success, tags = opt_get(:strip_tags, :default)
         tags.dup if success
     end
 
@@ -830,7 +853,7 @@ HTML
     # This checks either for a :match option for url or the extensions 
     # of path0 and path.
     def eligible_path?(url, path0, path)
-        rx = get(url, :match)
+        rx = url_get(url, :match)
         if rx
             return path =~ rx
         else
@@ -845,15 +868,15 @@ HTML
         begin
             $logger.debug "push_refs: #{url}"
             return if robots?(hpricot, 'nofollow') or is_excluded?(url)
-            depth = get(url, :depth)
+            depth = url_get(url, :depth)
             return if depth and depth <= 0
             uri0  = URI.parse(url)
             # pn0   = Pathname.new(guess_dir(File.expand_path(uri0.path)))
             pn0   = Pathname.new(guess_dir(uri0.path))
             (hpricot / 'a').each do |a|
                 next if a['rel'] == 'nofollow'
-                href = a['href']
-                next if href.nil? or href == url or href =~ /^\s*javascript:/ or href =~ /^\s*mailto:/ or is_excluded?(href)
+                href = clean_url(a['href'])
+                next if href.nil? or href == url or is_excluded?(href)
                 uri  = URI.parse(href)
                 pn   = guess_dir(uri.path)
                 href = rewrite_href(href, url, uri0, pn0, true)
@@ -869,7 +892,7 @@ HTML
                 opts[:title] = [opts[:title], File.basename(curl)].join(' - ')
                 opts[:depth] = depth - 1 if depth and depth >= 0
                 # opts[:sleep] = delay if delay
-                @urls[curl] = opts
+                url_set(curl, opts)
                 to_do curl
             end
         rescue Exception => e
@@ -887,7 +910,7 @@ HTML
         uri = URI.parse(url)
         urd = guess_dir(uri.path)
         (doc / 'a').each do |a|
-            href = a['href']
+            href = clean_url(a['href'])
             if is_excluded?(href)
                 comment_element(doc, a)
             else
@@ -896,7 +919,7 @@ HTML
             end
         end
         (doc / 'img').each do |a|
-            href = a['src']
+            href = clean_url(a['src'])
             if is_excluded?(href)
                 comment_element(doc, a)
             else
@@ -917,12 +940,14 @@ HTML
     # Try to make href an absolute url.
     def rewrite_href(href, url, uri=nil, urd=nil, local=false)
         begin
-            return if !href or href =~ /^\s*javascript:/
-            urh   = URI.parse(href)
+            return nil if !href or is_excluded?(href)
             uri ||= URI.parse(url)
+            if href =~ /^\s*\//
+                return uri.merge(href).to_s
+            end
+            urh   = URI.parse(href)
             urd ||= guess_dir(uri.path)
             rv    = nil
-            href  = href.strip
 
             # $logger.debug "DBG", uri, urh, #DBG#
             if href =~ /\w+:/
@@ -1026,7 +1051,7 @@ HTML
 
 
     def canonic_filename(filename)
-        call_cmd(get_optionvalue(:global, :canonic_filename), [filename], :default => filename)
+        call_cmd(optval_get(:global, :canonic_filename), [filename], :default => filename)
     end
 
 
@@ -1037,6 +1062,7 @@ HTML
                 :download_html => :openuri,
                 :encoding => 'ISO-8859-1',
                 :toggle_body => false,
+                :user_agent => "websitary/#{Websitary::VERSION}",
             },
         }
 
@@ -1052,11 +1078,11 @@ HTML
             },
 
             :binary => lambda {|old, new|
-                call_cmd(get_optionvalue(:diff, :diff), [old, new, '--binary -d -w'])
+                call_cmd(optval_get(:diff, :diff), [old, new, '--binary -d -w'])
             },
 
             :new => lambda {|old, new|
-                difftext = call_cmd(get_optionvalue(:diff, :binary), [old, new])
+                difftext = call_cmd(optval_get(:diff, :binary), [old, new])
                 difftext.empty? ? '' : new
             },
 
@@ -1067,7 +1093,7 @@ HTML
                 args = {
                     :oldhtml => File.read(old),
                     :newhtml => File.read(new),
-                    :ignore  => get(url, :ignore),
+                    :ignore  => url_get(url, :ignore),
                 }
                 difftext = Websitary::Htmldiff.new(args).diff
                 difftext
@@ -1130,7 +1156,7 @@ HTML
             # :download => 'w3m -no-cookie -S -F -dump "%s"'
 
         shortcut :lynx, :delegate => :diff,
-            :download => 'lynx -dump "%s"'
+            :download => 'lynx -dump -nolist "%s"'
 
         shortcut :links, :delegate => :diff,
             :download => 'links -dump "%s"'
@@ -1142,24 +1168,14 @@ HTML
             :download => 'wget -q -O - "%s"'
 
         shortcut :text, :delegate => :diff,
-            :download => lambda {|url| html_to_text(read_url(url, 'html'))}
+            :download => lambda {|url| doc_to_text(read_document(url))}
 
         shortcut :body_html, :delegate => :webdiff,
             :strip_tags => :default,
             :download => lambda {|url|
                 begin
-                    doc = Hpricot(read_url(url, 'html'))
-                    doc = doc.at('body')
-                    if doc
-                        doc  = rewrite_urls(url, doc)
-                        doc  = doc.inner_html
-                        if (tags = get(url, :strip_tags))
-                            doc = strip_tags(doc, :format => :hpricot, :tags => tags)
-                        end
-                    else
-                        $logger.warn 'inner html: No body'
-                    end
-                    doc.to_s
+                    doc = read_document(url)
+                    body_html(url, doc).to_s
                 rescue Exception => e
                     # $logger.error e  #DBG#
                     $logger.error e.message
@@ -1180,10 +1196,37 @@ HTML
                 end
             }
 
+        shortcut :mechanize, :delegate => :webdiff,
+            :download => lambda {|url|
+                require 'mechanize'
+                agent = WWW::Mechanize.new
+                proxy = get_proxy
+                if proxy
+                    agent.set_proxy(*proxy)
+                end
+                page = agent.get(url)
+                process = url_get(url, :mechanize)
+                if process
+                    uri = URI.parse(url)
+                    urd = guess_dir(uri.path)
+                    page.links.each {|link|
+                        href = link.node['href']
+                        if href
+                            href = rewrite_href(href, url, uri, urd, true)
+                            link.node['href'] = href if href
+                        end
+                    }
+                    process.call(url, agent, page)
+                else
+                    doc = url_document(url, page.content)
+                    body_html(url, doc).to_s
+                end
+            }
+
         shortcut :rss,
             :delegate => :openuri,
             :diff => lambda {|old, new|
-                success, rss_version = get_option(:rss, :version)
+                success, rss_version = opt_get(:rss, :version)
                 ro = RSS::Parser.parse(File.read(old), false)
                 if ro
                     rh = {}
@@ -1202,24 +1245,35 @@ HTML
                                     rnew << format_rss_item(item, rss_diff)
                                 else
                                     enc = item.respond_to?(:enclosure) && item.enclosure
-                                    if enc and (curl = enc.url)
-                                        url   = url_from_filename(new)
-                                        dir   = get(url, :rss_enclosure)
+                                    url = url_from_filename(new)
+                                    if !enc and item.description
+                                        scanner = url_get(url, :rss_find_enclosure)
+                                        if scanner
+                                            ddoc = Hpricot(item.description)
+                                            enc  = scanner.call(item, ddoc)
+                                            if enc
+                                                def enc.url
+                                                    self
+                                                end
+                                            else
+                                                $logger.warn "No embedded enclosure URL found: #{item.description}"
+                                            end
+                                        end
+                                    end
+                                    if enc and (curl = clean_url(enc.url))
+                                        dir   = url_get(url, :rss_enclosure)
                                         curl  = rewrite_href(curl, url, nil, nil, true)
                                         next unless curl
                                         if dir
-                                            if dir == true
-                                                dir = File.join(@cfgdir, 'attachments', encode(rn.channel.title))
-                                            end
-                                            @app.ensure_dir(dir)
-                                            $logger.debug "Enclosure URL: #{curl}"
+                                            dir = save_dir(url, dir, encode(rn.channel.title))
+                                            $logger.info "Enclosure: #{curl}"
                                             fname = File.join(dir, encode(File.basename(curl) || item.title || item.pubDate.to_s || Time.now.to_s))
-                                            $logger.debug "Enclosure save to: #{fname}"
+                                            $logger.debug "Save enclosure: #{fname}"
                                             enc   = read_url(curl, 'rss_enclosure')
                                             write_file(fname, 'wb') {|io| io.puts enc}
                                             furl = file_url(fname)
-                                            enclosure = %{<p class="enclosure"><a href="%s" class="enclosure" />Enclosure (local copy)</a></p>} % furl
-                                            if get(url, :rss_rewrite_enclosed_urls)
+                                            enclosure = rss_enclosure_local_copy(url, furl)
+                                            if url_get(url, :rss_rewrite_enclosed_urls)
                                                 item.description.gsub!(Regexp.new(Regexp.escape(curl))) {|t| furl}
                                             end
                                         else
@@ -1249,7 +1303,7 @@ HTML
                             opts = @urls[url].dup
                             opts[:download] = :rss
                             opts[:title] = elt['title'] || elt['text'] || elt['htmlurl'] || curl
-                            @urls[curl] = opts
+                            url_set(curl, opts)
                             to_do curl
                         else
                             $logger.warn "Unsupported type in OPML: #{elt.to_s}"
@@ -1266,10 +1320,10 @@ HTML
             :download => lambda {|url| get_website_below(:body_html, url)}
 
         shortcut :website_txt, :delegate => :default,
-            :download => lambda {|url| html_to_text(get_website(get(url, :download_html, :openuri), url))}
+            :download => lambda {|url| html_to_text(get_website(url_get(url, :download_html, :openuri), url))}
 
         shortcut :website_txt_below, :delegate => :default,
-            :download => lambda {|url| html_to_text(get_website_below(get(url, :download_html, :openuri), url))}
+            :download => lambda {|url| html_to_text(get_website_below(url_get(url, :download_html, :openuri), url))}
 
         shortcut :ftp, :delegate => :default,
             :download => lambda {|url| get_ftp(url).join("\n")}
@@ -1277,7 +1331,7 @@ HTML
         shortcut :ftp_recursive, :delegate => :default,
             :download => lambda {|url|
                 list = get_ftp(url)
-                depth = get(url, :depth)
+                depth = url_get(url, :depth)
                 if !depth or depth >= 0
                     dirs = list.find_all {|e| e =~ /^d/}
                     dirs.each do |l|
@@ -1287,7 +1341,7 @@ HTML
                         opts = @urls[url].dup
                         opts[:title] = [opts[:title], File.basename(curl)].join(' - ')
                         opts[:depth] = depth - 1 if depth and depth >= 0
-                        @urls[curl] = opts
+                        url_set(curl, opts)
                         to_do curl
                     end
                 end
@@ -1307,7 +1361,7 @@ HTML
 <html>
 <head>
 <title>%s</title>
-<meta http-equiv="Content-Type" content="text/html; charset=#{get_optionvalue(:global, :encoding)}">
+<meta http-equiv="Content-Type" content="text/html; charset=#{optval_get(:global, :encoding)}">
 <link rel="stylesheet" href="websitary.css" type="text/css">
 <link rel="alternate" href="websitary.rss" type="application/rss+xml" title="%s">
 </head>
@@ -1463,6 +1517,9 @@ CSS
         begin
             self.instance_eval(contents)
             return true
+        rescue Exception => e
+            $logger.fatal "Error when reading profile: #{profile_file}\n#{e}"
+            exit 5
         ensure
             @current_profile = nil
         end
@@ -1470,9 +1527,9 @@ CSS
 
 
     def get_website(download, url)
-        html = call_cmd(get_optionvalue(:download, download), [url], :url => url)
+        html = call_cmd(optval_get(:download, download), [url], :url => url)
         if html
-            doc = Hpricot(html)
+            doc = url_document(url, html)
             if doc
                 return if robots?(doc, 'noindex')
                 push_hrefs(url, doc) do |uri0, pn0, uri, pn|
@@ -1486,10 +1543,10 @@ CSS
 
 
     def get_website_below(download, url)
-        dwnl = get_optionvalue(:download, download)
+        dwnl = optval_get(:download, download)
         html = call_cmd(dwnl, [url], :url => url)
         if html
-            doc = Hpricot(html)
+            doc = url_document(url, html)
             if doc
                 return if robots?(doc, 'noindex')
                 push_hrefs(url, doc) do |uri0, pn0, uri, pn|
@@ -1547,8 +1604,26 @@ CSS
     end
 
 
+    def url_document(url, html)
+        doc = html && Hpricot(html)
+        if doc
+            unless url_get(url, :title)
+                ti = (doc / 'head > title').inner_html
+                url_set(url, :title => ti) unless ti.empty?
+            end
+        end
+        doc
+    end
+
+
+    def read_document(url)
+        html = read_url(url, 'html')
+        html && url_document(url, html)
+    end
+
+
     def read_url(url, type='html')
-        downloader = get(url, "download_#{type}".intern)
+        downloader = url_get(url, "download_#{type}".intern)
         if downloader
             call_cmd(downloader, [url], :url => url)
         else
@@ -1568,9 +1643,13 @@ CSS
         if uri.instance_of?(URI::Generic) or uri.scheme == 'file'
             open(url).read
         else
-            header = {"User-Agent" => @user_agent}
-            header.merge!(get(url, :header, {}))
-            open(url, header).read
+            args = {"User-Agent" => optval_get(:global, :user_agent)}
+            args.merge!(url_get(url, :header, {}))
+            # proxy = get_proxy
+            # if proxy
+            #     args[:proxy] = proxy[0,2].join(':')
+            # end
+            open(url, args).read
         end
     end
 
@@ -1579,7 +1658,7 @@ CSS
         bak = oldname(url)
         lst = latestname(url)
         if File.exist?(bak) and File.exist?(lst)
-            eval_arg(get(url, :format_annotation, '%s >>> %s'), [@mtimes.mtime(bak), @mtimes.mtime(lst)])
+            eval_arg(url_get(url, :format_annotation, '%s >>> %s'), [@mtimes.mtime(bak), @mtimes.mtime(lst)])
         end
     end
 
@@ -1597,6 +1676,21 @@ CSS
     end
 
 
+    def rss_enclosure_local_copy(url, furl)
+        t = url_get(url, :rss_format_local_copy) ||
+            %{<p class="enclosure"><a href="%s" class="enclosure" />Enclosure (local copy)</a></p>}
+        case t
+        when Proc
+            t.call(url, furl)
+        when String
+            t % furl
+        else
+            $logger.fatal 'Argument for :rss_format_local_copy must be String or Proc: %s' % t.inspect
+            exit 5
+        end
+    end
+
+
     def format_rss_item(item, body, enclosure='')
         ti = rss_field(item, :title)
         au = rss_field(item, :author)
@@ -1627,6 +1721,46 @@ EOT
     end
 
 
+    def get_proxy
+        proxy = optval_get(:global, :proxy)
+        if proxy
+            case proxy
+            when String
+                proxy = proxy.split(':', 2)
+                if proxy.size == 1
+                    proxy << 8080
+                else
+                    proxy[1] = proxy[1].to_i
+                end
+            when Array
+            else
+                raise ArgumentError, 'proxy must be String or Array'
+            end
+        end
+        proxy
+    end
+
+
+    def body_html(url, doc)
+        doc &&= doc.at('body')
+        if doc
+            doc  = rewrite_urls(url, doc)
+            doc  = doc.inner_html
+            if (tags = url_get(url, :strip_tags))
+                doc = strip_tags(doc, :format => :hpricot, :tags => tags)
+            end
+        else
+            $logger.warn 'inner html: No body'
+        end
+        doc
+    end
+
+
+    def doc_to_text(doc)
+        doc && doc.to_plain_text
+    end
+
+
     # Convert html to plain text using hpricot.
     def html_to_text(text)
         text && Hpricot(text).to_plain_text
@@ -1660,10 +1794,10 @@ EOT
                 return true if rurl.nil? or rurl.empty?
                 begin
                     robots_txt = read_url(rurl, 'robots')
-                    rules      = RobotRules.new(@user_agent)
+                    rules      = RobotRules.new(optval_get(:global, :user_agent))
                     rules.parse(rurl, robots_txt)
                     @robots[host] = rules
-                    $logger.info "Loaded #{rurl} for #{@user_agent}"
+                    $logger.info "Loaded #{rurl} for #{optval_get(:global, :user_agent)}"
                     $logger.debug robots_txt
                 rescue Exception => e
                     $logger.info "#{rurl}: #{e}"
@@ -1705,7 +1839,7 @@ EOT
         difftext.sort! do |a, b|
             aa = a[0]
             bb = b[0]
-            get(aa, :title, aa).downcase <=> get(bb, :title, bb).downcase
+            url_get(aa, :title, aa).downcase <=> url_get(bb, :title, bb).downcase
         end
     end
 
@@ -1713,7 +1847,7 @@ EOT
     def file_url(filename)
         # filename = File.join(File.basename(File.dirname(filename)), File.basename(filename))
         # "file://#{encode(filename, ':/')}"
-        filename = call_cmd(get_optionvalue(:global, :file_url), [filename], :default => filename)
+        filename = call_cmd(optval_get(:global, :file_url), [filename], :default => filename)
         encode(filename, ':/')
     end
 

metadata

@@ -1,7 +1,7 @@
 --- !ruby/object:Gem::Specification 
 name: websitary
 version: !ruby/object:Gem::Version 
-  version: "0.4"
+  version: "0.5"
 platform: ruby
 authors: 
 - Thomas Link
@@ -9,7 +9,7 @@ autorequire:
 bindir: bin
 cert_chain: []
 
-date: 2008-01-13 00:00:00 +01:00
+date: 2008-05-24 00:00:00 +02:00
 default_executable: 
 dependencies: 
 - !ruby/object:Gem::Dependency 
@@ -28,7 +28,7 @@ dependencies:
     requirements: 
     - - ">="
       - !ruby/object:Gem::Version 
-        version: 1.4.0
+        version: 1.5.1
     version: 
 description: "== DESCRIPTION: websitary (formerly known as websitiary with an extra \"i\") monitors  webpages, rss feeds, podcasts etc. It reuses other programs (w3m, diff  etc.) to do most of the actual work. By default, it works on an ASCII  basis, i.e. with the output of text-based webbrowsers like w3m (or lynx,  links etc.) as the output can easily be post-processed. It can also work  with HTML and highlight new items. This script was originally planned as  a ruby-based websec replacement.  By default, this script will use w3m to dump HTML pages and then run  diff over the current page and the previous backup. Some pages are  better viewed with lynx or links. Downloaded documents (HTML or ASCII)  can be post-processed (e.g., filtered through some ruby block that  extracts elements via hpricot and the like). Please see the  configuration options below to find out how to change this globally or  for a single source.  This user manual is also available as PDF[http://websitiary.rubyforge.org/websitary.pdf].  == FEATURES/PROBLEMS: * Handle webpages, rss feeds (optionally save attachments in podcasts  etc.) * Compare webpages with previous backups * Display differences between the current version and the backup * Provide hooks to post-process the downloaded documents and the diff * Display a one-page report summarizing all news * Automatically open the report in your favourite web-browser * Experimental: Download webpages on defined intervalls and generate  incremental diffs."
 email: micathom at gmail com
@@ -75,7 +75,7 @@ required_rubygems_version: !ruby/object:Gem::Requirement
 requirements: []
 
 rubyforge_project: websitiary
-rubygems_version: 1.0.1
+rubygems_version: 1.1.1
 signing_key: 
 specification_version: 2
 summary: A unified website news, rss feed, podcast monitor