bwkfanboy 0.0.1

data/LICENSE

@@ -0,0 +1,22 @@
+(The MIT License)
+
+Copyright (c) 2010 Alexander Gromnitsky.
+
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of this software and associated documentation files (the
+'Software'), to deal in the Software without restriction, including
+without limitation the rights to use, copy, modify, merge, publish,
+distribute, sublicense, and/or sell copies of the Software, and to
+permit persons to whom the Software is furnished to do so, subject to
+the following conditions:
+
+The above copyright notice and this permission notice shall be
+included in all copies or substantial portions of the Software.
+
+THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
+IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
+CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
+TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
+SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

data/README.rdoc

@@ -0,0 +1,88 @@
+= About
+
+bwkfanboy is a HTML to Atom feed converter that you can use to watch
+sites that do not provide its own feed.
+
+The converter is not a magick tool: you'll need to write a plugin (in
+Ruby) for each site you want to watch. bwkfanboy provides guidelines and
+general assistance.
+
+= Architecture
+
+== Plugins
+
+bwkfanboy comes with 1 exmple plugin that parses a search page of
+dailyprincetonian.com looking for bwk's articles.
+
+The plugin is a Ruby class +Page+ that inherits Bwkfanboy::Parse
+parent, overriding 1 method.
+
+The plugins can be in the system
+
+  `gem env gemdir`/gems/bwkfanboy-x.y.z/lib/bwkfanboy/plugins
+
+or user's home
+
+  ~/.bwkfanboy/plugins
+
+directories.
+
+== Pipeline
+
+The program consists of 4 parts:
+
+0. *bwkfanboy* script that takes 1 parameter: the name of a file in
+   plugins directories (without the .rb suffix). So, for example to get
+   an atom feed from dailyprincetonian.com you type:
+
+     % bwkfanboy bwk
+
+   and it will load
+   <tt>/usr/local/lib/ruby/gems/1.9/gems/bwkfanboy-0.0.1/lib/bwkfanboy/plugins/bwk.rb</tt>
+   file on my FreeBSD machine, fetch and parse html from
+   dailyprincetonian.com and generate the required feed, dumping it to
+   stdout.
+
+   The script is just a convinient wrapper for 3 separate utils.
+
+1. *bwkfanboy_fetch*
+
+   It reads 1 line from stdin for the URL to fetch from. The result will
+   be dumped to stdout.
+
+2. *bwkfanboy_parse*
+
+   It takes 1 parameter: <em>a full path</em> to a plugin file.
+
+   This util reads stdin expecting it to be a xhtml, parses it and dumps
+   the result to stdout in JSON-formatted object.
+
+3. *bwkfanboy_generate*
+
+   Reads stdin expecting it to be a proper JSON-formatted object.
+
+   The result will be an Atom feed dumped to stdout in UTF-8.
+
+So, without the wrapper all this together looks like:
+
+  % echo http://example.org | bwkfanboy_fetch | 
+  	bwkfanboy_parse /path/to/my/plugin.rb | bwkfanboy_generate
+
+== Log
+
+All utils write to <tt>/tmp/bwkfanboy/USER/log/general.log</tt> file if
+permissions allows it.
+
+== HTTP
+
+There are 2 method to get an Atom feed via HTTP:
+
+1. <tt>web/bwkfanboy.cgi</tt> (from the program tarball), which you may
+   copy to your Apache cgi directory and run it. This prohibits you from
+   using HOME directory for your own plugins. Also the cgi script
+   requires some manual editing (setting 1 variable in it) before even
+   you can start utilizing it.
+
+2. Small *bwkfanboy_server* HTTP server. It can run from any user and
+   thus is able to inherit env variables for discovering your HOME
+   directory. Read bin/bwkfanboy_server to know how to operate it.

data/Rakefile

@@ -0,0 +1,48 @@
+# -*-ruby-*-
+
+require 'rake'
+require 'rake/gempackagetask'
+require 'rake/clean'
+require 'rake/rdoctask'
+require 'rake/testtask'
+
+spec = Gem::Specification.new() {|i|
+  i.name = "bwkfanboy"
+  i.summary = 'A converter from HTML to Atom feed that you can use to watch sites that do not provide its own feed.'
+  i.version = '0.0.1'
+  i.author = 'Alexander Gromnitsky'
+  i.email = 'alexander.gromnitsky@gmail.com'
+  i.homepage = 'http://github.com/gromnitsky/bwkfanboy'
+  i.platform = Gem::Platform::RUBY
+  i.required_ruby_version = '>= 1.9'
+  i.files = FileList['lib/**/*', 'bin/*', 'doc/*', '[A-Z]*', 'test/**/*']
+
+  i.executables = FileList['bin/*'].gsub(/^bin\//, '')
+  i.default_executable = i.name
+  
+  i.has_rdoc = true
+  i.test_files = FileList['test/test_*.rb']
+  
+  i.rdoc_options << '-m' << 'Bwkfanboy' << '-x' << 'plugins'
+  i.extra_rdoc_files = FileList['bin/*']
+  
+  i.add_dependency('activesupport', '>= 3.0.0')
+  i.add_dependency('nokogiri', '>=  1.4.3')
+  i.add_dependency('open4', '>=  1.0.1')
+  i.add_dependency('jsonschema', '>= 2.0.0')
+}
+
+Rake::GemPackageTask.new(spec).define()
+
+task(default: %(repackage))
+
+Rake::RDocTask.new('doc') {|i|
+  i.main = "Bwkfanboy"
+  i.rdoc_files = FileList['doc/*', 'lib/**/*.rb', 'bin/*']
+  i.rdoc_files.exclude("lib/**/plugins", "test")
+}
+
+Rake::TestTask.new() {|i|
+  i.test_files = FileList['test/test_*.rb']
+  i.verbose = true
+}

data/TODO

@@ -0,0 +1,7 @@
+-*-text-*-
+
+0.0.2
+-----
+
+- Add plugin listing to bwkfanboy_server.
+- More tests.

data/bin/bwkfanboy

@@ -0,0 +1,128 @@
+#!/usr/bin/env ruby19
+# -*-ruby-*-
+
+# This program is executed by bin/bwkfanboy_server to do all dirty work:
+# fetch HTML, parse it and generate a pretty Atom feed.
+#
+# It is a wrapper which you can utilize for such common tasks as listing
+# all available plugins.
+#
+# Type:
+#
+#   % bwkfanboy -h
+#
+# to get some basic help & read about Bwkfanboy module.
+
+require_relative '../lib/bwkfanboy/parser'
+
+$conf = {
+  prog_name: 'bwkfanboy',
+  prog_ver: '0.0.1',
+  mode: 'pipe',
+  banner: "Usage: #{File.basename($0)} [options] plugin-name"
+}
+
+class Plugin # :nodoc: all
+  attr_reader :name, :path
+  
+  def initialize(name)
+    @name = name
+    @path = nil
+  end
+
+  def dirs()
+    # try to create user's home plugin directory
+    begin
+      ['~/.bwkfanboy', '~/.bwkfanboy/plugins'].each {|i|
+        Dir.mkdir(File.expand_path(i))
+      }
+    rescue
+      # empty
+    end
+    
+    r = []
+    dirs = ['~/.bwkfanboy/plugins', "#{Bwkfanboy::Utils.gem_dir_system}/plugins"]
+    begin
+      # this will fail for user's home directory under Apache CGI
+      # environment
+      dirs.map! {|i| File.expand_path(i) }
+    rescue
+    end
+    dirs.each {|i|
+      if File.readable?(i) then
+        r << i
+      else
+        Bwkfanboy::Utils.warnx("directory #{i} isn't readable");
+      end
+    }
+
+    if r.length == 0 then
+      Bwkfanboy::Utils.errx(1, "no dirs for plugins found: #{dirs.join(' ')}")
+    end
+    return r
+  end
+
+  def load()
+    abort($conf[:banner]) unless (@name && @name !~ /^\s*$/)
+
+    dirs.each {|i|
+      files = Dir.glob("#{i}/*.rb")
+      if (@path = files.index("#{i}/#{@name}.rb")) then
+        @path = files[@path]
+        break
+      end
+    }
+    Bwkfanboy::Utils.errx(1, "no such plugin '#{@name}'") if ! @path
+    Bwkfanboy::Utils.plugin_load(@path, Bwkfanboy::Meta::PLUGIN_CLASS)
+    
+    pn = Page.new()
+    pn.check()
+    return pn
+  end
+  
+end # class
+  
+# ----------------------------------------------------------------------
+
+o = Bwkfanboy::Utils.cl_parse(ARGV, $conf[:banner]) # create OptionParser object
+o.on('-i', 'Show some info about the plugin') { |i| $conf[:mode] = 'info' }
+o.on('-l', 'List all plugins') { |i| $conf[:mode] = 'list' }
+o.on('-p', 'List all plugins paths') { |i| $conf[:mode] = 'path' }
+o.on('-D', '(ignore this) Use URI_DEBUG const instead URI in plugins') { |i| $conf[:mode] = 'debug' }
+Bwkfanboy::Utils.cl_parse(ARGV, $conf[:banner], o) # run cl parser
+
+plugin = Plugin.new(ARGV[0])
+
+case $conf[:mode]
+when 'list'
+  plugin.dirs().each {|i|
+    puts "#{i}:"
+    Dir.glob("#{i}/*.rb").each {|j|
+      puts "\t#{File.basename(j, '.rb')}"
+    }
+  }
+when 'path'
+  plugin.dirs().each {|i| puts i}
+when 'info'
+  plugin.load().dump_info
+else
+  # A pipe mode
+  pn = plugin.load()
+  cmd = "./bwkfanboy_fetch | ./bwkfanboy_parse '#{plugin.path}' | ./bwkfanboy_generate"
+  if Bwkfanboy::Utils.cfg[:verbose] >= 2 then
+    puts ($conf[:mode] != 'debug' ? pn.class::Meta::URI : pn.class::Meta::URI_DEBUG)
+    puts cmd
+    exit 0
+  end
+
+  # go to the directory with current script
+  Dir.chdir(File.dirname(File.expand_path($0)))
+  
+  pipe = IO.popen(cmd, 'w+')
+  pipe.puts ($conf[:mode] != 'debug' ? pn.class::Meta::URI : pn.class::Meta::URI_DEBUG)
+  pipe.close_write
+  while line = pipe.gets
+    puts line
+  end
+  pipe.close
+end

data/bin/bwkfanboy_fetch

@@ -0,0 +1,30 @@
+#!/usr/bin/env ruby19
+# -*-ruby-*-
+
+# Read stdin for a URI or a full path to the local file, download it (or
+# read for the local file) and print to stdout.
+
+require 'open-uri'
+
+require_relative '../lib/bwkfanboy/utils'
+
+$conf = { banner: "Usage: #{File.basename($0)} [options] < uri" }
+
+Bwkfanboy::Utils.cl_parse(ARGV, $conf[:banner], nil, true)
+
+uri = gets.chomp()
+
+Bwkfanboy::Utils.veputs(1, "fetching #{uri}\n")
+
+begin
+  open(uri, "User-Agent" => Bwkfanboy::Meta::USER_AGENT) {|f|
+    if defined?(f.meta) && f.status[0] != '200' then
+      Bwkfanboy::Utils.errx(1, "cannot fetch #{uri} : HTTP responce: #{f.status[0]}")
+    end
+    Bwkfanboy::Utils.veputs(1, "charset = #{f.content_type_parse[1][1]}\n") if defined?(f.meta)
+    f.each_line {|i| puts i}
+  }
+rescue
+  # typically Errno::ENOENT
+  Bwkfanboy::Utils.errx(1, "cannot fetch: #{$!}");
+end

data/bin/bwkfanboy_generate

@@ -0,0 +1,80 @@
+#!/usr/bin/env ruby19
+# -*-ruby-*-
+
+# Read stdin for JSON, generate from it an Atom feed and print the
+# result to stdout in UTF-8.
+#
+# One can validate the JSON by providing '--check' command line option
+# (by default the validating is off).
+
+require 'rss/maker'
+require 'date'
+require 'json'
+require 'jsonschema'
+
+require_relative '../lib/bwkfanboy/utils'
+
+$conf = {
+  banner: "Usage: #{File.basename($0)} [options] < json",
+  check: false
+}
+
+o = Bwkfanboy::Utils.cl_parse(ARGV, $conf[:banner])
+o.on('--check', 'Validate the input (slow!)') { |i| $conf[:check] = true }
+Bwkfanboy::Utils.cl_parse(ARGV, $conf[:banner], o) # run cl parser
+
+begin
+  j = JSON.parse(STDIN.read)
+rescue
+  Bwkfanboy::Utils.errx(1, "stdin had invalid JSON");
+end
+
+# validate the input
+schema = Bwkfanboy::Utils.gem_dir_system() + '/schema.js'
+if $conf[:check] then
+  begin
+    JSON::Schema.validate(j, JSON.parse(File.read(schema)))
+  rescue
+    Bwkfanboy::Utils.errx(1, "JSON validation with schema (#{schema}) failed");
+  end
+end
+
+feed = RSS::Maker.make("atom") { |maker|
+  maker.channel.id = j['channel']['id']
+  maker.channel.updated = j['channel']['updated']
+  maker.channel.author = j['channel']['author']
+  maker.channel.title = j['channel']['title']
+  
+  maker.channel.links.new_link {|i|
+    i.href = j['channel']['link']
+    i.rel = 'alternate'
+    i.type = 'text/html' # eh
+  }
+
+  maker.items.do_sort = true
+
+  j['x_entries'].each { |i|
+    maker.items.new_item do |item|
+      item.links.new_link {|k|
+        k.href = i['link']
+        k.rel = 'alternate'
+        k.type = 'text/html' # only to make happy crappy pr2nntp gateway
+      }
+      item.title = i['title']
+      item.author = i['author']
+      item.updated = i['updated']
+      item.content.type = j['channel']['x_entries_content_type']
+      
+      case item.content.type
+      when 'text'
+        item.content.content = i['content']
+      when 'html'
+        item.content.content = i['content']
+      else
+        item.content.xhtml = i['content']
+      end
+    end
+  }
+}
+
+puts feed

data/bin/bwkfanboy_parse

@@ -0,0 +1,32 @@
+#!/usr/bin/env ruby19
+# -*-ruby-*-
+
+# Take 1 command line parameter: a full path to a plugin.
+#
+# Read stdin for a HTML, parse it and print the result to stdout in JSON
+# format. If '-vv' command line parameters were given, output will be in
+# 'key: value' pairs and <em>not</em> in JSON.
+
+require_relative '../lib/bwkfanboy/parser'
+
+$conf = {
+  banner: "Usage: #{File.basename($0)} [options] /path/to/my/plugin.rb < html"
+}
+
+Bwkfanboy::Utils.cl_parse(ARGV, $conf[:banner], nil, true)
+
+if ARGV.size == 0 then
+  abort($conf[:banner])
+else
+  Bwkfanboy::Utils.plugin_load(ARGV[0], Bwkfanboy::Meta::PLUGIN_CLASS)
+end;
+
+pn = Page.new()
+pn.check()
+pn.parse()
+
+if Bwkfanboy::Utils.cfg[:verbose] >= 2 then
+  pn.dump()
+else
+  puts pn.to_json()
+end

data/bin/bwkfanboy_server

@@ -0,0 +1,141 @@
+#!/usr/bin/env ruby19
+# -*-ruby-*-
+
+# Start a HTTP server (by default on 127.0.0.1:9042). To get Atom feeds
+# from it, initiate GET request with URI
+#   
+#   http://localhost:9042/?p=PLUGIN
+#
+# where +PLUGIN+ is a name of a bwkfanboy's plugin (without '.re' suffix).
+#
+# To list all available plugins, point you browser to
+#
+#   http://localhost:9042/list
+#
+# The server is intended to run from a non-root user from
+# <tt>~/.login</tt> file. It can detach from a terminal if you give it
+# '-d' command line option.
+#
+# For other help, type:
+#
+#  bwkfanboy_server -h
+#
+# The server maintains 2 logs:
+#
+#   /tmp/bwkfanboy/USER/log/bwkfanboy_server.log
+#   /tmp/bwkfanboy/USER/log/bwkfanboy_server-access.log
+#
+# The file with a pid:
+#
+#   /tmp/bwkfanboy/USER/bwkfanboy_server.pid
+
+require 'webrick'
+require_relative '../lib/bwkfanboy/utils'
+
+$conf = {
+  addr: '127.0.0.1',
+  port: 9042,
+  converter: "./#{Bwkfanboy::Meta::NAME}",
+  banner: "Usage: #{File.basename($0)} [options]",
+  server_type: WEBrick::SimpleServer,
+  workdir: File.dirname(File.expand_path($0)),
+  pidfile: "#{Bwkfanboy::Meta::DIR_TMP}/#{File.basename($0)}.pid",
+  log: "#{Bwkfanboy::Meta::DIR_LOG}/#{File.basename($0)}.log",
+  alog: "#{Bwkfanboy::Meta::DIR_LOG}/#{File.basename($0)}-access.log",
+  mode: 'pipe'
+}
+
+o = Bwkfanboy::Utils.cl_parse(ARGV, $conf[:banner]) # create OptionParser object
+o.on('-b VAL', 'BindAddress') { |i| $conf[:addr] = i }
+o.on('-p VAL', 'A port number') { |i| $conf[:port] = i }
+o.on('-c VAL', "A path to main #{Bwkfanboy::Meta::NAME} executable") { |i| $conf[:converter] = i }
+o.on('-d', 'Detach from a terminal') {|i| $conf[:server_type] = WEBrick::Daemon }
+o.on('-D', '(ignore this) Use URI_DEBUG const instead URI in plugins') { |i| $conf[:mode] = 'debug' }
+Bwkfanboy::Utils.cl_parse(ARGV, $conf[:banner], o) # run cl parser
+
+Bwkfanboy::Utils.dir_tmp_create()
+
+class FeedServlet < WEBrick::HTTPServlet::AbstractServlet # :nodoc: all
+  def do_GET(req, res)
+    if req.query['p'] && req.query['p'] =~ Bwkfanboy::Meta::PLUGIN_NAME
+      res['Content-Type'] = 'application/atom+xml; charset=UTF-8'
+      res['Content-Disposition'] = "inline; filename=\"#{Bwkfanboy::Meta::NAME}-#{req.query['p']}.xml"
+
+      cmd = "#{$conf[:converter]} #{$conf[:mode] == 'debug' ? '-D' : ''} #{req.query['p']}"
+      r = Bwkfanboy::Utils.cmd_run(cmd)
+      if r[0] != 0 then
+        raise WEBrick::HTTPStatus::InternalServerError.new("Errors in the pipeline:\n\n #{r[1]}")
+      end
+      
+      res.body = r[2]
+    else
+      raise WEBrick::HTTPStatus::InternalServerError.new("Parameter 'p' required")
+    end
+  end
+end
+
+class FeedListServlet < WEBrick::HTTPServlet::AbstractServlet # :nodoc: all
+  def do_GET(req, res)
+      cmd = "#{$conf[:converter]} -l"
+      r = Bwkfanboy::Utils.cmd_run(cmd)
+      if r[0] != 0 then
+        raise WEBrick::HTTPStatus::InternalServerError.new("Errors:\n\n #{r[1]}")
+      end
+      
+      res.body = r[2]
+  end
+end
+
+# create temporally files
+def start_callback()
+  Dir.chdir($conf[:workdir])
+  if ! File.executable?($conf[:converter]) then
+    Bwkfanboy::Utils.errx(1, "Missing executable file '#{$conf[:converter]}'")
+  end
+  
+  begin
+    File.open($conf[:pidfile], "w+") {|i| i.puts $$ }
+  rescue
+    Bwkfanboy::Utils.warnx("unable to create a pidfile " + $conf[:pidfile])
+  end
+end
+
+# remove temporally files
+def stop_callback()
+  begin
+    File.unlink $conf[:pidfile]
+  rescue
+    # ignore errors
+  end
+end
+
+def log_create(f)
+  begin
+    log = Logger.new(f, 2, Bwkfanboy::Meta::LOG_MAXSIZE)
+  rescue
+    Bwkfanboy::Utils.warnx("cannot open log #{f}");
+    return nil
+  end
+  log.datetime_format = "%H:%M:%S"
+  log
+end
+
+# ----------------------------------------------------------------------
+
+server_log = log_create($conf[:log])
+access_log = [[ log_create($conf[:alog]), WEBrick::AccessLog::COMBINED_LOG_FORMAT ]]
+
+s = WEBrick::HTTPServer.new(Port: $conf[:port],
+                            BindAddress: $conf[:addr],
+                            ServerType: $conf[:server_type],
+                            StartCallback: -> {start_callback},
+                            StopCallback: -> {stop_callback},
+                            Logger: server_log,
+                            AccessLog: access_log
+                            )
+s.mount("/", FeedServlet)
+s.mount("/list", FeedListServlet)
+['TERM', 'INT'].each {|i|
+  trap(i) { s.shutdown }
+}
+s.start

data/doc/README.rdoc

@@ -0,0 +1,88 @@
+= About
+
+bwkfanboy is a HTML to Atom feed converter that you can use to watch
+sites that do not provide its own feed.
+
+The converter is not a magick tool: you'll need to write a plugin (in
+Ruby) for each site you want to watch. bwkfanboy provides guidelines and
+general assistance.
+
+= Architecture
+
+== Plugins
+
+bwkfanboy comes with 1 exmple plugin that parses a search page of
+dailyprincetonian.com looking for bwk's articles.
+
+The plugin is a Ruby class +Page+ that inherits Bwkfanboy::Parse
+parent, overriding 1 method.
+
+The plugins can be in the system
+
+  `gem env gemdir`/gems/bwkfanboy-x.y.z/lib/bwkfanboy/plugins
+
+or user's home
+
+  ~/.bwkfanboy/plugins
+
+directories.
+
+== Pipeline
+
+The program consists of 4 parts:
+
+0. *bwkfanboy* script that takes 1 parameter: the name of a file in
+   plugins directories (without the .rb suffix). So, for example to get
+   an atom feed from dailyprincetonian.com you type:
+
+     % bwkfanboy bwk
+
+   and it will load
+   <tt>/usr/local/lib/ruby/gems/1.9/gems/bwkfanboy-0.0.1/lib/bwkfanboy/plugins/bwk.rb</tt>
+   file on my FreeBSD machine, fetch and parse html from
+   dailyprincetonian.com and generate the required feed, dumping it to
+   stdout.
+
+   The script is just a convinient wrapper for 3 separate utils.
+
+1. *bwkfanboy_fetch*
+
+   It reads 1 line from stdin for the URL to fetch from. The result will
+   be dumped to stdout.
+
+2. *bwkfanboy_parse*
+
+   It takes 1 parameter: <em>a full path</em> to a plugin file.
+
+   This util reads stdin expecting it to be a xhtml, parses it and dumps
+   the result to stdout in JSON-formatted object.
+
+3. *bwkfanboy_generate*
+
+   Reads stdin expecting it to be a proper JSON-formatted object.
+
+   The result will be an Atom feed dumped to stdout in UTF-8.
+
+So, without the wrapper all this together looks like:
+
+  % echo http://example.org | bwkfanboy_fetch | 
+  	bwkfanboy_parse /path/to/my/plugin.rb | bwkfanboy_generate
+
+== Log
+
+All utils write to <tt>/tmp/bwkfanboy/USER/log/general.log</tt> file if
+permissions allows it.
+
+== HTTP
+
+There are 2 method to get an Atom feed via HTTP:
+
+1. <tt>web/bwkfanboy.cgi</tt> (from the program tarball), which you may
+   copy to your Apache cgi directory and run it. This prohibits you from
+   using HOME directory for your own plugins. Also the cgi script
+   requires some manual editing (setting 1 variable in it) before even
+   you can start utilizing it.
+
+2. Small *bwkfanboy_server* HTTP server. It can run from any user and
+   thus is able to inherit env variables for discovering your HOME
+   directory. Read bin/bwkfanboy_server to know how to operate it.