bwkfanboy 1.4.1 → 2.0.0

data/.gitignore

@@ -0,0 +1,4 @@
+pkg
+html
+.bundle
+vendor

data/Gemfile

@@ -0,0 +1,7 @@
+# -*-ruby-*-
+source 'https://rubygems.org'
+
+gem "thin", "~> 1.3.1"
+
+# Specify your gem's dependencies in bwkfanboy.gemspec
+gemspec

data/Gemfile.lock

@@ -0,0 +1,51 @@
+PATH
+  remote: .
+  specs:
+    bwkfanboy (2.0.0)
+      haml (~> 3.1.4)
+      msgpack (~> 0.4.6)
+      nokogiri (~> 1.5.2)
+      open4 (~> 1.1.0)
+      rake (~> 0.9.2.2)
+      rdoc (~> 3.12)
+      sinatra (~> 1.3.2)
+
+GEM
+  remote: https://rubygems.org/
+  specs:
+    daemons (1.1.8)
+    eventmachine (0.12.10)
+    fakefs (0.4.0)
+    haml (3.1.4)
+    json (1.6.6)
+    minitest (2.12.1)
+    msgpack (0.4.6)
+    nokogiri (1.5.2)
+    open4 (1.1.0)
+    rack (1.4.1)
+    rack-protection (1.2.0)
+      rack
+    rack-test (0.6.1)
+      rack (>= 1.0)
+    rake (0.9.2.2)
+    rdoc (3.12)
+      json (~> 1.4)
+    sinatra (1.3.2)
+      rack (~> 1.3, >= 1.3.6)
+      rack-protection (~> 1.2)
+      tilt (~> 1.3, >= 1.3.3)
+    thin (1.3.1)
+      daemons (>= 1.0.9)
+      eventmachine (>= 0.12.6)
+      rack (>= 1.0.0)
+    tilt (1.3.3)
+
+PLATFORMS
+  ruby
+
+DEPENDENCIES
+  bwkfanboy!
+  fakefs (~> 0.4.0)
+  minitest (~> 2.12.1)
+  rack-test (~> 0.6.1)
+  thin (~> 1.3.1)

data/Procfile

@@ -0,0 +1 @@
+web: bundle exec rackup config.ru -p $PORT

data/README.rdoc

@@ -1,15 +1,17 @@
 = About
 
+A live example: http://bwkfanboy.herokuapp.com
+
 bwkfanboy is a converter from a raw HTML to an Atom feed. You can use it
 to watch sites that do not provide its own feed.
 
 The converter is not a magick tool: you'll need to write a plugin (in
-Ruby) for each site you want to watch. bwkfanboy gives guidelines and
-general assistance.
+Ruby) for an each site you want to watch. bwkfanboy gives guidelines and
+a general assistance.
 
-(Plugins from version 1.1.4 are *incompatible* with the 0.1.x
-series. Please reread in doc/plugin.rdoc the example of the skeleton
-plugin.)
+<b> Version 2.0.0 was rewritten form scratch. Plugins from previous
+versions will not work.</b> See http://github.com/gromnitsky/bwkfanboy_old
+repo for old versions.
 
 = Included Plugins
 
@@ -23,30 +25,36 @@ plugin.)
 
 = Installation
 
-The easiest way is to type:
+Install bundler:
+
+  # gem install bundler
+
+Install all dependencies. You have 2 choices:
+
+1. Usual via
+
+     # bundle install
+
+2. Or only for this program
 
-  # gem install bwkfanboy
+     % bundle install --path vendor/bundle
 
-Beware that plugins above are usually updated more frequently than the
-whole gem on rubygems.org, so grab the source before struggling with a
-particular plugin.
+   You'll have to prefix other commands with 'bundle exec' though.
 
-If the setup with <tt>gem install ...</tt> fails, go to
-https://rubygems.org/gems/bwkfanboy and click 'Download' link, save the
-file and run:
+Run application:
 
-  # gem install -l /where/is/your/bwkfanboy-x.y.z.gem
+  % rackup -E production
 
+Point a browser to http://127.0.0.1:9292
+
+  
 = Architecture
 
 == Plugins
 
-The plugin is a Ruby class +Page+ that inherits Bwkfanboy::Parse parent,
-overriding 1 method.
-
 Plugins can be in system
 
-  `gem env gemdir`/gems/bwkfanboy-x.y.z/lib/bwkfanboy/plugins
+  `gem env gemdir`/gems/bwkfanboy-x.y.z/plugins
 
 or user's home
 
@@ -56,86 +64,41 @@ directory.
 
 == Pipeline
 
-The program consists of 4 parts:
+The program consists of 3 parts:
 
 0. *bwkfanboy* script that takes 1 parameter: the name of a file in
-   plugins directories (without the .rb suffix). So, for example to get
+   plugin directories (without the .rb suffix). So, for example to get
    an atom feed from dailyprincetonian.com, you type:
 
      % bwkfanboy bwk
 
    and it will load
-   <tt>/usr/local/lib/ruby/gems/1.9/gems/bwkfanboy-0.0.1/lib/bwkfanboy/plugins/bwk.rb</tt>
+   <tt>/usr/local/lib/ruby/gems/1.9/gems/bwkfanboy-2.0.0/plugins/bwk.rb</tt>
    file on my FreeBSD machine, fetch and parse the html from
    dailyprincetonian.com and generate the required feed, dumping it to
    stdout.
 
-   The script is just a convenient wrapper for 3 separate utils.
-
-1. *bwkfanboy_fetch*
+   The script is just a convenient wrapper for 2 separate
+   utils below. (Although it doesn't executes them, but uses their internal
+   api.)
 
-   Reads 1 line from stdin for the URL to fetch from. The result will be
-   dumped to stdout.
+1. *bwkfanboy_parse*
 
-2. *bwkfanboy_parse*
+   Takes 1 parameter: a plugin name.
 
-   Takes 1 parameter: <em>a full path</em> to a plugin file.
+   This util reads stdin, expecting it to be a html, parses it and
+   dumps the result to stdout as MessagePack object in UTF-8.
 
-   This util reads stdin, expecting it to be a xhtml, parses it and
-   dumps the result to stdout in JSON-formatted object.
+2. *bwkfanboy_generate*
 
-3. *bwkfanboy_generate*
-
-   Reads stdin expecting it to be a proper JSON-formatted object.
+   Reads stdin expecting it to be a proper MessagePack object.
 
    The result will be an Atom feed dumped to stdout in UTF-8.
 
 So, without the wrapper all this together looks like:
 
-  % echo http://example.org | bwkfanboy_fetch | 
-  	bwkfanboy_parse /path/to/my/plugin.rb | bwkfanboy_generate
-
-== Log
-
-All utils write to <tt>/tmp/bwkfanboy/USER/log/general.log</tt> file if
-permissions allows it.
+  % curl http://example.org | bwkfanboy_parse myplugin | bwkfanboy_generate
 
 == HTTP
 
-There are 2 method to get an Atom feed via HTTP:
-
-1. <tt>web/bwkfanboy.cgi</tt> (from the program tarball), which you may
-   copy to your Apache cgi directory and run it. This prohibits you from
-   using HOME directory for your own plugins. Also the cgi script
-   requires some manual editing (setting 1 variable in it) before even
-   you can start utilizing it.
-
-2. Small *bwkfanboy_server* HTTP server. It can run from any user and
-   thus is able to inherit env variables for discovering your HOME
-   directory. Read doc/bwkfanboy_server.rdoc to know how to operate it.
-
-= License
-
-(The MIT License)
-
-Copyright (c) 2010 Alexander Gromnitsky.
-
-Permission is hereby granted, free of charge, to any person obtaining
-a copy of this software and associated documentation files (the
-'Software'), to deal in the Software without restriction, including
-without limitation the rights to use, copy, modify, merge, publish,
-distribute, sublicense, and/or sell copies of the Software, and to
-permit persons to whom the Software is furnished to do so, subject to
-the following conditions:
-
-The above copyright notice and this permission notice shall be
-included in all copies or substantial portions of the Software.
-
-THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
-EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
-MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
-IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
-CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
-TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
-SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
-
+bwkfanboy includes a small sinatra app. See the installation phase.

data/Rakefile

@@ -1,59 +1,24 @@
 # -*-ruby-*-
 
-require 'rake'
-require 'rake/gempackagetask'
 require 'rake/clean'
-require 'rake/rdoctask'
 require 'rake/testtask'
-require 'json'
+require 'bundler/gem_tasks'
+gem 'rdoc'
+require 'rdoc/task'
 
-require_relative 'test/rake_git'
-require_relative 'doc/rakefile'
+task default: [:test]
 
-spec = Gem::Specification.new {|i|
-  i.name = "bwkfanboy"
-  i.summary = "#{i.name} is a converter from a raw HTML to an Atom feed. You can use it to watch sites that do not provide its own feed."
-  i.description = i.summary
-  i.version = `bin/#{i.name} -V`
-  i.author = 'Alexander Gromnitsky'
-  i.email = 'alexander.gromnitsky@gmail.com'
-  i.homepage = "http://github.com/gromnitsky/#{i.name}"
-  i.platform = Gem::Platform::RUBY
-  i.required_ruby_version = '>= 1.9.2'
-  i.files = git_ls('.')
-
-  i.executables = FileList['bin/*'].gsub(/^bin\//, '')
-
-  i.test_files = FileList['test/test_*.rb']
-
-  i.rdoc_options << '-m' << 'doc/README.rdoc' << '-x' << 'plugins'
-  i.extra_rdoc_files = FileList['doc/*.rdoc']
-
-  i.add_dependency('open4', '>=  1.0.1')
-  i.add_dependency('activesupport', '>= 3.0.5')
-  i.add_dependency('nokogiri', '>=  1.4.4')
-  i.add_dependency('jsonschema', '>= 2.0.1')
-
-  i.add_development_dependency('git', '>= 1.2.5')
-}
-
-Rake::GemPackageTask.new(spec).define
-
-task gen_mydocs: ['mydocs:default'] do
-  spec.extra_rdoc_files.concat(MyDocs::RDOC_RELATIVE)
-end
-
-task default: [:gen_mydocs, :repackage]
-task doc: [:gen_mydocs, :rdoc]
-task clobber: ['mydocs:clean']
-
-Rake::RDocTask.new do |i|
-  i.rdoc_files = FileList['doc/*', 'lib/**/*.rb']
-  i.rdoc_files.exclude("lib/**/plugins")
-  i.main = 'doc/README.rdoc'
+RDoc::Task.new('html') do |i|
+  i.main = 'README.rdoc'
+  i.rdoc_files = FileList['README.rdoc', 'doc/*', 'lib/**/*.rb']
+#  i.rdoc_files.exclude("lib/**/some-nasty-staff")
 end
 
 Rake::TestTask.new do |i|
   i.test_files = FileList['test/test_*.rb']
-  i.libs << '.'
+end
+
+desc "Insert all 'requires' to speedup reload"
+task :shotgunreq do
+  sh "git grep 'require ' -- '*.rb' | awk -F: '{print $2}' | grep -v test | sort | uniq > shotgun.rb"
 end

data/bin/bwkfanboy

@@ -1,185 +1,66 @@
 #!/usr/bin/env ruby
-# -*-ruby-*-
 
-# This program is executed by bin/bwkfanboy_server to do all dirty work:
-# fetch HTML, parse it and generate a pretty Atom feed.
-#
-# It is a wrapper which you can utilize for such common tasks as listing
-# all available plugins.
-#
-# Type:
-#
-#   % bwkfanboy -h
-#
-# to get some basic help--read about Bwkfanboy module.
+require_relative '../lib/bwkfanboy/home'
+require_relative '../lib/bwkfanboy/utils'
+include Bwkfanboy
 
-require 'shellwords'
-require_relative '../lib/bwkfanboy/parser'
+### main
 
-$conf = {
-  mode: 'fast',
-  skeleton_name: 'bwkfanboy_plugin_skeleton.rb',
-  skeleton: Bwkfanboy::Utils.gem_dir_system() + '/plugin_skeleton.erb',
-  debug: false,
-  banner: "Usage: #{File.basename($0)} [options] plugin-name"
-}
-
-class Plugin # :nodoc: all
-  attr_reader :name, :path
-  
-  def initialize(name)
-    @name = name
-    @path = nil
-  end
-
-  def dirs()
-    # try to create user's home plugin directory
-    begin
-      ['~/.bwkfanboy', '~/.bwkfanboy/plugins'].each {|i|
-        Dir.mkdir(File.expand_path(i))
-      }
-    rescue
-      # empty
-    end
-    
-    r = []
-    dirs = ['~/.bwkfanboy/plugins', "#{Bwkfanboy::Utils.gem_dir_system}/plugins"]
-    begin
-      # this will fail for user's home directory under Apache CGI
-      # environment
-      dirs.map! {|i| File.expand_path(i) }
-    rescue
-    end
-    dirs.each {|i|
-      if File.readable?(i) then
-        r << i
-      else
-        Bwkfanboy::Utils.warnx("directory #{i} isn't readable");
-      end
-    }
-
-    if r.length == 0 then
-      Bwkfanboy::Utils.errx(1, "no dirs for plugins found: #{dirs.join(' ')}")
-    end
-    return r
-  end
-
-  def load(opt)
-    abort($conf[:banner]) unless (@name && @name !~ /^\s*$/)
+mode = nil
+$home = Home.new {|loader, o|
+  loader.conf[:banner] = "#{File.basename $0} [options] plugin [opt1 opt2 ...]"
 
-    dirs.each {|i|
-      files = Dir.glob("#{i}/*.rb")
-      if (@path = files.index("#{i}/#{@name}.rb")) then
-        @path = files[@path]
-        break
-      end
-    }
-    Bwkfanboy::Utils.errx(1, "no such plugin '#{@name}'") if ! @path
-    Bwkfanboy::Utils.plugin_load(@path, Bwkfanboy::Meta::PLUGIN_CLASS)
-    
-    pn = Page.new(opt)
-    pn.check()
-    return pn
-  end
-  
-end # class
-  
-# ----------------------------------------------------------------------
+  o.on('-l', 'List all plugins.') {
+    mode = :list
+  }
+  o.on('-i', 'Show some info about the plugin.') {
+    mode = :info
+  }
+  o.on('-s FILE', 'Write to the FILE a skeleton for a plugin.') {|i|
+    mode = :template
+    loader.conf[:skeleton] = i
+    loader.conf[:template] = (CliUtils::DIR_LIB_SRC + 'plugin_skeleton.erb').to_s
+  }
 
-o = Bwkfanboy::Utils.cl_parse(ARGV, $conf[:banner]) # create OptionParser object
-o.on('-i', 'Show some info about the plugin') { |i| $conf[:mode] = 'info' }
-o.on('-l', 'List all plugins') { |i| $conf[:mode] = 'list' }
-o.on('-p', 'List all plugins paths') { |i| $conf[:mode] = 'path' }
-o.on('-t FILE', 'Write to the FILE a skeleton for a plugin') { |i|
-  $conf[:mode] = 'template'
-  $conf[:skeleton_name] = i if i !~ /^\s*$/
 }
-o.on('-O', '(ignore this) Execute all bwkfanboy_* utils in a pipe') { |i| $conf[:mode] = 'pipe' }
-o.on('-D', '(ignore this) Use URI_DEBUG const instead URI in plugins') { |i| $conf[:debug] = true }
-Bwkfanboy::Utils.cl_parse(ARGV, $conf[:banner], o) # run cl parser
 
-plugin = Plugin.new(ARGV[0])
-opt = Bwkfanboy::Utils.plugin_opts(ARGV)
+if !mode
+  CliUtils.errx EX_USAGE, $home.conf[:banner] unless BH.all_set?(ARGV[0])
+end
 
-case $conf[:mode]
-when 'template'
-  require 'erb'
-  require 'digest/md5'
-  require 'etc'
+case mode
+when :list
+  list = PluginInfo.getList $home.conf[:plugins_path]
+  list.each {|idx|
+    puts idx.keys.first + ':'
+    idx[idx.keys.first].each {|file|
+      puts "\t#{file}"
+    }
+  }
 
-  t = ERB.new(File.read($conf[:skeleton]))
-  t.filename = $conf[:skeleton] # to report errors relative to this file
+when :info
   begin
-    md5_system = Digest::MD5.hexdigest(t.result(binding))
-  rescue Exception
-    Bwkfanboy::Utils.errx(1, "cannot read the template: #{$!}")
+    r = PluginInfo.about $home.conf[:plugins_path], ARGV[0], ARGV[1..-1]
+  rescue  PluginException
+    CliUtils.errx EX_DATAERR, $!.to_s
   end
 
-  if ! File.exists?($conf[:skeleton_name])
-    # write a skeleton
-    begin
-      File.open($conf[:skeleton_name], 'w+') { |fp| fp.puts t.result(binding) }
-    rescue
-      Bwkfanboy::Utils.errx(1, "cannot write the skeleton: #{$!}")
-    end
-  elsif
-    # warn a careless user
-    if md5_system != Digest::MD5.file($conf[:skeleton_name]).hexdigest
-      Bwkfanboy::Utils.errx(1, "#{$conf[:skeleton_name]} already exists")
-    end
-  end
-  
-when 'list'
-  plugin.dirs().each {|i|
-    puts "#{i}:"
-    Dir.glob("#{i}/*.rb").each {|j|
-      puts "\t#{File.basename(j, '.rb')}"
-    }
+  r.each {|key, val|
+    next if key == 'uri'
+    puts "%-9s : %s" % [key.upcase, val]
   }
-when 'path'
-  plugin.dirs().each {|i| puts i}
-when 'info'
-  plugin.load(opt).dump_info
-when 'version'
-  puts Bwkfanboy::Meta::VERSION
-when 'pipe'
-  pn = plugin.load(opt)
-  cmd = "./bwkfanboy_fetch | ./bwkfanboy_parse '#{plugin.path}' #{opt.size != 0 ? Shellwords.join(opt) : ''} | ./bwkfanboy_generate"
-  if Bwkfanboy::Utils.cfg[:verbose] >= 2 then
-    puts (!$conf[:debug] ? pn.uri() : pn.class::Meta::URI_DEBUG)
-    puts cmd
-    exit 0
-  end
-
-  # go to the directory with current script
-  Dir.chdir(File.dirname(File.expand_path($0)))
-  
-  pipe = IO.popen(cmd, 'w+')
-  pipe.puts (!$conf[:debug] ? pn.uri : pn.class::Meta::URI_DEBUG)
-  pipe.close_write
-  while line = pipe.gets
-    puts line
-  end
-  pipe.close
+  puts "URI       : #{r['uri'].size}\n\n"
+  r['uri'].each {|idx| puts idx }
   
-else
-  # a 'new', faster way
-  pn = plugin.load(opt)
 
-  require_relative '../lib/bwkfanboy/fetch'
-  require_relative '../lib/bwkfanboy/generate'
+when :template
+  exit EX_CANTCREAT unless Utils.skeletonCreate $home.conf[:template], "#{$home.conf[:skeleton]}.rb"
 
-  # 1. fetch & parse
-  pn = Page.new(opt)
-  Bwkfanboy::Fetch.cat(!$conf[:debug] ? pn.uri() : pn.class::Meta::URI_DEBUG) {|stream|
-    pn.parse(stream)
-  }
-  
-  if Bwkfanboy::Utils.cfg[:verbose] >= 2 then
-    pn.dump()
-    exit 0
+else
+  begin
+    puts Utils.atom($home.conf[:plugins_path], ARGV[0], ARGV[1..-1])
+  rescue FetchException, PluginException, GeneratorException
+    CliUtils.errx EX_DATAERR, $!.to_s
   end
-  
-  # 2.generate
-  puts Bwkfanboy::Generate.atom(JSON.parse(pn.to_json))
+
 end