bwkfanboy 1.4.1 → 2.0.0

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (81) hide show
  1. data/.gitignore +4 -0
  2. data/Gemfile +7 -0
  3. data/Gemfile.lock +51 -0
  4. data/Procfile +1 -0
  5. data/README.rdoc +40 -77
  6. data/Rakefile +13 -48
  7. data/bin/bwkfanboy +47 -166
  8. data/bin/bwkfanboy_generate +7 -19
  9. data/bin/bwkfanboy_parse +21 -17
  10. data/bwkfanboy.gemspec +40 -0
  11. data/config.ru +3 -0
  12. data/doc/NEWS.rdoc +21 -79
  13. data/doc/plugin.rdoc +63 -79
  14. data/etc/bwkfanboy.yaml +2 -0
  15. data/etc/sinatra.rb +34 -0
  16. data/lib/bwkfanboy/cliconfig.rb +141 -0
  17. data/lib/bwkfanboy/cliutils.rb +114 -0
  18. data/lib/bwkfanboy/fetch.rb +22 -24
  19. data/lib/bwkfanboy/generator.rb +78 -0
  20. data/lib/bwkfanboy/home.rb +53 -0
  21. data/lib/bwkfanboy/meta.rb +5 -2
  22. data/lib/bwkfanboy/plugin.rb +247 -0
  23. data/lib/bwkfanboy/plugin_skeleton.erb +19 -23
  24. data/lib/bwkfanboy/server.rb +73 -0
  25. data/lib/bwkfanboy/utils.rb +39 -129
  26. data/plugins/bwk.rb +25 -0
  27. data/plugins/econlib.rb +22 -0
  28. data/plugins/freebsd-ports-update.rb +73 -0
  29. data/plugins/inc.rb +29 -0
  30. data/plugins/test.rb +29 -0
  31. data/public/.gitattributes +1 -0
  32. data/public/favicon.ico +0 -0
  33. data/public/jquery-1.7.2.min.js +0 -0
  34. data/public/list.js +111 -0
  35. data/public/loading.gif +0 -0
  36. data/public/style.css +54 -0
  37. data/shotgun.rb +20 -0
  38. data/test/example/.gitattributes +1 -0
  39. data/test/example/.gitignore +1 -0
  40. data/test/example/02/plugins/bwk.html +0 -0
  41. data/test/{plugins → example/02/plugins}/empty.rb +0 -0
  42. data/test/example/02/plugins/garbage.rb +1 -0
  43. data/test/example/02/plugins/inc.html +0 -0
  44. data/test/helper.rb +30 -27
  45. data/test/helper_cliutils.rb +34 -0
  46. data/test/test_cli.rb +86 -0
  47. data/test/test_fetch.rb +49 -18
  48. data/test/test_generate.rb +43 -16
  49. data/test/test_home.rb +33 -0
  50. data/test/test_plugin.rb +141 -0
  51. data/test/test_server.rb +21 -32
  52. data/views/list.haml +38 -0
  53. metadata +223 -110
  54. data/bin/bwkfanboy_fetch +0 -13
  55. data/bin/bwkfanboy_server +0 -126
  56. data/doc/README.erb +0 -114
  57. data/doc/README.rdoc +0 -141
  58. data/doc/TODO +0 -7
  59. data/doc/bwkfanboy_fetch.rdoc +0 -4
  60. data/doc/bwkfanboy_generate.rdoc +0 -7
  61. data/doc/bwkfanboy_parse.rdoc +0 -7
  62. data/doc/bwkfanboy_server.rdoc +0 -35
  63. data/doc/rakefile.rb +0 -59
  64. data/lib/bwkfanboy/generate.rb +0 -63
  65. data/lib/bwkfanboy/parser.rb +0 -156
  66. data/lib/bwkfanboy/plugins/bwk.rb +0 -33
  67. data/lib/bwkfanboy/plugins/econlib.rb +0 -34
  68. data/lib/bwkfanboy/plugins/freebsd-ports-update.rb +0 -76
  69. data/lib/bwkfanboy/plugins/inc.rb +0 -37
  70. data/lib/bwkfanboy/schema.js +0 -39
  71. data/test/popen4.sh +0 -4
  72. data/test/rake_git.rb +0 -36
  73. data/test/semis/Rakefile +0 -35
  74. data/test/semis/bwk.html +0 -393
  75. data/test/semis/bwk.json +0 -82
  76. data/test/semis/econlib.html +0 -21
  77. data/test/semis/inc.html +0 -1067
  78. data/test/semis/links.txt +0 -4
  79. data/test/test_parse.rb +0 -27
  80. data/test/xml-clean.sh +0 -8
  81. data/web/bwkfanboy.cgi +0 -36
@@ -1,114 +0,0 @@
1
- = About
2
-
3
- bwkfanboy is a converter from a raw HTML to an Atom feed. You can use it
4
- to watch sites that do not provide its own feed.
5
-
6
- The converter is not a magick tool: you'll need to write a plugin (in
7
- Ruby) for each site you want to watch. bwkfanboy gives guidelines and
8
- general assistance.
9
-
10
- (Plugins from version 1.1.4 are *incompatible* with the 0.1.x
11
- series. Please reread in doc/plugin.rdoc the example of the skeleton
12
- plugin.)
13
-
14
- = Included Plugins
15
-
16
- <% MyDocs.plugins.each { |i| %> <%= i %>
17
- <% } %>
18
-
19
- = Installation
20
-
21
- The easiest way is to type:
22
-
23
- # gem install bwkfanboy
24
-
25
- Beware that plugins above are usually updated more frequently than the
26
- whole gem on rubygems.org, so grab the source before struggling with a
27
- particular plugin.
28
-
29
- If the setup with <tt>gem install ...</tt> fails, go to
30
- https://rubygems.org/gems/bwkfanboy and click 'Download' link, save the
31
- file and run:
32
-
33
- # gem install -l /where/is/your/bwkfanboy-x.y.z.gem
34
-
35
- = Architecture
36
-
37
- == Plugins
38
-
39
- The plugin is a Ruby class +Page+ that inherits Bwkfanboy::Parse parent,
40
- overriding 1 method.
41
-
42
- Plugins can be in system
43
-
44
- `gem env gemdir`/gems/bwkfanboy-x.y.z/lib/bwkfanboy/plugins
45
-
46
- or user's home
47
-
48
- ~/.bwkfanboy/plugins
49
-
50
- directory.
51
-
52
- == Pipeline
53
-
54
- The program consists of 4 parts:
55
-
56
- 0. *bwkfanboy* script that takes 1 parameter: the name of a file in
57
- plugins directories (without the .rb suffix). So, for example to get
58
- an atom feed from dailyprincetonian.com, you type:
59
-
60
- % bwkfanboy bwk
61
-
62
- and it will load
63
- <tt>/usr/local/lib/ruby/gems/1.9/gems/bwkfanboy-0.0.1/lib/bwkfanboy/plugins/bwk.rb</tt>
64
- file on my FreeBSD machine, fetch and parse the html from
65
- dailyprincetonian.com and generate the required feed, dumping it to
66
- stdout.
67
-
68
- The script is just a convenient wrapper for 3 separate utils.
69
-
70
- 1. *bwkfanboy_fetch*
71
-
72
- Reads 1 line from stdin for the URL to fetch from. The result will be
73
- dumped to stdout.
74
-
75
- 2. *bwkfanboy_parse*
76
-
77
- Takes 1 parameter: <em>a full path</em> to a plugin file.
78
-
79
- This util reads stdin, expecting it to be a xhtml, parses it and
80
- dumps the result to stdout in JSON-formatted object.
81
-
82
- 3. *bwkfanboy_generate*
83
-
84
- Reads stdin expecting it to be a proper JSON-formatted object.
85
-
86
- The result will be an Atom feed dumped to stdout in UTF-8.
87
-
88
- So, without the wrapper all this together looks like:
89
-
90
- % echo http://example.org | bwkfanboy_fetch |
91
- bwkfanboy_parse /path/to/my/plugin.rb | bwkfanboy_generate
92
-
93
- == Log
94
-
95
- All utils write to <tt>/tmp/bwkfanboy/USER/log/general.log</tt> file if
96
- permissions allows it.
97
-
98
- == HTTP
99
-
100
- There are 2 method to get an Atom feed via HTTP:
101
-
102
- 1. <tt>web/bwkfanboy.cgi</tt> (from the program tarball), which you may
103
- copy to your Apache cgi directory and run it. This prohibits you from
104
- using HOME directory for your own plugins. Also the cgi script
105
- requires some manual editing (setting 1 variable in it) before even
106
- you can start utilizing it.
107
-
108
- 2. Small *bwkfanboy_server* HTTP server. It can run from any user and
109
- thus is able to inherit env variables for discovering your HOME
110
- directory. Read doc/bwkfanboy_server.rdoc to know how to operate it.
111
-
112
- = License
113
-
114
- <%= File.read(MyDocs::PATH + 'LICENSE') %>
@@ -1,141 +0,0 @@
1
- = About
2
-
3
- bwkfanboy is a converter from a raw HTML to an Atom feed. You can use it
4
- to watch sites that do not provide its own feed.
5
-
6
- The converter is not a magick tool: you'll need to write a plugin (in
7
- Ruby) for each site you want to watch. bwkfanboy gives guidelines and
8
- general assistance.
9
-
10
- (Plugins from version 1.1.4 are *incompatible* with the 0.1.x
11
- series. Please reread in doc/plugin.rdoc the example of the skeleton
12
- plugin.)
13
-
14
- = Included Plugins
15
-
16
- NAME VER DESCRIPTION
17
-
18
- bwk 2 Brian Kernighan's articles from Daily Princetonian
19
- freebsd-ports-update 3 News from FreeBSD ports
20
- inc 1 Articles (per-user) from inc.com
21
- econlib 1 Latest articles from econlib.org
22
-
23
-
24
- = Installation
25
-
26
- The easiest way is to type:
27
-
28
- # gem install bwkfanboy
29
-
30
- Beware that plugins above are usually updated more frequently than the
31
- whole gem on rubygems.org, so grab the source before struggling with a
32
- particular plugin.
33
-
34
- If the setup with <tt>gem install ...</tt> fails, go to
35
- https://rubygems.org/gems/bwkfanboy and click 'Download' link, save the
36
- file and run:
37
-
38
- # gem install -l /where/is/your/bwkfanboy-x.y.z.gem
39
-
40
- = Architecture
41
-
42
- == Plugins
43
-
44
- The plugin is a Ruby class +Page+ that inherits Bwkfanboy::Parse parent,
45
- overriding 1 method.
46
-
47
- Plugins can be in system
48
-
49
- `gem env gemdir`/gems/bwkfanboy-x.y.z/lib/bwkfanboy/plugins
50
-
51
- or user's home
52
-
53
- ~/.bwkfanboy/plugins
54
-
55
- directory.
56
-
57
- == Pipeline
58
-
59
- The program consists of 4 parts:
60
-
61
- 0. *bwkfanboy* script that takes 1 parameter: the name of a file in
62
- plugins directories (without the .rb suffix). So, for example to get
63
- an atom feed from dailyprincetonian.com, you type:
64
-
65
- % bwkfanboy bwk
66
-
67
- and it will load
68
- <tt>/usr/local/lib/ruby/gems/1.9/gems/bwkfanboy-0.0.1/lib/bwkfanboy/plugins/bwk.rb</tt>
69
- file on my FreeBSD machine, fetch and parse the html from
70
- dailyprincetonian.com and generate the required feed, dumping it to
71
- stdout.
72
-
73
- The script is just a convenient wrapper for 3 separate utils.
74
-
75
- 1. *bwkfanboy_fetch*
76
-
77
- Reads 1 line from stdin for the URL to fetch from. The result will be
78
- dumped to stdout.
79
-
80
- 2. *bwkfanboy_parse*
81
-
82
- Takes 1 parameter: <em>a full path</em> to a plugin file.
83
-
84
- This util reads stdin, expecting it to be a xhtml, parses it and
85
- dumps the result to stdout in JSON-formatted object.
86
-
87
- 3. *bwkfanboy_generate*
88
-
89
- Reads stdin expecting it to be a proper JSON-formatted object.
90
-
91
- The result will be an Atom feed dumped to stdout in UTF-8.
92
-
93
- So, without the wrapper all this together looks like:
94
-
95
- % echo http://example.org | bwkfanboy_fetch |
96
- bwkfanboy_parse /path/to/my/plugin.rb | bwkfanboy_generate
97
-
98
- == Log
99
-
100
- All utils write to <tt>/tmp/bwkfanboy/USER/log/general.log</tt> file if
101
- permissions allows it.
102
-
103
- == HTTP
104
-
105
- There are 2 method to get an Atom feed via HTTP:
106
-
107
- 1. <tt>web/bwkfanboy.cgi</tt> (from the program tarball), which you may
108
- copy to your Apache cgi directory and run it. This prohibits you from
109
- using HOME directory for your own plugins. Also the cgi script
110
- requires some manual editing (setting 1 variable in it) before even
111
- you can start utilizing it.
112
-
113
- 2. Small *bwkfanboy_server* HTTP server. It can run from any user and
114
- thus is able to inherit env variables for discovering your HOME
115
- directory. Read doc/bwkfanboy_server.rdoc to know how to operate it.
116
-
117
- = License
118
-
119
- (The MIT License)
120
-
121
- Copyright (c) 2010 Alexander Gromnitsky.
122
-
123
- Permission is hereby granted, free of charge, to any person obtaining
124
- a copy of this software and associated documentation files (the
125
- 'Software'), to deal in the Software without restriction, including
126
- without limitation the rights to use, copy, modify, merge, publish,
127
- distribute, sublicense, and/or sell copies of the Software, and to
128
- permit persons to whom the Software is furnished to do so, subject to
129
- the following conditions:
130
-
131
- The above copyright notice and this permission notice shall be
132
- included in all copies or substantial portions of the Software.
133
-
134
- THE SOFTWARE IS PROVIDED 'AS IS', WITHOUT WARRANTY OF ANY KIND,
135
- EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF
136
- MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT.
137
- IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY
138
- CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT,
139
- TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE
140
- SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.
141
-
data/doc/TODO DELETED
@@ -1,7 +0,0 @@
1
- -*-org-*-
2
-
3
- - Add plugin listing to _server.
4
- - Several http requests to _fetch.
5
- + Inject falsework template.
6
- + econlib.org articles plugin.
7
- - Rewrite Utils module to be more compatible with falsework.
@@ -1,4 +0,0 @@
1
- = bwkfanboy_fetch
2
-
3
- Read stdin for a URI or a full path to the local file, download it (or
4
- read from the local file) and print the result to stdout.
@@ -1,7 +0,0 @@
1
- = bwkfanboy_generate
2
-
3
- Read stdin for JSON, generate from it an Atom feed and print the
4
- result to stdout in UTF-8.
5
-
6
- One can validate the JSON by providing '--check' command line option
7
- (by default the validating is off).
@@ -1,7 +0,0 @@
1
- = bwkfanboy_parse
2
-
3
- Takes 1 command line parameter: a full path to a plugin.
4
-
5
- Reads stdin for a HTML, parses it and prints the result to stdout in
6
- JSON format. If '-vv' command line parameters were given, output will
7
- be in 'key: value' pairs and <em>not</em> in JSON.
@@ -1,35 +0,0 @@
1
- = bwkfanboy_server
2
-
3
- Start a HTTP server (by default on 127.0.0.1:9042). To get Atom feeds
4
- from it, initiate GET request with URI
5
-
6
- http://localhost:9042/?p=PLUGIN
7
-
8
- where +PLUGIN+ is a name of a bwkfanboy's plugin (without '.re' suffix).
9
- If the plugin requires additional options you can specify them like:
10
-
11
- http://localhost:9042/?p=PLUGIN&o=opt1%20%22opt2%20has%20spaces%22
12
-
13
- where <tt>opt1%20%22opt2%20has%20spaces%22</tt> is a encoded string
14
- <tt>opt1 "opt2 has spaces"</tt>.
15
-
16
- To list all available plugins, point you browser to
17
-
18
- http://localhost:9042/list
19
-
20
- The server is intended to run from a non-root user from
21
- <tt>~/.login</tt> file. It can detach from a terminal if you give it
22
- '-d' command line option.
23
-
24
- For other help, type:
25
-
26
- bwkfanboy_server -h
27
-
28
- The server maintains 2 logs:
29
-
30
- /tmp/bwkfanboy/USER/log/bwkfanboy_server.log
31
- /tmp/bwkfanboy/USER/log/bwkfanboy_server-access.log
32
-
33
- The file with a pid:
34
-
35
- /tmp/bwkfanboy/USER/bwkfanboy_server.pid
@@ -1,59 +0,0 @@
1
- # -*-ruby-*-
2
-
3
- require 'erb'
4
- require 'json'
5
- require 'pathname'
6
-
7
- module MyDocs
8
- PATH = Pathname.new File.dirname(__FILE__)
9
-
10
- # our targets
11
- RDOC = FileList["#{PATH}/*.erb"].sub(/\.erb$/, '.rdoc')
12
- RDOC_RELATIVE = RDOC.pathmap '%-1d/%f'
13
-
14
- BWK = PATH.parent + 'bin/bwkfanboy'
15
- PLUGINS_DIR = PATH.parent + 'lib/bwkfanboy/plugins'
16
- PLUGINS = Dir["#{PLUGINS_DIR}/*.rb"].map {|i| i.pathmap '%{\.rb$}f' }
17
-
18
- # Return a list of plugins with description for each plugin.
19
- # This is used in .erb files.
20
- def self.plugins
21
- plugin_name_max = 20
22
-
23
- r = [format("%-#{plugin_name_max}s %s %s\n", 'NAME', 'VER', 'DESCRIPTION')]
24
- PLUGINS.each {|i|
25
- pr = `#{BWK} -i #{i} q w e r t y`.split("\n")
26
- ver = pr[0].split(':')[1..-1].join.strip
27
- desc = pr[2].split(':')[1..-1].join.strip
28
- r << format("%-#{plugin_name_max}s %3s %s", i, ver, desc)
29
- }
30
- r
31
- end
32
- end
33
-
34
- namespace 'mydocs' do
35
- rule '.rdoc' => ['.erb', MyDocs::PLUGINS_DIR.to_s] do |i|
36
- File.open(i.name, 'w+') {|fp|
37
- puts i.source.pathmap('%-1d/%f') + ' > ' + i.name.pathmap('%-1d/%f')
38
- fp.puts ERB.new(File.read(i.source)).result(binding)
39
- }
40
- end
41
-
42
- desc "Generate all"
43
- task :default => MyDocs::RDOC
44
-
45
- desc "Clean all crap"
46
- task :clean do
47
- rm(MyDocs::RDOC, verbose: true, force: true)
48
- end
49
-
50
- desc "Print a list of plugins"
51
- task :plugins do
52
- MyDocs.plugins.each {|i| puts i}
53
- end
54
-
55
- desc "Print all staff that _can_ be generated"
56
- task :print_gen do
57
- puts MyDocs::RDOC.to_a.to_json
58
- end
59
- end
@@ -1,63 +0,0 @@
1
- require 'rss/maker'
2
- require 'date'
3
- require 'json'
4
- require 'jsonschema'
5
-
6
- require_relative 'utils'
7
-
8
- module Bwkfanboy
9
- class Generate
10
-
11
- def self.validate(t)
12
- schema = Bwkfanboy::Utils.gem_dir_system() + '/schema.js'
13
- begin
14
- JSON::Schema.validate(t, JSON.parse(File.read(schema)))
15
- rescue
16
- Bwkfanboy::Utils.errx(1, "JSON validation with schema (#{schema}) failed");
17
- end
18
- end
19
-
20
- def self.atom(src)
21
- feed = RSS::Maker.make("atom") { |maker|
22
- maker.channel.id = src['channel']['id']
23
- maker.channel.updated = src['channel']['updated']
24
- maker.channel.author = src['channel']['author']
25
- maker.channel.title = src['channel']['title']
26
-
27
- maker.channel.links.new_link {|i|
28
- i.href = src['channel']['link']
29
- i.rel = 'alternate'
30
- i.type = 'text/html' # eh
31
- }
32
-
33
- maker.items.do_sort = true
34
-
35
- src['x_entries'].each { |i|
36
- maker.items.new_item do |item|
37
- item.links.new_link {|k|
38
- k.href = i['link']
39
- k.rel = 'alternate'
40
- k.type = 'text/html' # only to make happy crappy pr2nntp gateway
41
- }
42
- item.title = i['title']
43
- item.author = i['author']
44
- item.updated = i['updated']
45
- item.content.type = src['channel']['x_entries_content_type']
46
-
47
- case item.content.type
48
- when 'text'
49
- item.content.content = i['content']
50
- when 'html'
51
- item.content.content = i['content']
52
- else
53
- item.content.xhtml = i['content']
54
- end
55
- end
56
- }
57
- }
58
-
59
- return feed
60
- end
61
-
62
- end
63
- end