xmltv 0.8.1

Sign up to get free protection for your applications and to get access to all the features.
data/History.txt ADDED
@@ -0,0 +1,5 @@
1
+ == 1.0.0 / 2008-01-18
2
+
3
+ * 1 major enhancement
4
+ * Birthday!
5
+
data/Manifest.txt ADDED
@@ -0,0 +1,17 @@
1
+ History.txt
2
+ Manifest.txt
3
+ README.txt
4
+ Rakefile
5
+ bin/xmltv
6
+ lib/xmltv/sample/dumpids.rb
7
+ lib/xmltv/sample/mythtv_chns.yaml
8
+ lib/xmltv/sample/sample_output
9
+ lib/xmltv/sample/tvcat_spoolfiles.rb
10
+ lib/xmltv/sites/film1.rb
11
+ lib/xmltv/sites/rt.rb
12
+ lib/xmltv/sites/trivial.rb
13
+ lib/xmltv/sites/tvgids.rb
14
+ lib/xmltv/sites/tvtoday.rb
15
+ lib/xmltv/sites/upc.rb
16
+ lib/xmltv/sites/vpro.rb
17
+ lib/xmltv/xmltv.rb
data/README.txt ADDED
@@ -0,0 +1,123 @@
1
+ 0. Introduction
2
+
3
+ Xmltv consists of a single base grabber and a bunch of site specific grabbers.
4
+ The base grabber provides:
5
+ - a driver
6
+ - option parsing
7
+ - start/stop time fixing
8
+ - xml production
9
+ - helper functions that site specific grabbers might want to use
10
+
11
+ Currently grabbers are included for radiotimes.com, tvtoday.de, tvgids.nl,
12
+ upc.nl and film1.nl.
13
+ Grabbers for other sites are (hopefully) easy to add.
14
+
15
+ 1. Installation.
16
+
17
+ First get Ruby and Hpricot.
18
+ Then do 'gem install xmltv'
19
+
20
+ You could now run a site specific executable, e.g.:
21
+ xmltv upc
22
+ This will notice that there isn't yet any configuration for upc, and will fetch
23
+ and display all available channels and advice you to rerun the same command
24
+ with the --add switch
25
+ You the can do something like:
26
+ xmltv upc --add 211,212,216,215,214
27
+ (if you're a sports fan), and get back
28
+
29
+ Configured channels for upc
30
+ 211 Sport1
31
+ 212 Sport1+Extra
32
+ 216 Sport1+Extra+1
33
+ 215 Sport1+Extra+3
34
+ 214 Sport1+Extra+5
35
+
36
+ With --add and --del you can configure this grabber further.
37
+ Other grabbers can be added in the same way
38
+
39
+ Then you can either run the individual grabbers, one at a time, or run:
40
+ xmltv
41
+
42
+ which has the effect of running _all_ configured grabbers.
43
+
44
+ 2. What it does.
45
+
46
+ The grabbers produce xmltv files in spool directories. The locations of these are
47
+ configurable and overridable, but in the end that's all that the grabbers do at
48
+ the moment.
49
+ If you want to use these xmltv files for mythtv or something alike, you'll have
50
+ to take care of that yourself. In the sample directory are some scripts I use
51
+ for mythtv's mythfilldatabase but you'll have to edit these to make them useful.
52
+ At the moment you cannot configure mythtv to call these grabbers: I'm quite
53
+ willing to accept patches that add that ability, but I've never used any grabber
54
+ in this way, and I'm too lazy to study the mythtv documentation.
55
+
56
+ 3. What it does not do.
57
+
58
+ It doesn't add any <channel/> elements to the xmltv output files. It assumes you
59
+ have those channels configured somehow.
60
+ It also doesn't anything with icons. You don't need all this stuff for daily
61
+ program updates.
62
+
63
+ 4. Reasons why this is better than the Perl XMLTV suite
64
+
65
+ - Thanks to Hpricot, much more robust data gathering
66
+ - DTD driven xml production
67
+ - Shared code:
68
+ Option parsing
69
+ Xml generation
70
+ Start/stop time corrections
71
+ Configuration handling
72
+ Cacheing services
73
+
74
+ 5. Todo
75
+ - Write documentation (rdoc)
76
+ - Have it run under Windows and MacOS. I don't have any Windows machines or Macs
77
+ myself and I didn't write anything with portability in mind (after all I wrote
78
+ it for my own program listings) but it shouldn't be too hard.
79
+ I understand Ruby converts forward slashes to whatever is needed anyway.
80
+ Problem areas could be validation (which is a pipe through xmllint and tv_sort),
81
+ and sending exceptions via mail.
82
+
83
+ 6. What you need.
84
+
85
+ - Ruby obviously. I only tested under 1.8.6, but see no reason why it shouldn't
86
+ run on older versions. Didn't try 1.9.0 either.
87
+ - Hpricot. Just install the gem.
88
+ - If you want to validate you need xmllint (part of libxml2) and tv_sort (part
89
+ of the Perl XMLTV package).
90
+
91
+ 7. Adding other grabbers.
92
+
93
+ There no documentation yet, but there are 5 example implementations that should
94
+ help. The implemented sites are quite different as well:
95
+ - radiotimes doesn't have html (but a lot of encoding errors)
96
+ - film1.nl is so simple and small, there's no need for a cache. The structure of
97
+ the site is quite different from the others, and shown is how to fake the normal
98
+ pattern.
99
+ - tvgids.nl and tvtoday.de resemble each other in structure a lot. Probably some
100
+ extra code could be shared between these two, by deriving a subclass from
101
+ grabber, called DetailGrabber, and then deriving these two from that.
102
+ - gids.vpro.nl is dependent on cookies. Nice.
103
+
104
+ In general three thing have to be implemented:
105
+ - Getting all available channels (almost always there is a useful <SELECT>
106
+ element somewhere.)
107
+ - Grabbing program information and sometimes extra details (Firebug is very much
108
+ your friend here)
109
+ - Transforming data from site-specific, error-prone raw data to a fixed
110
+ structure the xml producer understands.
111
+
112
+ 8. Warning.
113
+
114
+ There are people on the Perl XMLTV mailing list who think that grabbing from
115
+ tvtoday.de might be illegal (in Germany). For that reason they withdrew the
116
+ tvtoday grabber.
117
+ If you think these people are right then don't use this implementation. I only
118
+ wrote it to check if I had the underlying code ready for creating new grabbers.
119
+ Writing and distributing this grabber isn't illegal in any way.
120
+
121
+ Han Holl
122
+
123
+
data/Rakefile ADDED
@@ -0,0 +1,18 @@
1
+ # -*- ruby -*-
2
+
3
+ require 'rubygems'
4
+ require 'hoe'
5
+ require './lib/xmltv/xmltv.rb'
6
+
7
+ Hoe.new('xmltv', XMLTV::VERSION) do |p|
8
+ p.rubyforge_name = 'xmltv'
9
+ p.author = 'Han Holl'
10
+ p.email = 'hanholl@rubyforge.org'
11
+ p.summary = 'fetches tv program data from various sites'
12
+ p.description = p.paragraphs_of('README.txt', 1..1).join("\n\n")
13
+ p.url = 'http://rubyforge.org/projects/xmltv'
14
+ p.extra_deps = %w{ hpriocot }
15
+ # p.changes = p.paragraphs_of('History.txt', 0..1).join("\n\n")
16
+ end
17
+
18
+ # vim: syntax=Ruby
data/bin/xmltv ADDED
@@ -0,0 +1,13 @@
1
+ #!/usr/bin/ruby -w
2
+
3
+ require 'rubygems' rescue nil
4
+ require 'xmltv/xmltv'
5
+
6
+ if ARGV[0] && XMLTV::Sites.include?(ARGV[0])
7
+ grabber = ARGV[0]
8
+ ARGV.replace(ARGV[1..-1])
9
+ require "xmltv/sites/#{grabber}"
10
+ else
11
+ XMLTV::go
12
+ end
13
+
@@ -0,0 +1,15 @@
1
+ #!/usr/bin/ruby
2
+ require 'mysql'
3
+ require 'pp'
4
+ require 'yaml'
5
+
6
+
7
+ DB = Mysql.new('localhost', 'mythtv', 'mythtv', 'mythconverg')
8
+
9
+ hash = Hash.new
10
+ DB.query('select sourceid, xmltvid from channel').each do |rsl|
11
+ hash[rsl[1]] = [ 'dummy', rsl[0]]
12
+ end
13
+ File.open('/tmp/sources.yaml', 'w') do |h|
14
+ h.puts hash.to_yaml
15
+ end
@@ -0,0 +1,183 @@
1
+ ---
2
+ nick.com:
3
+ - - 89.tvgids.nl
4
+ - "2"
5
+ mtv.nl:
6
+ - - 25.tvgids.nl
7
+ - "2"
8
+ at5.todo:
9
+ - - 40.tvgids.nl
10
+ - "2"
11
+ cnn.com:
12
+ - - 26.tvgids.nl
13
+ - "2"
14
+ arte-tv.com:
15
+ - - 38.tvgids.nl
16
+ - "2"
17
+ brabant.nl:
18
+ - - 114.tvgids.nl
19
+ - "4"
20
+ tv-oost.nl:
21
+ - - 111.tvgids.nl
22
+ - "4"
23
+ drenthe.nl:
24
+ - - 110.tvgids.nl
25
+ - "4"
26
+ choice.bbc.co.uk:
27
+ - - 45.rt.en
28
+ - "4"
29
+ knowledge.bbc.co.uk:
30
+ - - 47.rt.en
31
+ - "4"
32
+ prime.bbc.com:
33
+ - - 50.rt.en
34
+ - "4"
35
+ hollanddoc.vpro:
36
+ - - 309.upc.nl
37
+ - HOL.vpro.nl
38
+ - "4"
39
+ zdf.de:
40
+ - - ZDF.tvtoday.de
41
+ - "2"
42
+ nationalgeographic.co.uk:
43
+ - - 18.tvgids.nl
44
+ - "2"
45
+ south-east.bbc1.bbc.co.uk:
46
+ - - 94.rt.en
47
+ - "4"
48
+ bbc1.bbc.co.uk:
49
+ - - 7.tvgids.nl
50
+ - "2"
51
+ tmf.nl:
52
+ - - 35.tvgids.nl
53
+ - "2"
54
+ tv5.org:
55
+ - - 17.tvgids.nl
56
+ - "2"
57
+ flevo.nl:
58
+ - - 113.tvgids.nl
59
+ - "4"
60
+ ketnet.be:
61
+ - - 6.tvgids.nl
62
+ - "2"
63
+ rijnmond.nl:
64
+ - - 102.tvgids.nl
65
+ - "4"
66
+ fryslan.nl:
67
+ - - 109.tvgids.nl
68
+ - "4"
69
+ 4.film1.nl:
70
+ - - 4.film1.nl
71
+ - "4"
72
+ sbs6.nl:
73
+ - - 36.tvgids.nl
74
+ - "2"
75
+ utrecht.nl:
76
+ - - 100.tvgids.nl
77
+ - "4"
78
+ rtve.se:
79
+ - - 87.tvgids.nl
80
+ - "2"
81
+ talpa.tv:
82
+ - - 92.tvgids.nl
83
+ - "2"
84
+ 3.film1.nl:
85
+ - - 3.film1.nl
86
+ - "4"
87
+ nederland2.omroep.nl:
88
+ - - 2.tvgids.nl
89
+ - "2"
90
+ rtl5.nl:
91
+ - - 31.tvgids.nl
92
+ - "2"
93
+ discoveryeurope.com:
94
+ - - 29.tvgids.nl
95
+ - "2"
96
+ thebox.org:
97
+ - - 91.tvgids.nl
98
+ - "2"
99
+ cartoon.com:
100
+ - - 21.tvgids.nl
101
+ - "4"
102
+ rtl4.nl:
103
+ - - 4.tvgids.nl
104
+ - "2"
105
+ eurosport.sky.com:
106
+ - - 19.tvgids.nl
107
+ - "2"
108
+ bbcworld.com:
109
+ - - 86.tvgids.nl
110
+ - "2"
111
+ tcm.com:
112
+ - - 20.tvgids.nl
113
+ - "2"
114
+ cultura.nl:
115
+ - - 317.upc.nl
116
+ - "4"
117
+ een.be:
118
+ - - 5.tvgids.nl
119
+ - "2"
120
+ bbc2.bbc.co.uk:
121
+ - - 8.tvgids.nl
122
+ - "2"
123
+ veronica.nl:
124
+ - - 34.tvgids.nl
125
+ - "2"
126
+ trt.net.tr:
127
+ - - 32.tvgids.nl
128
+ - "2"
129
+ zeeland.nl:
130
+ - - 116.tvgids.nl
131
+ - "4"
132
+ tvnoord.nl:
133
+ - - 108.tvgids.nl
134
+ - "4"
135
+ nederland1.omroep.nl:
136
+ - - 1.tvgids.nl
137
+ - "2"
138
+ gelderland.nl:
139
+ - - 112.tvgids.nl
140
+ - "4"
141
+ sport1.nl:
142
+ - - 211.upc.nl
143
+ - "4"
144
+ geschiedenis.vpro:
145
+ - - 310.upc.nl
146
+ - GES.vpro.nl
147
+ - "4"
148
+ net5.nl:
149
+ - - 37.tvgids.nl
150
+ - "2"
151
+ rtl7.nl:
152
+ - - 46.tvgids.nl
153
+ - "2"
154
+ l1tv.nl:
155
+ - - 115.tvgids.nl
156
+ - "4"
157
+ tvwest.nl:
158
+ - - 101.tvgids.nl
159
+ - "4"
160
+ south-east.bbc2.bbc.co.uk:
161
+ - - 107.rt.en
162
+ - "4"
163
+ 2.film1.nl:
164
+ - - 2.film1.nl
165
+ - "4"
166
+ nederland3.omroep.nl:
167
+ - - 3.tvgids.nl
168
+ - "2"
169
+ ard.de:
170
+ - - ARD.tvtoday.de
171
+ - "2"
172
+ raiuno.rai.it:
173
+ - - 27.tvgids.nl
174
+ - "2"
175
+ tvnrh.nl:
176
+ - - 103.tvgids.nl
177
+ - "2"
178
+ 1.film1.nl:
179
+ - - 1.film1.nl
180
+ - "4"
181
+ mezzo.nl:
182
+ - - 73.tvgids.nl
183
+ - "4"
@@ -0,0 +1,73 @@
1
+ tvtoday: Removed 67 entries
2
+
3
+ 2008-01-11: tvtoday prgs hts days errs bytes rej
4
+ 05:02:24 ZDF ZDF 498 15 15 0 214444 0
5
+ 05:02:45 ARD ARD 515 15 15 0 218533 0
6
+
7
+ 2008-01-11: rt prgs hts days errs bytes rej
8
+ 05:02:53 94 BBC1 London & South East 465 1 14 2 279532 0
9
+ 05:03:01 107 BBC2 London & South East 503 1 15 0 301856 0
10
+ 05:03:04 45 BBC3 189 1 15 0 144878 0
11
+ 05:03:07 47 BBC4 139 1 15 0 102337 0
12
+ 05:03:16 50 BBC Prime 573 1 15 0 337125 0
13
+ upc: Removed 4 entries
14
+
15
+ 2008-01-11: upc prgs hts days errs bytes rej
16
+ 05:03:22 309 Holland+Doc 46 2 8 0 84227 0
17
+ 05:03:26 310 /Geschiedenis 47 2 9 0 70822 0
18
+ 05:03:28 317 NPS+Cultura 32 2 1 0 12421 0
19
+ 05:03:30 211 Sport1 19 2 2 0 5855 0
20
+
21
+ 2008-01-11: film1 prgs hts days errs bytes rej
22
+ 05:03:38 1 Film1 1 90 7 8 0 51509 0
23
+ 05:03:40 2 Film1 +1 90 0 8 0 51506 0
24
+ 05:03:42 3 Film1 2 93 0 8 0 49632 0
25
+ 05:03:44 4 Film1 3 90 0 8 0 50949 0
26
+ tvgids: Removed 1360 entries
27
+
28
+ 2008-01-11: tvgids prgs hts days errs bytes rej
29
+ 05:04:37 1 Nederland 1 328 47 9 13 141936 1
30
+ 05:05:02 2 Nederland 2 299 40 9 20 166591 1
31
+ 05:05:33 34 Veronica 352 51 9 0 94799 0
32
+ 05:05:53 5 E?n 241 33 9 0 84432 0
33
+ 05:06:23 6 KETNET/Canvas 347 46 9 0 112669 0
34
+ 05:07:05 3 Nederland 3 519 69 9 21 196135 0
35
+ 05:07:25 4 RTL 4 249 32 9 7 129258 0
36
+ 05:07:47 7 BBC 1 278 39 9 4 113613 3
37
+ 05:08:12 8 BBC 2 286 40 9 2 121388 0
38
+ 05:08:33 17 TV 5 269 36 9 0 58476 0
39
+ 05:08:50 18 National Geographic 228 28 9 0 59174 0
40
+ 05:08:59 19 Eurosport 122 14 9 0 34138 0
41
+ 05:09:09 20 TCM 112 15 9 1 39730 0
42
+ 05:09:32 21 Cartoon Network 292 37 8 0 57141 0
43
+ 05:09:49 25 MTV 194 27 9 0 48796 0
44
+ 05:10:06 26 CNN 199 28 9 4 36240 0
45
+ 05:10:24 27 Rai Uno 198 27 9 0 46917 0
46
+ 05:10:42 29 Discovery Channel 220 29 9 0 68186 0
47
+ 05:10:58 31 RTL 5 200 27 9 0 74434 0
48
+ 05:11:20 32 TRT int. 282 36 9 0 51836 0
49
+ 05:11:39 35 TMF 221 30 9 0 53925 0
50
+ 05:11:57 36 SBS 6 218 29 9 33 86888 1
51
+ 05:12:12 37 NET 5 186 23 9 21 105293 0
52
+ 05:12:34 38 ARTE 276 36 9 1 82944 0
53
+ 05:12:55 40 AT 5 279 37 9 0 51707 0
54
+ 05:13:08 46 RTL 7 160 18 9 3 67119 0
55
+ 05:13:21 73 Mezzo 136 18 9 3 31387 0
56
+ 05:13:39 87 TV E 221 29 9 1 47670 0
57
+ 05:14:01 89 Nickelodeon 298 40 9 1 55965 0
58
+ 05:14:32 91 Comedy Central 361 53 9 5 72189 0
59
+ 05:14:47 92 RTL 8 176 25 9 0 66487 0
60
+ 05:14:54 103 TV NOORD-HOLLAND 118 13 8 6 16988 1
61
+ 05:15:01 100 REGIO TV UTRECHT 77 10 8 0 11200 1
62
+ 05:15:26 101 TV WEST 265 35 9 0 38844 0
63
+ 05:15:29 102 TV RIJNMOND 41 6 8 0 6065 1
64
+ 05:16:12 108 TV NOORD 608 70 9 0 89193 0
65
+ 05:16:19 109 OMROP FRYSLAN 87 12 8 0 12581 1
66
+ 05:16:28 110 TV DRENTHE 125 15 8 0 18243 1
67
+ 05:16:52 111 TV OOST 324 46 9 0 48253 0
68
+ 05:17:12 112 TV GELDERLAND 257 36 9 0 39658 0
69
+ 05:17:19 113 TV FLEVOLAND 96 13 8 0 13685 1
70
+ 05:17:29 114 OMROEP BRABANT TV 143 18 8 0 20960 0
71
+ 05:17:48 115 L1 TV 268 32 9 0 37994 0
72
+ 05:18:04 116 OMROEP ZEELAND 192 30 8 0 27547 1
73
+ 05:18:23 86 BBC World 234 31 9 0 43575 0
@@ -0,0 +1,29 @@
1
+ #!/usr/bin/ruby -w
2
+
3
+ require 'yaml'
4
+
5
+ chns = YAML.load_file('/root/.xmltv/mythtv_chns.yaml')
6
+ allspools = Dir["/root/.xmltv/*/spool/*.xml"]
7
+
8
+ tvcat = Hash.new {|h,v| h[v] = Array.new }
9
+
10
+ chns.each_pair do |xmltv, data|
11
+ filenames, srcid = data
12
+ filenames.each do |filename|
13
+ source = filename.split('.')[1]
14
+ full = "/root/.xmltv/#{source}/spool/#{filename}.xml"
15
+ unless test(?f, full)
16
+ STDERR.puts "#{full} not found"
17
+ else
18
+ tvcat[srcid] << full
19
+ end
20
+ allspools.delete(full)
21
+ end
22
+ end
23
+ allspools.each do |fn|
24
+ STDERR.puts "#{fn} has no use"
25
+ end
26
+ tvcat.each_pair do |src, files|
27
+ system "tv_cat #{files.join(' ')} >/video/tv-#{src}"
28
+ end
29
+
@@ -0,0 +1,100 @@
1
+ #!/usr/bin/ruby -w
2
+
3
+ autoload :XMLTV, 'xmltv/xmltv'
4
+
5
+
6
+ module XMLTV
7
+
8
+ class Film1Grabber < Grabber
9
+ def initialize
10
+ super
11
+ @programs = nil
12
+ @all_channels = {
13
+ '1' => 'Film1 1',
14
+ '2' => 'Film1 +1',
15
+ '3' => 'Film1 2',
16
+ '4' => 'Film1 3'
17
+ }
18
+ end
19
+ def fetch_all_channels
20
+ all_channels
21
+ end
22
+
23
+ # def channel_list
24
+ # nil
25
+ # end
26
+ # def check_channel(chan_id)
27
+ # true
28
+ # end
29
+
30
+ ### Helpers
31
+
32
+ def fetch_programs
33
+ @programs = Hash.new
34
+ @n_progs = [ 0, 0, 0, 0]
35
+ %w{ sun mon tues wednes thurs fri satur }.each do |day|
36
+ url = "#{base_url}/tv/epg.php?day=#{day}day"
37
+ # url = '/home/han/epg.php'
38
+ page = fetch(url)
39
+ date = page.at('//span[@id="current"]').inner_text
40
+ STDERR.puts("#{url} #{date}") if XmltvOptions.verbose
41
+ @programs[date] = day_hash = Hash.new
42
+ %w{ 1 2 3 4 }.each do |col|
43
+ day_hash[col] = colarray = Array.new
44
+ page.search("//div.epg_block").each do |dagdeel|
45
+ period = dagdeel.previous_sibling.inner_text.strip
46
+ dagdeel.search("//dl.col#{col}").each do |el|
47
+ el.search('dd').each do |pg|
48
+ program = Hash.new
49
+ program['period'] = period
50
+ # STDERR.puts pg.class, pg, '===='
51
+ info = pg.previous_sibling
52
+ %w{ time title genre short_info }.each do |inf|
53
+ program[inf] = info.at("/.#{inf}").inner_text.to_utf
54
+ end
55
+ program['desc'] = pg.children[3].to_s.strip.to_utf
56
+ @n_progs[col.to_i - 1] += 1
57
+ colarray << program unless program.empty?
58
+ end
59
+ end
60
+ end
61
+ end
62
+ end
63
+ end
64
+
65
+ ### Must Implement
66
+
67
+ def grab_channel(chan_id)
68
+ fetch_programs unless @programs
69
+ @n_progs[chan_id.to_i - 1]
70
+ end
71
+
72
+ def transform(chan_id)
73
+ now = Time.new
74
+ jaar = now.year
75
+ progdata_array = Array.new
76
+ @programs.each_pair do |dag, schedule|
77
+ date = Date.dutch(dag)
78
+ jaar, maand, dag = date.year, date.month, date.day
79
+ schedule.each_pair do |ch, programs|
80
+ next unless ch == chan_id
81
+ programs.each do |p|
82
+ progdata = proghash(p, ch)
83
+ startuur, startmin, stopuur, stopmin = p['time'].split(/[:-]/).map{|x| x.to_i}
84
+
85
+ progdata['start'] = Time.local(jaar, maand, dag, startuur, startmin)
86
+ progdata['start'] += Dag if p['period'] == 'nacht'
87
+ date_stats(chan_id, progdata['start'])
88
+ progdata['stop'] = Time.local(jaar, maand, dag, stopuur, stopmin)
89
+ progdata['stop'] += Dag if progdata['stop'] < progdata['start']
90
+ progdata['category'] = 'Movies'
91
+ progdata['credits']['actor'] = p['short_info'].split(',').map {|x| x.strip}
92
+ progdata_array << progdata
93
+ end
94
+ end
95
+ end
96
+ progdata_array
97
+ end
98
+ end
99
+ end
100
+ XMLTV::Film1Grabber.new.run