feed-normalizer 1.5.1 → 1.5.2
Sign up to get free protection for your applications and to get access to all the features.
- data/History.txt +48 -48
- data/License.txt +27 -27
- data/Manifest.txt +18 -19
- data/README.txt +63 -63
- data/Rakefile +29 -25
- data/lib/feed-normalizer.rb +149 -149
- data/lib/html-cleaner.rb +181 -190
- data/lib/parsers/rss.rb +110 -95
- data/lib/parsers/simple-rss.rb +138 -137
- data/lib/structures.rb +245 -244
- data/test/data/atom03.xml +128 -127
- data/test/data/atom10.xml +114 -112
- data/test/data/rdf10.xml +1498 -1498
- data/test/data/rss20.xml +64 -63
- data/test/data/rss20diff.xml +59 -59
- data/test/data/rss20diff_short.xml +51 -51
- data/test/test_feednormalizer.rb +265 -267
- data/test/test_htmlcleaner.rb +156 -155
- metadata +99 -63
- data/test/test_all.rb +0 -6
data/test/data/atom03.xml
CHANGED
|
@@ -1,127 +1,128 @@
|
|
|
1
|
-
<?xml version="1.0" encoding="UTF-8"?>
|
|
2
|
-
<feed version="0.3"
|
|
3
|
-
xmlns="http://purl.org/atom/ns#"
|
|
4
|
-
xmlns:dc="http://purl.org/dc/elements/1.1/"
|
|
5
|
-
xml:lang="en">
|
|
6
|
-
<title>Cheap Stingy Bargains</title>
|
|
7
|
-
<link rel="alternate" type="text/html" href="http://www.cheapstingybargains.com" />
|
|
8
|
-
<tagline>Putting the "Squeeze" on high prices!</tagline>
|
|
9
|
-
<modified>2006-08-29T05:07:24Z</modified>
|
|
10
|
-
<copyright>Copyright 2006</copyright>
|
|
11
|
-
<generator url="http://wordpress.org/" version="2.0.3">WordPress</generator>
|
|
12
|
-
<entry>
|
|
13
|
-
<author>
|
|
14
|
-
<name>richard</name>
|
|
15
|
-
</author>
|
|
16
|
-
<title type="text/html" mode="escaped"><![CDATA[Levi Strauss Signature Girl’s Low Rise Slim Fit Flare Jeans $10]]></title>
|
|
17
|
-
<link rel="alternate" type="text/html" href="http://www.cheapstingybargains.com/24557/levi-strauss-signature-girls-low-rise-slim-fit-flare-jeans-10/" />
|
|
18
|
-
<id>http://www.cheapstingybargains.com/24557/levi-strauss-signature-girls-low-rise-slim-fit-flare-jeans-10/</id>
|
|
19
|
-
<modified>2006-08-29T02:31:03Z</modified>
|
|
20
|
-
<issued>2006-08-29T02:31:03Z</issued>
|
|
21
|
-
|
|
22
|
-
<dc:subject>Clothing</dc:subject>
|
|
23
|
-
<summary type="text/plain" mode="escaped"><![CDATA[
|
|
24
|
-
Kmart has the Levi Strauss Signature Girl’s Low Rise Flare Jean for $10 after $5 instant savings (ends 9/2)
|
|
25
|
-
Slim fit through hip and thigh, with zip-fly with button-through closure. Machine washable 99% Cotton/1% Spandex
|
|
26
|
-
]]></summary>
|
|
27
|
-
<content type="text/html" mode="escaped" xml:base="http://www.cheapstingybargains.com/24557/levi-strauss-signature-girls-low-rise-slim-fit-flare-jeans-10/"><![CDATA[<p><a href="/link/tplclick?lid=41000000011334249&pubid=21000000000053626" target=_"blank"><img src="/assets/images/product/productDetail/9990000058546711.jpg" width="150" height="150" border="0" style="float: right; margin: 0px 0px 5px 5px;" /></a><br />
|
|
28
|
-
<strong>Kmart has the <a href="http://clickserve.cc-dt.com/link/tplclick?lid=41000000011334249&pubid=21000000000053626" target=_"blank">Levi Strauss Signature Girl’s Low Rise Flare Jean</a> for $10 after $5 instant savings (ends 9/2)</strong></p>
|
|
29
|
-
<p>Slim fit through hip and thigh, with zip-fly with button-through closure. Machine washable 99% Cotton/1% Spandex</p>
|
|
30
|
-
]]></content>
|
|
31
|
-
</entry>
|
|
32
|
-
<entry>
|
|
33
|
-
<author>
|
|
34
|
-
<name>richard</name>
|
|
35
|
-
</author>
|
|
36
|
-
<title type="text/html" mode="escaped"><![CDATA[FragranceNet.com Coupon]]></title>
|
|
37
|
-
<link rel="alternate" type="text/html" href="http://www.cheapstingybargains.com/24556/fragrancenetcom-coupon/" />
|
|
38
|
-
<
|
|
39
|
-
<
|
|
40
|
-
<
|
|
41
|
-
|
|
42
|
-
|
|
43
|
-
<dc:subject>
|
|
44
|
-
<
|
|
45
|
-
|
|
46
|
-
|
|
47
|
-
|
|
48
|
-
|
|
49
|
-
<p>FragranceNet.com
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
<
|
|
59
|
-
<
|
|
60
|
-
<
|
|
61
|
-
<
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
<dc:subject>
|
|
65
|
-
<
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
<
|
|
72
|
-
<
|
|
73
|
-
<p>
|
|
74
|
-
<p>
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
<
|
|
84
|
-
<
|
|
85
|
-
<
|
|
86
|
-
<
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
<dc:subject>
|
|
90
|
-
<
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
Acer Aspire Laptop
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
<
|
|
98
|
-
<
|
|
99
|
-
<p><strong>Acer Aspire Laptop</strong
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
<
|
|
109
|
-
<
|
|
110
|
-
<
|
|
111
|
-
<
|
|
112
|
-
|
|
113
|
-
|
|
114
|
-
<dc:subject>
|
|
115
|
-
<
|
|
116
|
-
|
|
117
|
-
|
|
118
|
-
|
|
119
|
-
<
|
|
120
|
-
<
|
|
121
|
-
|
|
122
|
-
|
|
123
|
-
|
|
124
|
-
</
|
|
125
|
-
|
|
126
|
-
|
|
127
|
-
<!--
|
|
1
|
+
<?xml version="1.0" encoding="UTF-8"?>
|
|
2
|
+
<feed version="0.3"
|
|
3
|
+
xmlns="http://purl.org/atom/ns#"
|
|
4
|
+
xmlns:dc="http://purl.org/dc/elements/1.1/"
|
|
5
|
+
xml:lang="en">
|
|
6
|
+
<title>Cheap Stingy Bargains</title>
|
|
7
|
+
<link rel="alternate" type="text/html" href="http://www.cheapstingybargains.com" />
|
|
8
|
+
<tagline>Putting the "Squeeze" on high prices!</tagline>
|
|
9
|
+
<modified>2006-08-29T05:07:24Z</modified>
|
|
10
|
+
<copyright>Copyright 2006</copyright>
|
|
11
|
+
<generator url="http://wordpress.org/" version="2.0.3">WordPress</generator>
|
|
12
|
+
<entry>
|
|
13
|
+
<author>
|
|
14
|
+
<name>richard</name>
|
|
15
|
+
</author>
|
|
16
|
+
<title type="text/html" mode="escaped"><![CDATA[Levi Strauss Signature Girl’s Low Rise Slim Fit Flare Jeans $10]]></title>
|
|
17
|
+
<link rel="alternate" type="text/html" href="http://www.cheapstingybargains.com/24557/levi-strauss-signature-girls-low-rise-slim-fit-flare-jeans-10/" />
|
|
18
|
+
<id>http://www.cheapstingybargains.com/24557/levi-strauss-signature-girls-low-rise-slim-fit-flare-jeans-10/</id>
|
|
19
|
+
<modified>2006-08-29T02:31:03Z</modified>
|
|
20
|
+
<issued>2006-08-29T02:31:03Z</issued>
|
|
21
|
+
|
|
22
|
+
<dc:subject>Clothing</dc:subject>
|
|
23
|
+
<summary type="text/plain" mode="escaped"><![CDATA[
|
|
24
|
+
Kmart has the Levi Strauss Signature Girl’s Low Rise Flare Jean for $10 after $5 instant savings (ends 9/2)
|
|
25
|
+
Slim fit through hip and thigh, with zip-fly with button-through closure. Machine washable 99% Cotton/1% Spandex
|
|
26
|
+
]]></summary>
|
|
27
|
+
<content type="text/html" mode="escaped" xml:base="http://www.cheapstingybargains.com/24557/levi-strauss-signature-girls-low-rise-slim-fit-flare-jeans-10/"><![CDATA[<p><a href="/link/tplclick?lid=41000000011334249&pubid=21000000000053626" target=_"blank"><img src="/assets/images/product/productDetail/9990000058546711.jpg" width="150" height="150" border="0" style="float: right; margin: 0px 0px 5px 5px;" /></a><br />
|
|
28
|
+
<strong>Kmart has the <a href="http://clickserve.cc-dt.com/link/tplclick?lid=41000000011334249&pubid=21000000000053626" target=_"blank">Levi Strauss Signature Girl’s Low Rise Flare Jean</a> for $10 after $5 instant savings (ends 9/2)</strong></p>
|
|
29
|
+
<p>Slim fit through hip and thigh, with zip-fly with button-through closure. Machine washable 99% Cotton/1% Spandex</p>
|
|
30
|
+
]]></content>
|
|
31
|
+
</entry>
|
|
32
|
+
<entry>
|
|
33
|
+
<author>
|
|
34
|
+
<name>richard</name>
|
|
35
|
+
</author>
|
|
36
|
+
<title type="text/html" mode="escaped"><![CDATA[FragranceNet.com Coupon]]></title>
|
|
37
|
+
<link rel="alternate" type="text/html" href="http://www.cheapstingybargains.com/24556/fragrancenetcom-coupon/" />
|
|
38
|
+
<enclosure url="http://websrvr82il.audiovideoweb.com/ny60web16519/LTN/LawTechnologyNow/LawTechnologyNow_OCT08.mp3" type="audio/mpeg"/>
|
|
39
|
+
<id>http://www.cheapstingybargains.com/24556/fragrancenetcom-coupon/</id>
|
|
40
|
+
<modified>2006-08-29T02:21:08Z</modified>
|
|
41
|
+
<issued>2006-08-29T02:21:08Z</issued>
|
|
42
|
+
|
|
43
|
+
<dc:subject>Coupon</dc:subject>
|
|
44
|
+
<dc:subject>General</dc:subject>
|
|
45
|
+
<summary type="text/plain" mode="escaped"><![CDATA[New Coupon from FragranceNet.com - $10 off orders of $75+ use Coupon Code LSLBY at check-out (ends 9/5)
|
|
46
|
+
FragranceNet.com always offers deep discounts on all genuine brand name fragrances of up to 70% off retail with free shipping on all orders over $60
|
|
47
|
+
|
|
48
|
+
]]></summary>
|
|
49
|
+
<content type="text/html" mode="escaped" xml:base="http://www.cheapstingybargains.com/24556/fragrancenetcom-coupon/"><![CDATA[<p>New Coupon from FragranceNet.com - <a target="_blank" href="http://click.linksynergy.com/fs-bin/click?id=UnlJPFdznf8&offerid=47781.10000314&type=3&subid=0" >$10 off orders of $75+ </a> use Coupon Code <strong>LSLBY</strong> at check-out (ends 9/5)</p>
|
|
50
|
+
<p>FragranceNet.com always offers deep discounts on all genuine brand name fragrances of up to 70% off retail with free shipping on all orders over $60
|
|
51
|
+
</p>
|
|
52
|
+
]]></content>
|
|
53
|
+
</entry>
|
|
54
|
+
<entry>
|
|
55
|
+
<author>
|
|
56
|
+
<name>Ayusha</name>
|
|
57
|
+
</author>
|
|
58
|
+
<title type="text/html" mode="escaped"><![CDATA[Astar LTV-37LS 37″ Widescreen LCD TV with ATSC HD Tuner $1000]]></title>
|
|
59
|
+
<link rel="alternate" type="text/html" href="http://www.cheapstingybargains.com/23943/astar-ltv-37ls-37-widescreen-lcd-tv-1150-after-rebate/" />
|
|
60
|
+
<id>http://www.cheapstingybargains.com/23943/astar-ltv-37ls-37-widescreen-lcd-tv-1150-after-rebate/</id>
|
|
61
|
+
<modified>2006-08-29T02:13:38Z</modified>
|
|
62
|
+
<issued>2006-08-29T02:13:38Z</issued>
|
|
63
|
+
|
|
64
|
+
<dc:subject>Buy.com</dc:subject>
|
|
65
|
+
<dc:subject>TV</dc:subject>
|
|
66
|
+
<summary type="text/plain" mode="escaped"><![CDATA[New Low Price!
|
|
67
|
+
|
|
68
|
+
Buy.com has the Astar LTV-37LS 37″ Widescreen LCD TV for $1000 after $150 rebate (expires 9/9) w/ free shipping
|
|
69
|
+
The Astar LTV-37LS 37″ Widescreen LCD TV features 1366 x 768 resolution, 16:9 aspect ratio, 600cd/m2 brightness, 800:1 contrast ratio, built-in ATSC HD Tuner and 12ms response time.
|
|
70
|
+
New customers [...]]]></summary>
|
|
71
|
+
<content type="text/html" mode="escaped" xml:base="http://www.cheapstingybargains.com/23943/astar-ltv-37ls-37-widescreen-lcd-tv-1150-after-rebate/"><![CDATA[<p><strong>New Low Price!</strong><br />
|
|
72
|
+
<a href = "http://www.cheapstingybargains.com/jump.php?m=buy&id=202883003" target="_blank"><img src="http://ak.buy.com/db_assets/large_images/003/202883003.jpg" width="150" height="150" border="0" style="float: right; margin: 0px 0px 5px 5px;" /></a></p>
|
|
73
|
+
<p><strong>Buy.com has the <a href="http://www.cheapstingybargains.com/jump.php?m=buy&id=202883003" target="_blank" >Astar LTV-37LS 37″ Widescreen LCD TV</a> for $1000 after $150 rebate (expires 9/9) w/ free shipping</strong></p>
|
|
74
|
+
<p>The <strong>Astar LTV-37LS 37″ Widescreen LCD TV</strong> features 1366 x 768 resolution, 16:9 aspect ratio, 600cd/m2 brightness, 800:1 contrast ratio, built-in ATSC HD Tuner and 12ms response time. </p>
|
|
75
|
+
<p>New customers of Buy.com may take an additional $15 discount by clicking <a href="http://www.anrdoezrs.net/click-1547706-10391416?&URL=http%3A%2F%2Fwww%2Ebuy%2Ecom%2Fretail%2Fcoupon%2Easp%3Fprid%3D88301879" target=_"blank">$15 Coupon</a> and searching for <strong>202883003</strong>
|
|
76
|
+
</p>
|
|
77
|
+
]]></content>
|
|
78
|
+
</entry>
|
|
79
|
+
<entry>
|
|
80
|
+
<author>
|
|
81
|
+
<name>richard</name>
|
|
82
|
+
</author>
|
|
83
|
+
<title type="text/html" mode="escaped"><![CDATA[Acer Aspire Notebook (AS5003WLMI) w/ 15.4″ WXGA display $500]]></title>
|
|
84
|
+
<link rel="alternate" type="text/html" href="http://www.cheapstingybargains.com/24555/acer-aspire-notebook-as5003wlmi-w-154-wxga-display-500/" />
|
|
85
|
+
<id>http://www.cheapstingybargains.com/24555/acer-aspire-notebook-as5003wlmi-w-154-wxga-display-500/</id>
|
|
86
|
+
<modified>2006-08-29T02:02:42Z</modified>
|
|
87
|
+
<issued>2006-08-29T02:02:42Z</issued>
|
|
88
|
+
|
|
89
|
+
<dc:subject>CircuitCity.com</dc:subject>
|
|
90
|
+
<dc:subject>Laptops & Notebooks</dc:subject>
|
|
91
|
+
<summary type="text/plain" mode="escaped"><![CDATA[
|
|
92
|
+
|
|
93
|
+
Circuit City has a Acer Aspire Laptop (AS5003WLMI) for $500 after $100 instant savings and $150 rebate (expires 9/2) w/ free shipping
|
|
94
|
+
Acer Aspire Laptop:
|
|
95
|
+
AMD Turion 64 ML-32, 15.4″ WXGA display, 512MB RAM, 80GB hard drive, Double-layer DVD drive, 3 USB 2.0 ports, XP Home
|
|
96
|
+
]]></summary>
|
|
97
|
+
<content type="text/html" mode="escaped" xml:base="http://www.cheapstingybargains.com/24555/acer-aspire-notebook-as5003wlmi-w-154-wxga-display-500/"><![CDATA[<p><a href="http://www.cheapstingybargains.com/jump.php?m=circuit&id=156638" target="_blank" rel="nofollow"><br />
|
|
98
|
+
<img src="http://www.circuitcity.com/IMAGE/product/enlarged/aca/PC.ACA.AS5003WLMI.LT.JPG" width="210" height="140" border="0" style="float: right; margin: 0px 0px 5px 5px;" /></a></p>
|
|
99
|
+
<p><strong>Circuit City has a <a href="http://www.cheapstingybargains.com/jump.php?m=circuit&id=156638" target="_blank" rel="nofollow">Acer Aspire Laptop (AS5003WLMI) </a> for $500 after $100 instant savings and $150 rebate (expires 9/2) w/ free shipping</strong></p>
|
|
100
|
+
<p><strong>Acer Aspire Laptop</strong>:<br />
|
|
101
|
+
AMD Turion 64 ML-32, 15.4″ WXGA display, 512MB RAM, 80GB hard drive, Double-layer DVD drive, 3 USB 2.0 ports, XP Home </p>
|
|
102
|
+
]]></content>
|
|
103
|
+
</entry>
|
|
104
|
+
<entry>
|
|
105
|
+
<author>
|
|
106
|
+
<name>richard</name>
|
|
107
|
+
</author>
|
|
108
|
+
<title type="text/html" mode="escaped"><![CDATA[Astar LTV-32BG 32� LCD HD Television with built in HD Digital Tuner $700]]></title>
|
|
109
|
+
<link rel="alternate" type="text/html" href="http://www.cheapstingybargains.com/24554/astar-ltv-32bg-32%e2%80%9d-lcd-hd-television-with-built-in-hd-digital-tuner-700/" />
|
|
110
|
+
<id>http://www.cheapstingybargains.com/24554/astar-ltv-32bg-32%e2%80%9d-lcd-hd-television-with-built-in-hd-digital-tuner-700/</id>
|
|
111
|
+
<modified>2006-08-29T01:41:01Z</modified>
|
|
112
|
+
<issued>2006-08-29T01:41:01Z</issued>
|
|
113
|
+
|
|
114
|
+
<dc:subject>TV's & HDTV</dc:subject>
|
|
115
|
+
<dc:subject>PCConnection</dc:subject>
|
|
116
|
+
<summary type="text/plain" mode="escaped"><![CDATA[
|
|
117
|
+
PC Connection has the Astar LTV-32BG 32� LCD HDTV with built in HD Digital Tuner for $700 after $100 rebate (expires 9/9) w/ free shipping
|
|
118
|
+
The Astar LTV-32BG 32� LCD HDTV displays HDTV broadcast signals received through its Component Video inputs, DVI input, or built-in ATSC HD Tuner. It features contemporary styling, built-in stereo [...]]]></summary>
|
|
119
|
+
<content type="text/html" mode="escaped" xml:base="http://www.cheapstingybargains.com/24554/astar-ltv-32bg-32%e2%80%9d-lcd-hd-television-with-built-in-hd-digital-tuner-700/"><![CDATA[<p><a href="http://clickserve.cc-dt.com/link/click?lid=41000000011336478" target=_"blank"><img src="http://service.pcconnection.com/images/inhouse/6724384.jpg" width="150" height="150" border="0" style="float: right; margin: 0px 0px 5px 5px;" /></a><br />
|
|
120
|
+
<strong>PC Connection has the <a href="http://clickserve.cc-dt.com/link/click?lid=41000000011336478" target=_"blank">Astar LTV-32BG 32� LCD HDTV with built in HD Digital Tuner</a> for $700 after $100 rebate (expires 9/9) w/ free shipping </strong> </p>
|
|
121
|
+
<p>The <strong>Astar LTV-32BG 32� LCD HDTV</strong> displays HDTV broadcast signals received through its Component Video inputs, DVI input, or built-in ATSC HD Tuner. It features contemporary styling, built-in stereo speakers, classic easy-touch front-mounted control buttons, and a wide-screen 16:9 picture aspect ratio, 800:1 contrast ratio and 1366 x 768 resolution.
|
|
122
|
+
</p>
|
|
123
|
+
]]></content>
|
|
124
|
+
</entry>
|
|
125
|
+
</feed>
|
|
126
|
+
|
|
127
|
+
<!-- Dynamic Page Served (once) in 1.705 seconds -->
|
|
128
|
+
<!-- Cached page served by WP-Cache -->
|
data/test/data/atom10.xml
CHANGED
|
@@ -1,112 +1,114 @@
|
|
|
1
|
-
<?xml version="1.0" encoding="UTF-8"?>
|
|
2
|
-
<feed xml:lang="en-US" xmlns="http://www.w3.org/2005/Atom">
|
|
3
|
-
<title>~:caboose</title>
|
|
4
|
-
<id>tag:habtm.com,2005:Typo</id>
|
|
5
|
-
<generator version="4.0" uri="http://www.typosphere.org">Typo</generator>
|
|
6
|
-
<link href="http://habtm.com/xml/atom10/feed.xml" rel="self" type="application/atom+xml"/>
|
|
7
|
-
<link href="http://habtm.com/" rel="alternate" type="text/html"/>
|
|
8
|
-
<updated>2006-08-16T11:59:44-05:00</updated>
|
|
9
|
-
<entry>
|
|
10
|
-
<author>
|
|
11
|
-
<name>technoweenie</name>
|
|
12
|
-
</author>
|
|
13
|
-
<id>urn:uuid:3d9fa115-1bb1-420e-9fca-478a8cb520b5</id>
|
|
14
|
-
<published>2006-08-16T11:58:00-05:00</published>
|
|
15
|
-
<updated>2006-08-16T11:59:44-05:00</updated>
|
|
16
|
-
<title type="html">A forum on Rails</title>
|
|
17
|
-
<link href="http://habtm.com/articles/2006/08/16/a-forum-on-rails" rel="alternate" type="text/html"/>
|
|
18
|
-
<
|
|
19
|
-
<category term="
|
|
20
|
-
<
|
|
21
|
-
<
|
|
22
|
-
|
|
23
|
-
|
|
24
|
-
|
|
25
|
-
|
|
26
|
-
|
|
27
|
-
|
|
28
|
-
<
|
|
29
|
-
<
|
|
30
|
-
<
|
|
31
|
-
<
|
|
32
|
-
<
|
|
33
|
-
<category term="
|
|
34
|
-
<
|
|
35
|
-
<
|
|
36
|
-
|
|
37
|
-
|
|
38
|
-
|
|
39
|
-
|
|
40
|
-
|
|
41
|
-
|
|
42
|
-
<
|
|
43
|
-
<
|
|
44
|
-
<
|
|
45
|
-
<
|
|
46
|
-
<
|
|
47
|
-
<
|
|
48
|
-
|
|
49
|
-
|
|
50
|
-
|
|
51
|
-
|
|
52
|
-
|
|
53
|
-
|
|
54
|
-
|
|
55
|
-
|
|
56
|
-
|
|
57
|
-
|
|
58
|
-
require '
|
|
59
|
-
require '
|
|
60
|
-
require '
|
|
61
|
-
require '
|
|
62
|
-
|
|
63
|
-
|
|
64
|
-
|
|
65
|
-
|
|
66
|
-
|
|
67
|
-
|
|
68
|
-
|
|
69
|
-
|
|
70
|
-
|
|
71
|
-
|
|
72
|
-
|
|
73
|
-
|
|
74
|
-
|
|
75
|
-
|
|
76
|
-
|
|
77
|
-
|
|
78
|
-
|
|
79
|
-
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
83
|
-
|
|
84
|
-
|
|
85
|
-
|
|
86
|
-
|
|
87
|
-
|
|
88
|
-
|
|
89
|
-
|
|
90
|
-
|
|
91
|
-
|
|
92
|
-
|
|
93
|
-
|
|
94
|
-
|
|
95
|
-
|
|
96
|
-
|
|
97
|
-
|
|
98
|
-
|
|
99
|
-
|
|
100
|
-
|
|
101
|
-
|
|
102
|
-
|
|
103
|
-
|
|
104
|
-
|
|
105
|
-
|
|
106
|
-
|
|
107
|
-
|
|
108
|
-
|
|
109
|
-
|
|
110
|
-
|
|
111
|
-
|
|
112
|
-
|
|
1
|
+
<?xml version="1.0" encoding="UTF-8"?>
|
|
2
|
+
<feed xml:lang="en-US" xmlns="http://www.w3.org/2005/Atom">
|
|
3
|
+
<title>~:caboose</title>
|
|
4
|
+
<id>tag:habtm.com,2005:Typo</id>
|
|
5
|
+
<generator version="4.0" uri="http://www.typosphere.org">Typo</generator>
|
|
6
|
+
<link href="http://habtm.com/xml/atom10/feed.xml" rel="self" type="application/atom+xml"/>
|
|
7
|
+
<link href="http://habtm.com/" rel="alternate" type="text/html"/>
|
|
8
|
+
<updated>2006-08-16T11:59:44-05:00</updated>
|
|
9
|
+
<entry>
|
|
10
|
+
<author>
|
|
11
|
+
<name>technoweenie</name>
|
|
12
|
+
</author>
|
|
13
|
+
<id>urn:uuid:3d9fa115-1bb1-420e-9fca-478a8cb520b5</id>
|
|
14
|
+
<published>2006-08-16T11:58:00-05:00</published>
|
|
15
|
+
<updated>2006-08-16T11:59:44-05:00</updated>
|
|
16
|
+
<title type="html">A forum on Rails</title>
|
|
17
|
+
<link href="http://habtm.com/articles/2006/08/16/a-forum-on-rails" rel="alternate" type="text/html"/>
|
|
18
|
+
<enclosure url="http://habtm.com/podcasts/2008-10-08.mp3" type="audio/mpeg"/>
|
|
19
|
+
<category term="rails" scheme="http://habtm.com/articles/category/rails" label="rails"/>
|
|
20
|
+
<category term="ruby" scheme="http://habtm.com/articles/category/ruby" label="ruby"/>
|
|
21
|
+
<summary type="html"><plaintext><p>Josh Goebel and I took an evening to bang out a little project: <a href="http://beast.caboo.se/">Beast</a>. It&#8217;s our minimal no-fluff Rails forum. It&#8217;s no beast of an application either, clocking in at 285 LOC and a 1:1.5 test ratio. <a href="http://svn.techno-weenie.net/projects/beast/">Check it out</a>!</p></summary>
|
|
22
|
+
<content type="html"><plaintext><p>Josh Goebel and I took an evening to bang out a little project: <a href="http://beast.caboo.se/">Beast</a>. It&#8217;s our minimal no-fluff Rails forum. It&#8217;s no beast of an application either, clocking in at 285 LOC and a 1:1.5 test ratio. <a href="http://svn.techno-weenie.net/projects/beast/">Check it out</a>!</p></content>
|
|
23
|
+
</entry>
|
|
24
|
+
<entry>
|
|
25
|
+
<author>
|
|
26
|
+
<name>courtenay</name>
|
|
27
|
+
</author>
|
|
28
|
+
<id>urn:uuid:32816298-7855-4d65-a832-5f6a92b1a4ad</id>
|
|
29
|
+
<published>2006-08-11T12:11:13-05:00</published>
|
|
30
|
+
<updated>2006-08-11T12:11:13-05:00</updated>
|
|
31
|
+
<title type="html">Update on the documentation project</title>
|
|
32
|
+
<link href="http://habtm.com/articles/2006/08/11/update-on-the-documentation-project" rel="alternate" type="text/html"/>
|
|
33
|
+
<category term="rails" scheme="http://habtm.com/articles/category/rails" label="rails"/>
|
|
34
|
+
<category term="ruby" scheme="http://habtm.com/articles/category/ruby" label="ruby"/>
|
|
35
|
+
<summary type="html"><p>Please use the wiki - <a href="http://caboose.stikipad.com/documentation">http://caboose.stikipad.com/documentation</a> - to give us your ideas about how the documentation project can proceed. Thanks.</p></summary>
|
|
36
|
+
<content type="html"><p>Please use the wiki - <a href="http://caboose.stikipad.com/documentation">http://caboose.stikipad.com/documentation</a> - to give us your ideas about how the documentation project can proceed. Thanks.</p></content>
|
|
37
|
+
</entry>
|
|
38
|
+
<entry>
|
|
39
|
+
<author>
|
|
40
|
+
<name>KirinDave</name>
|
|
41
|
+
</author>
|
|
42
|
+
<id>urn:uuid:6c028f36-f87a-4f53-b7e3-1f943d2341f0</id>
|
|
43
|
+
<published>2006-08-10T01:09:02-05:00</published>
|
|
44
|
+
<updated>2006-08-10T01:09:02-05:00</updated>
|
|
45
|
+
<title type="html">Starfish - Easy Distribution of Site Maintenance</title>
|
|
46
|
+
<link href="http://habtm.com/articles/2006/08/10/starfish-easy-distribution-of-site-maintenance" rel="alternate" type="text/html"/>
|
|
47
|
+
<enclosure url="http://websrvr82il.audiovideoweb.com/ny60web16519/LTN/LawTechnologyNow/LawTechnologyNow_OCT08.mp3" type="audio/mpeg"/>
|
|
48
|
+
<summary type="html"><p>Lots of Rails apps are very simple <span class="caps">CRUD</span> mappings, which is something Rails excels at. But, as more and more large, complex sites go live with Rails, there is a demand for more complex &#8216;backend&#8217; components in the system. Already we&#8217;ve got <a href="http://brainspl.at/articles/2006/05/15/backgoundrb-initial-release">BackgrounDRB</a>, which is great, but enter Starfish, which is a tool for complex distributed tasks made easy.</p></summary>
|
|
49
|
+
<content type="html"><p>Lots of Rails apps are very simple <span class="caps">CRUD</span> mappings, which is something Rails excels at. But, as more and more large, complex sites go live with Rails, there is a demand for more complex &#8216;backend&#8217; components in the system. Already we&#8217;ve got <a href="http://brainspl.at/articles/2006/05/15/backgoundrb-initial-release">BackgrounDRB</a>, which is great, but enter Starfish, which is a tool for complex distributed tasks made easy.</p><p>Lucas Carlson (a.k.a. cardmagic) and I have been using Starfish for various distributed tasks at <a href="http://mog.com">mog.com</a> for awhile now. I&#8217;d like to show everyone how it works, as an incentive for Lucas to gem up the project and share it with the world.</p>
|
|
50
|
+
|
|
51
|
+
|
|
52
|
+
<p><strong>The Problem</strong>: Due to a bug, many albums in our music database were assigned the wrong artist. We can fix this pretty easily by checking against other data, but there are a lot of albums to run this fix against and not a lot of downtime available each day.</p>
|
|
53
|
+
|
|
54
|
+
|
|
55
|
+
<strong>The Solution</strong>: A starfish file that looks like this:
|
|
56
|
+
<pre>
|
|
57
|
+
ENV['RAILS_ENV'] ||= 'production'
|
|
58
|
+
require File.dirname(__FILE__) + '/../config/boot'
|
|
59
|
+
require File.dirname(__FILE__) + '/../config/environment'
|
|
60
|
+
require 'user'
|
|
61
|
+
require 'artist'
|
|
62
|
+
require 'album'
|
|
63
|
+
require 'user_collection_cleaning'
|
|
64
|
+
|
|
65
|
+
class ScrubAlbumsTask
|
|
66
|
+
def self.service
|
|
67
|
+
@@service ||= ScrubAlbumsTask.new
|
|
68
|
+
end
|
|
69
|
+
|
|
70
|
+
def initialize
|
|
71
|
+
@various_artists = Artist.find_by_name( "Various Artists" )
|
|
72
|
+
@albums = Album.find(:all, :conditions =&gt; ["artist_id = ?", @various_artists.id])
|
|
73
|
+
end
|
|
74
|
+
|
|
75
|
+
def next
|
|
76
|
+
@albums.shift.id
|
|
77
|
+
end
|
|
78
|
+
|
|
79
|
+
def has_data?
|
|
80
|
+
@albums.size &gt; 0
|
|
81
|
+
end
|
|
82
|
+
end
|
|
83
|
+
|
|
84
|
+
server do |object|
|
|
85
|
+
object = ScrubAlbumsTask.service
|
|
86
|
+
end
|
|
87
|
+
|
|
88
|
+
client do |object|
|
|
89
|
+
if object.has_data?
|
|
90
|
+
album = Album.find(object.next)
|
|
91
|
+
album.repair_artist_by_majority!
|
|
92
|
+
end
|
|
93
|
+
end
|
|
94
|
+
</pre>
|
|
95
|
+
|
|
96
|
+
<p><strong>The Explanation</strong>: We create a simple singleton-like object to do our dirty work. All it does is get ahold of the albums we need to fix (in <tt>initialize</tt>) and then provide a simple way to get ahold of the object id to repair.</p>
|
|
97
|
+
|
|
98
|
+
|
|
99
|
+
<p>The server object merely produces a drb proxy to our clients. The clients get an id from the queue, find the object in the database, then run the repair process on them.</p>
|
|
100
|
+
|
|
101
|
+
|
|
102
|
+
<p>You can notice a few things we <i>didn&#8217;t</i> do. We didn&#8217;t write any networking or distribution or threading code. We didn&#8217;t need to partition the table in any arbitrary fashion. We didn&#8217;t need to poll the database over and over to fetch new work units, the server keeps the state. We didn&#8217;t even need to worry about redoing work, the way we wrote our server makes sure that doesn&#8217;t happen.</p>
|
|
103
|
+
|
|
104
|
+
|
|
105
|
+
<p>To invoke the system, we simply run <tt>starfish my_albums_task &#38;</tt>. The first time we do this, it will output a message that says &#8220;Starting Server.&#8221; Starfish is smart enough to find that there are no servers for a given task, and if that is the case, it forks one off. It then invokes a client process. If you invoke the task again, a new client will be added. We can run them on multiple machines, so long as they share a local network. No fancy <span class="caps">CORBA</span>-ish code needed. Starfish handles the details, and you can just kill the clients when they are done. Starfish protects its client block from signals and closes after a client finishes a task.</p>
|
|
106
|
+
|
|
107
|
+
|
|
108
|
+
<p>The key to the speed of this system is that it parallelizes database access. Most database setups have very high read speeds, so running the repair processes in parallel isn&#8217;t a significant burden on the system. Starfish is most useful in situations where you need to go record by record through a table and perform some specific–possibly slow–task.</p>
|
|
109
|
+
|
|
110
|
+
|
|
111
|
+
<p>This is a very simple starfish task that I wrote in 20 minutes to handle this cleanup process. We use more sophsticated distributed systems for other aspects of our site, and starfish is scaling to work with them all. The most recent <a href="http://rufy.com/starfish">darcs repo</a> even has work to extend starfish to support MapReduce, of google fame.</p></content>
|
|
112
|
+
</entry>
|
|
113
|
+
</feed>
|
|
114
|
+
|