simplecrawler 0.1.7 → 0.1.8

Sign up to get free protection for your applications and to get access to all the features.
Files changed (11) hide show
  1. data/LICENSE +14 -0
  2. data/lib/simplecrawler.rb +179 -170
  3. data/tests/simplecrawler_test.rb +9 -4
  4. metadata +40 -46
  5. data/examples/accessibility_report.rb +0 -44
  6. data/examples/crawl.rb +0 -12
  7. data/examples/find_broken_links.rb +0 -21
  8. data/examples/find_pdfs.rb +0 -20
  9. data/examples/list_site_links.rb +0 -11
  10. data/examples/result.htm +0 -1282
  11. data/examples/riksdagen.txt +0 -66
data/examples/riksdagen.txt DELETED
@@ -1,66 +0,0 @@
1
- Added http://www.riksdagen.se/
2
- Opening http://www.riksdagen.se/
3
- Loading data from http://www.riksdagen.se/
4
- 3 http://www.riksdagen.se/
5
- Queuing links for http://www.riksdagen.se/
6
- Added http://www.riksdagen.se/om
7
- Added http://www.riksdagen.se/innehall
8
- Added http://www.riksdagen.se/fragor
9
- Added http://www.riksdagen.se/anvander
10
- Added http://www.riksdagen.se/kontakt
11
- Added /default____4.aspx
12
- Added /templates/R_Page____283.aspx
13
- Added /templates/R_OptionPage____285.aspx
14
- Added /shopping/R_Default____2508.aspx
15
- Added /webbnav/index.aspx?nid=53
16
- Added /templates/R_Page____275.aspx
17
- Added /templates/R_Page____2161.aspx
18
- Added /templates/R_Page____273.aspx
19
- Added /templates/R_Page____3618.aspx
20
- Added /templates/R_Page____271.aspx
21
- Added /templates/R_SubStartPage____5029.aspx
22
- Added /templates/R_SubStartPage____6694.aspx
23
- Added /templates/R_SubStartPage____272.aspx
24
- Added /templates/R_SubStartPage____282.aspx
25
- Added /templates/R_SubStartPage____238.aspx
26
- Added /templates/R_Page____1068.aspx
27
- Added /templates/R_Page____6531.aspx
28
- Added /templates/R_CustomCalendar____1765.aspx
29
- Added /Webbnav/index.aspx?nid=7800
30
- Added /templates/R_SubStartPage____4935.aspx
31
- Added /templates/R_Page____4335.aspx
32
- Added /templates/R_Page____13951.aspx
33
- Added /templates/R_Page____1595.aspx
34
- Added /templates/R_Page____16886.aspx
35
- Added /templates/R_Page____14748.aspx
36
- Added /templates/R_Page____12861.aspx
37
- Added #
38
- Added /templates/R_ExternalPage____3383.aspx?op=search&search_freetext=&search_parlamentary_session=2008%2F09&search_type=Interpellationsdebatt&search_committee=&search_speaker=&search_party=&search_sdate1=2009%2D01%2D16&search_sdate2=2009%2D01%2D16&rpage=0
39
- Added http://www.riksdagen.se/templates/R_HtmlCallPage____17356.aspx
40
- Added http://www.riksdagen.se/templates/R_Page____17944.aspx
41
- Added http://www.riksdagen.se/templates/R_Page____17958.aspx
42
- Added http://www.riksdagen.se/templates/R_HtmlCallPage____17361.aspx
43
- Added /templates/R_LopsedelArkiv____3388.aspx
44
- Added /templates/R_Page____1977.aspx
45
- Added /templates/R_LLSubStartPage____4303.aspx
46
- Added /default____56.aspx
47
- Added /templates/R_Page____1928.aspx
48
- Added /templates/R_Page____6546.aspx
49
- Added /templates/R_SubStartPage____257.aspx
50
- Added /templates/R_SubStartPage____448.aspx
51
- Added /templates/R_SubStartPage____4492.aspx
52
- Added /templates/R_Page____498.aspx
53
- Added /templates/R_Page____734.aspx
54
- Opening http://www.riksdagen.se/om
55
- Loading data from http://www.riksdagen.se/om
56
- 3 http://www.riksdagen.se/om
57
- Queuing links for http://www.riksdagen.se/om
58
- Added /templates/R_Page____735.aspx
59
- Added /templates/R_Page____5915.aspx
60
- Added /templates/R_Page____1791.aspx
61
- Added /templates/R_Page____1788.aspx
62
- Added /templates/R_PageFull____6580.aspx
63
- Added /templates/R_Page____6558.aspx
64
- Added /templates/R_Page____5674.aspx
65
- Opening http://www.riksdagen.se/innehall
66
- Loading data from http://www.riksdagen.se/innehall