simplecrawler 0.1.7 → 0.1.8

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
Files changed (11) hide show
  1. data/LICENSE +14 -0
  2. data/lib/simplecrawler.rb +179 -170
  3. data/tests/simplecrawler_test.rb +9 -4
  4. metadata +40 -46
  5. data/examples/accessibility_report.rb +0 -44
  6. data/examples/crawl.rb +0 -12
  7. data/examples/find_broken_links.rb +0 -21
  8. data/examples/find_pdfs.rb +0 -20
  9. data/examples/list_site_links.rb +0 -11
  10. data/examples/result.htm +0 -1282
  11. data/examples/riksdagen.txt +0 -66
data/examples/riksdagen.txt DELETED
@@ -1,66 +0,0 @@
1
- Added http://www.riksdagen.se/
2
- Opening http://www.riksdagen.se/
3
- Loading data from http://www.riksdagen.se/
4
- 3 http://www.riksdagen.se/
5
- Queuing links for http://www.riksdagen.se/
6
- Added http://www.riksdagen.se/om
7
- Added http://www.riksdagen.se/innehall
8
- Added http://www.riksdagen.se/fragor
9
- Added http://www.riksdagen.se/anvander
10
- Added http://www.riksdagen.se/kontakt
11
- Added /default____4.aspx
12
- Added /templates/R_Page____283.aspx
13
- Added /templates/R_OptionPage____285.aspx
14
- Added /shopping/R_Default____2508.aspx
15
- Added /webbnav/index.aspx?nid=53
16
- Added /templates/R_Page____275.aspx
17
- Added /templates/R_Page____2161.aspx
18
- Added /templates/R_Page____273.aspx
19
- Added /templates/R_Page____3618.aspx
20
- Added /templates/R_Page____271.aspx
21
- Added /templates/R_SubStartPage____5029.aspx
22
- Added /templates/R_SubStartPage____6694.aspx
23
- Added /templates/R_SubStartPage____272.aspx
24
- Added /templates/R_SubStartPage____282.aspx
25
- Added /templates/R_SubStartPage____238.aspx
26
- Added /templates/R_Page____1068.aspx
27
- Added /templates/R_Page____6531.aspx
28
- Added /templates/R_CustomCalendar____1765.aspx
29
- Added /Webbnav/index.aspx?nid=7800
30
- Added /templates/R_SubStartPage____4935.aspx
31
- Added /templates/R_Page____4335.aspx
32
- Added /templates/R_Page____13951.aspx
33
- Added /templates/R_Page____1595.aspx
34
- Added /templates/R_Page____16886.aspx
35
- Added /templates/R_Page____14748.aspx
36
- Added /templates/R_Page____12861.aspx
37
- Added #
38
- Added /templates/R_ExternalPage____3383.aspx?op=search&search_freetext=&search_parlamentary_session=2008%2F09&search_type=Interpellationsdebatt&search_committee=&search_speaker=&search_party=&search_sdate1=2009%2D01%2D16&search_sdate2=2009%2D01%2D16&rpage=0
39
- Added http://www.riksdagen.se/templates/R_HtmlCallPage____17356.aspx
40
- Added http://www.riksdagen.se/templates/R_Page____17944.aspx
41
- Added http://www.riksdagen.se/templates/R_Page____17958.aspx
42
- Added http://www.riksdagen.se/templates/R_HtmlCallPage____17361.aspx
43
- Added /templates/R_LopsedelArkiv____3388.aspx
44
- Added /templates/R_Page____1977.aspx
45
- Added /templates/R_LLSubStartPage____4303.aspx
46
- Added /default____56.aspx
47
- Added /templates/R_Page____1928.aspx
48
- Added /templates/R_Page____6546.aspx
49
- Added /templates/R_SubStartPage____257.aspx
50
- Added /templates/R_SubStartPage____448.aspx
51
- Added /templates/R_SubStartPage____4492.aspx
52
- Added /templates/R_Page____498.aspx
53
- Added /templates/R_Page____734.aspx
54
- Opening http://www.riksdagen.se/om
55
- Loading data from http://www.riksdagen.se/om
56
- 3 http://www.riksdagen.se/om
57
- Queuing links for http://www.riksdagen.se/om
58
- Added /templates/R_Page____735.aspx
59
- Added /templates/R_Page____5915.aspx
60
- Added /templates/R_Page____1791.aspx
61
- Added /templates/R_Page____1788.aspx
62
- Added /templates/R_PageFull____6580.aspx
63
- Added /templates/R_Page____6558.aspx
64
- Added /templates/R_Page____5674.aspx
65
- Opening http://www.riksdagen.se/innehall
66
- Loading data from http://www.riksdagen.se/innehall