omni_scrape 0.1.5 → 0.1.5.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +8 -8
- data/README.md +13 -5
- data/lib/omni_scrape/version.rb +1 -1
- metadata +1 -1
checksums.yaml
CHANGED
@@ -1,15 +1,15 @@
|
|
1
1
|
---
|
2
2
|
!binary "U0hBMQ==":
|
3
3
|
metadata.gz: !binary |-
|
4
|
-
|
4
|
+
NjQ5ZDBiODVkN2RkZDM0MWE1ZGQ2MGE1MmQ5ZTJlZDJjNjE2NWE3Yw==
|
5
5
|
data.tar.gz: !binary |-
|
6
|
-
|
6
|
+
MmI4MmQ0M2QwYWY1OTUwOWE0ZGY5MzQzYzRjZTk3MTdlMjVjOGE4MQ==
|
7
7
|
SHA512:
|
8
8
|
metadata.gz: !binary |-
|
9
|
-
|
10
|
-
|
11
|
-
|
9
|
+
YzViNThmMjk3OWQ2NTUzMDNiNWYwNTM5ZTMyZDk5MDM0M2EyZGI1NzY3MjA5
|
10
|
+
Y2QwYTM5MDgyNDVlNGU5MjFlMjBjYzMwYjU3MDM0NjM3NDhjZWViNjRlMDFl
|
11
|
+
MTRhODc1NjA5Yzc1MjQ0YmE2NDIyNDc4MTFiOTI2ODQ5Y2IxNDY=
|
12
12
|
data.tar.gz: !binary |-
|
13
|
-
|
14
|
-
|
15
|
-
|
13
|
+
NjAzM2Q0NmY5ZTYyZGU2ZDRiNjNlNDM3ZDNjYjM0MGMzMTVmZGFjYTllM2Q4
|
14
|
+
MDZhOGMzMjJmNTQ3YWJjNDg5MjkzOTQ4OTc3MTE0OGRlOWMxNmU2NzllMzAw
|
15
|
+
ZjU2YjMxYWY4NjBlOThiYmRmODQ2ZWM0M2E3YzVhZGQ2NjEzMDY=
|
data/README.md
CHANGED
@@ -1,12 +1,10 @@
|
|
1
1
|
# OmniScrape
|
2
2
|
|
3
|
-
|
4
|
-
|
5
|
-
TODO: Delete this and the text above, and describe your gem
|
3
|
+
This gem is an all-purpose web crawler and scraper in the works.
|
6
4
|
|
7
5
|
## Installation
|
8
6
|
|
9
|
-
Add
|
7
|
+
Add these lines to your application's Gemfile:
|
10
8
|
|
11
9
|
```ruby
|
12
10
|
gem 'omni_scrape'
|
@@ -21,8 +19,18 @@ Or install it yourself as:
|
|
21
19
|
$ gem install omni_scrape
|
22
20
|
|
23
21
|
## Usage
|
22
|
+
Add the lines : require 'omni_scrape' and include OmniScrape to your script file.
|
23
|
+
Method : CrawlScrape
|
24
|
+
example : OmniScrape.CrawlScrape("http://en.wikipedia.org/wiki/List_of_massively_multiplayer_online_role-playing_games", 0, "http://en.wikipedia.org")
|
25
|
+
|
26
|
+
This method takes three parameters the first should be the url to start at.
|
27
|
+
|
28
|
+
The second parameter is currently unimplemented but will be the depth to crawl. (just pass it 0)
|
29
|
+
|
30
|
+
The third is a sub-url for internal links.
|
31
|
+
|
24
32
|
|
25
|
-
|
33
|
+
description: CrawlScrape will follow every link from the page provided and scrape the html from those pages, storing it as html files where the script is located.
|
26
34
|
|
27
35
|
## Development
|
28
36
|
|
data/lib/omni_scrape/version.rb
CHANGED