medusa-crawler 1.0.0.pre.2 → 2.0.0.pre.2
This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
- checksums.yaml +4 -4
- data/CHANGELOG.md +46 -6
- data/CONTRIBUTORS.md +1 -1
- data/README.rdoc +32 -2
- data/Rakefile +0 -5
- data/VERSION +1 -1
- data/lib/medusa/core.rb +2 -4
- data/lib/medusa/http.rb +4 -14
- data/lib/medusa/page_store.rb +0 -53
- data/lib/medusa/storage/base.rb +0 -1
- data/lib/medusa/version.rb +1 -1
- data/spec/fakeweb_helper.rb +11 -6
- data/spec/medusa_spec.rb +2 -1
- metadata +83 -41
- checksums.yaml.gz.sig +0 -0
- data.tar.gz.sig +0 -0
- metadata.gz.sig +0 -2
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
2
|
SHA256:
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
3
|
+
metadata.gz: 93449f7eaf488828f47375c8a6f4507f935ad66bd8b87b9c0861883aa0c37ca3
|
|
4
|
+
data.tar.gz: bb31d351122effd895001db71c266759c860424ee23323cc2ceecadce89e1457
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: c4815f7e839fbe41bedd5e804452c3634504e40732b5eab1bad03215d2d6b858656eec9230b5a6e7ed3f6db7c5b8b43f8dbe6c110c039660effe5e931d947961
|
|
7
|
+
data.tar.gz: 00d26b931fbb6bd9c40ed23c661191cda398967c7c40f600a52cdf93ce396d4d69d815327f40f84513e12e66ef7214df5889bac127680c550552d2626f1154cf
|
data/CHANGELOG.md
CHANGED
|
@@ -1,8 +1,49 @@
|
|
|
1
|
+
## Pre-release v2.0.0.pre.2 (2026-08-26)
|
|
1
2
|
|
|
2
|
-
|
|
3
|
+
Fixes:
|
|
4
|
+
- Fix tag-version validation in the RubyGems publishing workflow
|
|
5
|
+
|
|
6
|
+
## Pre-release v2.0.0.pre.1 (2026-08-26)
|
|
7
|
+
|
|
8
|
+
Changes:
|
|
9
|
+
- Require Ruby 3.3 or newer and test against Ruby 3.3, 3.4, and 4.0
|
|
10
|
+
- Add explicit `ostruct` and `webrick` runtime dependencies for modern Ruby releases
|
|
11
|
+
- Add trusted publishing to RubyGems through GitHub Actions
|
|
12
|
+
- Add project guidance, roadmap, and agent-readable documentation
|
|
13
|
+
|
|
14
|
+
## Release v1.0.0 (2020-08-17)
|
|
15
|
+
Features:
|
|
16
|
+
- Remove `PageStore#pages_linking_to`, `PageStore#urls_linking_to`
|
|
17
|
+
- Remove `verbose` setting
|
|
18
|
+
|
|
19
|
+
Changes:
|
|
20
|
+
- Add an examples section to the [README](https://github.com/brutuscat/medusa-crawler/blob/main/README.md) file
|
|
21
|
+
- Update the [CONTRIBUTORS](https://github.com/brutuscat/medusa-crawler/blob/main/CONTRIBUTORS.mdd) file
|
|
22
|
+
- Update the [CHANGELOG](https://github.com/brutuscat/medusa-crawler/blob/main/CHANGELOG.md) file
|
|
23
|
+
|
|
24
|
+
## Pre-release v1.0.0.pre.2
|
|
25
|
+
Features:
|
|
26
|
+
- Remove CLI bins
|
|
27
|
+
- Remove `PageStore#shortest_paths!`
|
|
28
|
+
|
|
29
|
+
Fixes
|
|
30
|
+
- Skip link regex filter to consider the full URI [#1](https://github.com/brutuscat/medusa-crawler/issues/1)
|
|
3
31
|
|
|
32
|
+
## Pre-release v1.0.0.pre.1
|
|
4
33
|
Features:
|
|
34
|
+
- Switch to use `Moneta` instead of custom storage provider adapters
|
|
35
|
+
|
|
36
|
+
Fixes
|
|
37
|
+
- Fix link skip regex to include the full URI [#1](https://github.com/brutuscat/medusa-crawler/issues/1)
|
|
5
38
|
|
|
39
|
+
Dev
|
|
40
|
+
- Use webmock gem for testing
|
|
41
|
+
|
|
42
|
+
Changes:
|
|
43
|
+
- Rename Medusa to medusa-crawler gem
|
|
44
|
+
|
|
45
|
+
## Anemone forked into Medusa (2014-12-13)
|
|
46
|
+
Features:
|
|
6
47
|
- Switch to use `OpenURI` instead of `net/http`, gaining out of the box support for:
|
|
7
48
|
- Http basic auth options
|
|
8
49
|
- Proxy configuration options
|
|
@@ -11,10 +52,9 @@ Features:
|
|
|
11
52
|
- Ability to control the RETRY_LIMIT upon connection errors
|
|
12
53
|
|
|
13
54
|
Changes:
|
|
14
|
-
|
|
15
55
|
- Renamed Anemone to Medusa
|
|
16
|
-
- Revamped the [README](https://github.com/brutuscat/medusa/blob/
|
|
17
|
-
- Revamped the [CHANGELOG](https://github.com/brutuscat/medusa/blob/
|
|
18
|
-
- Revamped the [CONTRIBUTORS](https://github.com/brutuscat/medusa/blob/
|
|
56
|
+
- Revamped the [README](https://github.com/brutuscat/medusa-crawler/blob/main/README.md) file
|
|
57
|
+
- Revamped the [CHANGELOG](https://github.com/brutuscat/medusa-crawler/blob/main/CHANGELOG.md) file
|
|
58
|
+
- Revamped the [CONTRIBUTORS](https://github.com/brutuscat/medusa-crawler/blob/main/CONTRIBUTORS.mdd) file
|
|
19
59
|
|
|
20
|
-
> Refer to the [Anemone changelog](https://github.com/chriskite/anemone/blob/next/CHANGELOG.rdoc)
|
|
60
|
+
> Refer to the [Anemone changelog](https://github.com/chriskite/anemone/blob/next/CHANGELOG.rdoc) to go back to the past.
|
data/CONTRIBUTORS.md
CHANGED
data/README.rdoc
CHANGED
|
@@ -1,4 +1,4 @@
|
|
|
1
|
-
== Medusa: a ruby crawler framework {rdoc-image:https://badge.fury.io/rb/medusa-crawler.svg}[https://
|
|
1
|
+
== Medusa: a ruby crawler framework {rdoc-image:https://badge.fury.io/rb/medusa-crawler.svg}[https://rubygems.org/gems/medusa-crawler] rdoc-image:https://github.com/brutuscat/medusa-crawler/workflows/Ruby/badge.svg?event=push
|
|
2
2
|
|
|
3
3
|
Medusa is a framework for the ruby language to crawl and collect useful information about the pages
|
|
4
4
|
it visits. It is versatile, allowing you to write your own specialized tasks quickly and easily.
|
|
@@ -9,7 +9,6 @@ it visits. It is versatile, allowing you to write your own specialized tasks qui
|
|
|
9
9
|
* Multi-threaded design for high performance
|
|
10
10
|
* Tracks +301+ HTTP redirects
|
|
11
11
|
* Allows exclusion of URLs based on regular expressions
|
|
12
|
-
* HTTPS support
|
|
13
12
|
* Records response time for each page
|
|
14
13
|
* Obey _robots.txt_ directives (optional, but recommended)
|
|
15
14
|
* In-memory or persistent storage of pages during crawl, provided by Moneta[https://github.com/moneta-rb/moneta]
|
|
@@ -17,6 +16,37 @@ it visits. It is versatile, allowing you to write your own specialized tasks qui
|
|
|
17
16
|
|
|
18
17
|
<b>Do you have an idea or a suggestion? {Open an issue and talk about it}[https://github.com/brutuscat/medusa-crawler/issues/new]</b>
|
|
19
18
|
|
|
19
|
+
=== Examples
|
|
20
|
+
|
|
21
|
+
Medusa is versatile and to be used programatically, you can start with one or multiple URIs:
|
|
22
|
+
|
|
23
|
+
require 'medusa'
|
|
24
|
+
|
|
25
|
+
Medusa.crawl('https://www.example.com', depth_limit: 2)
|
|
26
|
+
|
|
27
|
+
Or you can pass a block and it will yield the crawler back, to manage configuration or drive its crawling focus:
|
|
28
|
+
|
|
29
|
+
require 'medusa'
|
|
30
|
+
|
|
31
|
+
Medusa.crawl('https://www.example.com', depth_limit: 2) do |crawler|
|
|
32
|
+
crawler.discard_page_bodies = some_flag
|
|
33
|
+
|
|
34
|
+
# Persist all the pages state across crawl-runs.
|
|
35
|
+
crawler.clear_on_startup = false
|
|
36
|
+
crawler.storage = Medusa::Storage.Moneta(:Redis, 'redis://redis.host.name:6379/0')
|
|
37
|
+
|
|
38
|
+
crawler.skip_links_like(/private/)
|
|
39
|
+
|
|
40
|
+
crawler.on_pages_like(/public/) do |page|
|
|
41
|
+
logger.debug "[public page] #{page.url} took #{page.response_time} found #{page.links.count}"
|
|
42
|
+
end
|
|
43
|
+
|
|
44
|
+
# Use an arbitrary logic, page by page, to continue customize the crawling.
|
|
45
|
+
crawler.focus_crawl(/public/) do |page|
|
|
46
|
+
page.links.first
|
|
47
|
+
end
|
|
48
|
+
end
|
|
49
|
+
|
|
20
50
|
---
|
|
21
51
|
|
|
22
52
|
=== Requirements
|
data/Rakefile
CHANGED
|
@@ -7,11 +7,6 @@ RSpec::Core::RakeTask.new(:rspec) do |spec|
|
|
|
7
7
|
spec.pattern = 'spec/**/*_spec.rb'
|
|
8
8
|
end
|
|
9
9
|
|
|
10
|
-
RSpec::Core::RakeTask.new(:rcov) do |spec|
|
|
11
|
-
spec.pattern = 'spec/**/*_spec.rb'
|
|
12
|
-
spec.rcov = true
|
|
13
|
-
end
|
|
14
|
-
|
|
15
10
|
task :default => :rspec
|
|
16
11
|
|
|
17
12
|
Rake::RDocTask.new(:rdoc) do |rdoc|
|
data/VERSION
CHANGED
|
@@ -1 +1 @@
|
|
|
1
|
-
|
|
1
|
+
2.0.0.pre.2
|
data/lib/medusa/core.rb
CHANGED
|
@@ -26,8 +26,6 @@ module Medusa
|
|
|
26
26
|
DEFAULT_OPTS = {
|
|
27
27
|
# run 4 Tentacle threads to fetch pages
|
|
28
28
|
:threads => 4,
|
|
29
|
-
# disable verbose output
|
|
30
|
-
:verbose => false,
|
|
31
29
|
# don't throw away the page response body after scanning it for links
|
|
32
30
|
:discard_page_bodies => false,
|
|
33
31
|
# identify self as Medusa/VERSION
|
|
@@ -40,7 +38,7 @@ module Medusa
|
|
|
40
38
|
:depth_limit => false,
|
|
41
39
|
# number of times HTTP redirects will be followed
|
|
42
40
|
:redirect_limit => 5,
|
|
43
|
-
# storage engine defaults to
|
|
41
|
+
# storage engine defaults to In-memory store in +process_options+ if none specified
|
|
44
42
|
:storage => nil,
|
|
45
43
|
# cleanups of the storage on every startup of the crawler
|
|
46
44
|
:clear_on_startup => true,
|
|
@@ -148,6 +146,7 @@ module Medusa
|
|
|
148
146
|
# Perform the crawl
|
|
149
147
|
#
|
|
150
148
|
def run
|
|
149
|
+
|
|
151
150
|
process_options
|
|
152
151
|
|
|
153
152
|
@urls.delete_if { |url| !visit_link?(url) }
|
|
@@ -165,7 +164,6 @@ module Medusa
|
|
|
165
164
|
loop do
|
|
166
165
|
page = page_queue.deq
|
|
167
166
|
@pages.touch_key page.url
|
|
168
|
-
puts "#{page.url} Queue: #{link_queue.size}" if @opts[:verbose]
|
|
169
167
|
do_page_blocks page
|
|
170
168
|
page.discard_doc! if @opts[:discard_page_bodies]
|
|
171
169
|
|
data/lib/medusa/http.rb
CHANGED
|
@@ -29,9 +29,9 @@ module Medusa
|
|
|
29
29
|
# including redirects
|
|
30
30
|
#
|
|
31
31
|
def fetch_pages(url, referer = nil, depth = nil)
|
|
32
|
+
pages = []
|
|
32
33
|
begin
|
|
33
34
|
url = URI(url) unless url.is_a?(URI)
|
|
34
|
-
pages = []
|
|
35
35
|
get(url, referer) do |response, headers, code, location, redirect_to, response_time|
|
|
36
36
|
pages << Page.new(location, :body => response,
|
|
37
37
|
:headers => headers,
|
|
@@ -43,13 +43,8 @@ module Medusa
|
|
|
43
43
|
end
|
|
44
44
|
|
|
45
45
|
return pages
|
|
46
|
-
rescue
|
|
47
|
-
|
|
48
|
-
puts e.inspect
|
|
49
|
-
puts e.backtrace
|
|
50
|
-
end
|
|
51
|
-
pages ||= []
|
|
52
|
-
return pages << Page.new(url, :error => e)
|
|
46
|
+
rescue StandardError => e
|
|
47
|
+
return pages << Page.new(url, error: e)
|
|
53
48
|
end
|
|
54
49
|
end
|
|
55
50
|
|
|
@@ -180,18 +175,13 @@ module Medusa
|
|
|
180
175
|
|
|
181
176
|
rescue Timeout::Error, EOFError, Errno::ECONNREFUSED, Errno::ETIMEDOUT, Errno::ECONNRESET => e
|
|
182
177
|
retries += 1
|
|
183
|
-
puts "[medusa] Retrying ##{retries} on url #{url} because of: #{e.inspect}" if verbose?
|
|
184
178
|
sleep(3 ^ retries)
|
|
185
179
|
retry unless retries > RETRY_LIMIT
|
|
186
180
|
ensure
|
|
187
|
-
resource
|
|
181
|
+
resource&.close unless resource&.closed?
|
|
188
182
|
end
|
|
189
183
|
end
|
|
190
184
|
|
|
191
|
-
def verbose?
|
|
192
|
-
@opts[:verbose]
|
|
193
|
-
end
|
|
194
|
-
|
|
195
185
|
#
|
|
196
186
|
# Allowed to connect to the requested url?
|
|
197
187
|
#
|
data/lib/medusa/page_store.rb
CHANGED
|
@@ -65,58 +65,5 @@ module Medusa
|
|
|
65
65
|
each_value { |page| delete page.url if page.redirect? }
|
|
66
66
|
self
|
|
67
67
|
end
|
|
68
|
-
|
|
69
|
-
#
|
|
70
|
-
# If given a single URL (as a String or URI), returns an Array of Pages which link to that URL
|
|
71
|
-
# If given an Array of URLs, returns a Hash (URI => [Page, Page...]) of Pages linking to those URLs
|
|
72
|
-
#
|
|
73
|
-
def pages_linking_to(urls)
|
|
74
|
-
unless urls.is_a?(Array)
|
|
75
|
-
urls = [urls]
|
|
76
|
-
single = true
|
|
77
|
-
end
|
|
78
|
-
|
|
79
|
-
urls.map! do |url|
|
|
80
|
-
unless url.is_a?(URI)
|
|
81
|
-
URI(url) rescue nil
|
|
82
|
-
else
|
|
83
|
-
url
|
|
84
|
-
end
|
|
85
|
-
end
|
|
86
|
-
urls.compact
|
|
87
|
-
|
|
88
|
-
links = {}
|
|
89
|
-
urls.each { |url| links[url] = [] }
|
|
90
|
-
values.each do |page|
|
|
91
|
-
urls.each { |url| links[url] << page if page.links.include?(url) }
|
|
92
|
-
end
|
|
93
|
-
|
|
94
|
-
if single and !links.empty?
|
|
95
|
-
return links[urls.first]
|
|
96
|
-
else
|
|
97
|
-
return links
|
|
98
|
-
end
|
|
99
|
-
end
|
|
100
|
-
|
|
101
|
-
#
|
|
102
|
-
# If given a single URL (as a String or URI), returns an Array of URLs which link to that URL
|
|
103
|
-
# If given an Array of URLs, returns a Hash (URI => [URI, URI...]) of URLs linking to those URLs
|
|
104
|
-
#
|
|
105
|
-
def urls_linking_to(urls)
|
|
106
|
-
unless urls.is_a?(Array)
|
|
107
|
-
urls = [urls] unless urls.is_a?(Array)
|
|
108
|
-
single = true
|
|
109
|
-
end
|
|
110
|
-
|
|
111
|
-
links = pages_linking_to(urls)
|
|
112
|
-
links.each { |url, pages| links[url] = pages.map{|p| p.url} }
|
|
113
|
-
|
|
114
|
-
if single and !links.empty?
|
|
115
|
-
return links[urls.first]
|
|
116
|
-
else
|
|
117
|
-
return links
|
|
118
|
-
end
|
|
119
|
-
end
|
|
120
|
-
|
|
121
68
|
end
|
|
122
69
|
end
|
data/lib/medusa/storage/base.rb
CHANGED
data/lib/medusa/version.rb
CHANGED
data/spec/fakeweb_helper.rb
CHANGED
|
@@ -29,6 +29,8 @@ module Medusa
|
|
|
29
29
|
@base = options[:base] if options.has_key?(:base)
|
|
30
30
|
@content_type = options[:content_type] || "text/html"
|
|
31
31
|
@body = options[:body]
|
|
32
|
+
@status = options[:status] || [200, 'OK']
|
|
33
|
+
@exception = options[:exception]
|
|
32
34
|
|
|
33
35
|
create_body unless @body
|
|
34
36
|
add_to_fakeweb
|
|
@@ -56,7 +58,7 @@ module Medusa
|
|
|
56
58
|
end
|
|
57
59
|
|
|
58
60
|
def add_to_fakeweb
|
|
59
|
-
options = {body: @body, status:
|
|
61
|
+
options = {body: @body, status: @status, headers: {'Content-Type' => @content_type}}
|
|
60
62
|
|
|
61
63
|
if @redirect
|
|
62
64
|
options[:status] = [301, 'Moved Permanently']
|
|
@@ -66,7 +68,7 @@ module Medusa
|
|
|
66
68
|
options[:headers]['Location'] = redirect_url
|
|
67
69
|
|
|
68
70
|
# register the page this one redirects to
|
|
69
|
-
WebMock.stub_request(:get, redirect_url).to_return(body: '', status:
|
|
71
|
+
WebMock.stub_request(:get, redirect_url).to_return(body: '', status: @status, headers: {'Content-Type' => @content_type})
|
|
70
72
|
end
|
|
71
73
|
|
|
72
74
|
if @auth
|
|
@@ -75,11 +77,14 @@ module Medusa
|
|
|
75
77
|
WebMock.stub_request(:get, url).to_return(unautorized_options)
|
|
76
78
|
WebMock.stub_request(:get, url).with(basic_auth: AUTH).to_return(options)
|
|
77
79
|
else
|
|
78
|
-
WebMock.stub_request(:get, url).
|
|
80
|
+
WebMock.stub_request(:get, url).tap do |req|
|
|
81
|
+
if @exception
|
|
82
|
+
req.to_raise(@exception)
|
|
83
|
+
else
|
|
84
|
+
req.to_return(options)
|
|
85
|
+
end
|
|
86
|
+
end
|
|
79
87
|
end
|
|
80
88
|
end
|
|
81
89
|
end
|
|
82
90
|
end
|
|
83
|
-
|
|
84
|
-
#default root
|
|
85
|
-
Medusa::FakePage.new
|
data/spec/medusa_spec.rb
CHANGED
|
@@ -1,3 +1,4 @@
|
|
|
1
|
+
require 'fakeweb_helper'
|
|
1
2
|
|
|
2
3
|
RSpec.describe Medusa do
|
|
3
4
|
|
|
@@ -6,9 +7,9 @@ RSpec.describe Medusa do
|
|
|
6
7
|
end
|
|
7
8
|
|
|
8
9
|
it "should return a Medusa::Core from the crawl, which has a PageStore" do
|
|
10
|
+
Medusa::FakePage.new
|
|
9
11
|
result = Medusa.crawl(SPEC_DOMAIN)
|
|
10
12
|
expect(result).to be_an_instance_of(Medusa::Core)
|
|
11
13
|
expect(result.pages).to be_an_instance_of(Medusa::PageStore)
|
|
12
14
|
end
|
|
13
|
-
|
|
14
15
|
end
|
metadata
CHANGED
|
@@ -1,41 +1,14 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: medusa-crawler
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version:
|
|
4
|
+
version: 2.0.0.pre.2
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Mauro Asprea
|
|
8
8
|
- Chris Kite
|
|
9
|
-
autorequire:
|
|
10
9
|
bindir: bin
|
|
11
|
-
cert_chain:
|
|
12
|
-
-
|
|
13
|
-
-----BEGIN CERTIFICATE-----
|
|
14
|
-
MIIERDCCAqygAwIBAgIBATANBgkqhkiG9w0BAQsFADAmMSQwIgYDVQQDDBttYXVy
|
|
15
|
-
b2FzcHJlYS9EQz1nbWFpbC9EQz1jb20wHhcNMjAwODA2MTEwNDAzWhcNMjEwODA2
|
|
16
|
-
MTEwNDAzWjAmMSQwIgYDVQQDDBttYXVyb2FzcHJlYS9EQz1nbWFpbC9EQz1jb20w
|
|
17
|
-
ggGiMA0GCSqGSIb3DQEBAQUAA4IBjwAwggGKAoIBgQDNsvJN5orxdj+eEUiSttpC
|
|
18
|
-
6N6IeTK/btk65TAbZlY7/9MVIHM/Ya3tUSWoQA6KURsksV+lUbJm0MC4pwm5y46T
|
|
19
|
-
p+Q6/eDz8gmofian4X+y3inyzYqmvbM2t2mPcO+Mj1NjSfGn56jqk0ey32xL7cz3
|
|
20
|
-
YZqx9LnkUUuqzn4QM+R8LfGUWXe9nlkMoLoNGw+X1XB8bPYoQo2YpofILEWcCLJl
|
|
21
|
-
NnFjcjIzQuE3NDaPIBX/cXPDpvuKNVXw51My35pQb5uW4Aa7sRZAz1MCDoFA1+XO
|
|
22
|
-
9czVf+zE+YsMtuqruWOI08TEPD2kMeiDqMc1fbPkLN9Rhe2Nxj8fTK1Ptyz0mQJL
|
|
23
|
-
x5cQZT2XK+nmZKiU/z+91XU17BNd+LEERU6ZmjkGQKUH/Gj060kQWMnw2sOsZJS9
|
|
24
|
-
VVU/raletJxhODeQbShcrxUR6VSGDkgQa4G/rqtPjUQ9AXYeNeVBg2aE4DRTRFhy
|
|
25
|
-
gYsyE9DbAluV7BcJWO+GKntyXqhu14dnATCfSFI3WMsCAwEAAaN9MHswCQYDVR0T
|
|
26
|
-
BAIwADALBgNVHQ8EBAMCBLAwHQYDVR0OBBYEFAjiCg2cEkNDWhG4P1PgDwATcmgR
|
|
27
|
-
MCAGA1UdEQQZMBeBFW1hdXJvYXNwcmVhQGdtYWlsLmNvbTAgBgNVHRIEGTAXgRVt
|
|
28
|
-
YXVyb2FzcHJlYUBnbWFpbC5jb20wDQYJKoZIhvcNAQELBQADggGBAG4S5uDseGtN
|
|
29
|
-
EqxvogGfJ+h7Pg5pdDsMVFWXOPN9IztSHE+88Ypv9b97zNk/vBztj3nwkUuhEczv
|
|
30
|
-
GZeI9/F72j1+8ARWdNgHalsrTHcFWaGEYNkm06EGh21dtO939Uqjg9auv9thDD/C
|
|
31
|
-
4jlEii1EvaGIaMhEdONAiRyOj0gEagw1AU5ItSxoWHFaaCEQSqXRskZMpw6fDZrE
|
|
32
|
-
jnJPYx5I8axku6D+/nlP3GeuMNwlqzTg1YblLDanzEdI7Yet41MgMwLcor6Z4BXb
|
|
33
|
-
xKDzHfGjkQfEpEvwzqSi0rAbrMv68FRybNxgB/3gwgSiEaeFeE4CFX0gm1F3m28m
|
|
34
|
-
Z9rgjNU1SA9dSTGLY14K3rjtn71PMlP64Ci+QJ0HA8V0+cpx1Lkn3WsywgILV16a
|
|
35
|
-
g4G6EZGbKCMwJDC0Wtmrygr7+THZVQlBs0ljTdrN8GXsuI9W52VlZctZQXEuoboH
|
|
36
|
-
mpXw1d3WewNciml1VaOG782DKqZvT0i19V5LnZzoGzmU2q3ZJw7jCw==
|
|
37
|
-
-----END CERTIFICATE-----
|
|
38
|
-
date: 2020-08-14 00:00:00.000000000 Z
|
|
10
|
+
cert_chain: []
|
|
11
|
+
date: 1980-01-02 00:00:00.000000000 Z
|
|
39
12
|
dependencies:
|
|
40
13
|
- !ruby/object:Gem::Dependency
|
|
41
14
|
name: moneta
|
|
@@ -77,6 +50,26 @@ dependencies:
|
|
|
77
50
|
- - ">="
|
|
78
51
|
- !ruby/object:Gem::Version
|
|
79
52
|
version: 1.3.0
|
|
53
|
+
- !ruby/object:Gem::Dependency
|
|
54
|
+
name: ostruct
|
|
55
|
+
requirement: !ruby/object:Gem::Requirement
|
|
56
|
+
requirements:
|
|
57
|
+
- - ">="
|
|
58
|
+
- !ruby/object:Gem::Version
|
|
59
|
+
version: '0.6'
|
|
60
|
+
- - "<"
|
|
61
|
+
- !ruby/object:Gem::Version
|
|
62
|
+
version: '1.0'
|
|
63
|
+
type: :runtime
|
|
64
|
+
prerelease: false
|
|
65
|
+
version_requirements: !ruby/object:Gem::Requirement
|
|
66
|
+
requirements:
|
|
67
|
+
- - ">="
|
|
68
|
+
- !ruby/object:Gem::Version
|
|
69
|
+
version: '0.6'
|
|
70
|
+
- - "<"
|
|
71
|
+
- !ruby/object:Gem::Version
|
|
72
|
+
version: '1.0'
|
|
80
73
|
- !ruby/object:Gem::Dependency
|
|
81
74
|
name: robotex
|
|
82
75
|
requirement: !ruby/object:Gem::Requirement
|
|
@@ -97,8 +90,28 @@ dependencies:
|
|
|
97
90
|
- - ">="
|
|
98
91
|
- !ruby/object:Gem::Version
|
|
99
92
|
version: 1.0.0
|
|
93
|
+
- !ruby/object:Gem::Dependency
|
|
94
|
+
name: webrick
|
|
95
|
+
requirement: !ruby/object:Gem::Requirement
|
|
96
|
+
requirements:
|
|
97
|
+
- - ">="
|
|
98
|
+
- !ruby/object:Gem::Version
|
|
99
|
+
version: '1.8'
|
|
100
|
+
- - "<"
|
|
101
|
+
- !ruby/object:Gem::Version
|
|
102
|
+
version: '2.0'
|
|
103
|
+
type: :runtime
|
|
104
|
+
prerelease: false
|
|
105
|
+
version_requirements: !ruby/object:Gem::Requirement
|
|
106
|
+
requirements:
|
|
107
|
+
- - ">="
|
|
108
|
+
- !ruby/object:Gem::Version
|
|
109
|
+
version: '1.8'
|
|
110
|
+
- - "<"
|
|
111
|
+
- !ruby/object:Gem::Version
|
|
112
|
+
version: '2.0'
|
|
100
113
|
description: |+
|
|
101
|
-
== Medusa: a ruby crawler framework {rdoc-image:https://badge.fury.io/rb/medusa-crawler.svg}[https://
|
|
114
|
+
== Medusa: a ruby crawler framework {rdoc-image:https://badge.fury.io/rb/medusa-crawler.svg}[https://rubygems.org/gems/medusa-crawler] rdoc-image:https://github.com/brutuscat/medusa-crawler/workflows/Ruby/badge.svg?event=push
|
|
102
115
|
|
|
103
116
|
Medusa is a framework for the ruby language to crawl and collect useful information about the pages
|
|
104
117
|
it visits. It is versatile, allowing you to write your own specialized tasks quickly and easily.
|
|
@@ -109,7 +122,6 @@ description: |+
|
|
|
109
122
|
* Multi-threaded design for high performance
|
|
110
123
|
* Tracks +301+ HTTP redirects
|
|
111
124
|
* Allows exclusion of URLs based on regular expressions
|
|
112
|
-
* HTTPS support
|
|
113
125
|
* Records response time for each page
|
|
114
126
|
* Obey _robots.txt_ directives (optional, but recommended)
|
|
115
127
|
* In-memory or persistent storage of pages during crawl, provided by Moneta[https://github.com/moneta-rb/moneta]
|
|
@@ -117,6 +129,37 @@ description: |+
|
|
|
117
129
|
|
|
118
130
|
<b>Do you have an idea or a suggestion? {Open an issue and talk about it}[https://github.com/brutuscat/medusa-crawler/issues/new]</b>
|
|
119
131
|
|
|
132
|
+
=== Examples
|
|
133
|
+
|
|
134
|
+
Medusa is versatile and to be used programatically, you can start with one or multiple URIs:
|
|
135
|
+
|
|
136
|
+
require 'medusa'
|
|
137
|
+
|
|
138
|
+
Medusa.crawl('https://www.example.com', depth_limit: 2)
|
|
139
|
+
|
|
140
|
+
Or you can pass a block and it will yield the crawler back, to manage configuration or drive its crawling focus:
|
|
141
|
+
|
|
142
|
+
require 'medusa'
|
|
143
|
+
|
|
144
|
+
Medusa.crawl('https://www.example.com', depth_limit: 2) do |crawler|
|
|
145
|
+
crawler.discard_page_bodies = some_flag
|
|
146
|
+
|
|
147
|
+
# Persist all the pages state across crawl-runs.
|
|
148
|
+
crawler.clear_on_startup = false
|
|
149
|
+
crawler.storage = Medusa::Storage.Moneta(:Redis, 'redis://redis.host.name:6379/0')
|
|
150
|
+
|
|
151
|
+
crawler.skip_links_like(/private/)
|
|
152
|
+
|
|
153
|
+
crawler.on_pages_like(/public/) do |page|
|
|
154
|
+
logger.debug "[public page] #{page.url} took #{page.response_time} found #{page.links.count}"
|
|
155
|
+
end
|
|
156
|
+
|
|
157
|
+
# Use an arbitrary logic, page by page, to continue customize the crawling.
|
|
158
|
+
crawler.focus_crawl(/public/) do |page|
|
|
159
|
+
page.links.first
|
|
160
|
+
end
|
|
161
|
+
end
|
|
162
|
+
|
|
120
163
|
email: mauroasprea@gmail.com
|
|
121
164
|
executables: []
|
|
122
165
|
extensions: []
|
|
@@ -151,9 +194,8 @@ licenses:
|
|
|
151
194
|
- MIT
|
|
152
195
|
metadata:
|
|
153
196
|
bug_tracker_uri: https://github.com/brutuscat/medusa-crawler/issues
|
|
154
|
-
source_code_uri: https://github.com/brutuscat/medusa-crawler/tree/
|
|
197
|
+
source_code_uri: https://github.com/brutuscat/medusa-crawler/tree/v2.0.0.pre.2
|
|
155
198
|
description_markup_format: rdoc
|
|
156
|
-
post_install_message:
|
|
157
199
|
rdoc_options:
|
|
158
200
|
- "-m"
|
|
159
201
|
- README.rdoc
|
|
@@ -165,19 +207,19 @@ required_ruby_version: !ruby/object:Gem::Requirement
|
|
|
165
207
|
requirements:
|
|
166
208
|
- - ">="
|
|
167
209
|
- !ruby/object:Gem::Version
|
|
168
|
-
version:
|
|
210
|
+
version: '3.3'
|
|
169
211
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
|
170
212
|
requirements:
|
|
171
|
-
- - "
|
|
213
|
+
- - ">="
|
|
172
214
|
- !ruby/object:Gem::Version
|
|
173
|
-
version:
|
|
215
|
+
version: '0'
|
|
174
216
|
requirements: []
|
|
175
|
-
rubygems_version:
|
|
176
|
-
signing_key:
|
|
217
|
+
rubygems_version: 4.0.16
|
|
177
218
|
specification_version: 4
|
|
178
219
|
summary: Medusa is a ruby crawler framework
|
|
179
220
|
test_files:
|
|
180
|
-
- spec/spec_helper.rb
|
|
181
|
-
- spec/medusa_helper.rb
|
|
182
221
|
- spec/fakeweb_helper.rb
|
|
222
|
+
- spec/medusa_helper.rb
|
|
183
223
|
- spec/medusa_spec.rb
|
|
224
|
+
- spec/spec_helper.rb
|
|
225
|
+
...
|
checksums.yaml.gz.sig
DELETED
|
Binary file
|
data.tar.gz.sig
DELETED
|
Binary file
|
metadata.gz.sig
DELETED