medusa-crawler 1.0.0.pre.2 → 2.0.0.pre.2

This diff represents the content of publicly available package versions that have been released to one of the supported registries. The information contained in this diff is provided for informational purposes only and reflects changes between package versions as they appear in their respective public registries.
checksums.yaml CHANGED
@@ -1,7 +1,7 @@
1
1
  ---
2
2
  SHA256:
3
- metadata.gz: 9daf1076b9f0528f797128f639affadc99c61aa7980f4de84f91b520bda7b305
4
- data.tar.gz: c95e174b0215473befb1d9437865bc5e20696efb245151ebd5e6b462fbadb099
3
+ metadata.gz: 93449f7eaf488828f47375c8a6f4507f935ad66bd8b87b9c0861883aa0c37ca3
4
+ data.tar.gz: bb31d351122effd895001db71c266759c860424ee23323cc2ceecadce89e1457
5
5
  SHA512:
6
- metadata.gz: e0afb9e4ac4cc5fdfd0a7c9bb3f01672bbdf41e711005eefffbd5d1b54a601e52868ad093e9985f49078942656551acca73de26d4b675e1442dec981e618bee3
7
- data.tar.gz: 1098323c312714b7d1d72abd762d49a97dc233dfd8fb38cd574ab5611112cdc73859050a6f8fefcda35315274fad939c36750ce8277c5560b72769e5e5051a5b
6
+ metadata.gz: c4815f7e839fbe41bedd5e804452c3634504e40732b5eab1bad03215d2d6b858656eec9230b5a6e7ed3f6db7c5b8b43f8dbe6c110c039660effe5e931d947961
7
+ data.tar.gz: 00d26b931fbb6bd9c40ed23c661191cda398967c7c40f600a52cdf93ce396d4d69d815327f40f84513e12e66ef7214df5889bac127680c550552d2626f1154cf
data/CHANGELOG.md CHANGED
@@ -1,8 +1,49 @@
1
+ ## Pre-release v2.0.0.pre.2 (2026-08-26)
1
2
 
2
- ## Anemone forked into Medusa (2014-12-13)
3
+ Fixes:
4
+ - Fix tag-version validation in the RubyGems publishing workflow
5
+
6
+ ## Pre-release v2.0.0.pre.1 (2026-08-26)
7
+
8
+ Changes:
9
+ - Require Ruby 3.3 or newer and test against Ruby 3.3, 3.4, and 4.0
10
+ - Add explicit `ostruct` and `webrick` runtime dependencies for modern Ruby releases
11
+ - Add trusted publishing to RubyGems through GitHub Actions
12
+ - Add project guidance, roadmap, and agent-readable documentation
13
+
14
+ ## Release v1.0.0 (2020-08-17)
15
+ Features:
16
+ - Remove `PageStore#pages_linking_to`, `PageStore#urls_linking_to`
17
+ - Remove `verbose` setting
18
+
19
+ Changes:
20
+ - Add an examples section to the [README](https://github.com/brutuscat/medusa-crawler/blob/main/README.md) file
21
+ - Update the [CONTRIBUTORS](https://github.com/brutuscat/medusa-crawler/blob/main/CONTRIBUTORS.mdd) file
22
+ - Update the [CHANGELOG](https://github.com/brutuscat/medusa-crawler/blob/main/CHANGELOG.md) file
23
+
24
+ ## Pre-release v1.0.0.pre.2
25
+ Features:
26
+ - Remove CLI bins
27
+ - Remove `PageStore#shortest_paths!`
28
+
29
+ Fixes
30
+ - Skip link regex filter to consider the full URI [#1](https://github.com/brutuscat/medusa-crawler/issues/1)
3
31
 
32
+ ## Pre-release v1.0.0.pre.1
4
33
  Features:
34
+ - Switch to use `Moneta` instead of custom storage provider adapters
35
+
36
+ Fixes
37
+ - Fix link skip regex to include the full URI [#1](https://github.com/brutuscat/medusa-crawler/issues/1)
5
38
 
39
+ Dev
40
+ - Use webmock gem for testing
41
+
42
+ Changes:
43
+ - Rename Medusa to medusa-crawler gem
44
+
45
+ ## Anemone forked into Medusa (2014-12-13)
46
+ Features:
6
47
  - Switch to use `OpenURI` instead of `net/http`, gaining out of the box support for:
7
48
  - Http basic auth options
8
49
  - Proxy configuration options
@@ -11,10 +52,9 @@ Features:
11
52
  - Ability to control the RETRY_LIMIT upon connection errors
12
53
 
13
54
  Changes:
14
-
15
55
  - Renamed Anemone to Medusa
16
- - Revamped the [README](https://github.com/brutuscat/medusa/blob/master/README.md) file
17
- - Revamped the [CHANGELOG](https://github.com/brutuscat/medusa/blob/master/CHANGELOG.md) file
18
- - Revamped the [CONTRIBUTORS](https://github.com/brutuscat/medusa/blob/master/CONTRIBUTORS.mdd) file
56
+ - Revamped the [README](https://github.com/brutuscat/medusa-crawler/blob/main/README.md) file
57
+ - Revamped the [CHANGELOG](https://github.com/brutuscat/medusa-crawler/blob/main/CHANGELOG.md) file
58
+ - Revamped the [CONTRIBUTORS](https://github.com/brutuscat/medusa-crawler/blob/main/CONTRIBUTORS.mdd) file
19
59
 
20
- > Refer to the [Anemone changelog](https://github.com/chriskite/anemone/blob/next/CHANGELOG.rdoc) for a travel in time.
60
+ > Refer to the [Anemone changelog](https://github.com/chriskite/anemone/blob/next/CHANGELOG.rdoc) to go back to the past.
data/CONTRIBUTORS.md CHANGED
@@ -1,6 +1,6 @@
1
1
  # Contributors
2
2
 
3
- Many thanks to the following folks who have contributed code to Medusa (a fork of Anemone).
3
+ Many thanks to the following people who have contributed code to Medusa (a fork of Anemone).
4
4
 
5
5
  In no particular order:
6
6
 
data/README.rdoc CHANGED
@@ -1,4 +1,4 @@
1
- == Medusa: a ruby crawler framework {rdoc-image:https://badge.fury.io/rb/medusa-crawler.svg}[https://badge.fury.io/rb/medusa-crawler] rdoc-image:https://github.com/brutuscat/medusa-crawler/workflows/Ruby/badge.svg?event=push
1
+ == Medusa: a ruby crawler framework {rdoc-image:https://badge.fury.io/rb/medusa-crawler.svg}[https://rubygems.org/gems/medusa-crawler] rdoc-image:https://github.com/brutuscat/medusa-crawler/workflows/Ruby/badge.svg?event=push
2
2
 
3
3
  Medusa is a framework for the ruby language to crawl and collect useful information about the pages
4
4
  it visits. It is versatile, allowing you to write your own specialized tasks quickly and easily.
@@ -9,7 +9,6 @@ it visits. It is versatile, allowing you to write your own specialized tasks qui
9
9
  * Multi-threaded design for high performance
10
10
  * Tracks +301+ HTTP redirects
11
11
  * Allows exclusion of URLs based on regular expressions
12
- * HTTPS support
13
12
  * Records response time for each page
14
13
  * Obey _robots.txt_ directives (optional, but recommended)
15
14
  * In-memory or persistent storage of pages during crawl, provided by Moneta[https://github.com/moneta-rb/moneta]
@@ -17,6 +16,37 @@ it visits. It is versatile, allowing you to write your own specialized tasks qui
17
16
 
18
17
  <b>Do you have an idea or a suggestion? {Open an issue and talk about it}[https://github.com/brutuscat/medusa-crawler/issues/new]</b>
19
18
 
19
+ === Examples
20
+
21
+ Medusa is versatile and to be used programatically, you can start with one or multiple URIs:
22
+
23
+ require 'medusa'
24
+
25
+ Medusa.crawl('https://www.example.com', depth_limit: 2)
26
+
27
+ Or you can pass a block and it will yield the crawler back, to manage configuration or drive its crawling focus:
28
+
29
+ require 'medusa'
30
+
31
+ Medusa.crawl('https://www.example.com', depth_limit: 2) do |crawler|
32
+ crawler.discard_page_bodies = some_flag
33
+
34
+ # Persist all the pages state across crawl-runs.
35
+ crawler.clear_on_startup = false
36
+ crawler.storage = Medusa::Storage.Moneta(:Redis, 'redis://redis.host.name:6379/0')
37
+
38
+ crawler.skip_links_like(/private/)
39
+
40
+ crawler.on_pages_like(/public/) do |page|
41
+ logger.debug "[public page] #{page.url} took #{page.response_time} found #{page.links.count}"
42
+ end
43
+
44
+ # Use an arbitrary logic, page by page, to continue customize the crawling.
45
+ crawler.focus_crawl(/public/) do |page|
46
+ page.links.first
47
+ end
48
+ end
49
+
20
50
  ---
21
51
 
22
52
  === Requirements
data/Rakefile CHANGED
@@ -7,11 +7,6 @@ RSpec::Core::RakeTask.new(:rspec) do |spec|
7
7
  spec.pattern = 'spec/**/*_spec.rb'
8
8
  end
9
9
 
10
- RSpec::Core::RakeTask.new(:rcov) do |spec|
11
- spec.pattern = 'spec/**/*_spec.rb'
12
- spec.rcov = true
13
- end
14
-
15
10
  task :default => :rspec
16
11
 
17
12
  Rake::RDocTask.new(:rdoc) do |rdoc|
data/VERSION CHANGED
@@ -1 +1 @@
1
- 1.0.0.pre.2
1
+ 2.0.0.pre.2
data/lib/medusa/core.rb CHANGED
@@ -26,8 +26,6 @@ module Medusa
26
26
  DEFAULT_OPTS = {
27
27
  # run 4 Tentacle threads to fetch pages
28
28
  :threads => 4,
29
- # disable verbose output
30
- :verbose => false,
31
29
  # don't throw away the page response body after scanning it for links
32
30
  :discard_page_bodies => false,
33
31
  # identify self as Medusa/VERSION
@@ -40,7 +38,7 @@ module Medusa
40
38
  :depth_limit => false,
41
39
  # number of times HTTP redirects will be followed
42
40
  :redirect_limit => 5,
43
- # storage engine defaults to Hash in +process_options+ if none specified
41
+ # storage engine defaults to In-memory store in +process_options+ if none specified
44
42
  :storage => nil,
45
43
  # cleanups of the storage on every startup of the crawler
46
44
  :clear_on_startup => true,
@@ -148,6 +146,7 @@ module Medusa
148
146
  # Perform the crawl
149
147
  #
150
148
  def run
149
+
151
150
  process_options
152
151
 
153
152
  @urls.delete_if { |url| !visit_link?(url) }
@@ -165,7 +164,6 @@ module Medusa
165
164
  loop do
166
165
  page = page_queue.deq
167
166
  @pages.touch_key page.url
168
- puts "#{page.url} Queue: #{link_queue.size}" if @opts[:verbose]
169
167
  do_page_blocks page
170
168
  page.discard_doc! if @opts[:discard_page_bodies]
171
169
 
data/lib/medusa/http.rb CHANGED
@@ -29,9 +29,9 @@ module Medusa
29
29
  # including redirects
30
30
  #
31
31
  def fetch_pages(url, referer = nil, depth = nil)
32
+ pages = []
32
33
  begin
33
34
  url = URI(url) unless url.is_a?(URI)
34
- pages = []
35
35
  get(url, referer) do |response, headers, code, location, redirect_to, response_time|
36
36
  pages << Page.new(location, :body => response,
37
37
  :headers => headers,
@@ -43,13 +43,8 @@ module Medusa
43
43
  end
44
44
 
45
45
  return pages
46
- rescue Exception => e
47
- if verbose?
48
- puts e.inspect
49
- puts e.backtrace
50
- end
51
- pages ||= []
52
- return pages << Page.new(url, :error => e)
46
+ rescue StandardError => e
47
+ return pages << Page.new(url, error: e)
53
48
  end
54
49
  end
55
50
 
@@ -180,18 +175,13 @@ module Medusa
180
175
 
181
176
  rescue Timeout::Error, EOFError, Errno::ECONNREFUSED, Errno::ETIMEDOUT, Errno::ECONNRESET => e
182
177
  retries += 1
183
- puts "[medusa] Retrying ##{retries} on url #{url} because of: #{e.inspect}" if verbose?
184
178
  sleep(3 ^ retries)
185
179
  retry unless retries > RETRY_LIMIT
186
180
  ensure
187
- resource.close if !resource.nil? && !resource.closed?
181
+ resource&.close unless resource&.closed?
188
182
  end
189
183
  end
190
184
 
191
- def verbose?
192
- @opts[:verbose]
193
- end
194
-
195
185
  #
196
186
  # Allowed to connect to the requested url?
197
187
  #
@@ -65,58 +65,5 @@ module Medusa
65
65
  each_value { |page| delete page.url if page.redirect? }
66
66
  self
67
67
  end
68
-
69
- #
70
- # If given a single URL (as a String or URI), returns an Array of Pages which link to that URL
71
- # If given an Array of URLs, returns a Hash (URI => [Page, Page...]) of Pages linking to those URLs
72
- #
73
- def pages_linking_to(urls)
74
- unless urls.is_a?(Array)
75
- urls = [urls]
76
- single = true
77
- end
78
-
79
- urls.map! do |url|
80
- unless url.is_a?(URI)
81
- URI(url) rescue nil
82
- else
83
- url
84
- end
85
- end
86
- urls.compact
87
-
88
- links = {}
89
- urls.each { |url| links[url] = [] }
90
- values.each do |page|
91
- urls.each { |url| links[url] << page if page.links.include?(url) }
92
- end
93
-
94
- if single and !links.empty?
95
- return links[urls.first]
96
- else
97
- return links
98
- end
99
- end
100
-
101
- #
102
- # If given a single URL (as a String or URI), returns an Array of URLs which link to that URL
103
- # If given an Array of URLs, returns a Hash (URI => [URI, URI...]) of URLs linking to those URLs
104
- #
105
- def urls_linking_to(urls)
106
- unless urls.is_a?(Array)
107
- urls = [urls] unless urls.is_a?(Array)
108
- single = true
109
- end
110
-
111
- links = pages_linking_to(urls)
112
- links.each { |url, pages| links[url] = pages.map{|p| p.url} }
113
-
114
- if single and !links.empty?
115
- return links[urls.first]
116
- else
117
- return links
118
- end
119
- end
120
-
121
68
  end
122
69
  end
@@ -18,7 +18,6 @@ module Medusa
18
18
  def [](key)
19
19
  @adap[key]
20
20
  rescue
21
- puts key
22
21
  raise RetrievalError
23
22
  end
24
23
 
@@ -1,3 +1,3 @@
1
1
  module Medusa
2
- VERSION = '1.0.0.pre.2'
2
+ VERSION = '2.0.0.pre.2'
3
3
  end
@@ -29,6 +29,8 @@ module Medusa
29
29
  @base = options[:base] if options.has_key?(:base)
30
30
  @content_type = options[:content_type] || "text/html"
31
31
  @body = options[:body]
32
+ @status = options[:status] || [200, 'OK']
33
+ @exception = options[:exception]
32
34
 
33
35
  create_body unless @body
34
36
  add_to_fakeweb
@@ -56,7 +58,7 @@ module Medusa
56
58
  end
57
59
 
58
60
  def add_to_fakeweb
59
- options = {body: @body, status: [200, 'OK'], headers: {'Content-Type' => @content_type}}
61
+ options = {body: @body, status: @status, headers: {'Content-Type' => @content_type}}
60
62
 
61
63
  if @redirect
62
64
  options[:status] = [301, 'Moved Permanently']
@@ -66,7 +68,7 @@ module Medusa
66
68
  options[:headers]['Location'] = redirect_url
67
69
 
68
70
  # register the page this one redirects to
69
- WebMock.stub_request(:get, redirect_url).to_return(body: '', status: [200, 'OK'], headers: {'Content-Type' => @content_type})
71
+ WebMock.stub_request(:get, redirect_url).to_return(body: '', status: @status, headers: {'Content-Type' => @content_type})
70
72
  end
71
73
 
72
74
  if @auth
@@ -75,11 +77,14 @@ module Medusa
75
77
  WebMock.stub_request(:get, url).to_return(unautorized_options)
76
78
  WebMock.stub_request(:get, url).with(basic_auth: AUTH).to_return(options)
77
79
  else
78
- WebMock.stub_request(:get, url).to_return(options)
80
+ WebMock.stub_request(:get, url).tap do |req|
81
+ if @exception
82
+ req.to_raise(@exception)
83
+ else
84
+ req.to_return(options)
85
+ end
86
+ end
79
87
  end
80
88
  end
81
89
  end
82
90
  end
83
-
84
- #default root
85
- Medusa::FakePage.new
data/spec/medusa_spec.rb CHANGED
@@ -1,3 +1,4 @@
1
+ require 'fakeweb_helper'
1
2
 
2
3
  RSpec.describe Medusa do
3
4
 
@@ -6,9 +7,9 @@ RSpec.describe Medusa do
6
7
  end
7
8
 
8
9
  it "should return a Medusa::Core from the crawl, which has a PageStore" do
10
+ Medusa::FakePage.new
9
11
  result = Medusa.crawl(SPEC_DOMAIN)
10
12
  expect(result).to be_an_instance_of(Medusa::Core)
11
13
  expect(result.pages).to be_an_instance_of(Medusa::PageStore)
12
14
  end
13
-
14
15
  end
metadata CHANGED
@@ -1,41 +1,14 @@
1
1
  --- !ruby/object:Gem::Specification
2
2
  name: medusa-crawler
3
3
  version: !ruby/object:Gem::Version
4
- version: 1.0.0.pre.2
4
+ version: 2.0.0.pre.2
5
5
  platform: ruby
6
6
  authors:
7
7
  - Mauro Asprea
8
8
  - Chris Kite
9
- autorequire:
10
9
  bindir: bin
11
- cert_chain:
12
- - |
13
- -----BEGIN CERTIFICATE-----
14
- MIIERDCCAqygAwIBAgIBATANBgkqhkiG9w0BAQsFADAmMSQwIgYDVQQDDBttYXVy
15
- b2FzcHJlYS9EQz1nbWFpbC9EQz1jb20wHhcNMjAwODA2MTEwNDAzWhcNMjEwODA2
16
- MTEwNDAzWjAmMSQwIgYDVQQDDBttYXVyb2FzcHJlYS9EQz1nbWFpbC9EQz1jb20w
17
- ggGiMA0GCSqGSIb3DQEBAQUAA4IBjwAwggGKAoIBgQDNsvJN5orxdj+eEUiSttpC
18
- 6N6IeTK/btk65TAbZlY7/9MVIHM/Ya3tUSWoQA6KURsksV+lUbJm0MC4pwm5y46T
19
- p+Q6/eDz8gmofian4X+y3inyzYqmvbM2t2mPcO+Mj1NjSfGn56jqk0ey32xL7cz3
20
- YZqx9LnkUUuqzn4QM+R8LfGUWXe9nlkMoLoNGw+X1XB8bPYoQo2YpofILEWcCLJl
21
- NnFjcjIzQuE3NDaPIBX/cXPDpvuKNVXw51My35pQb5uW4Aa7sRZAz1MCDoFA1+XO
22
- 9czVf+zE+YsMtuqruWOI08TEPD2kMeiDqMc1fbPkLN9Rhe2Nxj8fTK1Ptyz0mQJL
23
- x5cQZT2XK+nmZKiU/z+91XU17BNd+LEERU6ZmjkGQKUH/Gj060kQWMnw2sOsZJS9
24
- VVU/raletJxhODeQbShcrxUR6VSGDkgQa4G/rqtPjUQ9AXYeNeVBg2aE4DRTRFhy
25
- gYsyE9DbAluV7BcJWO+GKntyXqhu14dnATCfSFI3WMsCAwEAAaN9MHswCQYDVR0T
26
- BAIwADALBgNVHQ8EBAMCBLAwHQYDVR0OBBYEFAjiCg2cEkNDWhG4P1PgDwATcmgR
27
- MCAGA1UdEQQZMBeBFW1hdXJvYXNwcmVhQGdtYWlsLmNvbTAgBgNVHRIEGTAXgRVt
28
- YXVyb2FzcHJlYUBnbWFpbC5jb20wDQYJKoZIhvcNAQELBQADggGBAG4S5uDseGtN
29
- EqxvogGfJ+h7Pg5pdDsMVFWXOPN9IztSHE+88Ypv9b97zNk/vBztj3nwkUuhEczv
30
- GZeI9/F72j1+8ARWdNgHalsrTHcFWaGEYNkm06EGh21dtO939Uqjg9auv9thDD/C
31
- 4jlEii1EvaGIaMhEdONAiRyOj0gEagw1AU5ItSxoWHFaaCEQSqXRskZMpw6fDZrE
32
- jnJPYx5I8axku6D+/nlP3GeuMNwlqzTg1YblLDanzEdI7Yet41MgMwLcor6Z4BXb
33
- xKDzHfGjkQfEpEvwzqSi0rAbrMv68FRybNxgB/3gwgSiEaeFeE4CFX0gm1F3m28m
34
- Z9rgjNU1SA9dSTGLY14K3rjtn71PMlP64Ci+QJ0HA8V0+cpx1Lkn3WsywgILV16a
35
- g4G6EZGbKCMwJDC0Wtmrygr7+THZVQlBs0ljTdrN8GXsuI9W52VlZctZQXEuoboH
36
- mpXw1d3WewNciml1VaOG782DKqZvT0i19V5LnZzoGzmU2q3ZJw7jCw==
37
- -----END CERTIFICATE-----
38
- date: 2020-08-14 00:00:00.000000000 Z
10
+ cert_chain: []
11
+ date: 1980-01-02 00:00:00.000000000 Z
39
12
  dependencies:
40
13
  - !ruby/object:Gem::Dependency
41
14
  name: moneta
@@ -77,6 +50,26 @@ dependencies:
77
50
  - - ">="
78
51
  - !ruby/object:Gem::Version
79
52
  version: 1.3.0
53
+ - !ruby/object:Gem::Dependency
54
+ name: ostruct
55
+ requirement: !ruby/object:Gem::Requirement
56
+ requirements:
57
+ - - ">="
58
+ - !ruby/object:Gem::Version
59
+ version: '0.6'
60
+ - - "<"
61
+ - !ruby/object:Gem::Version
62
+ version: '1.0'
63
+ type: :runtime
64
+ prerelease: false
65
+ version_requirements: !ruby/object:Gem::Requirement
66
+ requirements:
67
+ - - ">="
68
+ - !ruby/object:Gem::Version
69
+ version: '0.6'
70
+ - - "<"
71
+ - !ruby/object:Gem::Version
72
+ version: '1.0'
80
73
  - !ruby/object:Gem::Dependency
81
74
  name: robotex
82
75
  requirement: !ruby/object:Gem::Requirement
@@ -97,8 +90,28 @@ dependencies:
97
90
  - - ">="
98
91
  - !ruby/object:Gem::Version
99
92
  version: 1.0.0
93
+ - !ruby/object:Gem::Dependency
94
+ name: webrick
95
+ requirement: !ruby/object:Gem::Requirement
96
+ requirements:
97
+ - - ">="
98
+ - !ruby/object:Gem::Version
99
+ version: '1.8'
100
+ - - "<"
101
+ - !ruby/object:Gem::Version
102
+ version: '2.0'
103
+ type: :runtime
104
+ prerelease: false
105
+ version_requirements: !ruby/object:Gem::Requirement
106
+ requirements:
107
+ - - ">="
108
+ - !ruby/object:Gem::Version
109
+ version: '1.8'
110
+ - - "<"
111
+ - !ruby/object:Gem::Version
112
+ version: '2.0'
100
113
  description: |+
101
- == Medusa: a ruby crawler framework {rdoc-image:https://badge.fury.io/rb/medusa-crawler.svg}[https://badge.fury.io/rb/medusa-crawler] rdoc-image:https://github.com/brutuscat/medusa-crawler/workflows/Ruby/badge.svg?event=push
114
+ == Medusa: a ruby crawler framework {rdoc-image:https://badge.fury.io/rb/medusa-crawler.svg}[https://rubygems.org/gems/medusa-crawler] rdoc-image:https://github.com/brutuscat/medusa-crawler/workflows/Ruby/badge.svg?event=push
102
115
 
103
116
  Medusa is a framework for the ruby language to crawl and collect useful information about the pages
104
117
  it visits. It is versatile, allowing you to write your own specialized tasks quickly and easily.
@@ -109,7 +122,6 @@ description: |+
109
122
  * Multi-threaded design for high performance
110
123
  * Tracks +301+ HTTP redirects
111
124
  * Allows exclusion of URLs based on regular expressions
112
- * HTTPS support
113
125
  * Records response time for each page
114
126
  * Obey _robots.txt_ directives (optional, but recommended)
115
127
  * In-memory or persistent storage of pages during crawl, provided by Moneta[https://github.com/moneta-rb/moneta]
@@ -117,6 +129,37 @@ description: |+
117
129
 
118
130
  <b>Do you have an idea or a suggestion? {Open an issue and talk about it}[https://github.com/brutuscat/medusa-crawler/issues/new]</b>
119
131
 
132
+ === Examples
133
+
134
+ Medusa is versatile and to be used programatically, you can start with one or multiple URIs:
135
+
136
+ require 'medusa'
137
+
138
+ Medusa.crawl('https://www.example.com', depth_limit: 2)
139
+
140
+ Or you can pass a block and it will yield the crawler back, to manage configuration or drive its crawling focus:
141
+
142
+ require 'medusa'
143
+
144
+ Medusa.crawl('https://www.example.com', depth_limit: 2) do |crawler|
145
+ crawler.discard_page_bodies = some_flag
146
+
147
+ # Persist all the pages state across crawl-runs.
148
+ crawler.clear_on_startup = false
149
+ crawler.storage = Medusa::Storage.Moneta(:Redis, 'redis://redis.host.name:6379/0')
150
+
151
+ crawler.skip_links_like(/private/)
152
+
153
+ crawler.on_pages_like(/public/) do |page|
154
+ logger.debug "[public page] #{page.url} took #{page.response_time} found #{page.links.count}"
155
+ end
156
+
157
+ # Use an arbitrary logic, page by page, to continue customize the crawling.
158
+ crawler.focus_crawl(/public/) do |page|
159
+ page.links.first
160
+ end
161
+ end
162
+
120
163
  email: mauroasprea@gmail.com
121
164
  executables: []
122
165
  extensions: []
@@ -151,9 +194,8 @@ licenses:
151
194
  - MIT
152
195
  metadata:
153
196
  bug_tracker_uri: https://github.com/brutuscat/medusa-crawler/issues
154
- source_code_uri: https://github.com/brutuscat/medusa-crawler/tree/v1.0.0.pre.2
197
+ source_code_uri: https://github.com/brutuscat/medusa-crawler/tree/v2.0.0.pre.2
155
198
  description_markup_format: rdoc
156
- post_install_message:
157
199
  rdoc_options:
158
200
  - "-m"
159
201
  - README.rdoc
@@ -165,19 +207,19 @@ required_ruby_version: !ruby/object:Gem::Requirement
165
207
  requirements:
166
208
  - - ">="
167
209
  - !ruby/object:Gem::Version
168
- version: 2.3.0
210
+ version: '3.3'
169
211
  required_rubygems_version: !ruby/object:Gem::Requirement
170
212
  requirements:
171
- - - ">"
213
+ - - ">="
172
214
  - !ruby/object:Gem::Version
173
- version: 1.3.1
215
+ version: '0'
174
216
  requirements: []
175
- rubygems_version: 3.1.2
176
- signing_key:
217
+ rubygems_version: 4.0.16
177
218
  specification_version: 4
178
219
  summary: Medusa is a ruby crawler framework
179
220
  test_files:
180
- - spec/spec_helper.rb
181
- - spec/medusa_helper.rb
182
221
  - spec/fakeweb_helper.rb
222
+ - spec/medusa_helper.rb
183
223
  - spec/medusa_spec.rb
224
+ - spec/spec_helper.rb
225
+ ...
checksums.yaml.gz.sig DELETED
Binary file
data.tar.gz.sig DELETED
Binary file
metadata.gz.sig DELETED
@@ -1,2 +0,0 @@
1
- �ԧ���Cm`�C�]�<��I�
2
- V,�� ܕ[+O�1�܃7����֦��L�L����Y,'M�@TAM7�g���50��N�cr@��d��;�ްW�q�f��D�T)]��A�]�H����,^%�!8���))P�I���ד�铎�f^�a&��� ]���FK��te=����^����b��37 ��L��4�����Fb�E���