prerender_rails 1.5.0 → 1.7.0
Sign up to get free protection for your applications and to get access to all the features.
- checksums.yaml +5 -5
- data/.gitignore +1 -0
- data/README.md +11 -20
- data/lib/prerender_rails.rb +8 -10
- data/prerender_rails.gemspec +3 -3
- data/test/lib/prerender_rails.rb +24 -16
- metadata +21 -22
checksums.yaml
CHANGED
|
@@ -1,7 +1,7 @@
|
|
|
1
1
|
---
|
|
2
|
-
|
|
3
|
-
metadata.gz:
|
|
4
|
-
data.tar.gz:
|
|
2
|
+
SHA256:
|
|
3
|
+
metadata.gz: 0b7bb52e4361db6e0c211ae8b2bc21abc3181e7004bc46d06f7f7363e7a2a797
|
|
4
|
+
data.tar.gz: 7fe960c2ced57ff8ea9a3810e3c8cea844fde97b65b4e5dda030deeecf758566
|
|
5
5
|
SHA512:
|
|
6
|
-
metadata.gz:
|
|
7
|
-
data.tar.gz:
|
|
6
|
+
metadata.gz: 1409773b53acd77e880007547a0dc5e5fbf07e2ccc06fde0f6ff0fc688d220c131e55d991e1f1810e7ca73990f3f5f844c49f110902406f0d43ae9d16189dc6b
|
|
7
|
+
data.tar.gz: 620548190f5384997a293b8a81f2cdc013b73a90adce2efcb4cee1a4a87e311c2451789260925cce841914067dc71ee15bc1304186e846405eb1383798fe88c1
|
data/.gitignore
CHANGED
data/README.md
CHANGED
|
@@ -1,14 +1,11 @@
|
|
|
1
|
-
Prerender Rails [](https://travis-ci.org/prerender/prerender_rails) [](http://badge.fury.io/rb/prerender_rails)
|
|
2
2
|
===========================
|
|
3
3
|
|
|
4
|
-
Google, Facebook, Twitter,
|
|
4
|
+
Google, Facebook, Twitter, and Bing are constantly trying to view your website... but Google is the only crawler that executes a meaningful amount of JavaScript and Google even admits that they can execute JavaScript weeks after actually crawling. Prerender allows you to serve the full HTML of your website back to Google and other crawlers so that they don't have to execute any JavaScript. [Google recommends using Prerender.io](https://developers.google.com/search/docs/guides/dynamic-rendering) to prevent indexation issues on sites with large amounts of JavaScript.
|
|
5
5
|
|
|
6
|
-
|
|
6
|
+
Prerender is perfect for Angular SEO, React SEO, Vue SEO, and any other JavaScript framework.
|
|
7
7
|
|
|
8
|
-
|
|
9
|
-
- Just add <meta name="fragment" content="!"> to the <head> of all of your pages
|
|
10
|
-
- If you use hash urls (#), change them to the hash-bang (#!)
|
|
11
|
-
- That's it! Perfect SEO on javascript pages.
|
|
8
|
+
This middleware intercepts requests to your Node.js website from crawlers, and then makes a call to the (external) Prerender Service to get the static HTML instead of the JavaScript for that page. That HTML is then returned to the crawler.
|
|
12
9
|
|
|
13
10
|
`Note` Make sure you have more than one webserver thread/process running because the prerender service will make a request to your server to render the HTML.
|
|
14
11
|
|
|
@@ -22,7 +19,7 @@ And in `config/environment/production.rb`, add this line:
|
|
|
22
19
|
config.middleware.use Rack::Prerender
|
|
23
20
|
```
|
|
24
21
|
|
|
25
|
-
or if you have an account on [prerender.io](
|
|
22
|
+
or if you have an account on [prerender.io](https://prerender.io/) and want to use your token:
|
|
26
23
|
|
|
27
24
|
```ruby
|
|
28
25
|
config.middleware.use Rack::Prerender, prerender_token: 'YOUR_TOKEN'
|
|
@@ -32,21 +29,15 @@ or if you have an account on [prerender.io](http://prerender.io) and want to use
|
|
|
32
29
|
|
|
33
30
|
## Testing
|
|
34
31
|
|
|
35
|
-
When testing make sure you're not using a single threaded application server like default WEBrick one, use Puma or Unicorn.
|
|
32
|
+
When testing make sure you're not using a single threaded application server like default WEBrick one, use Puma or Unicorn to prevent a deadlock when the Prerender Service needs to render your HTML on the fly.
|
|
36
33
|
|
|
37
|
-
|
|
34
|
+
The best way to test the prerendered page is to [set the User Agent of your browser to Googlebot's user agent](https://developers.google.com/web/tools/chrome-devtools/device-mode/override-user-agent) and visit your URL directly. If you View Source on that URL, you should see the static HTML version of the page with the `<script>` tags removed from the page. If you still see `<script>` tags then that means the middleware isn't set up properly yet.
|
|
38
35
|
|
|
39
|
-
|
|
40
|
-
Then go to `http://localhost:5000/?_escaped_fragment_=/profiles/1234`
|
|
41
|
-
|
|
42
|
-
If your URLs use push-state:
|
|
43
|
-
|
|
44
|
-
If you want to see `http://localhost:5000/profiles/1234`
|
|
45
|
-
Then go to `http://localhost:5000/profiles/1234?_escaped_fragment_=`
|
|
36
|
+
`Note` If you're testing locally, you'll need to run the [prerender server](https://github.com/prerender/prerender) locally so that it has access to your server.
|
|
46
37
|
|
|
47
38
|
## How it works
|
|
48
39
|
1. The middleware checks to make sure we should show a prerendered page
|
|
49
|
-
1. The middleware checks if the request is from a crawler
|
|
40
|
+
1. The middleware checks if the request is from a crawler by checking the user agent string against a default list of crawler user agents
|
|
50
41
|
2. The middleware checks to make sure we aren't requesting a resource (js, css, etc...)
|
|
51
42
|
3. (optional) The middleware checks to make sure the url is in the whitelist
|
|
52
43
|
4. (optional) The middleware checks to make sure the url isn't in the blacklist
|
|
@@ -137,7 +128,7 @@ config.middleware.use Rack::Prerender,
|
|
|
137
128
|
|
|
138
129
|
## Using your own prerender service
|
|
139
130
|
|
|
140
|
-
We host a Prerender server at [prerender.io](
|
|
131
|
+
We host a Prerender server at [prerender.io](https://prerender.io/) so that you can work on more important things, but if you've deployed the prerender service on your own... set the `PRERENDER_SERVICE_URL` environment variable so that this middleware points there instead. Otherwise, it will default to the service already deployed by [prerender.io](https://prerender.io/).
|
|
141
132
|
|
|
142
133
|
$ export PRERENDER_SERVICE_URL=<new url>
|
|
143
134
|
|
|
@@ -148,7 +139,7 @@ Or on heroku:
|
|
|
148
139
|
As an alternative, you can pass `prerender_service_url` in the options object during initialization of the middleware
|
|
149
140
|
|
|
150
141
|
``` ruby
|
|
151
|
-
config.middleware.use Rack::Prerender, prerender_service_url: '
|
|
142
|
+
config.middleware.use Rack::Prerender, prerender_service_url: '<new url>'
|
|
152
143
|
```
|
|
153
144
|
|
|
154
145
|
## License
|
data/lib/prerender_rails.rb
CHANGED
|
@@ -4,13 +4,10 @@ module Rack
|
|
|
4
4
|
require 'active_support'
|
|
5
5
|
|
|
6
6
|
def initialize(app, options={})
|
|
7
|
-
# googlebot, yahoo, and bingbot are not in this list because
|
|
8
|
-
# we support _escaped_fragment_ and want to ensure people aren't
|
|
9
|
-
# penalized for cloaking.
|
|
10
7
|
@crawler_user_agents = [
|
|
11
|
-
|
|
12
|
-
|
|
13
|
-
|
|
8
|
+
'googlebot',
|
|
9
|
+
'yahoo',
|
|
10
|
+
'bingbot',
|
|
14
11
|
'baiduspider',
|
|
15
12
|
'facebookexternalhit',
|
|
16
13
|
'twitterbot',
|
|
@@ -37,7 +34,8 @@ module Rack
|
|
|
37
34
|
'nuzzel',
|
|
38
35
|
'Discordbot',
|
|
39
36
|
'Google Page Speed',
|
|
40
|
-
'Qwantify'
|
|
37
|
+
'Qwantify',
|
|
38
|
+
'Chrome-Lighthouse'
|
|
41
39
|
]
|
|
42
40
|
|
|
43
41
|
@extensions_to_ignore = [
|
|
@@ -114,8 +112,8 @@ module Rack
|
|
|
114
112
|
|
|
115
113
|
def should_show_prerendered_page(env)
|
|
116
114
|
user_agent = env['HTTP_USER_AGENT']
|
|
117
|
-
buffer_agent = env['
|
|
118
|
-
prerender_agent = env['
|
|
115
|
+
buffer_agent = env['HTTP_X_BUFFERBOT']
|
|
116
|
+
prerender_agent = env['HTTP_X_PRERENDER']
|
|
119
117
|
is_requesting_prerendered_page = false
|
|
120
118
|
|
|
121
119
|
return false if !user_agent
|
|
@@ -215,7 +213,7 @@ module Rack
|
|
|
215
213
|
|
|
216
214
|
|
|
217
215
|
def build_rack_response_from_prerender(prerendered_response)
|
|
218
|
-
response = Rack::Response.new(prerendered_response.body, prerendered_response.code, prerendered_response
|
|
216
|
+
response = Rack::Response.new(prerendered_response.body, prerendered_response.code, prerendered_response)
|
|
219
217
|
|
|
220
218
|
@options[:build_rack_response_from_prerender].call(response, prerendered_response) if @options[:build_rack_response_from_prerender]
|
|
221
219
|
|
data/prerender_rails.gemspec
CHANGED
|
@@ -2,7 +2,7 @@
|
|
|
2
2
|
|
|
3
3
|
Gem::Specification.new do |spec|
|
|
4
4
|
spec.name = "prerender_rails"
|
|
5
|
-
spec.version = "1.
|
|
5
|
+
spec.version = "1.7.0"
|
|
6
6
|
spec.authors = ["Todd Hooper"]
|
|
7
7
|
spec.email = ["todd@prerender.io"]
|
|
8
8
|
spec.description = %q{Rails middleware to prerender your javascript heavy pages on the fly by a phantomjs service}
|
|
@@ -15,10 +15,10 @@ Gem::Specification.new do |spec|
|
|
|
15
15
|
spec.test_files = spec.files.grep(%r{^(test|spec|features)/})
|
|
16
16
|
spec.require_paths = ["lib"]
|
|
17
17
|
|
|
18
|
-
spec.add_dependency 'rack', '
|
|
18
|
+
spec.add_dependency 'rack', '~> 2.2.2'
|
|
19
19
|
spec.add_dependency 'activesupport', '>= 0'
|
|
20
20
|
|
|
21
|
-
spec.add_development_dependency "bundler", "~> 1.
|
|
21
|
+
spec.add_development_dependency "bundler", "~> 2.1.2"
|
|
22
22
|
spec.add_development_dependency "rake"
|
|
23
23
|
spec.add_development_dependency "webmock"
|
|
24
24
|
end
|
data/test/lib/prerender_rails.rb
CHANGED
|
@@ -19,9 +19,9 @@ describe Rack::Prerender do
|
|
|
19
19
|
stub_request(:get, @prerender.build_api_url(request)).with(:headers => { 'User-Agent' => bot }).to_return(:body => "<html></html>", :status => 301, :headers => { 'Location' => 'http://google.com'})
|
|
20
20
|
response = Rack::Prerender.new(@app).call(request)
|
|
21
21
|
|
|
22
|
-
assert_equal response[2]
|
|
23
|
-
assert_equal response[
|
|
24
|
-
assert_equal( { 'location' => 'http://google.com'
|
|
22
|
+
assert_equal response[2], ["<html></html>"]
|
|
23
|
+
assert_equal response[0], 301
|
|
24
|
+
assert_equal( { 'location' => 'http://google.com'}, response[1] )
|
|
25
25
|
end
|
|
26
26
|
|
|
27
27
|
|
|
@@ -30,7 +30,7 @@ describe Rack::Prerender do
|
|
|
30
30
|
stub_request(:get, @prerender.build_api_url(request)).with(:headers => { 'User-Agent' => user }).to_return(:body => "<html></html>")
|
|
31
31
|
response = Rack::Prerender.new(@app).call(request)
|
|
32
32
|
|
|
33
|
-
assert_equal ["<html></html>"], response[2]
|
|
33
|
+
assert_equal ["<html></html>"], response[2]
|
|
34
34
|
end
|
|
35
35
|
|
|
36
36
|
|
|
@@ -58,6 +58,14 @@ describe Rack::Prerender do
|
|
|
58
58
|
end
|
|
59
59
|
|
|
60
60
|
|
|
61
|
+
it "should continue to app routes if contains X-Prerender header" do
|
|
62
|
+
request = Rack::MockRequest.env_for "/path?_escaped_fragment_=", "HTTP_USER_AGENT" => user, "HTTP_X_PRERENDER" => "1"
|
|
63
|
+
response = Rack::Prerender.new(@app).call(request)
|
|
64
|
+
|
|
65
|
+
assert_equal "", response[2]
|
|
66
|
+
end
|
|
67
|
+
|
|
68
|
+
|
|
61
69
|
it "should continue to app routes if user is a bot, but the bot is requesting a resource file" do
|
|
62
70
|
request = Rack::MockRequest.env_for "/main.js?anyQueryParam=true", "HTTP_USER_AGENT" => bot
|
|
63
71
|
response = Rack::Prerender.new(@app).call(request)
|
|
@@ -75,13 +83,13 @@ describe Rack::Prerender do
|
|
|
75
83
|
|
|
76
84
|
|
|
77
85
|
it "should set use_ssl to true for https prerender_service_url" do
|
|
78
|
-
|
|
86
|
+
@prerender = Rack::Prerender.new(@app, prerender_service_url: 'https://service.prerender.io/')
|
|
79
87
|
|
|
80
|
-
|
|
81
|
-
|
|
82
|
-
|
|
88
|
+
request = Rack::MockRequest.env_for "/search/things/123/page?_escaped_fragment_=", "HTTP_USER_AGENT" => bot
|
|
89
|
+
stub_request(:get, @prerender.build_api_url(request)).to_return(:body => "<html></html>")
|
|
90
|
+
response = @prerender.call(request)
|
|
83
91
|
|
|
84
|
-
|
|
92
|
+
assert_equal ["<html></html>"], response[2]
|
|
85
93
|
end
|
|
86
94
|
|
|
87
95
|
|
|
@@ -90,7 +98,7 @@ describe Rack::Prerender do
|
|
|
90
98
|
stub_request(:get, @prerender.build_api_url(request)).to_return(:body => "<html></html>")
|
|
91
99
|
response = Rack::Prerender.new(@app, whitelist: ['^/search.*page', '/help']).call(request)
|
|
92
100
|
|
|
93
|
-
assert_equal ["<html></html>"], response[2]
|
|
101
|
+
assert_equal ["<html></html>"], response[2]
|
|
94
102
|
end
|
|
95
103
|
|
|
96
104
|
|
|
@@ -113,7 +121,7 @@ describe Rack::Prerender do
|
|
|
113
121
|
stub_request(:get, @prerender.build_api_url(request)).to_return(:body => "<html></html>")
|
|
114
122
|
response = Rack::Prerender.new(@app, blacklist: ['^/search', '/help']).call(request)
|
|
115
123
|
|
|
116
|
-
assert_equal ["<html></html>"], response[2]
|
|
124
|
+
assert_equal ["<html></html>"], response[2]
|
|
117
125
|
end
|
|
118
126
|
|
|
119
127
|
|
|
@@ -130,7 +138,7 @@ describe Rack::Prerender do
|
|
|
130
138
|
stub_request(:get, @prerender.build_api_url(request)).to_return(:body => "<html></html>")
|
|
131
139
|
response = Rack::Prerender.new(@app, blacklist: ['^/search', '/help']).call(request)
|
|
132
140
|
|
|
133
|
-
assert_equal ["<html></html>"], response[2]
|
|
141
|
+
assert_equal ["<html></html>"], response[2]
|
|
134
142
|
end
|
|
135
143
|
|
|
136
144
|
|
|
@@ -138,7 +146,7 @@ describe Rack::Prerender do
|
|
|
138
146
|
request = Rack::MockRequest.env_for "/", "HTTP_USER_AGENT" => bot
|
|
139
147
|
response = Rack::Prerender.new(@app, before_render: Proc.new do |env| '<html>cached</html>' end).call(request)
|
|
140
148
|
|
|
141
|
-
assert_equal ["<html>cached</html>"], response[2]
|
|
149
|
+
assert_equal ["<html>cached</html>"], response[2]
|
|
142
150
|
end
|
|
143
151
|
|
|
144
152
|
|
|
@@ -146,9 +154,9 @@ describe Rack::Prerender do
|
|
|
146
154
|
request = Rack::MockRequest.env_for "/", "HTTP_USER_AGENT" => bot
|
|
147
155
|
response = Rack::Prerender.new(@app, before_render: Proc.new do |env| Rack::Response.new('<html>cached2</html>', 200, { 'test' => 'test2Header'}) end).call(request)
|
|
148
156
|
|
|
149
|
-
assert_equal ["<html>cached2</html>"], response[2]
|
|
150
|
-
assert_equal response[
|
|
151
|
-
assert_equal( { 'test' => 'test2Header'
|
|
157
|
+
assert_equal ["<html>cached2</html>"], response[2]
|
|
158
|
+
assert_equal response[0], 200
|
|
159
|
+
assert_equal( { 'test' => 'test2Header'}, response[1] )
|
|
152
160
|
end
|
|
153
161
|
|
|
154
162
|
|
metadata
CHANGED
|
@@ -1,83 +1,83 @@
|
|
|
1
1
|
--- !ruby/object:Gem::Specification
|
|
2
2
|
name: prerender_rails
|
|
3
3
|
version: !ruby/object:Gem::Version
|
|
4
|
-
version: 1.
|
|
4
|
+
version: 1.7.0
|
|
5
5
|
platform: ruby
|
|
6
6
|
authors:
|
|
7
7
|
- Todd Hooper
|
|
8
8
|
autorequire:
|
|
9
9
|
bindir: bin
|
|
10
10
|
cert_chain: []
|
|
11
|
-
date:
|
|
11
|
+
date: 2020-06-03 00:00:00.000000000 Z
|
|
12
12
|
dependencies:
|
|
13
13
|
- !ruby/object:Gem::Dependency
|
|
14
14
|
name: rack
|
|
15
15
|
requirement: !ruby/object:Gem::Requirement
|
|
16
16
|
requirements:
|
|
17
|
-
- -
|
|
17
|
+
- - "~>"
|
|
18
18
|
- !ruby/object:Gem::Version
|
|
19
|
-
version:
|
|
19
|
+
version: 2.2.2
|
|
20
20
|
type: :runtime
|
|
21
21
|
prerelease: false
|
|
22
22
|
version_requirements: !ruby/object:Gem::Requirement
|
|
23
23
|
requirements:
|
|
24
|
-
- -
|
|
24
|
+
- - "~>"
|
|
25
25
|
- !ruby/object:Gem::Version
|
|
26
|
-
version:
|
|
26
|
+
version: 2.2.2
|
|
27
27
|
- !ruby/object:Gem::Dependency
|
|
28
28
|
name: activesupport
|
|
29
29
|
requirement: !ruby/object:Gem::Requirement
|
|
30
30
|
requirements:
|
|
31
|
-
- -
|
|
31
|
+
- - ">="
|
|
32
32
|
- !ruby/object:Gem::Version
|
|
33
33
|
version: '0'
|
|
34
34
|
type: :runtime
|
|
35
35
|
prerelease: false
|
|
36
36
|
version_requirements: !ruby/object:Gem::Requirement
|
|
37
37
|
requirements:
|
|
38
|
-
- -
|
|
38
|
+
- - ">="
|
|
39
39
|
- !ruby/object:Gem::Version
|
|
40
40
|
version: '0'
|
|
41
41
|
- !ruby/object:Gem::Dependency
|
|
42
42
|
name: bundler
|
|
43
43
|
requirement: !ruby/object:Gem::Requirement
|
|
44
44
|
requirements:
|
|
45
|
-
- - ~>
|
|
45
|
+
- - "~>"
|
|
46
46
|
- !ruby/object:Gem::Version
|
|
47
|
-
version:
|
|
47
|
+
version: 2.1.2
|
|
48
48
|
type: :development
|
|
49
49
|
prerelease: false
|
|
50
50
|
version_requirements: !ruby/object:Gem::Requirement
|
|
51
51
|
requirements:
|
|
52
|
-
- - ~>
|
|
52
|
+
- - "~>"
|
|
53
53
|
- !ruby/object:Gem::Version
|
|
54
|
-
version:
|
|
54
|
+
version: 2.1.2
|
|
55
55
|
- !ruby/object:Gem::Dependency
|
|
56
56
|
name: rake
|
|
57
57
|
requirement: !ruby/object:Gem::Requirement
|
|
58
58
|
requirements:
|
|
59
|
-
- -
|
|
59
|
+
- - ">="
|
|
60
60
|
- !ruby/object:Gem::Version
|
|
61
61
|
version: '0'
|
|
62
62
|
type: :development
|
|
63
63
|
prerelease: false
|
|
64
64
|
version_requirements: !ruby/object:Gem::Requirement
|
|
65
65
|
requirements:
|
|
66
|
-
- -
|
|
66
|
+
- - ">="
|
|
67
67
|
- !ruby/object:Gem::Version
|
|
68
68
|
version: '0'
|
|
69
69
|
- !ruby/object:Gem::Dependency
|
|
70
70
|
name: webmock
|
|
71
71
|
requirement: !ruby/object:Gem::Requirement
|
|
72
72
|
requirements:
|
|
73
|
-
- -
|
|
73
|
+
- - ">="
|
|
74
74
|
- !ruby/object:Gem::Version
|
|
75
75
|
version: '0'
|
|
76
76
|
type: :development
|
|
77
77
|
prerelease: false
|
|
78
78
|
version_requirements: !ruby/object:Gem::Requirement
|
|
79
79
|
requirements:
|
|
80
|
-
- -
|
|
80
|
+
- - ">="
|
|
81
81
|
- !ruby/object:Gem::Version
|
|
82
82
|
version: '0'
|
|
83
83
|
description: Rails middleware to prerender your javascript heavy pages on the fly
|
|
@@ -88,8 +88,8 @@ executables: []
|
|
|
88
88
|
extensions: []
|
|
89
89
|
extra_rdoc_files: []
|
|
90
90
|
files:
|
|
91
|
-
- .gitignore
|
|
92
|
-
- .travis.yml
|
|
91
|
+
- ".gitignore"
|
|
92
|
+
- ".travis.yml"
|
|
93
93
|
- Gemfile
|
|
94
94
|
- LICENSE.txt
|
|
95
95
|
- README.md
|
|
@@ -108,17 +108,16 @@ require_paths:
|
|
|
108
108
|
- lib
|
|
109
109
|
required_ruby_version: !ruby/object:Gem::Requirement
|
|
110
110
|
requirements:
|
|
111
|
-
- -
|
|
111
|
+
- - ">="
|
|
112
112
|
- !ruby/object:Gem::Version
|
|
113
113
|
version: '0'
|
|
114
114
|
required_rubygems_version: !ruby/object:Gem::Requirement
|
|
115
115
|
requirements:
|
|
116
|
-
- -
|
|
116
|
+
- - ">="
|
|
117
117
|
- !ruby/object:Gem::Version
|
|
118
118
|
version: '0'
|
|
119
119
|
requirements: []
|
|
120
|
-
|
|
121
|
-
rubygems_version: 2.0.14.1
|
|
120
|
+
rubygems_version: 3.1.2
|
|
122
121
|
signing_key:
|
|
123
122
|
specification_version: 4
|
|
124
123
|
summary: Prerender your backbone/angular/javascript rendered application on the fly
|