RubyGems - prerender_rails - Versions diffs - 1.3.0 → 1.9.0 - Mend

prerender_rails 1.3.0 → 1.9.0

Files changed (8) hide show

checksums.yaml +5 -5
data/.gitignore +1 -0
data/.travis.yml +3 -1
data/README.md +11 -18
data/lib/prerender_rails.rb +54 -12
data/prerender_rails.gemspec +3 -3
data/test/lib/prerender_rails.rb +73 -11
metadata +21 -22

checksums.yaml CHANGED Viewed

@@ -1,7 +1,7 @@
 ---
-SHA1:
-  metadata.gz: 0dab2fb1c0fb1a232b5901dde71f986ca4000c65
-  data.tar.gz: 675b2dfbdede57ad5f448377aa964e304f5fea30
+SHA256:
+  metadata.gz: 31eaea5bcb5ca7912fe4702e55d5ad48834c5c4912026b3c7c5b80e8b67c2273
+  data.tar.gz: b93d84f1616b9338507cbd976dcf0b0e5e2b1e8676fb9543364f606bd8df0652
 SHA512:
-  metadata.gz: 748be26bea44cae70d7c0f5197f1e54aeede93c8951868e70d5acc7d6a620fb86deac649a8a71b4bd07c81e26ab73af9b79c09316e5362ec8a0d60afaa35bfc2
-  data.tar.gz: e34eb14973e15ec46bbbf9214aaad0bc688a8d9edc58b4148ec8828d2d447d55700bd7b0ebf7f8c8af56138340d1510a8d070bc7eba6379dd3dbc3b6e4293b5c
+  metadata.gz: ea3b84d1fa613248dace19ffee4b66ef4014b42da4c558db1e22663a5de8c7d3c8969abb84ac3ab18743155a5e5026e5b54e1b6a72f803c5917a64d98f6a7b77
+  data.tar.gz: d51a0feaa238dd3b88e87aa83a6388c96f06bb1d4313fa6ba2273a43e8c9b04250f5cd9523c9b5dbf9a0123eea87c9247ad3a99c3d80fb4b1a50e174488869fd

data/.gitignore CHANGED Viewed

@@ -15,3 +15,4 @@ spec/reports
 test/tmp
 test/version_tmp
 tmp
+.DS_Store

data/.travis.yml CHANGED Viewed

@@ -1,3 +1,5 @@
 language: ruby
 rvm:
-  - 2.0.0
+  - 2.5
+before_install:
+  - gem install bundler

data/README.md CHANGED Viewed

@@ -1,14 +1,11 @@
-Prerender Rails [![Stories in Ready](https://badge.waffle.io/prerender/prerender_rails.png?label=ready&title=Ready)](https://waffle.io/prerender/prerender_rails) [![Build Status](https://travis-ci.org/prerender/prerender_rails.png)](https://travis-ci.org/prerender/prerender_rails) [![Gem Version](https://badge.fury.io/rb/prerender_rails.png)](http://badge.fury.io/rb/prerender_rails)
+Prerender Rails [![Build Status](https://travis-ci.org/prerender/prerender_rails.png)](https://travis-ci.org/prerender/prerender_rails) [![Gem Version](https://badge.fury.io/rb/prerender_rails.png)](http://badge.fury.io/rb/prerender_rails)
 ===========================
-Google, Facebook, Twitter, Yahoo, and Bing are constantly trying to view your website... but they don't execute javascript. That's why we built Prerender. Prerender is perfect for AngularJS SEO, BackboneJS SEO, EmberJS SEO, and any other javascript framework.
+Google, Facebook, Twitter, and Bing are constantly trying to view your website... but Google is the only crawler that executes a meaningful amount of JavaScript and Google even admits that they can execute JavaScript weeks after actually crawling. Prerender allows you to serve the full HTML of your website back to Google and other crawlers so that they don't have to execute any JavaScript. [Google recommends using Prerender.io](https://developers.google.com/search/docs/guides/dynamic-rendering) to prevent indexation issues on sites with large amounts of JavaScript.
-This middleware intercepts requests to your Rails website from crawlers, and then makes a call to the (external) Prerender Service to get the static HTML instead of the javascript for that page.
+Prerender is perfect for Angular SEO, React SEO, Vue SEO, and any other JavaScript framework.
-Prerender adheres to google's `_escaped_fragment_` proposal, which we recommend you use. It's easy:
-- Just add &lt;meta name="fragment" content="!"> to the &lt;head> of all of your pages
-- If you use hash urls (#), change them to the hash-bang (#!)
-- That's it! Perfect SEO on javascript pages.
+This middleware intercepts requests to your Node.js website from crawlers, and then makes a call to the (external) Prerender Service to get the static HTML instead of the JavaScript for that page. That HTML is then returned to the crawler.
 `Note` Make sure you have more than one webserver thread/process running because the prerender service will make a request to your server to render the HTML.
@@ -22,7 +19,7 @@ And in `config/environment/production.rb`, add this line:
 	config.middleware.use Rack::Prerender
 ```
-or if you have an account on [prerender.io](http://prerender.io) and want to use your token:
+or if you have an account on [prerender.io](https://prerender.io/) and want to use your token:
 ```ruby
 	config.middleware.use Rack::Prerender, prerender_token: 'YOUR_TOKEN'
@@ -32,19 +29,15 @@ or if you have an account on [prerender.io](http://prerender.io) and want to use
 ## Testing
-If your URLs use a hash-bang:
+When testing make sure you're not using a single threaded application server like default WEBrick one, use Puma or Unicorn to prevent a deadlock when the Prerender Service needs to render your HTML on the fly.
-	If you want to see `http://localhost:5000/#!/profiles/1234`
-	Then go to `http://localhost:5000/?_escaped_fragment_=/profiles/1234`
+The best way to test the prerendered page is to [set the User Agent of your browser to Googlebot's user agent](https://developers.google.com/web/tools/chrome-devtools/device-mode/override-user-agent) and visit your URL directly. If you View Source on that URL, you should see the static HTML version of the page with the `<script>` tags removed from the page. If you still see `<script>` tags then that means the middleware isn't set up properly yet.
-If your URLs use push-state:
-	If you want to see `http://localhost:5000/profiles/1234`
-	Then go to `http://localhost:5000/profiles/1234?_escaped_fragment_=`
+`Note` If you're testing locally, you'll need to run the [prerender server](https://github.com/prerender/prerender) locally so that it has access to your server.
 ## How it works
 1. The middleware checks to make sure we should show a prerendered page
-	1. The middleware checks if the request is from a crawler (`_escaped_fragment_` or agent string)
+	1. The middleware checks if the request is from a crawler by checking the user agent string against a default list of crawler user agents
 	2. The middleware checks to make sure we aren't requesting a resource (js, css, etc...)
 	3. (optional) The middleware checks to make sure the url is in the whitelist
 	4. (optional) The middleware checks to make sure the url isn't in the blacklist
@@ -135,7 +128,7 @@ config.middleware.use Rack::Prerender,
 ## Using your own prerender service
-We host a Prerender server at [prerender.io](http://prerender.io) so that you can work on more important things, but if you've deployed the prerender service on your own... set the `PRERENDER_SERVICE_URL` environment variable so that this middleware points there instead. Otherwise, it will default to the service already deployed by [prerender.io](http://prerender.io).
+We host a Prerender server at [prerender.io](https://prerender.io/) so that you can work on more important things, but if you've deployed the prerender service on your own... set the `PRERENDER_SERVICE_URL` environment variable so that this middleware points there instead. Otherwise, it will default to the service already deployed by [prerender.io](https://prerender.io/).
 	$ export PRERENDER_SERVICE_URL=<new url>
@@ -146,7 +139,7 @@ Or on heroku:
 As an alternative, you can pass `prerender_service_url` in the options object during initialization of the middleware
 ``` ruby
-config.middleware.use Rack::Prerender, prerender_service_url: '&lt;new url>'
+config.middleware.use Rack::Prerender, prerender_service_url: '<new url>'
 ```
 ## License

data/lib/prerender_rails.rb CHANGED Viewed

@@ -4,13 +4,24 @@ module Rack
     require 'active_support'
     def initialize(app, options={})
-      # googlebot, yahoo, and bingbot are not in this list because
-      # we support _escaped_fragment_ and want to ensure people aren't
-      # penalized for cloaking.
+      @connection_header_values = [
+        'close',
+        'keep-alive'
+      ].freeze
+      @hop_by_hop_headers = [
+        'Connection',
+        'Keep-Alive',
+        'Public',
+        'Proxy-Authenticate',
+        'Transfer-Encoding',
+        'Upgrade'
+      ].freeze
       @crawler_user_agents = [
-        # 'googlebot',
-        # 'yahoo',
-        # 'bingbot',
+        'googlebot',
+        'Google-InspectionTool',
+        'yahoo',
+        'bingbot',
         'baiduspider',
         'facebookexternalhit',
         'twitterbot',
@@ -21,13 +32,25 @@ module Rack
         'quora link preview',
         'showyoubot',
         'outbrain',
-        'pinterest',
+        'pinterest/0.',
         'developers.google.com/+/web/snippet',
+        'www.google.com/webmasters/tools/richsnippets',
         'slackbot',
         'vkShare',
         'W3C_Validator',
         'redditbot',
-        'Applebot'
+        'Applebot',
+        'WhatsApp',
+        'flipboard',
+        'tumblr',
+        'bitlybot',
+        'SkypeUriPreview',
+        'nuzzel',
+        'Discordbot',
+        'Google Page Speed',
+        'Qwantify',
+        'Chrome-Lighthouse',
+        'TelegramBot'
       ]
       @extensions_to_ignore = [
@@ -104,7 +127,8 @@ module Rack
     def should_show_prerendered_page(env)
       user_agent = env['HTTP_USER_AGENT']
-      buffer_agent = env['X-BUFFERBOT']
+      buffer_agent = env['HTTP_X_BUFFERBOT']
+      prerender_agent = env['HTTP_X_PRERENDER']
       is_requesting_prerendered_page = false
       return false if !user_agent
@@ -120,13 +144,16 @@ module Rack
       #if it is BufferBot...show prerendered page
       is_requesting_prerendered_page = true if buffer_agent
+      #if it is Prerender...don't show prerendered page
+      is_requesting_prerendered_page = false if prerender_agent
       #if it is a bot and is requesting a resource...dont prerender
       return false if @extensions_to_ignore.any? { |extension| request.fullpath.include? extension }
       #if it is a bot and not requesting a resource and is not whitelisted...dont prerender
       return false if @options[:whitelist].is_a?(Array) && @options[:whitelist].all? { |whitelisted| !Regexp.new(whitelisted).match(request.fullpath) }
-      #if it is a bot and not requesting a resource and is not blacklisted(url or referer)...dont prerender
+      #if it is a bot and not requesting a resource and is blacklisted(url or referer)...dont prerender
       if @options[:blacklist].is_a?(Array) && @options[:blacklist].any? { |blacklisted|
           blacklistedUrl = false
           blacklistedReferer = false
@@ -155,12 +182,27 @@ module Rack
         headers['X-Prerender-Token'] = @options[:prerender_token] if @options[:prerender_token]
         req = Net::HTTP::Get.new(url.request_uri, headers)
         req.basic_auth(ENV['PRERENDER_USERNAME'], ENV['PRERENDER_PASSWORD']) if @options[:basic_auth]
-        response = Net::HTTP.start(url.host, url.port) { |http| http.request(req) }
+        http = Net::HTTP.new(url.host, url.port)
+        http.use_ssl = true if url.scheme == 'https'
+        response = http.request(req)
         if response['Content-Encoding'] == 'gzip'
           response.body = ActiveSupport::Gzip.decompress(response.body)
           response['Content-Length'] = response.body.length
           response.delete('Content-Encoding')
         end
+        hop_by_hop_headers = @hop_by_hop_headers
+        connection = response['Connection']
+        if connection
+          connection_hop_by_hop_headers = connection.split(',').
+                                            map(&:strip).
+                                            map(&:downcase).
+                                            difference(@connection_header_values)
+          hop_by_hop_headers = connection_hop_by_hop_headers.
+                                 concat(hop_by_hop_headers)
+        end
+        hop_by_hop_headers.each { |h| response.delete(h) }
         response
       rescue
         nil
@@ -199,7 +241,7 @@ module Rack
     def build_rack_response_from_prerender(prerendered_response)
-      response = Rack::Response.new(prerendered_response.body, prerendered_response.code, prerendered_response.header)
+      response = Rack::Response.new(prerendered_response.body, prerendered_response.code, prerendered_response)
       @options[:build_rack_response_from_prerender].call(response, prerendered_response) if @options[:build_rack_response_from_prerender]

data/prerender_rails.gemspec CHANGED Viewed

@@ -2,7 +2,7 @@
 Gem::Specification.new do |spec|
   spec.name          = "prerender_rails"
-  spec.version       = "1.3.0"
+  spec.version       = "1.9.0"
   spec.authors       = ["Todd Hooper"]
   spec.email         = ["todd@prerender.io"]
   spec.description   = %q{Rails middleware to prerender your javascript heavy pages on the fly by a phantomjs service}
@@ -15,10 +15,10 @@ Gem::Specification.new do |spec|
   spec.test_files    = spec.files.grep(%r{^(test|spec|features)/})
   spec.require_paths = ["lib"]
-  spec.add_dependency 'rack', '>= 0'
+  spec.add_dependency 'rack', '~> 2.2.2'
   spec.add_dependency 'activesupport', '>= 0'
-  spec.add_development_dependency "bundler", "~> 1.3"
+  spec.add_development_dependency "bundler", "~> 2.1.2"
   spec.add_development_dependency "rake"
   spec.add_development_dependency "webmock"
 end

data/test/lib/prerender_rails.rb CHANGED Viewed

@@ -19,9 +19,9 @@ describe Rack::Prerender do
     stub_request(:get, @prerender.build_api_url(request)).with(:headers => { 'User-Agent' => bot }).to_return(:body => "<html></html>", :status => 301, :headers => { 'Location' => 'http://google.com'})
     response = Rack::Prerender.new(@app).call(request)
-    assert_equal response[2].body, ["<html></html>"]
-    assert_equal response[2].status, 301
-    assert_equal( { 'location' => 'http://google.com', 'Content-Length' => '13'}, response[2].headers )
+    assert_equal response[2], ["<html></html>"]
+    assert_equal response[0], 301
+    assert_equal( { 'location' => 'http://google.com'}, response[1] )
   end
@@ -30,7 +30,7 @@ describe Rack::Prerender do
     stub_request(:get, @prerender.build_api_url(request)).with(:headers => { 'User-Agent' => user }).to_return(:body => "<html></html>")
     response = Rack::Prerender.new(@app).call(request)
-    assert_equal ["<html></html>"], response[2].body
+    assert_equal ["<html></html>"], response[2]
   end
@@ -58,6 +58,14 @@ describe Rack::Prerender do
   end
+  it "should continue to app routes if contains X-Prerender header" do
+    request = Rack::MockRequest.env_for "/path?_escaped_fragment_=", "HTTP_USER_AGENT" => user, "HTTP_X_PRERENDER" => "1"
+    response = Rack::Prerender.new(@app).call(request)
+    assert_equal "", response[2]
+  end
   it "should continue to app routes if user is a bot, but the bot is requesting a resource file" do
     request = Rack::MockRequest.env_for "/main.js?anyQueryParam=true", "HTTP_USER_AGENT" => bot
     response = Rack::Prerender.new(@app).call(request)
@@ -74,12 +82,23 @@ describe Rack::Prerender do
   end
+  it "should set use_ssl to true for https prerender_service_url" do
+    @prerender = Rack::Prerender.new(@app, prerender_service_url: 'https://service.prerender.io/')
+    request = Rack::MockRequest.env_for "/search/things/123/page?_escaped_fragment_=", "HTTP_USER_AGENT" => bot
+    stub_request(:get, @prerender.build_api_url(request)).to_return(:body => "<html></html>")
+    response = @prerender.call(request)
+    assert_equal ["<html></html>"], response[2]
+  end
   it "should return a prerendered response if the url is part of the regex specific whitelist" do
     request = Rack::MockRequest.env_for "/search/things/123/page?_escaped_fragment_=", "HTTP_USER_AGENT" => bot
     stub_request(:get, @prerender.build_api_url(request)).to_return(:body => "<html></html>")
     response = Rack::Prerender.new(@app, whitelist: ['^/search.*page', '/help']).call(request)
-    assert_equal ["<html></html>"], response[2].body
+    assert_equal ["<html></html>"], response[2]
   end
@@ -102,7 +121,7 @@ describe Rack::Prerender do
     stub_request(:get, @prerender.build_api_url(request)).to_return(:body => "<html></html>")
     response = Rack::Prerender.new(@app, blacklist: ['^/search', '/help']).call(request)
-    assert_equal ["<html></html>"], response[2].body
+    assert_equal ["<html></html>"], response[2]
   end
@@ -119,7 +138,7 @@ describe Rack::Prerender do
     stub_request(:get, @prerender.build_api_url(request)).to_return(:body => "<html></html>")
     response = Rack::Prerender.new(@app, blacklist: ['^/search', '/help']).call(request)
-    assert_equal ["<html></html>"], response[2].body
+    assert_equal ["<html></html>"], response[2]
   end
@@ -127,7 +146,7 @@ describe Rack::Prerender do
     request = Rack::MockRequest.env_for "/", "HTTP_USER_AGENT" => bot
     response = Rack::Prerender.new(@app, before_render: Proc.new do |env| '<html>cached</html>' end).call(request)
-    assert_equal ["<html>cached</html>"], response[2].body
+    assert_equal ["<html>cached</html>"], response[2]
   end
@@ -135,11 +154,47 @@ describe Rack::Prerender do
     request = Rack::MockRequest.env_for "/", "HTTP_USER_AGENT" => bot
     response = Rack::Prerender.new(@app, before_render: Proc.new do |env| Rack::Response.new('<html>cached2</html>', 200, { 'test' => 'test2Header'}) end).call(request)
-    assert_equal ["<html>cached2</html>"], response[2].body
-    assert_equal response[2].status, 200
-    assert_equal( { 'test' => 'test2Header', "Content-Length"=>"20"}, response[2].headers )
+    assert_equal ["<html>cached2</html>"], response[2]
+    assert_equal response[0], 200
+    assert_equal( { 'test' => 'test2Header'}, response[1] )
   end
+  it "should return a prerendered response stripped of hop-by-hop headers" do
+    request = Rack::MockRequest.env_for "/", "HTTP_USER_AGENT" => bot
+    stub_request(:get, @prerender.build_api_url(request)).
+      with(:headers => { 'User-Agent' => bot }).
+      to_return(:body => "<html></html>", :status => 401, :headers => {
+                  'Content-Type' => 'text/html',
+                  'Transfer-Encoding' => 'Chunked',
+                  'Connection' => 'Keep-Alive',
+                  'Keep-Alive' => 'timeout=5, max=100',
+                  'Public' => 'GET HEAD',
+                  'Proxy-Authenticate' => 'Basic',
+                  'X-Drop-Test' => 'ShouldAlwaysHappen',
+                  'Upgrade' => 'dummy'
+                })
+    response = Rack::Prerender.new(@app).call(request)
+    assert_equal response[2], ["<html></html>"]
+    assert_equal response[0], 401
+    assert_equal( { 'content-type' => 'text/html', 'x-drop-test' => 'ShouldAlwaysHappen'}, response[1] )
+  end
+it "should return a prerendered response stripped of custom-defined hop-by-hop headers" do
+    request = Rack::MockRequest.env_for "/", "HTTP_USER_AGENT" => bot
+    stub_request(:get, @prerender.build_api_url(request)).
+      with(:headers => { 'User-Agent' => bot }).
+      to_return(:body => "<html></html>", :status => 200, :headers => {
+                  'Content-Type' => 'text/html',
+                  'Connection' => 'Close, X-Drop-Test',
+                  'X-Drop-Test' => 'ShouldNeverHappen'
+                })
+    response = Rack::Prerender.new(@app).call(request)
+    assert_equal response[2], ["<html></html>"]
+    assert_equal response[0], 200
+    assert_equal( { 'content-type' => 'text/html' }, response[1] )
+  end
   describe '#buildApiUrl' do
     it "should build the correct api url with the default url" do
@@ -164,6 +219,13 @@ describe Rack::Prerender do
     end
+    it "should build the correct https api url with an initialization variable url" do
+      @prerender = Rack::Prerender.new(@app, prerender_service_url: 'https://prerenderurl.com')
+      request = Rack::MockRequest.env_for "https://google.com/search?q=javascript"
+      assert_equal 'https://prerenderurl.com/https://google.com/search?q=javascript', @prerender.build_api_url(request)
+    end
     # Check CF-Visitor header in order to Work behind CloudFlare with Flexible SSL (https://support.cloudflare.com/hc/en-us/articles/200170536)
     it "should build the correct api url for the Cloudflare Flexible SSL support" do
       request = Rack::MockRequest.env_for "http://google.com/search?q=javascript", { 'CF-VISITOR' => '"scheme":"https"'}

metadata CHANGED Viewed

@@ -1,83 +1,83 @@
 --- !ruby/object:Gem::Specification
 name: prerender_rails
 version: !ruby/object:Gem::Version
-  version: 1.3.0
+  version: 1.9.0
 platform: ruby
 authors:
 - Todd Hooper
 autorequire:
 bindir: bin
 cert_chain: []
-date: 2016-02-17 00:00:00.000000000 Z
+date: 2023-05-22 00:00:00.000000000 Z
 dependencies:
 - !ruby/object:Gem::Dependency
   name: rack
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - '>='
+    - - "~>"
       - !ruby/object:Gem::Version
-        version: '0'
+        version: 2.2.2
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - '>='
+    - - "~>"
       - !ruby/object:Gem::Version
-        version: '0'
+        version: 2.2.2
 - !ruby/object:Gem::Dependency
   name: activesupport
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - '>='
+    - - ">="
       - !ruby/object:Gem::Version
         version: '0'
   type: :runtime
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - '>='
+    - - ">="
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
   name: bundler
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - ~>
+    - - "~>"
       - !ruby/object:Gem::Version
-        version: '1.3'
+        version: 2.1.2
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - ~>
+    - - "~>"
       - !ruby/object:Gem::Version
-        version: '1.3'
+        version: 2.1.2
 - !ruby/object:Gem::Dependency
   name: rake
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - '>='
+    - - ">="
       - !ruby/object:Gem::Version
         version: '0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - '>='
+    - - ">="
       - !ruby/object:Gem::Version
         version: '0'
 - !ruby/object:Gem::Dependency
   name: webmock
   requirement: !ruby/object:Gem::Requirement
     requirements:
-    - - '>='
+    - - ">="
       - !ruby/object:Gem::Version
         version: '0'
   type: :development
   prerelease: false
   version_requirements: !ruby/object:Gem::Requirement
     requirements:
-    - - '>='
+    - - ">="
       - !ruby/object:Gem::Version
         version: '0'
 description: Rails middleware to prerender your javascript heavy pages on the fly
@@ -88,8 +88,8 @@ executables: []
 extensions: []
 extra_rdoc_files: []
 files:
-- .gitignore
-- .travis.yml
+- ".gitignore"
+- ".travis.yml"
 - Gemfile
 - LICENSE.txt
 - README.md
@@ -108,17 +108,16 @@ require_paths:
 - lib
 required_ruby_version: !ruby/object:Gem::Requirement
   requirements:
-  - - '>='
+  - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 required_rubygems_version: !ruby/object:Gem::Requirement
   requirements:
-  - - '>='
+  - - ">="
     - !ruby/object:Gem::Version
       version: '0'
 requirements: []
-rubyforge_project:
-rubygems_version: 2.0.14
+rubygems_version: 3.0.3.1
 signing_key:
 specification_version: 4
 summary: Prerender your backbone/angular/javascript rendered application on the fly