Hi, @reimertz and @icapulet. If you have an example domain I would be happy to describe this issue in more detail. For this example, I’m using testmysite.io as an example.
If I make a request for just https://testmysite.io/ with the default user-agent that curl sends, then the page is not prerendered:
$ curl -vs https://testmysite.io/
* Trying 165.227.0.164...
* TCP_NODELAY set
* Connected to testmysite.io (165.227.0.164) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=testmysite.io
* start date: Nov 11 12:00:26 2020 GMT
* expire date: Feb 9 12:00:26 2021 GMT
* subjectAltName: host "testmysite.io" matched cert's "testmysite.io"
* issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fe56400f600)
> GET / HTTP/2
> Host: testmysite.io
> User-Agent: curl/7.64.1
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 150)!
< HTTP/2 200
< cache-control: public, max-age=0, must-revalidate
< content-length: 1360
< content-type: text/html; charset=UTF-8
< date: Fri, 20 Nov 2020 14:34:51 GMT
< etag: "7a447848c1659e5d69c18599458b16db-ssl"
< strict-transport-security: max-age=31536000
< age: 400499
< server: Netlify
< x-nf-request-id: 52faad11-20f7-456f-ac92-d9037f81a3b0-2596346
<
<!DOCTYPE html>
<html lang="en">
<head>
<meta charset="UTF-8" />
<meta name="viewport" content="width=device-width, initial-scale=1" />
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<title>Website Speed Test Tool - Testmysite.io by Netlify</title>
<meta name="description" content="Is your site as fast and secure as it could be? Testmysite.io rates your site based on load times, performance, and security settings. Get your website's speed test results!">
<meta name="robots" content="index, follow">
<link rel="shortcut icon" href="favicon.ico">
<link rel="stylesheet" href="/styles.css">
<script> window.prerenderReady = false; </script>
</head>
<body>
<div id="app" class="container"></div>
<script src="/app.bundle.js"></script>
<div class="footer">
<a href="https://www.netlify.com/">
<img src="https://www.netlify.com/img/global/badges/netlify-color-bg.svg" />
</a>
</div>
<script>
(function(i,s,o,g,r,a,m){i['GoogleAnalyticsObject']=r;i[r]=i[r]||function(){
(i[r].q=i[r].q||[]).push(arguments)},i[r].l=1*new Date();a=s.createElement(o),
m=s.getElementsByTagName(o)[0];a.async=1;a.src=g;m.parentNode.insertBefore(a,m)
})(window,document,'script','https://www.google-analytics.com/analytics.js','ga');
ga('create', 'UA-42258181-10', 'auto');
ga('send', 'pageview');
</script></body>
</html>
* Connection #0 to host testmysite.io left intact
* Closing connection 0
If I change the user-agent to “Twitterbot”, it is prerendered:
$ curl -vs -A Twitterbot https://testmysite.io/
* Trying 165.227.12.111...
* TCP_NODELAY set
* Connected to testmysite.io (165.227.12.111) port 443 (#0)
* ALPN, offering h2
* ALPN, offering http/1.1
* successfully set certificate verify locations:
* CAfile: /etc/ssl/cert.pem
CApath: none
* TLSv1.2 (OUT), TLS handshake, Client hello (1):
* TLSv1.2 (IN), TLS handshake, Server hello (2):
* TLSv1.2 (IN), TLS handshake, Certificate (11):
* TLSv1.2 (IN), TLS handshake, Server key exchange (12):
* TLSv1.2 (IN), TLS handshake, Server finished (14):
* TLSv1.2 (OUT), TLS handshake, Client key exchange (16):
* TLSv1.2 (OUT), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (OUT), TLS handshake, Finished (20):
* TLSv1.2 (IN), TLS change cipher, Change cipher spec (1):
* TLSv1.2 (IN), TLS handshake, Finished (20):
* SSL connection using TLSv1.2 / ECDHE-RSA-AES256-GCM-SHA384
* ALPN, server accepted to use h2
* Server certificate:
* subject: CN=testmysite.io
* start date: Nov 11 12:00:26 2020 GMT
* expire date: Feb 9 12:00:26 2021 GMT
* subjectAltName: host "testmysite.io" matched cert's "testmysite.io"
* issuer: C=US; O=Let's Encrypt; CN=Let's Encrypt Authority X3
* SSL certificate verify ok.
* Using HTTP2, server supports multi-use
* Connection state changed (HTTP/2 confirmed)
* Copying HTTP/2 data in stream buffer to connection buffer after upgrade: len=0
* Using Stream ID: 1 (easy handle 0x7fa1c880f600)
> GET / HTTP/2
> Host: testmysite.io
> User-Agent: Twitterbot
> Accept: */*
>
* Connection state changed (MAX_CONCURRENT_STREAMS == 150)!
< HTTP/2 200
< content-type: text/html;charset=UTF-8
< date: Wed, 25 Nov 2020 05:51:02 GMT
< etag: W/"772-/ApNc54+OzHvjF6ziSherETS14E"
< content-length: 1906
< age: 0
< server: Netlify
< vary: Accept-Encoding
< x-nf-request-id: ea4a4aee-9753-4a46-9073-05e2233caa8d-5794462
<
<!DOCTYPE html><html lang="en"><head>
<meta charset="UTF-8">
<meta name="viewport" content="width=device-width, initial-scale=1">
<meta http-equiv="X-UA-Compatible" content="IE=edge">
<title>Website Speed Test Tool - Testmysite.io by Netlify</title>
<meta name="description" content="Is your site as fast and secure as it could be? Testmysite.io rates your site based on load times, performance, and security settings. Get your website's speed test results!">
<meta name="robots" content="index, follow">
<link rel="shortcut icon" href="favicon.ico">
<link rel="stylesheet" href="/styles.css">
</head>
<body>
<div id="app" class="container"><div data-reactroot="" class="app is-step-0"><form class="search"><div class="search__title"><img class="pen-arrow left" src="/b22715183c068b0120ae96598b621657.svg"><!-- react-text: 5 -->Test Your Site Here<!-- /react-text --><img class="pen-arrow right" src="/b22715183c068b0120ae96598b621657.svg"></div><input type="text" class="search__input" placeholder="Enter URL here..." value=""><div class="search__hook">Is your site as fast and secure as it could be?</div><div class="search__intro"><p class="intro-text-paragraph"><strong>Testmysite.io rates your site based on its initial global load times and its security settings.</strong></p><p class="intro-text-paragraph">Even a very optimized site made on a legacy stack like Wordpress can often become several seconds faster by being built in a modern way and served via a service such as Netlify. By using HTTPS and HTTP2 you can improve both speed and search ranking.</p></div></form><div class="app__steps"><span class="app__steps__transitions"><div class="app__steps__item"></div></span></div></div></div>
<div class="footer">
<a href="https://www.netlify.com/">
<img src="https://www.netlify.com/img/global/badges/netlify-color-bg.svg">
</a>
</div>
* Connection #0 to host testmysite.io left intact
</body></html>* Closing connection 0
However, if I make the same two requests (with and without the Twitterbot user-agent) but this time without SSL (http:// instead of https://), then something different happens:
For the non-prerendered version (without the special user-agent), a 301 redirect occurs:
$ curl -vs http://testmysite.io/
* Trying 138.68.244.143...
* TCP_NODELAY set
* Connected to testmysite.io (138.68.244.143) port 80 (#0)
> GET / HTTP/1.1
> Host: testmysite.io
> User-Agent: curl/7.64.1
> Accept: */*
>
< HTTP/1.1 301 Moved Permanently
< Cache-Control: public, max-age=0, must-revalidate
< Content-Length: 38
< Content-Type: text/plain
< Date: Mon, 23 Nov 2020 16:17:05 GMT
< Location: https://testmysite.io/
< Age: 135467
< Connection: keep-alive
< Server: Netlify
< X-NF-Request-ID: 227e1b02-1217-4b5c-a2a3-2a4e0152d036-22772393
<
Redirecting to https://testmysite.io/
* Connection #0 to host testmysite.io left intact
* Closing connection 0
However, if prerendering is enabled and a user-agent which triggers prerendering is used, then a 307 redirect happens and there is HTML returned:
$ curl -vs -A Twitterbot http://testmysite.io/
* Trying 167.172.221.254...
* TCP_NODELAY set
* Connected to testmysite.io (167.172.221.254) port 80 (#0)
> GET / HTTP/1.1
> Host: testmysite.io
> User-Agent: Twitterbot
> Accept: */*
>
< HTTP/1.1 307 Temporary Redirect
< Content-Length: 39
< Content-Type: text/html; charset=utf-8
< Date: Wed, 25 Nov 2020 05:56:05 GMT
< Etag: W/"27-ghawzGh2y9RPAcFY59/zgzzszUE"
< Location: https://testmysite.io/
< Age: 0
< Connection: keep-alive
< Server: Netlify
< Vary: Accept-Encoding
< X-NF-Request-ID: a76337e4-f65b-440c-bb78-270818e94b6a-8283499
<
* Connection #0 to host testmysite.io left intact
<html><head></head><body></body></html>* Closing connection 0
You can see the HTML is returned when a 301 redirect without HTML would be preferrable.
We have an issue filed to change this but the behavior hasn’t been changed at this time.
The only workaround for now is to not type in the apex domain alone and to manually include https:// when typing the URL. Similar, if these services would default to HTTPS (instead of HTTP only) when the apex domain is used alone, that would also resolve the issue.
If/when this issue is known to be resolved on our side we will post an update here to let you know.