Turning on pre-rendering multiplies bandwidth 20x times! We measured bandwith with and without pre-rendering turned on. You can see the last test on the chart. The bandwidth got back to normal instantly after turning off pre-rendering in Netlify. Any ideas how to fix this?
1 Like
Hi, @maintopbiz. Maybe it would help to explain this to talk about the difference of what happens when a “bot” requests your page with and without prerendering.
The example scenario I’m going to use is that someone posts a link to example.com on Twitter and a Twitter user looks at that Tweet. (Also, I’m pretending that example.com is hosted at Netlify but it isn’t. Let’s pretend it does for the sake of this example though.)
Without Prerendering
If there prerendering is not enabled, the following happens:
- Twitter’s “bots”, not the end user’s browser, make an HTTP request to
http://example.com/(which might 301 redirect a few times depending on what domain name is the “primary custom domain” for the site). - Once the 200 response is returned, Twitter will look for open graph (OG) meta tags in the HTML returned and display a “card” based on those tags.
Note, the only site traffic generated for this request isare the HTTP requests for only the index.html file itself. For the 301s (if there are any) there are only headers returned, no HTML. So, the majority (like 99%) of the bandwidth used will be for a single 200 response of a single HTML file.
To summarize “without prerendering”, viewing the site’s Twitter card on Twitter only uses the bandwidth of a single HTML file being sent (plus the headers for any 301s before that which will only be a few hundred bytes at most).
With Prerendering
If prerendering is enabled the first step is identical but, after that, things differ greatly:
- Twitter’s “bots”, not the end user’s browser, make an HTTP request to
http://example.com/(which might 301 redirect a few times depending on what domain name is the “primary custom domain” for the site). - The routing service will see the “Twitterbot” user agent header and activate prerendering for this page.
- The prerendering service create a Chrome headless browser instance. That browser instance will load index.html just like a real browser would. It will then request all the resources on the page. This includes any images, CSS, javascript, videos, audio files, or other media on the page. It then runs the site javascript which might load other files. Each additional file/URL loaded are real HTTP requests and all of those assets being loaded count towards the bandwidth.
With the prerendering enabled, the bot traffic is no longer making one to three HTTP requests and only one of which will contribute to bandwidth. The bot traffic is now causing all the other media and supporting files on the page to be loaded.
To summarize “with prerendering”, the bandwidth used will be the bandwidth of all the site assets loaded in the Chrome headless browser inside the prerendering service. There are likely many files load and all of them will count towards the bandwidth.
This is almost certainly going to be orders of magnitude more traffic than non-prerendered traffic.
If a large portion of your traffic is comes from these “cards” in social media sites like Facebook, Twitter, etc. then, yes, enabling prerendering will, by its very definition, increase the bandwidth your site is using.
So, how do you prevent this? The answer depends on your site’s javascript.
Ideally, the site javascript would be optimized for the prerendering and do as little as possible to create the required OG tags for the card to be shown. In other words, the javascript would send an early signal to the prerendering service that the page is “done” before loading things a real person would need to view the page. This would cause the prerender to exit before loading the other assets and this would save some bandwidth.
So, how does the site javascript tell the prerendering service it is ready?
Our prerendering service is a fork of prerender.io’s public repo (and you can also use prerender.io directly with your Netlify site, using their service instead of ours if you prefer). Their documentation about how to tell the prerender service that the page is ready can be found here:
https://docs.prerender.io/article/11-best-practices
Quoting that page:
Tell us exactly when your pages are ready to be saved.
We try our very best to detect when a page is done loading by counting the number of requests in flight and saving the page when the number of requests in flight reaches zero (after a slight delay). You can tell us when your pages are ready so that we can be more accurate.
Put this in your HTML as early as you can (preferably the top of the section):
When we see window.prerenderReady set to false, we’ll wait until it’s set to true to save the HTML. Execute the following when your page will be ready (usually after ajax calls).
window.prerenderReady = true;
Those instructions apply to our prerendering as well.
If there are other questions about this, please let us know.
That seems to be a bug right there:
No images or videos should be downloaded during prerendering process! They are not needed for pre-rendering and they are disabled by default by prerender.io. See Disable loading images · Issue #175 · prerender/prerender · GitHub
1 Like
@maintopbiz Have you noticed higher traffic than usual?
We turned on prerendering aswell and now have about 13k requests/day coming from the prerenderer.
Previously we had about 500 requests/day from bots and 230GB/month used bandwidth (now at about 1TB/month).
Unfortunately we can’t use Netlify Analytics to see where the requests originally came from, since it doesn’t show user_agent metrics.
I was actually wondering myself why I suddenly used so much bandwidth for a small website, didn’t think about the pre-rendering I enabled at all.
This is a serious issue, when a website goes from 230 GB monthly bandwidth to 1TB, something is not working like it should, Google is perfectly capable of indexing a javascript site when you do things right, and it’s fast enough, and they don’t consume this kind of bandwidth by doing so, why would a prerender service, a service intended to make things faster and better, quadruple the bandwidth usage?
Netlify is doing something wrong here; 20x bandwidth consumption or 4x bandwidth consumption defeats the purpose of pre-rendering. Like mentioned above, images and videos are not needed, neither is CSS when you specifically only use pre-rendering for indexing.
1 Like
Hi, @maintopbiz, @randomname, and @freddy. There is a lot of finger pointing occurring here saying the the prerendering at Netlify is the cause of the issues. However, I haven’t seen any facts provided to back this up yet.
For example, if the site javascript loads additional content and that is the cause of the increased bandwidth then the root cause is the site’s javascript, not our prerendering service. The prerender only runs the javascript and it cannot control what the javascript is doing.
I’m the first to admit it is possible there is a bug but bug reports (AKA “issues”) require reproduction steps or other proof. I don’t have that information yet and I need it to file an issue.
If you have proof there is a bug, please share that information with our support team and we will be happy to examine that information. If we can validate your findings, we can get a bug filed to address this.
On the other hand, if no one has data to prove there is a bug, I’d be happy to assist with helping you prove that there is a bug so we can get an issue filed for it. Again, without proof, I cannot file a bug.
Would someone like to volunteer to have their site being the one used to research this? If so, what is the site subdomain at Netlify or the site’s API id?
You can post that information publicly or you can private message (PM) that to one of our support staff. I’ve confirmed that PMs are enabled for your community logins (all three). Please keep in mind that only one person can see the PM and this will likely mean a slower reply than posting the information publicly. Please feel free to reply to however you prefer though.
So which one is true? These 2 descriptions of how your prerendering works are conflicting:
A)
B)
The proof is kind of obvious when the resource usage goes from 3-4 GB to 40 GB. Google is not stopping the indexing just because you disable the pre-rendering. Google is not failing the indexing because you stop the prerender. Google works perfectly fine with javascript driven websites, mostly, but it depends on loading times and speed to index properly and not “cut off” mid-process. Which is why a prerender is a better alternative, to ensure availability to the search engine.
The prerender is not doing anything different then what Google is doing already, it’s the same content, same website, same images, same javascript, only difference is that Netlify is handling it before Google does. And as soon as you add Netlify as a middleman, the bandwidth resource jumps through the roof.
Correlation is not causation. It proves only that an increase occurred. It does not prove what caused the increase. As I’ve said before, the increase can be caused by things in your javascript or HTML and we don’t control that.
Most people do not report an increase bandwidth when prerendering is enabled.
Maybe it will help if I revisit the following:
It turns out that this is not all true. The prerendering service does not load images by default. The prerendering service does load your javascript files and does execute the code they contain just like a real browser would. Prerendering also loads CSS files referenced in the HTML if they are for the current domain and it loads any link tags with a rel="prefetch" attribute.
@randomname, you did contact me with a site to test but that site no longer has prerendering enabled. I did test it with the prerendering service run locally (which anyone can do and there are instructions in the open source repo at GitHub about this can be done).
Also, @randomname, I’ve noticed that the site you sent me includes a large number of links in the HTML with option of rel="prefetch". I did test your site with the prerendering service locally and I can confirm it loads all those files.
Again, I have tested and confirmed that the prerendering service does not load images. It does not load CSS from other domains. CSS from the current domain is loaded as are any “preload” links in the HTML.
If others are using prefetch links that would likely explain the increased bandwidth for your prerendering as well.
Note, you can run the prerendering service locally to test it and those instructions can be found here:
https://github.com/netlify/prerender
Note, the prerendering services uses basic auth by default and it can be disabled by deleting this line (quoted below) from server.js:
server.use(require("./lib/plugins/basicAuth"));
I’m going to summarize my understanding of what is known:
- Most people do not report an increase in bandwidth use when prerendering is enabled.
- Some people (three so far) are reporting they do see an increase in bandwidth use when prerendering is enabled.
- One site tested included numerous prefetch links and I can confirm links of that nature are loaded by the prerendering service.
If there are other questions about this, please let us know.
1 Like
The links you provided do not work. They require a login.
Hi, @maintopbiz. I have deleted the links you mentioned now. They were intended as a private comment for other members of the support team if they wanted to pick up troubleshooting where I left off. You were never supposed to see those links and, yes, they do require authentication to view.
There is still a comment above which contains a public response intended for the people in this thread and it does have links that do work.
Quite nice, didn’t know about this.
No CSS is preloaded, or JS for that matter, because of how the assets are defined without the domain, as in, just /css/app.css and not example.com/css/app.css, leading to a bunch of 400’s. Same goes for the netlify website where most fails.
Does Netlify’s prerender work exactly the same way? And if so, how exactly do you remedy this?
This looks like a serious bug too. As per W3C specification, prefetch attribute should not be used for any kind of pre-processing. It indicates resources that might be used in the future upon navigation. But prerendering generates only current page, there is no future navigation hence this resource should not be downloaded.
See HTML Standard
Hi, @maintopbiz. It isn’t pre-processing though and I don’t know why you think it is. Are you saying Chrome shouldn’t prefetch prefetch links? Because that is what Chrome does and will always do. The prerendering service is just running headless Chrome and headless Chrome prefetches when told to by HTML.
If you would like to see the behavior changed, that repo is public and you are free to file issues there to request a change.
Our prerendering service is in beta. You are welcome to use it if it meets your requirements. If you would prefer to use a third-party prerendering service we support several other services as well.
@freddy, I’m not sure what you are saying? Did you test the prerendering service locally? If so, what did behavior did you see?
@luke I am just saying that downloading links with “prefetch” attribute for prerendering purposes is wrong. It just artificially increases bandwidth. These assets shouldn’t be downloaded. I don’t know if and how it can be changed in headless Chrome. I would expect it to be your job to find out. You are selling prerendering under Netlify brand.
Prerendering was a major reason why we chose Netlify in the first place. Now you are telling us we should either deteriorate the experience for real users by removing “prefetch” attributes or keep paying those unnecessary huge bills.
Hi, @maintopbiz. You are welcome to file a feature request on that repo. Did you do that?
No, we are not. Prerendering is a free feature which is clearly noted as being in beta and available at all plan levels. We do not charge for it in any way.
I explained why you see what you see. I explained the technologies involved in prerendering. I explained the behavior of those technologies when using these HTML tag attributes (rel="prefetch").
I also told you what the remedy is if you would like to see this changed (filing a feature request). If there are other questions about this, please reply here anytime.
1 Like
Yes, you are charging for it heavily via bandwidth.
Our monthly charge without prerendering - $19
Our monthly charge with prerendering - $319 !!!
I don’t how else to explain that this is a serious issue.
Not sure why you can’t tell you colleagues yourself but ok, I will post an issue in the repo myself.
Hey @maintopbiz,
Sorry to hear that prerendering isn’t a good fit for your use case. We’ll reach out to you separately to grab a few more details and take a better look-see at your site(s).
Just an update. Removing prefetch attributes helped. The prerendering is no longer causing huge bandwidth costs. However, the website now feels much slower for the users because the assets cannot be prefetched. This is still a bug in prerendering service that needs to be fixed. Resources with prefetch attribute should not be downloaded during prerendering. We just decided for now to deteriorate the user experience in order to save the bandwidth costs.
Appreciate the loop-back, @maintopbiz. Fingers crossed, the dev team can consider a better approach for prefetch attributes when the beta feature is revised!
1 Like