I just have to add my +1 to the fact that this migration has been handled extremely poorly. I made an effort to test this as early as possible, and 4 months ago reported discrepancies between the recognized user agents in the legacy prerendering service, and those that the new one recognizes, and my detailed feedback has fallen on deaf ears. Now we find ourselves left with no choice but to experience a serious regression in functionality, or do an expensive migration to a 3rd-party service.
Hello @swrobel,
Your feedback has not fallen on deaf ears. The missing user-agents have been added and tested a few months ago. To my knowledge, all clients you’ve mentioned in the past are served with the pre-rendered variant of a site (some user-agents were missing, and were added at that time). Since then, we’ve worked with multiple users to optimize the extension based on their feedback.
From the screenshots you’ve shared at the time, it’s not clear to me what your expectation is for the simple demo site - the pre-rendered variant returns exactly the content that is dynamically rendered in the demo site. For example, the demo site does not have a different title or og: tags per breed page.
If you have a specific user-agent string and website to verify behavior on, we’d be glad to look into it.
@elad.rosenheim thanks for the reply. My continued confusion seems to involve a number of factors:
- Lack of a clearly published supported UA list: I requested the supported UA list, which had previously been provided for the legacy prerendering extension, and you provided it, but it’s difficult to find, because for reasons that are unclear to me, my thread was unlisted by statusbot, preventing it from being found on this forum.
- The aforementioned limitations with the demo site you’ve created. It frequently does show image previews for individual breed pages, which is what I understood to be an indication of successful pre-rendering, as for us, image previews are the most important pre-rendering factor for our users.
- Most importantly, the
getBrowserwarm up continues to be long, in the order of 5-10 seconds, which seems to cause timeouts with the Facebook crawler, which seems to power previews in FB posts, Messenger.com, WhatsApp, and Instagram DMs. This timeout results in a preview without the image, despite the pre-rendering extension eventually responding with one:
Incorrect result
Correct result
In Summary
I have attempted to communicate early & often with your team, providing as much information as I’m able to, being a helpful beta tester of this tool, and still felt blindsided by this announcement of a forced migration. Without a shared understanding of expected results for various scenarios, it’s led to responses that have felt confusing & dismissive to me. I hope you will take this feedback to heart.
Another thing that’s important to mention, that I don’t think has come up before, is that the new prerendering extension, for whatever reason, uses significantly more bandwidth than the legacy one. Our daily usage basically doubled upon enabling it:
Update
After the Headless Browser Cache flag was enabled for me in early testing, bandwidth usage dropped considerably, back to historical levels. Thanks @elad.rosenheim for working with me on this!
Hi @swrobel,
Looking at the data from before the extension was enabled, the no. of prerendering requests handled for your site was (and is) very very high: above the p99.9% of accounts using prerendering in your tier (in other words: less than 1 of 1000 of comparable accounts, with prerendering enabled, have this high a value), and 27x the p95 value for requests per account.
The cost to use an external prerendering service at this volume is above $1k/month (and as reminder, the legacy service was provided completely free of charge).
The reason this is so high seems to do with a high rate of cache misses, requiring constant fresh pre-rendering (while the common case is depending on caching on the CDN edge - which was much improved with the new extension; this is also directly related to the bootstrap time of the function, which should be much less frequent), plus a number of errors requiring retries.
I’m pretty sure it’s not intended for your site to have so many uncached prerender requests. We can work with you to figure out why that is, focusing on cache misses and other issues. But having looked at the data, I’m pretty certain the site represents an extreme edge case in how many rendering requests are invoked, vs. serving quickly from cache.
Thanks @elad.rosenheim! I’ve opened ticket 521421 to work on this with your team.
@swrobel I’ll be on it with our support team, and hopefully we get some learnings we could share back here.
Two important changes:
1. Behavior for crawlers that support client-side rendering:
These include: Googlebot / GoogleOther, bingbot, Amazonbot, Petalbot, and AppleBot.
By default, these have been excluded from prerendering, since they perform their own rendering as needed.
However, you can control this in the extension settings:
2. Caching in the headless browser:
Client-side caching in the headless browser is now enabled by default, which reduces the bandwidth usage related to repeatedly loading common assets - mostly JS bundle files.
This was found to dramatically reduce network bandwidth in sites with a very high number of prerender invocations (thanks @swrobel for working with us on this).
Browser caching can be toggled on or off in the extension settings as well.
1 Like
@phillip please see the post just above - whether Googlebot is skipped or not is now easily configurable.