We’ve enabled the prerendering service on Netlify. I have an issue where the bot for ahrefs (an SEO service) is not being recognized as a crawler. This causes it to be served the user-facing single-page app when it requests pages from my site (and messes up any reporting). I saw that you previously enabled this for SemrushBot – can you do the same here?
We’ve begun the process of adding this UA. We’re now waiting for the developers to confirm and release it into production. We’d let you know once that happens. Thank you for your patience.
Hi, am I allowed to disagree with this decision to add Ahref or a way to specifically turn off crawling by them? I do not believe this is the right decision. They are not an SEO service by any stretch of imagination - they crawl sites so that they can sell that data to anybody who wants to compare multiple services. I certainly do not want to allow Ahref to crawl through any of my sites. They are a nuisance.
H, I thought of that - but it depends on Netlify processing flow, yes? Since this is based on UA for prerendering, wouldn’t the prerendered response be returned before checking robots.txt. Happy to be wrong about this…
I asked the developers and according to them, in theory, a bot should check for robots.txt before trying to crawl the site. However, respecting that file or not is up to the bot. At least their docs suggest that they respect that file:
Since robots.txt is not a HTML file, Netlify won’t pre-render it and thus, it should be easily scanned by the bot before crawling the rest of the content.