My netlify site name is remoteambition
We’ve enabled the prerendering service on Netlify. I have an issue where the bot for ahrefs (an SEO service) is not being recognized as a crawler. This causes it to be served the user-facing single-page app when it requests pages from my site (and messes up any reporting). I saw that you previously enabled this for SemrushBot – can you do the same here?
According to their docs here (AhrefsBot. Learn About the Ahrefs' Web Crawler), they’re the second most active crawler after Googlebot.
They use one of 3 user agent strings…
- Mozilla/5.0 (compatible; AhrefsBot/7.0; +http://ahrefs.com/robot/)
- Mozilla/5.0 (compatible; AhrefsSiteAudit/6.1; +http://ahrefs.com/robot/)
- Mozilla/5.0 (Linux; Android 6.0.1; Nexus 5X Build/MMB29P) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2272.96 Mobile Safari/537.36 (compatible; AhrefsSiteAudit/6.1; +http://ahrefs.com/robot/)
Can you add AhrefsBot and AhrefsSiteAudit to your regexp matcher for crawlers?