Hello Netlify Support Team,
I hope you can help me resolve an issue with my website, drlevi.co.il. Although the robots.txt file is accessible and returns a proper 200 OK response when accessed via a browser or using curl, Google Search Console continues to report an error stating “Failed: Could not access robots.txt” during live tests and indexing requests.
Here are the details of my setup and troubleshooting steps:
robots.txt File:
The file is located in the root of my project and its contents are as follows:
makefile
Copy
www.drlevi.co.il robots.txt
User-agent: *
Allow: /
Allow: /sitemap.xml
Disallow admin and private areas
Disallow: /admin/
Disallow: /private/
Disallow: /wp-admin/
Disallow: /wp-includes/
Allow crawling of important content
Allow: /מאמרים/
Allow: /assets/
Crawl-delay for better server performance
Crawl-delay: 10
Sitemap location
Sitemap: https://www.drlevi.co.il/sitemap.xml
I have confirmed that this file is accessible at https://drlevi.co.il/robots.txt.
_headers File:
To ensure that the file is served with the correct Content-Type, I have added a _headers file at the root with the following content:
bash
Copy
/robots.txt
Content-Type: text/plain; charset=utf-8
DNS and SSL Configuration:
Both the apex domain (drlevi.co.il) and the www subdomain are properly configured with valid SSL certificates, and my DNS settings are correct. Testing with curl (using system certificates) returns a proper response.
Google Search Console:
Despite the above, Google Search Console still shows the error “Failed: Could not access robots.txt” when attempting a live test of the file.
Could you please investigate whether there might be any edge caching or server configuration issues on Netlify’s side that could be causing Googlebot to be unable to access the robots.txt file? Any assistance or recommendations to resolve this issue would be greatly appreciated.
Thank you for your time and support.
Sincerely,
Roi levy