Prevent Google from indexing netlify URL

Support
1

After connecting a custom domain, how do we prevent Google from indexing the netlify URL (example.netlify.app)?
I’m using Netlify DNS.

Can someone confirm these are the steps?

Step 1. Save a plain text file called _redirects in my root directory. (Note: my project is just an index.html file in my root directory. Not using a build command or static site generator).
Step 2. Inside of the _redirects file write:
https://example.netlify.app https://example.com 301

Is that it? Do I need to write the http and www versions of these lines too?

Is the netlify.toml file required too?

Thanks

2

Hi @flynet,

You could try using a custom headers rule in a _headers file, like this:

https://example.netlify.app/*
   X-Robots-Tag: noindex

This will tell google not to index that url at all. Let me know if that works for you.

1 Like
3

I created a _redirects file based on the solution described here. The file says:

https://example.netlify.app https://example.com 301!
http://example.netlify.app  https://example.com 301!

Do I also need to add the custom headers rule? According to this Google article: "If you need to change the URL of a page as it is shown in search engine results, we recommend that you use a server-side 301 redirect. "

Please clarify if the current setup is enough or if the _headers file is required too.

Thank you!

4

I followed the solution here.

Do I also need the _headers file? This Google article suggests that 301 is enough to change the URL in search results.

Please advise. Thank you.

5

Hi, @flynet, the 301! rules will force all traffic to the custom domain. This alone is enough to prevent Google from indexing those pages.

However, if you have a site without a custom domain, the _headers rules are another way to prevent indexing. This can be helpful when you have a demo/testing site without a custom domain and you don’t want it indexed.

There is a third method as well. The “robots meta tag”:

To summarize, the 301! rules can only be used with custom domains. The X-Robots-Tag HTTP header and robots meta tag can be used on any sites, custom domain or not.

6

@Dennis

Sorry for hijacking this. But I have a question.

I am using Hugo. How can I put this (the _herders file) in my Hugo site?

7

By creating that file in the public folder.

8

Hey, sorry for reviving this conversation, but I have a question about this:
If I have a custom domain connected in Netlify, does Netlify automatically use headers in the ***.netlify.app doimain to tell Google to not index it? I’m a bit worried that having duplicate content (since it’s the same website on 2 different domains) will hurt SEO.

Thanks!

9

No, we don’t. You can setup 301 redirects to redirect to the custom domain.