How do I prevent Google from crawling domains that are not my custom domain?

[nathanmartin]

@davidf Ideally to achieve the indexing of the main branch, but not the staging/preview deploys, you should output different robots.txt files depending on the “Deploy Context”.

I explain the approach here regarding passwords, but it’s much the same for adjusting the output for any purpose:

How to set a password for deploy previews

You can do this with a combination of Deploy Contexts and Basic Authentication via _headers. There’s a blog article explaining the premise here: The pieces are: The context in the netlify.toml lets you identify which branch is building. You run a different script when you need to. The ‘branch only’ script copies a _headers file (with credentials) into the output. They can be. However you can also use the above idea to build a different robots.txt file into the output of your branch buil…

Effectively you would configure your contexts to run a build command that outputs a permissive robots.txt for your main branch and a restricted one for any other branch.