Thank for all for the feedback. At the moment, the way our service works, this would not be possible in the near future. It might need heavy rework of the underlying systems. Not saying it will never happen, it might, but trying to set expectations here.
Hi there! Just commenting if this can help anyone, as I didn’t find obvious why branch deploys
in contrast to preview deploys
doesn’t get the x-robots-tag: noindex
header. I found this subtlety confusing.
According to the referenced article (Site deploys overview | Netlify Docs), “non-recent” branch-deploys are not indexed. But I don’t see what’s considered a «recent branch deploy». Which period is recent?
So to avoid ambiguities here I opted to add the custom x-robots-tag: noindex
header in the deployed branch in netlify.toml
:
# Note that I'm only adding this to the deploy-branch, not production branch!
[[headers]]
for = "/*"
[headers.values]
X-Robots-Tag = "noindex"
Things to notice:
-
I think it would be nice to allow define headers conditionally on branch/context, e.g. nesting
[[headers]]
under[context.branch-deploy]
, like requested here TOML different headers for different contexts –but reading that thread I understand this is not implemented, is it? -
I’d also like to have the possibility to remove that deploy (at least from DNS) likewise other people in this tread, as it was just an experiment.
Cheers!
P.S. I will stick to preview-deploy
using PRs as in this case the noindex
comes out-of-the-box.
+1 to deleting deploy previews when the PR is closed and the branch is deleted.
Yup, we are still working to make that possible within our system. We’ll certainly follow up in this thread when deploys become deletable
It’s honestly pretty outrageous that this has stayed open for so long in this way, and is arguably illegal under GDPR and the various right to be forgotten laws. Both the predictable naming of URLs and inability to delete publicly visible content are critical security flaws and require urgent action by Netlify.
Hi, @maxhodak. All deploys can always be deleted. You must delete the site to do this but it is always possible to do so.
To prevent downtime, you can spin up a new version of the site and move any custom domains to the new site before deleting the original.
You can also always use two sites. The production site with no deploy previews and a second site linked to the same repo where you do allow deploy previews. The second deploy preview site could then be deleted from time to time to clear up the old deploy previews without impacting the production site in any way.
No one will argue that being able to delete individual deploys would be better but it is untrue that deploys cannot be deleted. All deploys can be deleted which means the statements above about security issues and GDPR violations are also untrue. You can always delete deploys.
About this:
It’s honestly pretty outrageous that this has stayed open for so long in this way
While I do understand why you have that opinion, I don’t think it is outrageous at all. There are reasons why this change is happening slowly.
In my time working in enterprise software development I have come to understand that technical debt is all too real and reworking early fundamental design choices is the trickiest of all technical debt to handle.
The early design choices in this case were that deploys would be atomic and immutable and there is a blog post about that very topic here:
The decision to make deploys immutable also led to the decision that deploys cannot be deleted the site itself is deleted (as deleting a site deploy is changing it and changing things isn’t allowed for immutable objects). Since the time these design decisions were made, a huge amount of code was written that relies on the deploys not being deletable. There are certain efficiencies to be gained when you can count on deploys always existing until the site is deleted. The code that was written took advantage of those efficiencies.
We are also going to have the make the changes on a live service without breaking anything. So, huge changes must be made, in parallel, to many interconnected software systems while keeping all the existing sites working and all currently running builds deploying. This will require extensive testing and a coordinated rollout of the changes.
So, making this change far from being as simple as just making an API call to delete deploys and adding a button in the UI to trigger it. There are a number of interdependent systems that will all require fundamental changes to their design to allow deploys to become deletable. That work is non-trivial in terms of both the complexity of the changes required and the number of person hours of work required to make the those changes.
I hope this clarifies that we are working on this feature and why those changes are slow in arriving.
You also stated that the URLs are predictable which true; they are predictable by design.
Would you be willing to explain more about how would you prefer they worked instead? Would all deploy preview URLs be random? Would the site subdomain (which is currently part of the URL) continue to be used?
Hey @luke fabulous explanation! I would love to read a blog post about the challenges faced, the impact and redesign after this has been implemented
It seems a really huge change under the surface.
Deleting the whole site to remove deploy previews is a pretty burdensome suggested workaround. If I have to delete the site for this purpose I (& associated companies which currently use Netlify) will simply move to Vercel.
Re: predictable URLs, yes, it should be random, or at least have a random component. Otherwise any content published as “preview” must be assumed to be publicly known, and it is indeed a practical concern that there are bots which watch and wait for predictable URLs like this. If we wanted it to be public content, we would have simply deployed it.
Let me back up a bit. I was reminded of this thread recently after having confidential information compromised which was contained on the first deploy preview I ever made, created when it was far from clear that these were permanent. When I first found this thread last year I figured it would eventually get fixed, even if it took a while. I don’t want to be unkind or unfair, though text-over-internet often makes that too easy. But it feels very wrong and worrying to be told that either there is no problem here, or that it is an extremely difficult one.
I’ve worked in tech for over 15 years now and I’ve been there. I’ve dealt with the tech debt, the architecture decisons, the complexities of caching, distributed databases; I get it. But if this is really a deep issue in Netlify that makes me pretty worried about what we’re building on and what the future looks like. I don’t say this to be mean, but it’s just a fact.
Is it really Netlify’s position here that there is no issue here worth solving in a timely manner? If so, I really will invest the time to think about switching versus deal with the complexity of multiple apps etc, and I would recommend others do as well.
No one at Netlify said that, @maxhodak. I already explained that it is being actively worked on and why it is taking so long in my post yesterday.
I don’t know if this has been suggested as a workaround, but here’s how I “hide” my old branch deployments:
In the branch of the deployment I want to hide, I add a redirect in my /static/_redirects file:
https://branch--site.netlify.app/* https://production-url.com/:splat 301!
Hello
Just looping in to say that this workaround will not work on Deploy Permalinks. Once these are generated, they are available if one knows about the unique build ID.
This will be very hard to predict of course so it will most likely not be an issue.
It can work (I have tested this) but it completely breaks deploy previews as they become inaccessible so not advised.
Is there a way to also lock a deploy preview to specific people inside the team?
I don’t believe that’s possible, Nightdev. Maybe in the future!
Is there any news on progress on this feature, or at least making the URLs less trivially guessable?
@maxhodak See: Automated deploy cleanup and new deploy retention policies starting late October 2022
It appears that automated deletion of old preview deploys is not working. The following screenshot was taken on 2022-11-28, or roughly one month after this “automatic deletion” was supposed to happen.
I’m grateful that there’s a way to delete deploys through the UI now! However, with about 1600 staged previews to delete for our site, this needs to be done in bulk, and deleting / recreating the site is not an option for us. Is there an API endpoint available which can do deploy deletion?
Hi, @sptramer. Any API endpoints used via the web UI can also be used directly. We have a support guide that covers how to examine the API calls in the browser devtools here:
Using that workflow you can see how the API call is made (the URL format for the endpoint) and examine the format of the JSON being sent to delete deploys.
Once you have the example API call and JSON format information, it will be possible to both call the API to get a list of deploy ids to delete and also to then iterate over that list of ids deleting the ones you want to remove.
Using the API directly in this way is both permitted and encouraged. It does not break our terms of use to directly call the API.