However, when I try to trigger a build/deploy from admin using the “Trigger Deploy” > “Deploy Site” feature, this ignore always causes the build to abort - which seems like not the default behavior that the docs would seem to indicate: Ignore builds | Netlify Docs I believe in the past when an ignore command was not specified, triggering a build from admin worked properly regardless of changes. How can I reference whether or not this build was triggered in admin from the ignore script or otherwise get around the ignore script when a build is triggered from admin?
I just confirmed this behavior with the following:
create a new nextJS site using yarn create next-app
link this to netlify
build the site
trigger a build from admin, it builds normally
add a netlify.toml to the root of the repository and include only the ignore example from the docs that should “Mimic default behavior” - I just copied that code block
site rebuilt on change in repository automatically
trigger a rebuild from admin, site fails to build due to no content change
How can I get around this to know a build was triggered from admin?
Well, at least for now I’m going with the following:
#! /bin/bash
if [ "$CACHED_COMMIT_REF" != "$COMMIT_REF" ]; then
echo "New Commit - Checking for changes..."
git diff --quiet $CACHED_COMMIT_REF $COMMIT_REF
exit
fi
echo "Looks like we already deployed this commit, proceeding with forced re-deploy..."
exit 1
slightly adapted for our purposes, but this would be a “closer to default” behavior script based on the one from the docs.
This actually won’t work properly. If a commit is aborted (either via this script or canceled) the commit hashes won’t line up when you try to re-deploy. Example:
commit A - deploys
commit B - canceled by user
trigger rebuild from admin - script compares commit A hash (cached) to commit B hash (current) and sees they are different. Proceeds with git diff which shows no changes. Rebuild from admin does not occur.
Not sure what is happening in your situation based on the notes you left, but we have guided some customers in forcing a single build using an ignore command of /bin/false which may be needed on even a new site if you use the base directory setting in our UI or your toml file.
For those customers, thereafter, the (netlify-default, no ignore script needed) ignored builds worked better once they got a first one under their belt.
Mind trying that to see if it helps on the new site you set up just to report the bug?
I’m a bit confused by that suggestion - it seems like that would always trigger a build (even if there are no changes)? Which, for this particular situation (triggering a rebuild from admin) would work - but it also would have a lot of other negative consequences. We are deploying a monorepo to 3x different Netlify sites, so this would case many commits to cause 2 of those 3 sites to deploy when they otherwise wouldn’t need to and would eventually lead to using a lot of extra build minutes. Is there no way to detect that a build was triggered from the admin panel?
I checked with our team and we never had this behaviour in the past. Builds using the ignore command will be ignored if no code changes are seen (git diff) regardless of them coming from the UI or CI/CD.
If you want to “force” a build using the UI, I’m afraid the ignore logic will not support it at the moment. Let’s take a few steps back, what are you trying to achieve exactly? Why would you need to force a deploy on a folder that has no changes?
web, sanity, and storybook all deploy separately to netlify. When something in one of the libaries (libs) changes, we need to trigger rebuilds of various apps that consume them which seems to necessitate a custom ignore script that will look outside the base directory of a deploy for changes in other directories/files. Consistent with the default ignore behavior, if there is a change in only one app we do not want to re-deploy the others.
The reason we need to be able to trigger a build from admin is because our web app (and possibly others in the future) consume data from APIs. If these external data sources change, we need to rebuild. we plan to utilize webhooks to trigger rebuilds, but we also need (for reassurance of uptime/data accuracy) to be able to manually trigger a rebuild in case of a webhook failure, a botched ENV var entry, etc - who knows. not being able to control deploys properly is simply not acceptable in an enterprise environment. Also since netlify doesn’t support NextJS’s ISR, we will be relying on rebuilds for any and all content changes unfortunately.
Let me know if you need more info, looking forward to a solution.
I guess the only other solution I see is to use very strict semantic versioning on all of our libraries so that there is a “change” present in the directory in question (via the package.json) but this is really not ideal as we don’t need to version anything and will simply add overhead. We are not particularly interested in polluting commits with netlify specific information either to accomplish this goal.
Actually, scratch that. The versioning solution would only partly work as that doesn’t account for loose files (such as configs) in the root directory that would not be versioned as a dependency of the individual projects. I guess those could also be moved to a “package” that is depended upon by various applications but this is all increasing complexity significantly for a relatively simple problem.
That isn’t correct. Netlify does support the Next.js incremental static regeneration (ISR) feature and I have some questions about this at the end of this reply.
Before I get side-tracked with the ISR questions, I also want to comment about this:
When you trigger a build via the web UI using the “Retry with latest branch commit” in the web UI, if there have been no new commits between the last successful build and the current build then the CACHED_COMMIT_REF and the COMMIT_REF will be identical.
The identical commit refs will only happen when the button in the web UI is used (because any builds triggered by a new git commit will, by definition, have different commit refs).
Based on this, I would recommend changing that script logic to always build if the CACHED_COMMIT_REF and COMMIT_REF match and use the git diff checks only when they differ . If that is done it should have the required behavior (meaning always build the button in the UI is used).
Would that meet the requirements for this build ignore script?
Okay, back to the Next.js ISR feature questions! It should work without any special requirements other that the required build plugins which are, by defailt, automatically installed for all Next.js sites.
Are you running into an issue using that feature at Netlify? If so, we can open a support ticket and troubleshoot that issue privately (or we can troubleshoot via a new topic here in the support forum if you prefer).
Re: ISR, i believe I used the wrong terminology. We were hoping to leverage “on demand revalidation” so that stakeholders could “refresh” particular pages on the site as needed. For instance, if some information is dated or was found to be incorrect, we could need that corrected ASAP. without on demand revalidation, my understanding is that could only happen via a full rebuild (or waiting for the cache time on ISR, which we would like to set to be relatively lax). I was under the impression on demand revalidation was not available based on this GH issue: Support for on-Demand Revalidation · Issue #1288 · netlify/next-runtime · GitHub
Re: triggering a build via the web UI, from my testing it did not work as you described if the previous build failed or was cancelled. In my previous post I did my best to describe this. The behavior I saw (just based on logging commit hashes from the ignore script) was that the “cached hash” is always what is deployed. So in the case of a canceled or failed build, the admin button would no longer trigger a redeploy going forward with that ignore logic because the “new commit hash” would not line up with the deployed one AND there would be no changes, resulting in aborted build repeatedly. The script I tested this with is described in the previous post. It’s possible that there is a bug in this script or I am not quite understanding.
Aha, yes, true, this is not a feature we yet support. Our best advice for today is to set a lowish revalidate timeout using “normal” ISR, for the content that needs frequent refreshes - something like 5 minutes or 10 minutes. Then your natural site traffic will cause revalidation to happen somewhat organically as folks browse. If this is something critical like a stock ticker where dollars are made based on content being STRICTLY CORRECT AT ALL TIMES FOR EVERY VISITOR (which is not likely to be the exact behavior of ISR even with revalidation, on Netlify - we intentionally serve the stale content for up to a minute as/after revalidation happens), you’d need to use SSR instead. I see you’ve already considered this, but it’s our best advice for today (in addition to tracking that issue so you see when your desired feature becomes available).
If the previous build was failed or cancelled, we do not save cache from it, so to clarify - you are working with an even older cache in this case.
It is possible - certainly some folks have reported it working oddly before. However, since most haven’t reported problems - that’s millions of customers - we presume it generally does work right :). Regardless, it’s a bit hard for us to debug what is happening for you, without access to your source code and repository.
I do wonder if you followed my earlier advice despite questioning it:
forcing a single build using an ignore command of /bin/false […]
In many of the cases reported - usually of folks setting up new sites that won’t build for the first time - yes, this setting causes every build to go through, but I was suggesting you try it once to get unblocked and see if later builds work right after removing it rather than suggesting it as a long-term solution.
Hopefully a more clear description of the situation I am trying to explain:
Consider the folder structure outlined in this post and the following ignore script in the web directory and storybook directory:
#! /bin/bash
if [ "$CACHED_COMMIT_REF" != "$COMMIT_REF" ]; then
echo "New Commit - Checking for changes..."
git diff --quiet $CACHED_COMMIT_REF $COMMIT_REF . ../../libs/ui-library
exit
fi
echo "Looks like we already deployed this commit, proceeding with forced re-deploy..."
exit 1
The intention of this ignore script being that a build would succeed in either of the following cases:
A site is re-deployed from Netlify admin (“Retry deploy” from netlify admin)
A commit is building that has changes in either the deployed apps’ directory or the libs/ui-library directory (which both source)
Consider the following scenario:
Connection to Github initialized with Commit A
web deploys as this is the first commit
sanity deploys as this is the first commit
storybook deploys as this is the first commit
Commit B is pushed to GH - modifies only files under the storybook directory
web does not deploy based on the custom ignore script
sanity does not deploy based on default ignore behavior from Netlify
storybook deploys based on custom ignore script
We now have an outside data source change some data and require manually triggering a rebuild of web. When we “Retry Deploy” the ignore script runs on the build. For the web deploy it behaves as follows:
$CACHED_COMMIT_REF is “Commit A” since this was the only commit ever deployed for web
$COMMIT_REF is “Commit B” since that is the most recent commit in the repository
This causes the ignore script to look for changes using the git diff in the above script logic. This finds no changes since “Commit B” had no changes to web or the ui-components folder. This build is then aborted. There appears to be no way to get web to deploy again unless we push a useless commit that modifies the web folder (or ui-library) or trigger a build via a webhook as they skip the ignore script according to the docs.
Does this sound correct? This was my experience based on setting up an end to end test and logging commit refs inside the above script.
Your suggestion does work and we are not currently blocked from working (the site is not actually live yet) but we do need to figure this out before we are production ready and the solution you suggest would not be acceptable for our use case since it would use an excessive amount of build minutes.
Hi, @paul.mourer. It sounds like you want to handle the case where the the commit SHAs have changed but the directory content has not (hence the build ignore script won’t detect changes). This is a situation where outside data source has changes but nothing in the repo directories has changed.
The recommended workflow in cases like this is to have a trigger configured on the outside data source to call the web hook. For example, if someone is using Contentful as a CMS, it can be configured to call the webhook at Netlify anytime the data at Contentful changes.
The keystone of that workflow is the third-party service telling Netlify the change occurred. Netlify does not try to detect the change in the outside data source. The outside data source tells Netlify that the change occurred (as the outside data source will always know when changes occurs).
Also, the outside data source itself doesn’t need to the be source of the webhook. The client that submitted changes to the outside data source can also call the webhook to trigger a build.
So, client or server, something in the outside data source workflow calls the webhook.
The only other option that I can see is to have Netlify constantly polling the data source to see if changes have occurred. While this is possible hypothetically (and we can enter a feature request for that), it would be far less efficient (in terms of CPU cycles and network data transfer) than the outside data source self-reporting changes by calling the webhook.
We plan to have webhooks set up for our main data sources, but we also anticipate having some sources of data that do not have webhooks or are unable to have those implemented. That said, webhooks are not 100% reliable so we don’t want to have those be the only way to trigger a build. Also, a situation could arise where we update an environment variable for the build environment and would need a re-deploy that otherwise has no changes and could not come from a “natural” webhook arising from a data change.
With all that in mind, I was hoping to have a “failsafe” that is in the netlify admin to re-deploy the site. With many stakeholders and team members, an externally hosted button would work in a technical sense but viewed from an organizational lense, it is a bit counterintuitive that the most powerful controls for our netlify deploys are outside the platform and could lead to friction and confusion in situations that are very time sensitive. It also would really need to be hosted outside netlify/our monorepo, since we would otherwise have a single point of failure so it would require additional hosting and tooling to accomplish this goal.
If this truly is not possible with Netlify, we will do what we have to do to work around it but this is far from ideal.
Edit: Actually with or without the ignore scripts it seems as though an ENV change would never deploy, I tested this with a current site using the Retry button and a dummy ENV variable. So, still stumped on that one as well. There really should be a master button built into Netlify that gets past anything.
I’m not sure why you would say webhooks aren’t reliable. They are, since you can:
see if no response is returned and retry or notify
see if some error response is returned and retry or notify
otherwise, success is returned because the build was launched successfully
What I think about webhooks: They’re a lot more reliable at triggering builds here than any git provider. For instance, GitHub, which has outages on average more than 1 day each month, (cf GitHub Status - Incident History), 100% prevents builds triggered during those outages with regularity. And of course you can’t rely on its reporting during an outage. But you should use your own standards, not mine.
TL;DR The only errors we’ve ever had with webhooks are when we have an outage that affects everything, which will happen less frequently than we experience issues with GitHub triggering builds while they are down.
So, for me, it is pretty easy to rely on them, and also pretty easy to change your workflows to accommodate them since all you need to use them is ability to hit an endpoint and parse a return code. If you can’t, of course, you are welcome to build whatever functionality you need as you are doing
Re: a failsafe, it’s there already you can hit the “trigger deploy” button in our UI at any time, so I think that covers your use case of:
I was hoping to have a “failsafe” that is in the netlify admin
So hopefully we are in a place that works for you with all of that.
My issue with webhooks is that they can only ever be as reliable as the two systems they are connecting, so in essence less reliable than either system on their own. They also create more for us to maintain and debug. This is all possible of course as you note, I was just hoping that there was a way within Netlify to accomplish this without us creating additional systems to maintain.
This is actually not true per this discussion unless we choose to not use any ignore scripts in our repo, which we will have to in order to conserve build minutes. if there was a way to break past the ignore scripts from admin (this is my ask for netlify) then yes, we would have a failsafe in admin. As it stands now, we are unable to trigger rebuilds from admin. we can pause this discussion as I feel like we’ve come up with all the possible solutions and it will be up to us to implement the one that best fits our use case.
you can hit the “trigger deploy” button in our UI at any time, so I think that covers your use case of: