Latest Gatsby builds broke our contact forms (serverless Netlify function)

Support
, ,
1

We have about 40 Gatsby websites (using Sanity headless CMS) that are all essentially identical and hosted on Netlify. This morning I noticed that the contact forms on some of them were no longer working and returning 502 responses. These forms are all routing to a serverless Netlify function and have been working fine for many months.

Upon digging, I noticed a few things:

  • The sites that were working had their builds triggered from within the Netlify dashboard or had not been built in many weeks
  • The sites that were not working had their most recent build triggered via the Netlify build widget within Sanity and were built within the last few weeks.

Upon reviewing the differences between the two builds, I primarily noticed 2 things:

  1. @netlify/plugin-gatsby was loading on the non-working version
  2. .netlify/functions-internal were being loaded on the non-working version

From there, I reviewed the NPM documentation for @netlify/plugin-gatsby and found an environment variable called NETLIFY_SKIP_GATSBY_FUNCTIONS to disable Gatsby functions. Since i’m not using any Gatsby functions, I set this variable to false team wide and rebuilt using the widget which restored functionality of the contact forms across all of the sites.

Although I have found this solution, I was hoping to understand a few things to try and avoid this type of outage in the future. Since the builds were completing successfully in this case it could potentially go unnoticed for a period of time which is catastrophic for us.

  1. Why is the plugin being injected into the builds if we do not have the plugin in our projects packages?
  2. Did something change on Netlify’s end to begin injecting this plugin recently? If so was there a notification that we missed?
  3. Any idea why it was not being injected when the builds were started from within the Netlify portal but was injected using the Sanity widget to initiate the builds?

Any additional context or details that you have surrounding this is appreciated. Again my goal is to prevent outages like this going forward and try to uncover what caused this seemingly out of the blue.

Thanks in advance!

3

Hi @RYNO-PPC,

About a month (or at least a few weeks back), we had started automatically installing Gatsby build plugins on sites that used Gatsby - however, due to several reports, this rollout was quickly stopped and I believe it’s now installed automatically only on nwe websites. So, while this sounds as a related change, I don’t think this is exactly what could have caused it.

Do you think you can share the links to the deploys in the Netlify UI where this got auto-installed and the problems started?

4

Sure thing, the first build I noticed was Feb 8:
Build Preview here
Example Build Report Here

5

Hi @RYNO-PPC,

Feb 8 is about the time-period when we were rolling out the changes I mentioned. Since you created this post recently, I thought this was a new issue, but based on your date, it’s just related to the old incident.