Netlify Functions suddenly returning 500 errors, whilst Search Plugin function returns 404

[Landor_Publications]

Hello. I’ve been using Netlify’s Serverless Functions for a blog website for nearly two years. Some of them are used to load comments from a FaunaDB database, whilst another is created automatically during deployment (as I have the Netlify Search Index Plugin installed).

All of these have been working flawlessly, but since mid-February the Netlify functions for FaunaDB have been returning 500 errors, whilst the function for the search plugin now returns a 404. The errors occur 100% of the time.

Obviously something has happened that has affected both entities, so to keep this thread focussed I’d like to concentrate on just the Search Plugin (as it’s a public Netlify plugin, and therefore should be easier to debug compared to my own custom functions).

My account name is festive-swirles-0f46c8, using a custom domain of discussion.lttmagazine.co.uk and the search page can be found here (you’ll notice from the browser console logs that it immediately struggles to find the search function, as it returns a 404).

You can see from my build logs below that the Netlify Search Index Plugin installs fine during deployment…

11:28:29 AM: ────────────────────────────────────
11:28:29 AM:   3. netlify-plugin-search-index (onPostBuild event)            
11:28:29 AM: ────────────────────────────────────
11:28:29 AM: ​
11:28:31 AM: Search Index JSON generated at /searchIndex.json!
11:28:33 AM: Netlify Function generated at /.netlify/functions/search!
11:28:33 AM: ​
11:28:33 AM: (netlify-plugin-search-index onPostBuild completed in 3.9s)
11:28:33 AM: ​
11:28:33 AM: ────────────────────────────────────

As the logs correctly point out, the searchIndex.json file is created successfully (as can be seen here). But it incorrectly implies that the function has been created inside /.netlify/functions/search, whereas in reality this path generates a 404 error.

Below is a screenshot from my browser’s console demonstrating this when viewing the search page…

I’ve double checked the generated JSON file to make sure the search plugin is generating it properly and it validates fine. But then I noticed that my serverless functions elsewhere on the website are now generating 500 errors, so I figured there must be something else going on!

The only programmatical change that has occurred recently was to remove a custom domain name localtransport.today, which basically was a ‘domain alias’. Bearing in mind that it wasn’t the primary domain, could this still potentially be the cause of the issue? Below is how my domains are currently setup…

Default subdomain: festive-swirles-0f46c8.netlify.app
Primary domain: discussion.lttmagazine.co.uk

Below you can see the SSL certificate…

Certificate: Let’s Encrypt
Domains: discussion.lttmagazine.co.uk
Created: May 9, 2020 at 6:09 PM
Updated: Feb 20 at 8:48 PM
Auto-renews before: May 21 (in 2 months)

It might be a coincidence, but the date of the last SSL certificate renewal (20th Feb 2022) is roughly the same time these problems started occurring. I therefore tried clicking the ‘Renew Certificate’ button, but it never instigates a renewal (and fails silently).

If it helps below is my netlify.toml file, although it hasn’t been edited for a long time…

[dev]
  command = "npm run dev"

[build]
  command = "npm run generate"
  publish = "dist"
  functions = "functions"

[[plugins]]
  package = 'netlify-plugin-search-index'
  [plugins.inputs]
    exclude = ['/sitemap.xml']

[[headers]]
  for = "/*"
  [headers.values]
    Access-Control-Allow-Origin = "*"

I’m at a total loss to work out where to look next. So if anyone has any pointers (or if someone at Netlify is able to take a look at my account) then I’d be eternally grateful. I think I’ve divulged enough information to explain my problem, but if I have missed anything important then please let me know. Thanks everyone!

[audrey]

Hi, @Landor_Publications and welcome to the community !

It looks like the function isn’t being recognized by Netlify, you can see in the site’s functions page that search is not listed as a function. To fix this, the plugin creator will need to change onPostBuild to onBuild in their code so that it installs before function bundling during the build process and not after as seen in this build. To learn more about build plugin events, please see our docs. Perhaps you can reach out to them or create a PR?

To help us debug the 500 error, would it be possible for you to share the function code?

[Landor_Publications]

Hi @audrey thank you so much for getting in touch, it’s very much appreciated.

That’s a very good observation regarding the Search plugin’s build process, so I’ll let the plugin creator know, thanks.

As requested below is one of the function’s that’s returning a 500 error.

For context, this function basically uses GraphQL via FaunaDB to grab a bunch of blog comments. Please note that this function has always worked fine previously, and still works fine when I run it on my dev environment using Netlify CLI.

const axios = require('axios')
const print = require('graphql').print
const gql = require('graphql-tag')

exports.handler = async function(event, context) {

  const { FAUNA_READONLY_KEY, FAUNA_GRAPHQL_URL } = process.env

  const body = JSON.parse(event.body)
  const postSlug = body.postSlug

  const ALL_COMMENTS_BY_POST_SLUG = gql`
  query allCommentsByPostSlug($postSlug: String!) {
    allCommentsByPostSlug(postSlug: $postSlug) {
      data {
        _id
        user {
          role
          email
          forename
          surname
          occupation
          organisation
        }
        message
        timestamp
      }
    }
  }`

  let comments = await axios.post(FAUNA_GRAPHQL_URL,
    {
      query: print(ALL_COMMENTS_BY_POST_SLUG),
      variables: {
        postSlug: postSlug
      }
    },
    {
      headers: {
        Authorization: 'Bearer ' + FAUNA_READONLY_KEY,
        'Content-Type': 'application/json'
      }
    }
  ).then((response) => {
    return {
      body: JSON.stringify(response.data.data.allCommentsByPostSlug.data),
      statusCode: response.status
    }
  }).catch((error) => {
    console.log('Error', error)
    return {
      statusCode: 400,
      body: JSON.stringify(error)
    }
  })

  return comments
}

[hrishikesh]

Hey @Landor_Publications,

Could you additionally check your function logs and share the error message that might be appearing there? I tried to do it right now, but there are no records currently, so we can’t just guess why you’re getting an error by looking at the code.

[Landor_Publications]

Hello @hrishikesh thanks for reaching out.

I’ve tried as you suggested, but I’m afraid nothing ever appears in the logs. Even if you try accessing one of the functions directly in your browser you receive a 500 error…

https://discussion.lttmagazine.co.uk/.netlify/functions/faunadb-get-comments

…so therefore I don’t even think the request is even reaching the functions in the first place (hence the reason for the empty logs). If you try accessing Netlify functions directly from the browser via my other websites (that are working fine) then they correctly show a 403 instead (as obviously doing it in this way lacks the correct permissions etc…but this does demonstrate that they’re reachable).

Do you think that @audrey’s suggestion (regarding the incorrect build process for the Search Plugin) is having a detrimental effect on the other functions being built during deployment too?

[hrishikesh]

Hey @Landor_Publications,

I actually checked our internal logs for this one and thankfully, we seem to have an error message in there. It says:

InvalidZipFileException: Lambda was not able to unzip the file

So for some reason AWS is not able to unzip the zip that got deployed. Would you be able to make a simple change to the function so it would be considered as a new function and deployed accordingly to see if it fixes this?

About the second question, the 2 issues are totally isolated. So yeah, we can take that one out of equation.

[Landor_Publications]

Thanks for the suggestion @hrishikesh

Unfortunately this hasn’t worked. I’ve just tried adding a few extra lines into each function, but sadly with no success. A few weeks back I also tried the idea of changing the name of the functions, but alas no joy there either.

[hillary]

Hey there, @Landor_Publications

Thanks for your patience here. Would you be able to share your project repo with us? This will help us look into the setup and also reproduce your error locally. If your repo is private, please share a reproduction of the issue.

Thanks so much!

[Landor_Publications]

Thanks @hillary for reaching out.

Sorry for the delay, I was on annual leave.

Yes of course, you can find the repo from here:

https://github.com/landortravelpublications/local_transport_today

If you have any queries then just let me know, and thanks again.

[amelia]

Thanks for providing the repo! It looks like we will need to take this to our engineering team to investigate further. We handle the zip/unzip on Functions on our end, so it makes sense that you’ve hit a roadblock on debugging. We’ll provide an update as soon as we receive word back from the engineering team, but definitely let us know if you have any other questions or concerns in the mean time!

[audrey]

Hi @Landor_Publications , thanks for your patience while our developers investigated!

It appears that removing the netlify-plugin-search-index plugin from the netlify.toml resolves the zip issue and we were able to see the functions invoked. Would you be willing to give that a try and let us know if that helps or not?

[Landor_Publications]

Hello @audrey thanks for the update.

I’ve tested your suggestion. It didn’t immediately work on its own, as I also had to disable the netlify-plugin-search-index plugin via the Netlify dashboard too. But once both edits were made everything started working.

Thank you so much for your help, it’s very much appreciated.

I hope this thread is useful for others who may have installed this plugin in the past.