Thanks for your patience, Sander - we finally have an answer!
Your site is massive: almost 200k files. Thus, your redirect rules are massive. So massive it takes 15 seconds to transmit them across our network, since they cannot be cached for very long. This will make any non-file access (and some file accesses, depending on your setup) take quite a bit of time to run.
Your options to fix are to split into several smaller sites (perhaps using the workflow here: [Support Guide] Can I deploy multiple repositories in a single site? - you don’t have multiple repos, but you can stitch sites together linked to the same repo), or to move the functions to a separate site and call them by full path: https://othersite.netlify.app/.netlify/functions/yourfunc