(Very) slow loading of some js: TTBF / download / revalidation

hi there @zehawki - we were able to make some small changes that you might see making some improvement, but we are also planning a more long-term remediation (updating some much more central underlying dependencies) that should hopefully fix things for good.

1 Like

Hi @perry, thanks a lot for the update. I havent been seeing this myself in a while - so I think certainly something has changed for the better.

I spoke too soon. Things are far far worse today than ever. Sites are not loading and taking 1+ MINUTE to load bundles.

x-nf-request-ids:

01FSKGDS6KH0CR5Z2S0KKW3JDW
01FSKGDS6KXVCFR3KVMVHWCVWM

x-nf-request-ids:

01FSKGF4QKX2WPW66WFM9ECQQP
01FSKGFBDE282C9EAF6TJK01TR
01FSKGFBDFDKK550Y0EM8TQT6H
01FSKGFBDG5Y6TCMZFQHYSTTQ4

hey there zehawki, i’m sorry the sites seem to be loading so slowly. we will look into this further as soon as we can. i have passed the info you supplied back to the team.

Documenting more cases as they are happening:

x-nf-request-id: 01FT6877V3SQMMQ868N6WMRZ4A

And then suddenly:

Hey @zehawki and thanks for your patience while we worked on this! I believe that we have (this week, since your last post) resolved the slow static asset loading situation - what I think you show in all screenshots except for your very last one. Please let us know if that slow load - 6+ seconds on a static file - recurs!

Your last screenshot seems to be something different - you don’t show the status, but since the assets are named in red, I expect it was some kind of HTTP error (400+ return status). That would almost certainly be a separate issue (issue we resolved:
slow but successful loads in the 6-8 second range), which we are of course happy to work with you on but do need to know either a request ID, or at least a timestamp with timezone, complete URL, and status, to dig in more deeply on those.

Hiya,

or at least a timestamp with timezone, complete URL, and status, to dig in more deeply on those.

Oops terribly sorry about that. Will provide that next time.

In the meanwhile, the earlier issue continues, hit multiple times today again (but nevertheless sporadic as earlier):

x-nf-request-id: 01FVAHA7D3HYGR0SMGB6935XB4

Thanks, I can see that in our internal logs with the same timing, and don’t understand what led it to be so slow, so I’ve asked our dev team to look into this presumably new-cause situation for us and will follow up as soon as I get some details!

1 Like

Documenting more cases as they arise:

x-nf-request-id: 01FVJ2H7W68XMTDDR2SGHKEPWP

thank you, zehawki. I passed that on, and it looks like we will get a chance to talk about this in more detail soon. Keep the request IDs coming I’d say, and we will definitely let you know as we figure out more information. thanks for your patience.

I hit another massive slowdown (or outage) today a while ago, across sites, with sites not loading. Lasted for about ~2 mins.

Started with this:

timestamp with timezone
13/02/22, around 12:30GMT (Once again, apologies, next time I’ll record precise TS)
complete URL
MainCross Social+
status
404 on the js chunk

And then I tried to reload this site and load others that are on Netlify - but nothing was loading. Then just as suddenly, all OK.

PS: Please note that whenever I hit this issue on Netlify, I immediately crosscheck vs our instances on Cloudflare (eg https://www.dhanada.in, https://www.hoten.life) to be sure that its not across the board.

Thank you so much for providing this additional information! We shared it on the issue where we are tracking this.

1 Like

Another one:

URL: MainCross Social+
Time: Wed, 02 Mar 2022 18:51 GMT

image

hi there zehawki, we didn’t forget and i am trying to get an update for you.

1 Like

@perry No worries, its rare enough these days.

2 Likes

And we are back.

I certainly hear Netlify when it says that all these troubles happen for “sites that are not in cache”, ie they are not busy enough. So do we have to live with this? And how quickly are assets moving out of cache - are they getting dropped every 30 seconds?? Why only certain assets, and not all? Very mysterious.

x-nf-request-id: 01G0VXEKC41XE8K5F580PEPAKF

Note that the above was after reloading the same site for the 4th time. Should certainly be in cache by then or?

This is the first:

x-nf-request-id: 01G0VX8M4CVF7B9J2Y9WDVAGSP
x-nf-request-id: 01G0VX8M4DVXJT0RT12HN670CT

Similar delays seen on load 2 and load 3.

So this part is rather frustrating. I really don’t see any reason why hashed static resources that are already in browser cache should have max-age=0 and then end up with such grand delays.

More (another site):

x-nf-request-id: 01G0VXWW4HVYF6YVBC454HB4FR

Thank you for reaching back out to us on this! The engineering team is still investigating, so the additional request IDs and info you’ve provided will be very helpful as they continue to research. I’m sending this over to them now, and I will follow up with you as soon as I receive word back from the engineering team.

We’re also noticing extremely slow download times, randomly.
For instance x-nf-request-id: 01G3YVPS40833YNGR4J2CTYER7, downloaded a 1.3mb bundle in 44s.
Is there a setting we need to adjust on our end? Thank you.

Hi, @ldiqual. Thank you for sharing that x-nf-request-id. I see the same thing in our logs. I’ve written up an analysis of the slow HTTP responses for your sites and escalated this to our developers to take a look at this.

Once we have a response from our developers, will post another update here to let you know what we discover.

1 Like

Thanks for your patience while we reviewed in more details, @ldiqual !

I do see exactly 43 requests that took over 5 seconds, over the course of the week 24 May to 31 May across all sites within your account. Those were exactly .02% of your total traffic, and some of them were fairly large (2-3MByte) assets which we would expect it to occasionally take over 5 seconds to send while our CDN is working normally - for instance to a slow client, or if the asset is not cached on the CDN node the client connected to. Our CDN has dozens of nodes, so to answer your question:

Note that the above was after reloading the same site for the 4th time. Should certainly be in cache by then or?

No, it might not be. Each CDN node maintains its own cache and we usually have around a dozen in our points of presence, so even the 12th reload for the same browser from the same location in the same 5 minutes, may still be bringing an asset into cache.

In short, while I do agree that the occasional slow loads aren’t ideal, our CDN is behaving within the parameters we intend, in aggregate. Or, put another way, since it is such a small percentage of requests, we will not be able to investigate more deeply in this case. I do recommend reading this article about using forever-changing names in your javascript files, which could help the assets be more cacheable across deploys, though: [Support Guide] Making the most of Netlify's CDN cache

I also didn’t see anyone respond to you directly about “was there some misconfiguration here?” and I do not see anything that seems wrong or that I would improve in your configuration (besides not changing filenames when content doesn’t change, per the article I linked above, to improve cacheability).

So, while I understand this isn’t the answer you wanted, we aren’t saying we’ll never investigate again. But in this case, the incidence is so low across your account, that we will not be able to investigate this incidence further, unless it happens rather more frequently.