Server sometimes responds with HTTP 200 even when If-None-Match matches ETag

Support
1

Hi,

We’re experiencing intermittent slow page loads on our Netlify website. Some debugging points to what seems to be an intermittent bug in Netlify web servers related to caching and compression. I’ve searched the forums but couldn’t find a post on this issue.

Specifically, we have a large JavaScript resource at the following URL:
https://cantamus.app/packages/verovio-3.3.0/verovio.js

Usually the browser correctly uses the cached version, since the server returns HTTP 304. However in some cases, the browser re-downloads the resource since the server returns HTTP 200, even though the browser sent a correct If-None-Match header. It seems that in those ‘cache miss’ requests, the server returns a wrong ETag header which does not include the -df suffix (even though the request was for the compressed resource).

I’ve verified this in two ways:

  1. Accessing the above URL in my browser (Chrome) with the DevTools open, and seeing that sometimes HTTP 200 is sent instead of HTTP 304, even though the browser sent If-None-Match correctly.

  2. Using curl with the following command:
    curl --compressed "https://cantamus.app/packages/verovio-3.3.0/verovio.js" -H "If-None-Match: f1ee2cc0ca9c51f2d05575f8c1436407-ssl-df" -I
    And observing that this request intermittently gives 200 and 304 responses. See screenshot below of two such requests (but I’ve reproduced this dozens of times).

Needless to say the resource itself is not being updated.

image
image2878×1380 354 KB

2

Further debugging:

I fixed the curl command to include double quotes in If-None-Match, but the problem can still be reproduced:

When requesting the non-compressed version (with the corresponding ETag - omitting -df), I have till now NOT managed to reproduce the problem:

When requesting the non-compressed version without quotes in If-None-Match, I seem to always get HTTP 200 (never 304):

So in summary, the problem only seems to be for compressed resource requests.

3

Further debugging: It seems that curl’s --compressed flag passes an Accept-Encoding header slightly different from Chrome’s.

curl: Accept-Encoding: deflate, gzip
Chrome: accept-encoding: gzip, deflate, br

If I use the header from Chrome in curl, I can’t right now reproduce the issue. Although it definitely can be reproduced this way, because I first noticed the problem in Chrome in the first place.

Also, the server sometimes returns a different ETag for the same request…
At this point I’m giving up and waiting for the support team’s reponse :slight_smile:

image
image2288×1678 391 KB

4

Just caught a slow page load with Chrome DevTools open, where HTTP 200 was returned for the script even though the resource ETag matched the browser’s If-None-Match. Also, for some reason the download was very slow (normally it would take a few hundred milliseconds but here it took 12.44 seconds for the content download).

verovio-request-200-headers1
verovio-request-200-headers12880×1800 593 KB

5

hi there @matan, thanks for sharing - i am still hoping to get some :eyes: on this soon.

6

Hey there, thanks for your patience!

So, I delved in to this and I could see that the d42d7fe7-a8e6-411d-bb9c-51fac47f983f request was uncached. This can happen for several reasons:

  • Our cache opted to evict this from the store
    • The asset may not have been requested for some time, or
    • The asset is of considerable size
  • The fingerprinted asset changed (a new build, file change)
    • You seem to have this accounted for

Just a head’s up RE: fingerprinted assets, it’s not advisable when using our CDN. By fingerprinting your assets, it’s likely that file names for all of your assets change after every build, even if the file does not change. This means that the asset becomes uncached after each build. You can read more about how we identify file changes without needing file name hashes here.

But, in your case, the eviction is likely due to the asset’s size (~1.5MB). If this particular JS asset is blocking your initial render, you should definitely try to break it up and load some of it asynchronously/lazily – the smaller, the better, with sub-500kb being the sweet spot and sub-1MB being a good benchmark.

Larger files are more regularly contested and subsequently evicted from our cache.

To add to this, and in general, you may experience slow loads on our CDN, if a CDN node hasn’t cached the file before. In each region where we have a CDN presence, we have multiple nodes and they each have their own cache. If the node which responds to your request hasn’t cached the asset, the node must request the asset (uncompressed) from our west-coast US origin server. The asset is then sent uncompressed to the CDN node, compressed, then sent to the browser. This is a one-off longer load. Each node will preserve the asset until your site is re-deployed and the asset changes.

I hope that this helps!

7

Hi, thanks for your response! This was really helpful.

I guess one simple workaround/solution for us could be to override the caching headers to remove max-age=0, must-revalidate. Then we would be relying much less on the CDN cache. This is an external library and basically a static resource (unless we upgrade to a newer version of the library and then the name changes).

The loading of this library is not blocking initial render, but we do have to rely on it for displaying the main page content (the library is a WASM-compiled musical score renderer, and the main page content is the musical score). We will try to reduce the library size though.

P.S. Maybe your CDN should compress assets when sending them between nodes that are far away :slight_smile:

8

Btw also I’m not sure it’s the best strategy to evict large files from the cache, for them the cache is most important!

9

Depends how often they’re called! Low traffic sites might see their content booted more often. Not to mention the warm-up time if a new node enters rotation.

Content is compressed, btw, during transit from CDN to requester (just clarifying for anyone who ignores the wall of text in future!)

10

I’m replying here rather than opening a new issue because this seems to be exactly what we’re seeing. I understand that large assets might not be in Netlify’s cache, but it seems like returning 200 instead of 304 is still a bug, even when a file is not cached. The server is still returning an identical etag header, so it’s clearly able to calculate the correct etag before starting to send the file. Once the etag is known, Netlify should be able to send a 304, and save bandwidth and not send the file contents. Is there any plan to fix this? Thanks!

1 Like
11

Hi, @dicta. I don’t see any bug filed for this and it was never reproduced by our support team. Long story short, the root cause of the issue wasn’t found for this topic.

The first step in troubleshooting this is getting enough information to see the issue happening when we test.

To this end, if you are seeing an if-none-match header not return a 304, would you please send us the following details?

  • the if-none-match header being used in the request
  • the exact URL being requested

That information will should allow us to see the issue happening so we can get a bug filed for it.

12

Sure, thanks.
I made two requests for:
https://dicta-library-data.netlify.app/Turei_Zahav_on_Shulchan_Arukh_Orach_Chayim/Halakhah.Shulchan_Arukh.Commentary.Turei_Zahav.Turei_Zahav_on_Shulchan_Arukh_Orach_Chayim.1.zip
in Chrome.

There was a brief delay of a few seconds between the requests, probably long enough for the object to leave the Netlify cache. The second request was sent with:
if-none-match: "46b404b0c2c84591ae7fda2ced1cb12d-ssl-df"

I was expecting a delay for the object to be retrieved, but a 304 response. Instead, I got a 200 response, and the response also included:
etag: "46b404b0c2c84591ae7fda2ced1cb12d-ssl-df"
!
If it’s helpful, the response included x-nf-request-id: "01FTR41DBTT9F99P66Y3SGX890"

14

Hey there, @dicta :wave:

Thanks for the patience here. We have tried to reproduce this, but are seeing 304 errors and cannot reproduce the 200 error. Are you still seeing this on your end? If you are, can you please share a HAR file with us?

15

Hi, yes, we are still seeing the error, but no longer on that link, since the file at that URL is now smaller than whatever it takes to trigger the bug; we replaced it since the time I originally posted. As of right now, it’s easy to reproduce on larger files - I just reproduced it with:
https://dicta-library-data.netlify.app/Turei_Zahav_on_Shulchan_Arukh_Orach_Chayim/Halakhah.Shulchan_Arukh.Commentary.Turei_Zahav.Turei_Zahav_on_Shulchan_Arukh_Orach_Chayim.1__nakdan.zip

I don’t see a way to attach the HAR so I’ll just post it a new comment.

16 
{

  "log": {

    "version": "1.2",

    "creator": {

      "name": "WebInspector",

      "version": "537.36"

    },

    "pages": [

      {

        "startedDateTime": "2022-02-10T17:00:31.938Z",

        "id": "page_1",

        "title": "https://dicta-library-data.netlify.app/Turei_Zahav_on_Shulchan_Arukh_Orach_Chayim/Turei_Zahav_on_Shulchan_Arukh_Orach_Chayim/Halakhah.Shulchan_Arukh.Commentary.Turei_Zahav.Turei_Zahav_on_Shulchan_Arukh_Orach_Chayim.1__nakdan.zip",

        "pageTimings": {

          "onContentLoad": 326.52199999574805,

          "onLoad": 336.18399999977555

        }

      }

    ],

    "entries": [

      {

        "_initiator": {

          "type": "other"

        },

        "_priority": "VeryHigh",

        "_resourceType": "document",

        "cache": {},

        "connection": "109925",

        "pageref": "page_1",

        "request": {

          "method": "GET",

          "url": "https://dicta-library-data.netlify.app/Turei_Zahav_on_Shulchan_Arukh_Orach_Chayim/Halakhah.Shulchan_Arukh.Commentary.Turei_Zahav.Turei_Zahav_on_Shulchan_Arukh_Orach_Chayim.1__nakdan.zip",

          "httpVersion": "http/2.0",

          "headers": [

            {

              "name": ":method",

              "value": "GET"

            },

            {

              "name": ":authority",

              "value": "dicta-library-data.netlify.app"

            },

            {

              "name": ":scheme",

              "value": "https"

            },

            {

              "name": ":path",

              "value": "/Turei_Zahav_on_Shulchan_Arukh_Orach_Chayim/Halakhah.Shulchan_Arukh.Commentary.Turei_Zahav.Turei_Zahav_on_Shulchan_Arukh_Orach_Chayim.1__nakdan.zip"

            },

            {

              "name": "sec-ch-ua",

              "value": "\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"98\", \"Google Chrome\";v=\"98\""

            },

            {

              "name": "sec-ch-ua-mobile",

              "value": "?0"

            },

            {

              "name": "sec-ch-ua-platform",

              "value": "\"Windows\""

            },

            {

              "name": "upgrade-insecure-requests",

              "value": "1"

            },

            {

              "name": "user-agent",

              "value": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.82 Safari/537.36"

            },

            {

              "name": "accept",

              "value": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"

            },

            {

              "name": "sec-fetch-site",

              "value": "none"

            },

            {

              "name": "sec-fetch-mode",

              "value": "navigate"

            },

            {

              "name": "sec-fetch-user",

              "value": "?1"

            },

            {

              "name": "sec-fetch-dest",

              "value": "document"

            },

            {

              "name": "accept-encoding",

              "value": "gzip, deflate, br"

            },

            {

              "name": "accept-language",

              "value": "en-US,en;q=0.9"

            },

            {

              "name": "if-none-match",

              "value": "\"77998a638031650099887d4ee5e41337-ssl-df\""

            }

          ],

          "queryString": [],

          "cookies": [],

          "headersSize": -1,

          "bodySize": 0

        },

        "response": {

          "status": 304,

          "statusText": "",

          "httpVersion": "http/2.0",

          "headers": [

            {

              "name": "date",

              "value": "Thu, 10 Feb 2022 17:00:50 GMT"

            },

            {

              "name": "etag",

              "value": "\"77998a638031650099887d4ee5e41337-ssl-df\""

            },

            {

              "name": "cache-control",

              "value": "public, max-age=0, must-revalidate"

            },

            {

              "name": "x-nf-request-id",

              "value": "01FVJ995WBN01KBSEW59TRNVZ2"

            },

            {

              "name": "server",

              "value": "Netlify"

            },

            {

              "name": "vary",

              "value": "Accept-Encoding"

            }

          ],

          "cookies": [],

          "content": {

            "size": 0,

            "mimeType": "application/zip"

          },

          "redirectURL": "",

          "headersSize": -1,

          "bodySize": 0,

          "_transferSize": 95,

          "_error": "net::ERR_ABORTED"

        },

        "serverIPAddress": "3.125.252.47",

        "startedDateTime": "2022-02-10T17:00:50.296Z",

        "time": 198.30399999773363,

        "timings": {

          "blocked": 4.6589999984502795,

          "dns": -1,

          "ssl": -1,

          "connect": -1,

          "send": 0.129,

          "wait": 189.0619999975683,

          "receive": 4.4540000017150305,

          "_blocked_queueing": 4.035999998450279

        }

      },

      {

        "_initiator": {

          "type": "other"

        },

        "_priority": "VeryHigh",

        "_resourceType": "document",

        "cache": {},

        "connection": "110651",

        "pageref": "page_1",

        "request": {

          "method": "GET",

          "url": "https://dicta-library-data.netlify.app/Turei_Zahav_on_Shulchan_Arukh_Orach_Chayim/Halakhah.Shulchan_Arukh.Commentary.Turei_Zahav.Turei_Zahav_on_Shulchan_Arukh_Orach_Chayim.1__nakdan.zip",

          "httpVersion": "http/2.0",

          "headers": [

            {

              "name": ":method",

              "value": "GET"

            },

            {

              "name": ":authority",

              "value": "dicta-library-data.netlify.app"

            },

            {

              "name": ":scheme",

              "value": "https"

            },

            {

              "name": ":path",

              "value": "/Turei_Zahav_on_Shulchan_Arukh_Orach_Chayim/Halakhah.Shulchan_Arukh.Commentary.Turei_Zahav.Turei_Zahav_on_Shulchan_Arukh_Orach_Chayim.1__nakdan.zip"

            },

            {

              "name": "sec-ch-ua",

              "value": "\" Not A;Brand\";v=\"99\", \"Chromium\";v=\"98\", \"Google Chrome\";v=\"98\""

            },

            {

              "name": "sec-ch-ua-mobile",

              "value": "?0"

            },

            {

              "name": "sec-ch-ua-platform",

              "value": "\"Windows\""

            },

            {

              "name": "upgrade-insecure-requests",

              "value": "1"

            },

            {

              "name": "user-agent",

              "value": "Mozilla/5.0 (Windows NT 10.0; Win64; x64) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/98.0.4758.82 Safari/537.36"

            },

            {

              "name": "accept",

              "value": "text/html,application/xhtml+xml,application/xml;q=0.9,image/avif,image/webp,image/apng,*/*;q=0.8,application/signed-exchange;v=b3;q=0.9"

            },

            {

              "name": "sec-fetch-site",

              "value": "none"

            },

            {

              "name": "sec-fetch-mode",

              "value": "navigate"

            },

            {

              "name": "sec-fetch-user",

              "value": "?1"

            },

            {

              "name": "sec-fetch-dest",

              "value": "document"

            },

            {

              "name": "accept-encoding",

              "value": "gzip, deflate, br"

            },

            {

              "name": "accept-language",

              "value": "en-US,en;q=0.9"

            },

            {

              "name": "if-none-match",

              "value": "\"77998a638031650099887d4ee5e41337-ssl-df\""

            }

          ],

          "queryString": [],

          "cookies": [],

          "headersSize": -1,

          "bodySize": 0

        },

        "response": {

          "status": 200,

          "statusText": "",

          "httpVersion": "http/2.0",

          "headers": [

            {

              "name": "accept-ranges",

              "value": "bytes"

            },

            {

              "name": "access-control-allow-origin",

              "value": "*"

            },

            {

              "name": "cache-control",

              "value": "public, max-age=0, must-revalidate"

            },

            {

              "name": "content-type",

              "value": "application/zip"

            },

            {

              "name": "date",

              "value": "Thu, 10 Feb 2022 17:02:03 GMT"

            },

            {

              "name": "etag",

              "value": "\"77998a638031650099887d4ee5e41337-ssl-df\""

            },

            {

              "name": "strict-transport-security",

              "value": "max-age=31536000; includeSubDomains; preload"

            },

            {

              "name": "content-encoding",

              "value": "br"

            },

            {

              "name": "vary",

              "value": "Accept-Encoding"

            },

            {

              "name": "age",

              "value": "0"

            },

            {

              "name": "x-nf-request-id",

              "value": "01FVJ9BD98YCA6J8TSVVDXBGGC"

            },

            {

              "name": "server",

              "value": "Netlify"

            }

          ],

          "cookies": [],

          "content": {

            "size": 0,

            "mimeType": "application/zip"

          },

          "redirectURL": "",

          "headersSize": -1,

          "bodySize": -1,

          "_transferSize": 229,

          "_error": "net::ERR_ABORTED"

        },

        "serverIPAddress": "3.67.153.12",

        "startedDateTime": "2022-02-10T17:02:03.197Z",

        "time": 488.2499999966747,

        "timings": {

          "blocked": 2.2199999966477044,

          "dns": 72.983,

          "ssl": 73.67500000000001,

          "connect": 218.528,

          "send": 0.24500000000000455,

          "wait": 189.82199999898904,

          "receive": 4.452000001037959,

          "_blocked_queueing": 1.6879999966477044

        }

      }

    ]

  }

}
19

Hey there, @dicta :wave:

Thanks so much for your patience here. We have been a bit underwater the past week. Can you please try inspecting their browser cache from here: How To See Cached Pages And Files From Your Browser to see if the file exists? Let us know and we will go from there!

20

Thanks for following up. Yes, it’s in the browser cache.

(You can verify this yourself from both requests in the HAR I posted earlier - the browser will only send if-none-match if the file is cached.)

Perhaps I wasn’t clear enough. The Netlify bug is this:
The browser sends if-none-match. Netlify sometimes responds with 200 and an etag that is the same as the one in the if-none-match in the request. It should send a 304 instead of 200. That’s the bug, and the HAR shows the correct behavior in the first request, and the bug in the second request.

The rest of what I wrote was speculation - I am guessing that Netlify only shows the buggy behavior when the file is large, and was evicted from the Netlify cache, because as the HAR shows, sometimes it behaves correctly. It’s not just zip files; I just had the same bug occur for Javascript at this URL: https://generalclassify.dicta.org.il/static/js/app.5a874f69ec2141edd53e.js

1 Like
23

Hey there, @dicta :wave:

My sincere apologies for the delay! I have looped in a Support Engineer who can look into this further for you. Thanks for your patience. If anything has changed since you last posted, please provide an update in the thread!

-Hillary

26

Hi, @dicta. Sorry about the delay on this, I’m seeing exactly what you see on some CDN nodes but not others.

This is how it should work (and most of the time this is what the site does):

$ curl --compressed -svo /dev/null -H 'if-none-match: 1213b3799e61100c3aa74ae6f89589b4-ssl-df' https://generalclassify.dicta.org.il/static/js/app.5a874f69ec2141edd53e.js  2>&1 | egrep '^< '
< HTTP/2 304
< cache-control: public, max-age=0, must-revalidate
< date: Fri, 11 Mar 2022 11:56:29 GMT
< etag: "1213b3799e61100c3aa74ae6f89589b4-ssl-df"
< server: Netlify
< vary: Accept-Encoding
< x-nf-request-id: 01FXWDCQREDWT76M5GD2FG02MX
<

The if-none-match has the correct etag value so a 304 is sent.

However, some CDN nodes are returning a 200 response and, even then, they only do so sometimes. I cannot reproduce it all the time.

Please also note, it is typically a fleeting error. By this I mean that I can get a CDN node to give a 200 incorrectly once but it will start sending the 304 correctly after that.

I’ve filed an issue for this so our developers can find the root cause. We will follow-up here as soon as we have more information or if the issue is known to be resolved.

27

Thanks for the update. As I mentioned, I have a guess that all the CDN nodes will return 304 correctly while the file is in their cache, but when they need to retrieve it from the origin, they return 200 incorrectly. If this is true, then you should be able to see a 200 again if you wait for a while to let it expire from the CDN node’s cache.