Download speed through proxy is really slow

My site is porter.netlify.app and I am using it with a custom domain, cdn.porter.sh. I have binaries and xml file stored in Azure Blob Storage that I am proxying through Netlify.

Here are a few sample curl requests

$ curl -I  https://cdn.porter.sh/canary/porter-darwin-amd64
HTTP/2 200
accept-ranges: bytes
cache-control: max-age=300
content-disposition:
content-length: 52551088
content-md5: k+WV/r+uzTJ15le20GZvhA==
content-type: application/octet-stream
date: Tue, 16 Feb 2021 23:44:15 GMT
etag: "0x8D8D2D4C20B5048"
last-modified: Tue, 16 Feb 2021 23:44:10 GMT
server: Netlify
x-ms-blob-type: BlockBlob
x-ms-lease-state: available
x-ms-lease-status: unlocked
x-ms-request-id: d285b1d2-501e-0171-7ebd-04d2df000000
x-ms-version: 2014-02-14
age: 0
x-nf-request-id: 0621b3eb-952c-4f09-a54f-f1aebf9464a2-5637843
$ curl -I  https://cdn.porter.sh/atom.xml
HTTP/2 200
accept-ranges: bytes
cache-control: no-cache
content-length: 109946
content-md5: 8Qq2wIYrUEjo/Nqf4osoPA==
content-type: application/xml
date: Tue, 16 Feb 2021 23:40:54 GMT
etag: "0x8D8D2D0C13FD145"
last-modified: Tue, 16 Feb 2021 23:15:31 GMT
server: Netlify
x-ms-blob-type: BlockBlob
x-ms-lease-state: available
x-ms-lease-status: unlocked
x-ms-request-id: 4723e394-b01e-00de-74bd-04b61a000000
x-ms-version: 2014-02-14
age: 0
x-nf-request-id: 0621b3eb-952c-4f09-a54f-f1aebf9464a2-5591467
$ curl -I  https://cdn.porter.sh/v0.33.1/porter-darwin-amd64
HTTP/2 200
accept-ranges: bytes
content-length: 52551088
content-md5: U6bBwjXDFx8LwJ+m8QjsMw==
content-type: application/octet-stream
date: Tue, 16 Feb 2021 23:41:35 GMT
etag: "0x8D8D2C51AE9C660"
last-modified: Tue, 16 Feb 2021 21:52:07 GMT
server: Netlify
x-ms-blob-type: BlockBlob
x-ms-lease-state: available
x-ms-lease-status: unlocked
x-ms-request-id: 48c3cb3c-c01e-0174-70bd-0426a0000000
x-ms-version: 2014-02-14
age: 1
x-nf-request-id: 28990145-8909-46f4-a392-6941bbee5438-4749347

I am seeing very slow downloads for not so large files (19s for 50MB) and am hoping you can help either spot my config problem or tell me that I shouldn’t use Netlify for this. :sweat_smile: Here’s an example:

$ time curl -O https://cdn.porter.sh/canary/porter-darwin-amd64
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 50.1M  100 50.1M    0     0  2812k      0  0:00:18  0:00:18 --:--:-- 2817k

real	0m18.255s
user	0m0.134s
sys	0m0.267s

~$ time curl -O https://deislabs.blob.core.windows.net/porter/canary/porter-darwin-amd64
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100 50.1M  100 50.1M    0     0  17.6M      0  0:00:02  0:00:02 --:--:-- 17.6M

real	0m2.872s
user	0m0.088s
sys	0m0.200s

I’ve noticed that even for files with max-age set that the age returned in the headers is always 0/1. I’m not sure if that’s normal or if it indicates that my files aren’t making it into the Netlify CDN cache?

Thanks for your help!

Hey there @carolynvs!

I came across this article that may be of interest to you : [Support Guide] Why not proxy to Netlify?

I tried to understand if you are in fact proxying TO netlify, which I think you are? (Please correct me if I am wrong). If so then it seems this approach invalidates the Netlify caching mechanisms.

I don’t believe that I am proxying TO netlify? :thinking:

I have some files stored in Azure. My DNS and website is hosted with Netlify. I have a domain alias and CNAME, cdn.porter.sh, pointed to porter.netlify.app. In my _redirects I rewrite requests to that subdomain to Azure storage. Requests to the main domain just go to my static website on Netlify.

I am not using Azure CDN.

Here is the redirects in case that helps porter/_redirects at 87fda49278ad48ff20ee4f6d1daa8110c69e4c6e · getporter/porter · GitHub

I don’t think that’s the same thing that the article is describing?

1 Like

Hi, @carolynvs. I also agree this site is proxying from (not to Netlify) so that support guide would not apply to this situation.

About the transfer speed, files of that size (anything over 1 MB) will never be cached by the ADN nodes, proxied or not. They are so big they will never be kept in memory by our service.

Also, yes, content being proxied to is now sharing bandwidth on the ADN node and it will not be served as quickly as accessing the content directly.

To summarize, the performance you are seeing is normal and expected. (We also have a High Performance Edge ADN available for Enterprise level plans which would likely give better performance.)

1 Like

Thanks for your help, Luke! That’s the info I was looking for. Much appreciated!

1 Like

Great to hear! Thanks @luke for the detailed reply, good to know about the 1MB limit on caching. Still learning so thanks @carolynvs for making me understand it more as well :blush:

Have a good day!

HI CAROLYN!!! :netliheart:

1 Like

PERRY! :sparkling_heart:

Seriously, my team loves y’all at Netlify. When we realized that we needed a different CDN you were first on our list (all of our sites and domains are here already). It’s too bad that my go binaries are chonky. :joy:

1 Like

One workaround might be to host the large file downloads elsewhere (Cloudfront/S3?) and just change the links to them in the HTML itself on the site at Netlify. You can even keep the files as part of the Netlify site (but just not link to them or use those URLs as a backup link).

On the other hand, you are now managing deployments across two systems (two different web hosting platforms) so that workaround does come with some negatives as well.

I am trying out a few different providers now to do just that. Continue hosting the main site on Netlify and host the downloads separately.

1 Like