@luke i realized that i am hijacking my own issue 
I created a separate one for the cache-contol header: Improve Netlify Large Media cache-control header
Ok, this information helps a lot. Let’s try to figure this out.
CDN POP location: Frankfurt sounds like the closest one to me. So thats fine.
Cache hits/misses: That ratio is quite good. So that is not the issue. But if a cache hit has a response time of ~600ms, that is quite bad. Please give me an update once you have feedback on your bug report.
FYI: I see very similar response times on your NLM demo: https://netlify-photo-gallery.netlify.app/