Serving large dynamic files

Support
,
1

Hi,

We are migrating a site from Aerobatic (https://www.aerobatic.com/) which has been a great service for us, but it unfortunately is shutting down.

The site allows users to log in to be able to view daily updated data. The data resides in several large JSON files (about 50 files of 2.5 MB JSON each) which is currently stored gzip compressed in S3.

We are currently leveraging the Auth0 and S3 plugins of Aerobatic. I’ve been able to integrate with Auth0 and Netlify RBAC using Netlify functions, but I’m struggling to find a good way to serve the JSON files.

First approach was to use Git LFS and Netlify Large Media, but that requires the backend that generates the files to log in both to GitHub and Netlify, which is very cumbersome, as this runs as a Kubernetes cron job using a Docker image.

Second approach was to write a Netlify function to serve the files from S3, similar to the Aerobatic S3 plugin (aerobatic.com is for sale | www.oxley.com), taking care to forward cache related headers in both the request and response, but it I hit two problems:

  1. The response has to be a JSON string, while I just want to stream the binary (gzip’ed) body from S3.
  2. RBAC cannot be applied to Netlify Functions (which I can work around by implementing authentication).

Does anybody have a suggestion for how to accomplish this?

Best regards,
Øyvind Matheson Wergeland

2

Hi @omw,

I’m trying to wrap my head around your problem, but I’m not 100% sure I’ve got it. From what I’ve understood, you just need to connect to S3 to serve the JSON files from there. Just that, these requests need to be authenticated.

If that’s the case, your second approach might work. You can write a Netlify Function to handle the connection to S3 and fetch the URL from there. It can then return the JSON string that you need to return along with the URL that could help the client stream the binary from S3. Functions would be short lived, so would not be too much useful to stream the data from S3. Thus, you’d have to use the function only to pass the streaming credentials back to the client.

1 Like
3

Got it! Since we already use a function in the front-end to request the JSON data with XHR, a Netlify function to generate a signed URL with limited lifetime works. We’re playing around a bit to make sure we cache properly.

Thanks!

1 Like