Hi, @smcgregor. I believe I found the other community topic and added an update to it with the root cause and solution.
The same root cause is happening here. Site wide password protection is incompatible with our Large Media service and this is covered in the limitations documentation for Large Media, quoting:
Uploading tracked files to the Netlify Large Media storage service requires Git to have access to the /.netlify/large-media path on the connected site. This will not work with site-wide password protection, but will work with other forms of visitor access control, as long as you leave access open to the /.netlify/large-media path.
The solutions for this are:
disable Large Media
or:
disable site-wide password protection
If there are other questions or concerns, please let us know. If you want to remove the Large Media add-on for this site, please read the following guide first:
Please let us know if there are any questions and/or if you would like to uninstall Large Media for this site.
Thanks Luke. When I look in the Netlify user interface, it does not say I have site-wide authentication enabled, but I do use gatsby-plugin-netlify with the config below. Can this really be the culprit? It seems odd that the hosted site would need to have a .netlify directory available in the clear to GitHub. I think I must be misinterpreting things here.
Hi, @smcgregor. This a basic-auth header not identical to our site-wide password protection feature but it does create the same effect. The Large Media LFS endpoint is now password protected behind this basic authentication so the git lfs push fails.
The limitation is the same for basic-auth as it is for the site-wide password protection. Either remove the password protection or disable Large Media.
It might also be possible to exclude the /.netlify/large-media path from the basic-auth header as well. I’m not familiar with that plugin so I’m not sure what the exact syntax required to do this would be.
If there are other questions about this, please let us know.
Hi, @smcgregor. You removed the authentication from the branch but not the site. It must be removed from the production site to resolve the issue.
It doesn’t appear you are using the image transformation feature of Large Media. If that is not being used, another Git repo host’s Git LFS service will work just as well.
Have you considered using Git LFS without Large Media?
We plan on using image transformations, but we need to develop the feature before opening the site. How do you suggest I develop the build with LFS without opening the site?
Hi, @smcgregor. With Large Media, you won’t be able to upload files to the Git LFS service Large Media provides while the basic-auth header rule is in place.
How do you suggest I develop the build with LFS without opening the site?
This is not possible. If you want to remove the Large Media add-on and use Git LFS by itself, that is possible with the basic-auth but using Large Media is not.
If you decide to remove the Large Media add-on, please read the support guide linked to above and let us know when you are ready for us to remove it.
As I’ve said before, if you want to keep the Large Media add-on, then disabling the basic-auth headers for any/all Large Media API paths should also work.
A rule _headers rule like this should match the API endpoints:
/.netlify/large-media/*
foo: bar
The foo: bar header server no purpose other than to prevent other header rules from applying. The headers used don’t matter (as long as there is no basic-auth header).
Note, this rule must added before any other rules which would also match the path. Putting it first would be my recommendation.
If there are other questions or you would like to proceed with the Large Media add-on uninstall, please let us know.
I had to avoid any wildcards in selecting headers at the root path because it always seemed to cause problems when the service deployed. After blacklisting a bunch of paths manually, I am able to push large files directly to Netlify upon pushing to GitHub. However, even after all the large files are apparently uploaded to Netlify, I do not see them on Netlify App . There is no transparency into where the files were uploaded and where the files may be hosted if I visit the production site’s hosting path. Is there documentation somewhere on the hosting of the large media files outside the transformation service? I would like to get the file host working, then switch over to the transformation service.
Also note, my large media files render fine in the browser when I host them, but they do not named with their file extensions. Please let me know if this could be one of the issues in activating the image transformation service.
Hi, @smcgregor. Large Media is a Git LFS service. The image transformation feature is something Large Media provides which other Git LFS services usually do not (because most Git LFS services are not provided by a company that also specializes in web hosting).
The image transformations occur when the deployed file is requested via HTTP by supplying the transformation requirements in the GET parameters of the HTTP request. Before any files (Large Media or not) can be requested via HTTP, they must be deployed.
To summarize, all files must be deployed before they can be requested via HTTP - Large Media included.
The same is true for the file list in the web UI for the Large Media add-on for a site. In order for a file to be listed on the Large Media add-on page, the file must be deployed first (instead of just uploaded via Git but not deployed.)
What does it mean for a file to be deployed?
A file is deployed if it exists in the publish directory for the site after the site build (if any) completes.
Are you deploying the files? If so, what is a URL for a file using Git LFS after the deployment is complete?
If you are not deploying the files, please let us know there are questions about how to do so.
Thanks @luke, I can confirm that the media is now showing up in the Netlify UI. Thank you for the help and extremely timely feedback.
One comment about the docs: everything tends to direct me to the image transformation service, but I don’t see much about the hosting service that backs the image transformation service. I don’t know, for instance, how large of files will be accepted by this hosting API. Are there docs anywhere breaking out the non-transformation file hosting?
Hi, @smcgregor. The Large Media besides the image transformation feature of Large Media, it is simply a Git LFS service. You interact with the files in the Git LFS service (meaning the files in Large Media) using Git.
The files uploaded to Large Media using Git are not intended to be used/accessed in any other way (besides Git itself) so we do not document that API further. Now, once the files are deployed, there is the image transformation feature but otherwise they are served via HTTP like all other deployed files.
To summarize, deployed files are accessed using HTTP. Files tracked using Large Media can use the image transformation feature. Besides that, Large Media is a just like any other Git LFS backend service for storing the LFS tracked files in a Git repo.
Now, is it possible to access the Git LFS API directly and not using Git? Yes, but I cannot think of an reason to do so for a site deployed at Netlify and depending on what you are doing this use might break our Terms of Service.
Example 1 (good API use and not breaking the ToS): You are manually constructing commits somehow and pushing commits programmatically to our Git LFS API.
Is that a violation of our ToS? No. Because these are “real” commits which would be visible in the Git history to anyone cloning the repo - this would not break the ToS.
Example 2 (bad API use which breaks our ToS): You use the Git LFS API to download files for display on the website or a third-party service.
The files must be requested using HTTP request to the site and not using the Git LFS API. Using the Git LFS API for any other purpose than creating or cloning commits for this repo is a violation of the ToS.
Thanks @luke, I can report that everything is working and your last comment brought needed clarity on the differences between the supported APIs/services/systems. The greatest sticking point for me was separating Git LFS as an API and Netlify’s hosting capacity both within Git LFS and the transformations add on. All good now.
