I’m struggling to install netlify:lm on a docker image:
This is my error. I’ve not seen any documentation about a SHELL environmental variable.
I’ve been unable to find the ~/.netlify/helper/ file. Perhaps there is a problem about running as root which is kind of standard in docker.
# netlify lm:setup
✔ Checking Git version [2.11.0]
✔ Checking Git LFS version [2.12.0]
✔ Checking Git LFS filters
✖ Installing Netlify's Git Credential Helper for Linux
→ Unable to detect SHELL type, make sure the variable is defined in your environment
Configuring Git to use Netlify's Git Credential Helper
Error: Unable to detect SHELL type, make sure the variable is defined in your environment
at shellVariables (/root/.local/share/netlify-cli/node_modules/netlify-lm-plugin/lib/install.js:232:15)
at setupUnixPath (/root/.local/share/netlify-cli/node_modules/netlify-lm-plugin/lib/install.js:138:23)
at Task.task (/root/.local/share/netlify-cli/node_modules/netlify-lm-plugin/lib/install.js:74:19)
at runMicrotasks (<anonymous>)
at processTicksAndRejections (internal/process/task_queues.js:93:5) {
context: [Object: null prototype] {}
}
Thanks Perry. I there may be some instances where it’s not possible to switch user. I’m trying to use in a gitlab ci runner for example and su appears to be ignored.
I got a step further installing via a package manager
PACKAGE="git-credential-netlify-linux-amd64.deb"
curl -L -o $PACKAGE https://github.com/netlify/netlify-credential-helper/releases/download/v0.1.9/git-credential-netlify-linux-amd64.deb
dpkg -i $PACKAGE
netlify lm:install
...
✖ Configuring Git to use Netlify's Git Credential Helper
→ ENOENT: no such file or directory, open '/root/.netlify/helper/git-config'
Error: ENOENT: no such file or directory, open '/root/.netlify/helper/git-config'
at Object.openSync (fs.js:474:3)
at Object.writeFileSync (fs.js:1436:35)
at configureGitConfig (/root/.local/share/netlify-cli/node_modules/netlify-lm-plugin/lib/install.js:212:8)
at processTicksAndRejections (internal/process/task_queues.js:93:5) {
errno: -2,
syscall: 'open',
code: 'ENOENT',
path: '/root/.netlify/helper/git-config',
context: [Object: null prototype] {}
}
mkdir /root/.netlify/helper/
echo '# The first line resets the list of helpers so we can check Netlify`s first.\n[credential]\n helper = ""\n\n[credential]\n helper = netlify' > /root/.netlify/helper/git-config
netlify lm:info
✔ Checking Git version [2.11.0]
✔ Checking Git LFS version [2.12.0]
✔ Checking Git LFS filters
✔ Checking Netlify's Git Credentials version [0.1.9]
The script is completing but when I try and pull a file I get:
fatal: could not read Username
If I try a command such as netlify status I’m asked to login. git-credential-netlify is in my path:
Are you using an access token environment variable? If so, netlify status should not require a login.
Finally, once the Git credential helper is installed using the Netlify CLI tool, please confirm the “source” line is added to the .bashrc file for this user. Here is an example below:
# The next line updates PATH for Netlify's Git Credential Helper.
if [ -f '/root/.netlify/helper/path.bash.inc' ]; then source '/root/.netlify/helper/path.bash.inc'; fi
If all of this above is successful, what is the output of this command when those requirements are met?
It is the netlify lm:install step that fails, unless I install from the package manager and provide the missing config file.
There is almost certainly other things missing. The path helper for one.
ls /root/.netlify/helper/
git-config
I do have an access token in environmental variables as per reproduction steps.
Have you access to docker? Reproduction might take ten mins tops if you don’t have the image cached. I was using node:latest but image that any debian based container would act similarly.
root@1ab0ba8f795a:/nlm-fake-ssg-site-copy-1# netlify lm:setup
Logging into your Netlify account...
Opening https://app.netlify.com/authorize?response_type=ticket&ticket=CODE_HERE
---------------------------
Error: Unable to open browser automatically
Please open your browser & open the URL below to login:
https://app.netlify.com/authorize?response_type=ticket&ticket=CODE_HERE
---------------------------
TypeError: Cannot read property 'slug' of undefined
First, I think you are going to run into an issue with the netlify lm:setup period. This is interactive and there is no way to automate it that I can find. I don’t think this is currently possible because of the manual login step required.
However, even if you found a way around the manual login issue, there is the error issue. There is another example of this error in this open issue on the Netlify CLI repo:
So, there is some error blocking adding the Large Media add-on to a site using this Docker container and it is identical to the error reported above.
Would you please confirm what the end goal is here? Is it the creation of new Large Media sites from within a Docker container? If so, what is the reason for automating that process with Docker?
If you are not trying to add Large Media to a new site though, it should be possible to use existing Large Media sites/repos from within Docker. If that is the goal, install Large Media did work for me. Please let us know that is the case and I’ll be happy to keep troubleshooting with you.
Yes so I want to download some original images and convert user uploaded content such as .png, .JPG, .jpeg, .Tiff. Convert them all to .jpg and output webp files.
It would be nice to use the image transform service but I don’t think it standardizes file types or serves webp.
I run a diff against git commit time of the pointers, pull the binary if required and process the newly updated images, and put them back in the repo. If I can access the Large media without all of the netlify plugins that would be great, it would be great to be able to simply use git lfs with some setup.
#GIT_LFS_SKIP_SMUDGE=1 git clone git@gitlab.com:myRepo lfsTest
#cd lfsTest
#GIT_LFS_SKIP_SMUDGE=1 git checkout Branch
#docker run --rm --volume="$PWD:/srv/repo" -it node "/bin/sh"
# /srv/repo/setup.sh
TEST_FILE="src/uploads/img.jpg"
SECRET_NETLIFY_AUTH_TOKEN="xxxxxxxxxxxxxxxxx"
mkdir /srv/temp
cd /srv/temp
curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | bash
apt-get install -y git-lfs
npm install netlify-cli -g
netlify plugins:install netlify-lm-plugin
PACKAGE="git-credential-netlify-linux-amd64.deb"
curl -L -o $PACKAGE https://github.com/netlify/netlify-credential-helper/releases/download/v0.1.9/git-credential-netlify-linux-amd64.deb
dpkg -i $PACKAGE
cd /srv/repo
mkdir /root/.netlify/helper/
echo '# The first line resets the list of helpers so we can check Netlify`s first.\n[credential]\n helper = ""\n\n[credential]\n helper = netlify' > /root/.netlify/helper/git-config
netlify lm:install
git lfs fetch -I "$TEST_FILE"
if (( du "$TEST_FILE" > 4 )); then
echo "file has been downloaded" else
echo "file is still a pointer"
fi
This gets me in a loop asking for credentials in the terminal
Username for 'https://xxxxxxx.netlify.app':
Password for 'https://xxxxxxx.netlify.app':
I’ll read the attached issue now. I think I got this to work just fine as a normal user. It was root user that made it so hard.
So yes, I’d love to hear of an alternative, if I don’t need to install all the plugins to checkout and commit large media great.
Or if the media service has more features than I realize and can convert file formats, even if webp isn’t possible yet or is on a feature list for implementation later.
Yes so I want to download some original images and convert user uploaded content such as .png, .JPG, .jpeg, .Tiff. Convert them all to .jpg and output webp files.
Large Media cannot do this at this time.
Our Large Media service allows for browse time image transformations and there is more information about this at the documentation link below:
However, at this time, there is no transformation between image formats with our Large Media service.
To summarize, the image transformations are performed by making an HTTP request to the deployed file on the Netlify site with the image transformation settings as GET parameters which are part of the URL itself. There is no reason to use Docker to use these transformations and if you are using docker, Netlify CLI and the Large Media add on are not required. It is all done with HTTPs request so any HTTP client (wget, curl, a language library, etc) can be used. Second, the transformation that Large Media does not convert between image types.
Hi Luke, I can see there are a few options for getting the images downloaded. Is there any way of committing processed files back to Netlify Large Media Service without the cli?
Hi, @CraigC. The image transformations happen to the deployed files on the site, not files in the repository itself. To change the files served by the site, make a new deploy with those changes.
Thanks for the link. I was really wanting to know if I could do the following without the cli installed or if there was some other api?
git clone x
downloadAndProcessFilesOnGitLogTimeStamp
git commit -am "process new images"
git push
This wouldn’t be part of a build step it would run after each commit or as a precursor to a merge.
I have several GB of images so uploading a zip is a nonstarter and netlify pulling all of the lfs files from another repo each build is pretty slow too. The incremental nature of Netlify Large Media is it’s major selling point it’s just not easy to access.
I’m tempted to but all images into assets.domain.com and then do some separate process of Azure or something and maybe leave the site with netlify.
Is there not a lower level API, we use forestry for CMS and it can upload media to Netlify Large Media through the image gallery. It uses and API key not an interactive shell.
You can use the API for deploys but all deploy objects must define all files in each deploy. The API will require the paths and SHA checksums for all the files in each deploy. If the API doesn’t have the contents for any of the file checksums it will request that data. However, there is no escaping the “all deploys must contain all files” requirement. There is no way upload a single file without at least including the checksums and paths of all other files as well (and then only if none of the other files are new or have changed).
If you don’t use Large Media you can always push to the Git repo without Netlify CLI. If you do use Large Media, the Netlify CLI is required.
Note, Large Media and Git LFS are not identical. Large Media is a Git LFS service but using it is not required.
In other words, you don’t have to use Large Media to use Git LFS. You can use Git LFS for Netlify sites and skip Large Media entirely.
Have you tried just using Git LFS and not using Large Media?
That would be my recommended workflow since you are not using the image transformations Large Media makes possible (and it doesn’t make the type of transformations you require even if you were trying to use it).
To use Git LFS without Large Media, we must uninstall that add-on from the site and then GIT_LFS_ENABLED must be set to “true” in the site build environment variables.
If you would like to try using Git LFS without Large Media, would you please confirm you have read the following Support Guide?
If you are sure you have a backup of your files and you are ready for us to remove the Large Media add-on for this site, please let us know. Likewise, if there are other questions, we will be happy to answer.