Netlify large media on docker image

[CraigC]

I’m struggling to install netlify:lm on a docker image:

This is my error. I’ve not seen any documentation about a SHELL environmental variable.

I’ve been unable to find the ~/.netlify/helper/ file. Perhaps there is a problem about running as root which is kind of standard in docker.

# netlify lm:setup
  ✔ Checking Git version [2.11.0]
  ✔ Checking Git LFS version [2.12.0]
  ✔ Checking Git LFS filters
  ✖ Installing Netlify's Git Credential Helper for Linux
    → Unable to detect SHELL type, make sure the variable is defined in your environment
    Configuring Git to use Netlify's Git Credential Helper
Error: Unable to detect SHELL type, make sure the variable is defined in your environment
    at shellVariables (/root/.local/share/netlify-cli/node_modules/netlify-lm-plugin/lib/install.js:232:15)
    at setupUnixPath (/root/.local/share/netlify-cli/node_modules/netlify-lm-plugin/lib/install.js:138:23)
    at Task.task (/root/.local/share/netlify-cli/node_modules/netlify-lm-plugin/lib/install.js:74:19)
    at runMicrotasks (<anonymous>)
    at processTicksAndRejections (internal/process/task_queues.js:93:5) {
  context: [Object: null prototype] {}
}

Reproduction instructions.

#On host
docker pull node:latest
docker run --rm  -it node  "/bin/sh"

#In container
SECRET_NETLIFY_AUTH_TOKEN="XXXXXXXXXXXXXXXXXXXXXXXXXXXXXXXX"
curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | bash && apt-get install -y git-lfs
npm install netlify-cli -g
netlify plugins:install netlify-lm-plugin
netlify lm:install

[CraigC]

Confirmed. Must install netlify cli plugins as a user

useradd -ms /bin/bash newuser
su newuser

netlify plugins:install netlify-lm-plugin
netlify lm:install

[perry]

thanks for sharing your progress - i’ll see if we can make instructions more clear.

[CraigC]

Thanks Perry. I there may be some instances where it’s not possible to switch user. I’m trying to use in a gitlab ci runner for example and su appears to be ignored.

I got a step further installing via a package manager

PACKAGE="git-credential-netlify-linux-amd64.deb"
curl -L -o $PACKAGE https://github.com/netlify/netlify-credential-helper/releases/download/v0.1.9/git-credential-netlify-linux-amd64.deb
dpkg -i $PACKAGE

netlify lm:install
...
✖ Configuring Git to use Netlify's Git Credential Helper
    → ENOENT: no such file or directory, open '/root/.netlify/helper/git-config'
Error: ENOENT: no such file or directory, open '/root/.netlify/helper/git-config'
    at Object.openSync (fs.js:474:3)
    at Object.writeFileSync (fs.js:1436:35)
    at configureGitConfig (/root/.local/share/netlify-cli/node_modules/netlify-lm-plugin/lib/install.js:212:8)
    at processTicksAndRejections (internal/process/task_queues.js:93:5) {
  errno: -2,
  syscall: 'open',
  code: 'ENOENT',
  path: '/root/.netlify/helper/git-config',
  context: [Object: null prototype] {}
}

[perry]

thanks craig. I’ve asked luke, our large media pro, to take a look at this when he comes on shift later today.

[CraigC]

Having a go creating the missing files manually.

mkdir /root/.netlify/helper/ 
echo '# The first line resets the list of helpers so we can check Netlify`s first.\n[credential]\n  helper = ""\n\n[credential]\n  helper = netlify' > /root/.netlify/helper/git-config

netlify lm:info
  ✔ Checking Git version [2.11.0]
  ✔ Checking Git LFS version [2.12.0]
  ✔ Checking Git LFS filters
  ✔ Checking Netlify's Git Credentials version [0.1.9]

The script is completing but when I try and pull a file I get:

fatal: could not read Username

If I try a command such as netlify status I’m asked to login. git-credential-netlify is in my path:

git-credential-netlify
time="2020-10-12T21:32:40Z" level=error msg="Aborting Netlify credential helper execution" error="Usage: git-credential-netlify <get|version>"
git-credential-netlify get #this hangs indefinitely

[luke]

Hi, @CraigC. The Git credential helper is normally installed with this:

netlify plugins:install netlify-lm-plugin
netlify lm:install

Is that preparation being done in the Docker image prior to running netlify lm:info? If not, then please try that and let us know your results.

Also, how are you handling authentication with the Netlify CLI in the docker image?

https://docs.netlify.com/cli/get-started/#authentication

Are you using an access token environment variable? If so, netlify status should not require a login.

Finally, once the Git credential helper is installed using the Netlify CLI tool, please confirm the “source” line is added to the .bashrc file for this user. Here is an example below:

# The next line updates PATH for Netlify's Git Credential Helper.
if [ -f '/root/.netlify/helper/path.bash.inc' ]; then source '/root/.netlify/helper/path.bash.inc'; fi

If all of this above is successful, what is the output of this command when those requirements are met?

git config -l

[CraigC]

Hi Luke,

Thanks for getting back with this.

In my reproduction steps in the first post I do indeed include the installation steps requested:

netlify plugins:install netlify-lm-plugin
netlify lm:install

It is the netlify lm:install step that fails, unless I install from the package manager and provide the missing config file.

There is almost certainly other things missing. The path helper for one.

ls /root/.netlify/helper/             
git-config

I do have an access token in environmental variables as per reproduction steps.

Have you access to docker? Reproduction might take ten mins tops if you don’t have the image cached. I was using node:latest but image that any debian based container would act similarly.

[luke]

Hi, @CraigC. This is as far as I got:

docker pull node:latest
docker run --rm  -it node  "/bin/bash"
export NETLIFY_AUTH_TOKEN="foo"
export SHELL="/bin/bash"
curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | bash && apt-get install -y git-lfs
npm install netlify-cli -g
netlify lm:install
source /root/.netlify/helper/path.bash.inc
netlify link --name=some-existing-site
netlify lm:setup # <-- this fails

This is the output of the final step above:

root@1ab0ba8f795a:/nlm-fake-ssg-site-copy-1# netlify lm:setup
Logging into your Netlify account...
Opening https://app.netlify.com/authorize?response_type=ticket&ticket=CODE_HERE
---------------------------
Error: Unable to open browser automatically

Please open your browser & open the URL below to login:
https://app.netlify.com/authorize?response_type=ticket&ticket=CODE_HERE
---------------------------
    TypeError: Cannot read property 'slug' of undefined

First, I think you are going to run into an issue with the netlify lm:setup period. This is interactive and there is no way to automate it that I can find. I don’t think this is currently possible because of the manual login step required.

However, even if you found a way around the manual login issue, there is the error issue. There is another example of this error in this open issue on the Netlify CLI repo:

`netlify lm:setup` crashes

So, there is some error blocking adding the Large Media add-on to a site using this Docker container and it is identical to the error reported above.

Would you please confirm what the end goal is here? Is it the creation of new Large Media sites from within a Docker container? If so, what is the reason for automating that process with Docker?

If you are not trying to add Large Media to a new site though, it should be possible to use existing Large Media sites/repos from within Docker. If that is the goal, install Large Media did work for me. ​Please let us know that is the case and I’ll be happy to keep troubleshooting with you.

[CraigC]

Thanks for taking this on.

Yes so I want to download some original images and convert user uploaded content such as .png, .JPG, .jpeg, .Tiff. Convert them all to .jpg and output webp files.

It would be nice to use the image transform service but I don’t think it standardizes file types or serves webp.

I run a diff against git commit time of the pointers, pull the binary if required and process the newly updated images, and put them back in the repo. If I can access the Large media without all of the netlify plugins that would be great, it would be great to be able to simply use git lfs with some setup.

#GIT_LFS_SKIP_SMUDGE=1 git clone git@gitlab.com:myRepo lfsTest
#cd lfsTest
#GIT_LFS_SKIP_SMUDGE=1 git checkout Branch

#docker run --rm  --volume="$PWD:/srv/repo" -it node  "/bin/sh" 
# /srv/repo/setup.sh
TEST_FILE="src/uploads/img.jpg"
SECRET_NETLIFY_AUTH_TOKEN="xxxxxxxxxxxxxxxxx"

mkdir /srv/temp
cd /srv/temp
curl -s https://packagecloud.io/install/repositories/github/git-lfs/script.deb.sh | bash 
apt-get install -y git-lfs
npm install netlify-cli -g
netlify plugins:install netlify-lm-plugin
PACKAGE="git-credential-netlify-linux-amd64.deb"
curl -L -o $PACKAGE https://github.com/netlify/netlify-credential-helper/releases/download/v0.1.9/git-credential-netlify-linux-amd64.deb
dpkg -i $PACKAGE
cd /srv/repo
mkdir /root/.netlify/helper/ 
echo '# The first line resets the list of helpers so we can check Netlify`s first.\n[credential]\n  helper = ""\n\n[credential]\n  helper = netlify' > /root/.netlify/helper/git-config
netlify lm:install
git lfs fetch -I "$TEST_FILE"
if (( du "$TEST_FILE" > 4 )); then
    echo "file has been downloaded" else
    echo "file is still a pointer"
fi

This gets me in a loop asking for credentials in the terminal

Username for 'https://xxxxxxx.netlify.app':                                                                             
Password for 'https://xxxxxxx.netlify.app': 

I’ll read the attached issue now. I think I got this to work just fine as a normal user. It was root user that made it so hard.

[CraigC]

So yes, I’d love to hear of an alternative, if I don’t need to install all the plugins to checkout and commit large media great.

Or if the media service has more features than I realize and can convert file formats, even if webp isn’t possible yet or is on a feature list for implementation later.

[luke]

Hi @CraigC. Earlier you wrote this:

Yes so I want to download some original images and convert user uploaded content such as .png, .JPG, .jpeg, .Tiff. Convert them all to .jpg and output webp files.

Large Media cannot do this at this time.

Our Large Media service allows for browse time image transformations and there is more information about this at the documentation link below:

https://docs.netlify.com/large-media/transform-images/#request-transformations

However, at this time, there is no transformation between image formats with our Large Media service.

To summarize, the image transformations are performed by making an HTTP request to the deployed file on the Netlify site with the image transformation settings as GET parameters which are part of the URL itself. There is no reason to use Docker to use these transformations and if you are using docker, Netlify CLI and the Large Media add on are not required. It is all done with HTTPs request so any HTTP client (wget, curl, a language library, etc) can be used. Second, the transformation that Large Media does not convert between image types.

If there are other questions, please let us know.

[CraigC]

Hi Luke, I can see there are a few options for getting the images downloaded. Is there any way of committing processed files back to Netlify Large Media Service without the cli?

[luke]

Hi, @CraigC. The image transformations happen to the deployed files on the site, not files in the repository itself. To change the files served by the site, make a new deploy with those changes.

There are several ways to deploy:

[Support Guide] Deploying a site to Netlify - different approaches

So you are ready to put your work live? Great! Here are 3 ways to deploy a site on Netlify: Drag and drop. This is the simplest & quickest deployment method. There is a "drop zone" at https://app.netlify.com/sites (once you have logged in). You can drop a folder or zip file in the specified box at the bottom of the page. Then to update your site, just go to your Deploys page for that site and drag and drop right there. Command Line Interface (CLI). This is often the easiest way to integrat…

[CraigC]

Thanks for the link. I was really wanting to know if I could do the following without the cli installed or if there was some other api?

git clone x
downloadAndProcessFilesOnGitLogTimeStamp
git commit -am "process new images"
git push

This wouldn’t be part of a build step it would run after each commit or as a precursor to a merge.

I have several GB of images so uploading a zip is a nonstarter and netlify pulling all of the lfs files from another repo each build is pretty slow too. The incremental nature of Netlify Large Media is it’s major selling point it’s just not easy to access.

I’m tempted to but all images into assets.domain.com and then do some separate process of Azure or something and maybe leave the site with netlify.

Is there not a lower level API, we use forestry for CMS and it can upload media to Netlify Large Media through the image gallery. It uses and API key not an interactive shell.

[CraigC]

So there is this:

Netlify

[luke]

Hi, @CraigC.

You can use the API for deploys but all deploy objects must define all files in each deploy. The API will require the paths and SHA checksums for all the files in each deploy. If the API doesn’t have the contents for any of the file checksums it will request that data. However, there is no escaping the “all deploys must contain all files” requirement. There is no way upload a single file without at least including the checksums and paths of all other files as well (and then only if none of the other files are new or have changed).

If you don’t use Large Media you can always push to the Git repo without Netlify CLI. If you do use Large Media, the Netlify CLI is required.

Note, Large Media and Git LFS are not identical. Large Media is a Git LFS service but using it is not required.

In other words, you don’t have to use Large Media to use Git LFS. You can use Git LFS for Netlify sites and skip Large Media entirely.

Have you tried just using Git LFS and not using Large Media?

That would be my recommended workflow since you are not using the image transformations Large Media makes possible (and it doesn’t make the type of transformations you require even if you were trying to use it).

To use Git LFS without Large Media, we must uninstall that add-on from the site and then GIT_LFS_ENABLED must be set to “true” in the site build environment variables.

If you would like to try using Git LFS without Large Media, would you please confirm you have read the following Support Guide?

[Support Guide] Uninstalling Large Media

Important This guide covers the process of uninstalling the Large Media add-on for a site. Notice: Once our support team uninstalls the Large Media add-on any copies of those files are no longer retrievable using our service. Please be absolutely certain you have local backups of all required files before directing our support team to uninstall the Large Media add-on from your site. Uninstalling Large Media Two considerations before we begin There are several ways to approach removing the add-…

If you are sure you have a backup of your files and you are ready for us to remove the Large Media add-on for this site, please let us know. Likewise, if there are other questions, we will be happy to answer.