A can't upload image to lfs in netlify

Thanks @luke that makes alot of sense,
After running:
tree ~/.netlify/helper

The response I get is
zsh: command not found: tree

Would it be handy if I just did a ls, or cat some files for you or something?

I tried this in both the mac terminal and the vs code build in terminal

About this:

It would but I didn’t know which file to tell you to cat. The tree command wasn’t installed. This be done using Homebrew with:

brew install tree

You mentioned Mac and I see zsh is the shell. You probably need to add a helper to the configuration file for zsh. This is likely already installed and the tree command would tell me the correct path to add to the .zshrc or .profile file (whichever your system is using).

This is what was added on a Linux system for testing:

# The next line updates PATH for Netlify's Git Credential Helper.
if [ -f '/home/username/.netlify/helper/path.zsh.inc' ]; then source '/home/username/.netlify/helper/path.zsh.inc'; fi

For MacOS, this would likely be the following instead:

# The next line updates PATH for Netlify's Git Credential Helper.
if [ -f '/Users/username/.netlify/helper/path.zsh.inc' ]; then source '/Users/username/.netlify/helper/path.zsh.inc'; fi

Note, that change won’t affect any existing terminal windows (aka shell sessions). To make the change take effect in an existing window, use the following:

source /Users/username/.netlify/helper/path.zsh.inc

If adding that line doesn’t help, please let us know.

Hey there @luke,

Here’s the response I got from tree
/Users/ericmikkelsen/.netlify/helper
├── bin
│ └── git-credential-netlify
├── git-config
└── path.zsh.inc

Also I added a .bash_profile and this line (but with my user name)
if [ -f '/home/username/.netlify/helper/path.zsh.inc' ]; then source '/home/username/.netlify/helper/path.zsh.inc'; fi

I’m still getting this after closing the terminal and re-opening it:

batch response: Repository or object not found: https://8af78b07-6f98-4b04-9a19-2daa8040e79e.netlify.app/.netlify/large-media/objects/batch

Check that it exists and that you have proper access to it

Thanks for all the help so far

Hi, @dev-boldium-ebce. Did you also change /home to /Users like so:

 if [ -f '/Users/ericmikkelsen/.netlify/helper/path.zsh.inc' ]; then source '/Users/ericmikkelsen/.netlify/helper/path.zsh.inc'; fi

Also, you are using zsh and not bash so this line would added to .zshrc because zsh won’t read .bash_profile.

Would you please try both those changes as well?

Hey @luke, I just updated it, then tried pushing it hung for a long time then rejected in the same way with a bunch of:

Check that it exists and that you have proper access to it
batch response: Repository or object not found: https://8af78b07-6f98-4b04-9a19-2daa8040e79e.netlify.app/.netlify/large-media/objects/batch

errors, but the most interesting thing is when I run tree again, it has all my tracked files listed out in there.

Hi, @dev-boldium-ebce. It sounds much like the Git credential helper still isn’t working.

What does netlify lm:info show now? For example:

$ netlify lm:info
  ✔ Checking Git version [2.28.0]
  ✔ Checking Git LFS version [2.12.0]
  ✔ Checking Git LFS filters
  ✔ Checking Netlify's Git Credentials version [0.1.9]

Hey @luke Here’s what I get for my output on the netlify lm:info

  ✔ Checking Git version [Git-128)]
  ✔ Checking Git LFS version [2.12.0]
  ✔ Checking Git LFS filters
  ✔ Checking Netlify's Git Credentials version [0.1.9]

Hey there, Just wanted to drop a couple pieces of additional context

  1. we’re not seeing anything under our Netlify Large media tab on our account
  2. We’ve validated that netlify credential helper is in my path and is accessible
  3. We had configured git lfs, but hadn’t pushed any lfs files to the repo yet, and we followed these steps to make sure they were tracked by netlify large media
  4. and our .lfsconfig file is present and looks correct.

Hi, @dev-boldium-ebce. This sounds a lot like the issue here in this earlier topic:

Just like in the comment above, it looks like you are also using the default Git client with MacOS (from the version “Git-128”).

To summarize, I think there is some combination of three things happening:

  1. an issue with multiple Git clients being installed and different clients being used in different contexts
  2. an issue with the presence of the Git credential helper
  3. an issue with the ordering of the Git credential helpers

I think you will be much more successful once the version reported by netlify lm:info is some 2.x.x version and not “Git-128”.

Resolving that can be done with:

  • installing the Homebrew version of Git
  • setting the shell PATH environment variable so that the correct Git client is consistently used (the Homebrew version)

​Please let us know if there are questions about how to make those changes.

Once the netlify lm:info reports a 2.x.x version, you might find the git push issue to be resolved.

If getting the Git version updated and consistent does not resolve the git push issues, then we can move on to troubleshooting issues two and three above.

If you are that that point, would you please send us the output of these two commands below?

git config -l
env | grep "PATH"

As always, if there are questions about any of this, please let us know.

Hey @luke, it looks like git update didn’t fix it, here’s my netlify lm:info output

  ✔ Checking Git version [2.28.0]
  ✔ Checking Git LFS version [2.12.0]
  ✔ Checking Git LFS filters
  ✔ Checking Netlify's Git Credentials version [0.1.9]

After running the git config -l output is:

credential.helper=osxkeychain
user.email=eric.mikkelsen@boldium.com
include.path=/Users/ericmikkelsen/.netlify/helper/git-config
credential.helper=
credential.helper=netlify
filter.lfs.clean=git-lfs clean -- %f
filter.lfs.smudge=git-lfs smudge -- %f
:...skipping...
credential.helper=osxkeychain
user.email=eric.mikkelsen@boldium.com
include.path=/Users/ericmikkelsen/.netlify/helper/git-config
credential.helper=
credential.helper=netlify
filter.lfs.clean=git-lfs clean -- %f
filter.lfs.smudge=git-lfs smudge -- %f
filter.lfs.process=git-lfs filter-process
filter.lfs.required=true
include.path=/Users/ericmikkelsen/.netlify/helper/git-config
credential.helper=
credential.helper=netlify
core.repositoryformatversion=0
core.filemode=true
core.bare=false
core.logallrefupdates=true
core.ignorecase=true
core.precomposeunicode=true
remote.origin.url=git@github.com:devboldium/ebce.org-11ty.git
remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*
branch.trunk.remote=origin
branch.trunk.merge=refs/heads/trunk
lfs.https://8af78b07-6f98-4b04-9a19-2daa8040e79e.netlify.app/.netlify/large-media.locksverify=false
credential.helper=netlify

And the env | grep "PATH" output is:

PATH=/Users/ericmikkelsen/.netlify/helper/bin:/Users/ericmikkelsen/.netlify/helper/bin:/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin:/Library/Apple/usr/bin:/Users/ericmikkelsen/.netlify/helper/bin
1 Like

I’m seeing the issue in the git config -l output.

We need the output to look like this:

credential.helper=
credential.helper=netlify
credential.helper=osxkeychain

It can also look like this (if these are the only two lines for “credential.helper” in the file):

credential.helper=netlify
credential.helper=osxkeychain

The “empty value” line below is important (or it can be skpped):

credential.helper=

Let’s take a look at the contents of ~/.netlify/helper/git-config for an explanation:

$ cat ~/.netlify/helper/git-config

# The first line resets the list of helpers so we can check Netlify's first.
[credential]
  helper = ""

[credential]
  helper = netlify
  helper = osxkeychain

Setting the credential helper to empty string is resetting the list. Then the order after that must be “netlify” then “osxkeychain”.

Let’s take a look at your system’s git config -l output filtered for just the “helper” lines (but otherwise left in order):

$ grep helper git-config-output.txt
credential.helper=osxkeychain
include.path=/Users/ericmikkelsen/.netlify/helper/git-config
credential.helper=
credential.helper=netlify
credential.helper=osxkeychain
include.path=/Users/ericmikkelsen/.netlify/helper/git-config
credential.helper=
credential.helper=netlify
include.path=/Users/ericmikkelsen/.netlify/helper/git-config
credential.helper=
credential.helper=netlify
credential.helper=netlify

So, to summarize, your ordering is:

credential.helper=
credential.helper=netlify

This is because only the lines after the most recent empty string line matter. So, your credential helpers are not in order.

Would you please confirm your ~/.netlify/helper/git-config file contains just this below?

# The first line resets the list of helpers so we can check Netlify's first.
[credential]
  helper = ""

[credential]
  helper = netlify
  helper = osxkeychain

If so, please close your terminal window, open a new window, and then run git config -l again. If this ordering isn’t fixed or if the git push still doesn’t work, please let us know.

Also, if it is still not working, would you please send us the exact command being run and output of that command?

Hey @luke thanks for the walk deep dive I super appreciate it.

It looks like I’m missing osxkeychain in my helper file

# The first line resets the list of helpers so we can check Netlify's first.
[credential]
  helper = ""

[credential]
  helper = netlify

Here’s the total output when I try to git push origin trunk

batch response: Repository or object not found: https://8af78b07-6f98-4b04-9a19-2daa8040e79e.netlify.app/.netlify/large-media/objects/batch
Check that it exists and that you have proper access to it
batch response: Repository or object not found: https://8af78b07-6f98-4b04-9a19-2daa8040e79e.netlify.app/.netlify/large-media/objects/batch
Check that it exists and that you have proper access to it
batch response: Repository or object not found: https://8af78b07-6f98-4b04-9a19-2daa8040e79e.netlify.app/.netlify/large-media/objects/batch
Check that it exists and that you have proper access to it
batch response: Repository or object not found: https://8af78b07-6f98-4b04-9a19-2daa8040e79e.netlify.app/.netlify/large-media/objects/batch
Check that it exists and that you have proper access to it
batch response: Repository or object not found: https://8af78b07-6f98-4b04-9a19-2daa8040e79e.netlify.app/.netlify/large-media/objects/batch
Check that it exists and that you have proper access to it
Uploading LFS objects:   0% (0/101), 0 B | 0 B/s, done.
batch response: Repository or object not found: https://8af78b07-6f98-4b04-9a19-2daa8040e79e.netlify.app/.netlify/large-media/objects/batch
Check that it exists and that you have proper access to it
batch response: Repository or object not found: https://8af78b07-6f98-4b04-9a19-2daa8040e79e.netlify.app/.netlify/large-media/objects/batch
Check that it exists and that you have proper access to it
batch response: Repository or object not found: https://8af78b07-6f98-4b04-9a19-2daa8040e79e.netlify.app/.netlify/large-media/objects/batch
Check that it exists and that you have proper access to it
batch response: Repository or object not found: https://8af78b07-6f98-4b04-9a19-2daa8040e79e.netlify.app/.netlify/large-media/objects/batch
Check that it exists and that you have proper access to it
batch response: Repository or object not found: https://8af78b07-6f98-4b04-9a19-2daa8040e79e.netlify.app/.netlify/large-media/objects/batch
Check that it exists and that you have proper access to it
error: failed to push some refs to 'github.com:devboldium/ebce.org-11ty.git'

As for updating my gitconfig, should I do that by hand and same question for adding the osxkeychain to my netlify config.

Hi, @dev-boldium-ebce. Yes, if you are using MacOS, add this line to that file:

  helper = osxkeychain

Then close and reopen new terminal window. Then the git config -l output should show the updated configuration. (If not, please reply anytime.)

Hey @luke,

So I added helper osxkeychain to my ~/.netlify/helper/git-config
So it now looks like this

# The first line resets the list of helpers so we can check Netlify's first.
[credential]
  helper = ""

[credential]
  helper = netlify
  helper = osxkeychain

Then when I run git config -l
I get this

include.path=/Users/ericmikkelsen/.netlify/helper/git-config
credential.helper=
credential.helper=netlify
credential.helper=osxkeychain
filter.lfs.clean=git-lfs clean -- %f
filter.lfs.smudge=git-lfs smudge -- %f
filter.lfs.process=git-lfs filter-process
filter.lfs.required=true
include.path=/Users/ericmikkelsen/.netlify/helper/git-config
credential.helper=
credential.helper=netlify
credential.helper=osxkeychain

Then when I tried to push my branch using git push origin trunk
I got this again

batch response: Repository or object not found: https://8af78b07-6f98-4b04-9a19-2daa8040e79e.netlify.app/.netlify/large-media/objects/batch                   
Check that it exists and that you have proper access to it
batch response: Repository or object not found: https://8af78b07-6f98-4b04-9a19-2daa8040e79e.netlify.app/.netlify/large-media/objects/batch
Check that it exists and that you have proper access to it
Uploading LFS objects:   0% (0/100), 0 B | 0 B/s, done.
batch response: Repository or object not found: https://8af78b07-6f98-4b04-9a19-2daa8040e79e.netlify.app/.netlify/large-media/objects/batch
Check that it exists and that you have proper access to it
batch response: Repository or object not found: https://8af78b07-6f98-4b04-9a19-2daa8040e79e.netlify.app/.netlify/large-media/objects/batch
Check that it exists and that you have proper access to it
batch response: Repository or object not found: https://8af78b07-6f98-4b04-9a19-2daa8040e79e.netlify.app/.netlify/large-media/objects/batch
Check that it exists and that you have proper access to it
batch response: Repository or object not found: https://8af78b07-6f98-4b04-9a19-2daa8040e79e.netlify.app/.netlify/large-media/objects/batch
Check that it exists and that you have proper access to it
batch response: Repository or object not found: https://8af78b07-6f98-4b04-9a19-2daa8040e79e.netlify.app/.netlify/large-media/objects/batch
Check that it exists and that you have proper access to it
batch response: Repository or object not found: https://8af78b07-6f98-4b04-9a19-2daa8040e79e.netlify.app/.netlify/large-media/objects/batch
Check that it exists and that you have proper access to it
batch response: Repository or object not found: https://8af78b07-6f98-4b04-9a19-2daa8040e79e.netlify.app/.netlify/large-media/objects/batch
Check that it exists and that you have proper access to it
batch response: Repository or object not found: https://8af78b07-6f98-4b04-9a19-2daa8040e79e.netlify.app/.netlify/large-media/objects/batch
Check that it exists and that you have proper access to it
error: failed to push some refs to 'github.com:devboldium/ebce.org-11ty.git'

then when I run git config -l again I get this output

credential.helper=osxkeychain
user.email=eric.mikkelsen@boldium.com
include.path=/Users/ericmikkelsen/.netlify/helper/git-config
credential.helper=
credential.helper=netlify
credential.helper=osxkeychain
filter.lfs.clean=git-lfs clean -- %f
filter.lfs.smudge=git-lfs smudge -- %f
filter.lfs.process=git-lfs filter-process
filter.lfs.required=true
include.path=/Users/ericmikkelsen/.netlify/helper/git-config
credential.helper=
credential.helper=netlify
credential.helper=osxkeychain
core.repositoryformatversion=0
core.filemode=true
core.bare=false
core.logallrefupdates=true
core.ignorecase=true
core.precomposeunicode=true
remote.origin.url=git@github.com:devboldium/ebce.org-11ty.git
remote.origin.fetch=+refs/heads/*:refs/remotes/origin/*
branch.trunk.remote=origin
branch.trunk.merge=refs/heads/trunk
lfs.https://8af78b07-6f98-4b04-9a19-2daa8040e79e.netlify.app/.netlify/large-media.locksverify=false
credential.helper=netlify

and just for posterity here’s the output of my current netlify lm:info

  ✔ Checking Git version [2.28.0]
  ✔ Checking Git LFS version [2.12.0]
  ✔ Checking Git LFS filters
  ✔ Checking Netlify's Git Credentials version [0.1.9]

A post was split to a new topic: Git push issues with Git LFS and/or Large Media

Hi, @dev-boldium-ebce. It seems silly to ask but just to be sure, are you sure you are logged in (or using a token)? In other words, does the following command show you logged in as yourself?

netlify status

I’m guessing the answer is “yes” but I want to confirm to be sure. Assuming “yes”, would you please private message PM the output of the command below to me?

GIT_TRACE=true GIT_CURL_VERBOSE=true git lfs push --all origin

It is probably safe to share the output publicly. Again, I recommend PM-ing to be safe but please do whichever you prefer.

Now if, on the other hand, the output of netlify status does not show you logged in using the Netlify CLI, that is a problem and we will need to start there first. If that is happening, please let us know.

Hey @luke it looks like I’m logged in, so I’ll private message you, Here’s the output of netlify status

──────────────────────┐
 Current Netlify User │
──────────────────────┘
Name:  dev-boldium
Email: dev+ebce@boldium.com
Teams: 
  dev-ebce's team: Collaborator 

Hi, @dev-boldium-ebce. I’m going to share a bit of output here in two examples of the GIT_TRACE=true GIT_CURL_VERBOSE=true git lfs push --all origin.

This first example is a working push from my system (and don’t worry, that isn’t my real password in the logs).

Working example:

20:26:42.069608 trace git-lfs: tq: sending batch of size 4
20:26:42.070874 trace git-lfs: api: batch 4 files
20:26:42.071024 trace git-lfs: creds: git credential fill ("https", "fe21b1ce-1d9a-4899-bbff-a0d7bcb55ad0.netlify.app", "")
20:26:42.075218 git.c:444               trace: built-in: git credential fill
20:26:42.076172 run-command.c:663       trace: run_command: 'git credential-netlify get'
20:26:42.087781 git.c:704               trace: exec: git-credential-netlify get
20:26:42.088516 run-command.c:663       trace: run_command: git-credential-netlify get
time="2020-10-15T20:26:42-07:00" level=debug msg="Initializing Netlify credential helper" args="[/Users/username/.netlify/helper/bin/git-credential-netlify get]"
time="2020-10-15T20:26:42-07:00" level=debug msg="Git input received" host=fe21b1ce-1d9a-4899-bbff-a0d7bcb55ad0.netlify.app protocol=https
time="2020-10-15T20:26:42-07:00" level=debug msg="Writing output data" host=fe21b1ce-1d9a-4899-bbff-a0d7bcb55ad0.netlify.app password="12345" protocol=https username=access-token
20:26:42.570702 trace git-lfs: Filled credentials for https://fe21b1ce-1d9a-4899-bbff-a0d7bcb55ad0.netlify.app/.netlify/large-media
20:26:42.571156 trace git-lfs: exec: /usr/bin/security 'list-keychains'
20:26:42.594467 trace git-lfs: exec: /usr/bin/security 'find-certificate' '-a' '-p' '-c' 'fe21b1ce-1d9a-4899-bbff-a0d7bcb55ad0.netlify.app' '/Library/Keychains/System.keychain'
20:26:42.621249 trace git-lfs: HTTP: POST https://fe21b1ce-1d9a-4899-bbff-a0d7bcb55ad0.netlify.app/.netlify/large-media/objects/batch

Those logs contain lines referencing the Git credential helper (“git-credential-netlify”):

20:26:42.076172 run-command.c:663       trace: run_command: 'git credential-netlify get'
20:26:42.087781 git.c:704               trace: exec: git-credential-netlify get
20:26:42.088516 run-command.c:663       trace: run_command: git-credential-netlify get
time="2020-10-15T20:26:42-07:00" level=debug msg="Initializing Netlify credential helper" args="[/Users/luke/.netlify/helper/bin/git-credential-netlify get]"

Here is a sample of the output you sent me (and your doesn’t contain a password line:

21:56:30.319299 trace git-lfs: tq: running as batched queue, batch size of 100
21:56:30.320097 trace git-lfs: run_command: git rev-list --objects --do-walk --stdin --
21:56:30.323175 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'cat-file' '--batch-check'
21:56:30.326077 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' '--git-common-dir'
21:56:30.352981 trace git-lfs: tq: sending batch of size 100
21:56:30.354457 trace git-lfs: api: batch 100 files
21:56:30.354871 trace git-lfs: exec: /usr/bin/security 'list-keychains'
21:56:30.378748 trace git-lfs: exec: /usr/bin/security 'find-certificate' '-a' '-p' '-c' '8af78b07-6f98-4b04-9a19-2daa8040e79e.netlify.app' '/Library/Keychains/System.keychain'
21:56:30.404320 trace git-lfs: HTTP: POST https://8af78b07-6f98-4b04-9a19-2daa8040e79e.netlify.app/.netlify/large-media/objects/batch

We know the issue is definitely that your Git client is not using the credential helper. Now we need to figure out why.

If you run git version does it output 2.28.0? (I suspect it does.)

I think the real clue is that the credential.helper lines change after you try the git push. They effectively become this:

credential.helper=
credential.helper=netlify
credential.helper=osxkeychain
credential.helper=netlify

I think this is causing the “netlify” helper to be used last or not at all but this is just a guess.

Would you please run the following command and share the output here?

 git help -a | grep credential-

For example, this is what I see:

$ git help -a | grep credential-
   credential-cache     Helper to temporarily store passwords in memory
   credential-store     Helper to store credentials on disk
   credential-gcloud.sh
   credential-netlify

This will tell us if it is an ordering issue or if the credential-netlify helper is missing entirely.

I work at an agency and I’m under a time crunch and so the feedback rounds being a whole day between messages are just too much. We’ve decided to go in a different direction and move our media hosting to Cloudinary instead. Thanks for all the work, I’m sure given enough time we would’ve figured it out.

Hi, @dev-boldium-ebce. If you want to troubleshoot this further in the future, please reply here anytime and we will be happy to do so.