Support Forums

Large Media: git lfs push fails with 503

Site name: https://bruggisserpartner-ch.netlify.app/

I set up LFS properly which is confirmed by:

$ netlify lm:info                                               
  ✔ Checking Git version [2.33.1]
  ✔ Checking Git LFS version [3.0.2]
  ✔ Checking Git LFS filters
  ✔ Checking Netlify's Git Credentials version [0.1.10]

When trying to push I get this:

$ git push
Enumerating objects: 65, done.
Counting objects: 100% (65/65), done.
Delta compression using up to 12 threads
Compressing objects: 100% (39/39), done.
Writing objects: 100% (48/48), 4.96 KiB | 725.00 KiB/s, done.
Total 48 (delta 17), reused 0 (delta 0), pack-reused 0
remote: GitLab: LFS objects are missing. Ensure LFS is properly set up or try a manual "git lfs push --all".
To gitlab.com:SITE-ID.git
 ! [remote rejected] master -> master (pre-receive hook declined)
error: failed to push some refs to 'gitlab.com:

So I ran this for some debugging information:

$ GIT_TRACE=true GIT_CURL_VERBOSE=true git lfs push --all origin
11:49:54.553176 git.c:743               trace: exec: git-lfs push --all origin
11:49:54.553201 run-command.c:666       trace: run_command: git-lfs push --all origin
11:49:54.563922 trace git-lfs: exec: git 'version'
11:49:54.568718 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'remote'
11:49:54.570378 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' 'HEAD' '--symbolic-full-name' 'HEAD'
11:49:54.572112 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' '--git-dir' '--show-toplevel'
11:49:54.573611 trace git-lfs: exec: git 'config' '--includes' '-l'
11:49:54.574757 trace git-lfs: exec: git 'rev-parse' '--is-bare-repository'
11:49:54.575794 trace git-lfs: exec: git 'config' '--includes' '-l' '-f' '/.lfsconfig'
11:49:54.578883 trace git-lfs: Upload refs [] to remote origin
11:49:54.579115 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'show-ref'
11:49:54.584160 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'show-ref'
11:49:54.588887 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'ls-remote' '--heads' '--tags' '-q' 'origin'
11:49:56.160522 trace git-lfs: tq: running as batched queue, batch size of 100
11:49:56.160864 trace git-lfs: run_command: git rev-list --objects --do-walk --stdin --
11:49:56.161928 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'cat-file' '--batch-check'
11:49:56.162879 trace git-lfs: exec: git '-c' 'filter.lfs.smudge=' '-c' 'filter.lfs.clean=' '-c' 'filter.lfs.process=' '-c' 'filter.lfs.required=false' 'rev-parse' '--git-common-dir'
11:49:56.173925 trace git-lfs: tq: sending batch of size 1                                                                                                                                    
11:49:56.174133 trace git-lfs: api: batch 1 files
11:49:56.174441 trace git-lfs: HTTP: POST https://SITE-ID.netlify.app/.netlify/large-media/objects/batch
> POST /.netlify/large-media/objects/batch HTTP/1.1
> Host: SITE-ID.netlify.app
> Accept: application/vnd.git-lfs+json; charset=utf-8
> Content-Length: 229
> Content-Type: application/vnd.git-lfs+json; charset=utf-8
> User-Agent: git-lfs/3.0.2 (GitHub; linux amd64; go 1.16.8)
{"operation":"upload","objects":[{"oid":"a30d0b136e98fc9162ccacf080a3aed30d94c9ef677fa5cd632522ff2d8faee5","size":505185}],"transfers":["lfs-standalone-file","basic","ssh"],"ref":{"name":"refs/heads/master"},"hash_algo":"sha256"}11:50:24.319273 trace git-lfs: HTTP: 503

< HTTP/2.0 503 Service Unavailable
< Content-Length: 0
< Age: 28
< Date: Thu, 16 Dec 2021 10:50:24 GMT
< Server: Netlify
11:50:24.319530 trace git-lfs: api error: Fatal error: Server error: https://SITE-ID.netlify.app/.netlify/large-media/objects/batch from HTTP 503
batch response: Fatal error: Server error: https://SITE-ID.netlify.app/.netlify/large-media/objects/batch from HTTP 503
Uploading LFS objects:   0% (0/1), 0 B | 0 B/s, done.

Errors logged to /.git/lfs/logs/20211216T115024.319778833.log
Use `git lfs logs last` to view the log.

I did a successful deployment before running all those commands.

Ok I found the issue. I forgot to disable LFS in Gitlab.

Hi, @luksak. Thank you for taking the time to post the solution you discovered. We (the Netlify support team) appreciate it and people searching here in the future will likely appreciate the answer even more than we do!

Thanks again for the follow-up! :+1: