`git push` and `git lfs push origin master` keeps failing

I keep running git push on my GitHub repository with Netlify Large Media enabled, but the command fails with multiple messages saying:

batch response: Expected json type, got: "text/plain; charset=utf-8"

I was able to incrementally push 80% of lfs tracked files up to my large media object while getting this error message. Now it stopped working and every time I run git push I get:

Uploading LFS objects:   0% (0/100), 0 B | 0 B/s, done.
batch response: Expected json type, got: "text/plain; charset=utf-8"
batch response: Expected json type, got: "text/plain; charset=utf-8"
batch response: Expected json type, got: "text/plain; charset=utf-8"
...
batch response: Expected json type, got: "text/plain; charset=utf-8"
batch response: Expected json type, got: "text/plain; charset=utf-8"
batch response: Expected json type, got: "text/plain; charset=utf-8"
error: failed to push some refs to 'git@github.com:<ORG>/<REPO_NAME>.git'

I was wondering if others are also experiencing similar issues.

If it helps, I got this log once when my git push failed:

git-lfs/2.9.2 (GitHub; windows amd64; go 1.12.7; git 0274d856)
git version 2.25.0.windows.1

$ git-lfs pre-push origin git@github.com:[GIT_USER]/[REPO].git
batch response: Expected json type, got: "text/plain; charset=utf-8"
batch response: Expected json type, got: "text/plain; charset=utf-8"
batch response: Expected json type, got: "text/plain; charset=utf-8"
batch response: Expected json type, got: "text/plain; charset=utf-8"
batch response: Expected json type, got: "text/plain; charset=utf-8"
batch response: Expected json type, got: "text/plain; charset=utf-8"
batch response: Expected json type, got: "text/plain; charset=utf-8"
batch response: Expected json type, got: "text/plain; charset=utf-8"
batch response: Expected json type, got: "text/plain; charset=utf-8"
batch response: Expected json type, got: "text/plain; charset=utf-8"
batch response: Expected json type, got: "text/plain; charset=utf-8"
batch response: Expected json type, got: "text/plain; charset=utf-8"
batch response: Expected json type, got: "text/plain; charset=utf-8"
batch response: Expected json type, got: "text/plain; charset=utf-8"
batch response: Fatal error: Server error: https://[SITE_ID].netlify.com/.netlify/large-media/objects/batch

Server error: https://[SITE_ID].netlify.com/.netlify/large-media/objects/batch
github.com/git-lfs/git-lfs/errors.Errorf
	d:/a/git-lfs/git-lfs/errors/errors.go:69
github.com/git-lfs/git-lfs/lfshttp.defaultError
	d:/a/git-lfs/git-lfs/lfshttp/errors.go:128
github.com/git-lfs/git-lfs/lfshttp.(*Client).handleResponse
	d:/a/git-lfs/git-lfs/lfshttp/errors.go:51
github.com/git-lfs/git-lfs/lfshttp.(*Client).DoWithRedirect
	d:/a/git-lfs/git-lfs/lfshttp/client.go:311
github.com/git-lfs/git-lfs/lfsapi.(*Client).doWithCreds
	d:/a/git-lfs/git-lfs/lfsapi/auth.go:101
github.com/git-lfs/git-lfs/lfsapi.(*Client).doWithAuth
	d:/a/git-lfs/git-lfs/lfsapi/auth.go:67
github.com/git-lfs/git-lfs/lfsapi.(*Client).DoWithAuth
	d:/a/git-lfs/git-lfs/lfsapi/auth.go:25
github.com/git-lfs/git-lfs/lfsapi.(*Client).DoAPIRequestWithAuth
	d:/a/git-lfs/git-lfs/lfsapi/auth.go:56
github.com/git-lfs/git-lfs/tq.(*tqClient).Batch
	d:/a/git-lfs/git-lfs/tq/api.go:74
github.com/git-lfs/git-lfs/tq.Batch
	d:/a/git-lfs/git-lfs/tq/api.go:40
github.com/git-lfs/git-lfs/tq.(*TransferQueue).enqueueAndCollectRetriesFor
	d:/a/git-lfs/git-lfs/tq/transfer_queue.go:520
github.com/git-lfs/git-lfs/tq.(*TransferQueue).collectBatches.func1
	d:/a/git-lfs/git-lfs/tq/transfer_queue.go:432
runtime.goexit
	C:/Go1.12.7/src/runtime/asm_amd64.s:1337
Fatal error
github.com/git-lfs/git-lfs/errors.newWrappedError
	d:/a/git-lfs/git-lfs/errors/types.go:198
github.com/git-lfs/git-lfs/errors.NewFatalError
	d:/a/git-lfs/git-lfs/errors/types.go:242
github.com/git-lfs/git-lfs/lfshttp.(*Client).handleResponse
	d:/a/git-lfs/git-lfs/lfshttp/errors.go:75
github.com/git-lfs/git-lfs/lfshttp.(*Client).DoWithRedirect
	d:/a/git-lfs/git-lfs/lfshttp/client.go:311
github.com/git-lfs/git-lfs/lfsapi.(*Client).doWithCreds
	d:/a/git-lfs/git-lfs/lfsapi/auth.go:101
github.com/git-lfs/git-lfs/lfsapi.(*Client).doWithAuth
	d:/a/git-lfs/git-lfs/lfsapi/auth.go:67
github.com/git-lfs/git-lfs/lfsapi.(*Client).DoWithAuth
	d:/a/git-lfs/git-lfs/lfsapi/auth.go:25
github.com/git-lfs/git-lfs/lfsapi.(*Client).DoAPIRequestWithAuth
	d:/a/git-lfs/git-lfs/lfsapi/auth.go:56
github.com/git-lfs/git-lfs/tq.(*tqClient).Batch
	d:/a/git-lfs/git-lfs/tq/api.go:74
github.com/git-lfs/git-lfs/tq.Batch
	d:/a/git-lfs/git-lfs/tq/api.go:40
github.com/git-lfs/git-lfs/tq.(*TransferQueue).enqueueAndCollectRetriesFor
	d:/a/git-lfs/git-lfs/tq/transfer_queue.go:520
github.com/git-lfs/git-lfs/tq.(*TransferQueue).collectBatches.func1
	d:/a/git-lfs/git-lfs/tq/transfer_queue.go:432
runtime.goexit
	C:/Go1.12.7/src/runtime/asm_amd64.s:1337
batch response
github.com/git-lfs/git-lfs/errors.newWrappedError
	d:/a/git-lfs/git-lfs/errors/types.go:198
github.com/git-lfs/git-lfs/errors.Wrap
	d:/a/git-lfs/git-lfs/errors/errors.go:74
github.com/git-lfs/git-lfs/tq.(*tqClient).Batch
	d:/a/git-lfs/git-lfs/tq/api.go:77
github.com/git-lfs/git-lfs/tq.Batch
	d:/a/git-lfs/git-lfs/tq/api.go:40
github.com/git-lfs/git-lfs/tq.(*TransferQueue).enqueueAndCollectRetriesFor
	d:/a/git-lfs/git-lfs/tq/transfer_queue.go:520
github.com/git-lfs/git-lfs/tq.(*TransferQueue).collectBatches.func1
	d:/a/git-lfs/git-lfs/tq/transfer_queue.go:432
runtime.goexit
	C:/Go1.12.7/src/runtime/asm_amd64.s:1337

Current time in UTC: 
2020-03-03 06:41:23

ENV:
LocalWorkingDir=path/to/working-dir
LocalGitDir=path/to/git-dir
LocalGitStorageDir=path/to/git-storage-dir
LocalMediaDir=path/to/media-dir
LocalReferenceDirs=
TempDir=path/to/temp-dir
ConcurrentTransfers=3
TusTransfers=false
BasicTransfersOnly=false
SkipDownloadErrors=false
FetchRecentAlways=false
FetchRecentRefsDays=7
FetchRecentCommitsDays=0
FetchRecentRefsIncludeRemotes=true
PruneOffsetDays=3
PruneVerifyRemoteAlways=false
PruneRemoteName=origin
LfsStorageDir=path/to/lfs-storage-dir
AccessDownload=basic
AccessUpload=basic
DownloadTransfers=basic,lfs-standalone-file
UploadTransfers=basic,lfs-standalone-file
GIT_EXEC_PATH=path/to/git-exec
GIT_PREFIX=

I’d be happy to share the SITE_ID on DM with the staff.

This is a sample of the type of responses I get from Netlify Large Media servers when I run GIT_CURL_VERBOSE=1 git push:

12:02:05.557263 trace git-lfs: tq: sending batch of size 100
12:02:05.558263 trace git-lfs: api: batch 100 files
12:02:05.558263 trace git-lfs: creds: git credential cache ("https", "[SITE_ID].netlify.com", "")
12:02:05.559263 trace git-lfs: Filled credentials for https://[SITE_ID].netlify.com/.netlify/large-media
12:02:05.559263 trace git-lfs: HTTP: POST https://[SITE_ID].netlify.com/.netlify/large-media/objects/batch

> POST /.netlify/large-media/objects/batch HTTP/1.1
> Host: [SITE_ID].netlify.com
> Accept: application/vnd.git-lfs+json; charset=utf-8
> Authorization: Basic * * * * *
> Content-Length: 8914
> Content-Type: application/vnd.git-lfs+json; charset=utf-8
> User-Agent: git-lfs/2.9.2 (GitHub; windows amd64; go 1.12.7; git 0274d856)
>
{
   "operation":"upload",
   "objects": [
      {
         "oid": "[HASH_VALUE]",
         "size": [SIZE]
      }, 
      ...
   ],
   "transfers":[ "lfs-standalone-file", "basic" ],
   "ref": { "name": "refs/heads/master" }
}

12:02:05.675270 trace git-lfs: HTTP: 200

< HTTP/2.0 200 OK
< Content-Length: 1346
< Age: 0
< Cache-Control: public, max-age=0, must-revalidate
< Content-Type: text/plain; charset=utf-8
< Date: Tue, 03 Mar 2020 20:02:05 GMT
< Server: Netlify
< X-Bb-Proxy: https://lm.services.netlify.com/lfs/objects/batch
< X-Bb-Proxy-Version: v2
< X-Nf-Request-Id: 02187845-bfb4-4242-ad90-b7c7169d9390-660364
< X-Robots-Tag: noindex
<

I deleted my site and recreated it again. The same thing happens again. Although I got a little more detailed message for one of my early failed git push commands:

Uploading LFS objects:  37% (1094/2968), 220 MB | 767 KB/s, done.
Repository or object not found: https://[SITE_ID].netlify.com/.netlify/large-media/verify
Check that it exists and that you have proper access to it
Repository or object not found: https://[SITE_ID].netlify.com/.netlify/large-media/verify
Check that it exists and that you have proper access to it
Post https://[SITE_ID].netlify.com/.netlify/large-media/verify: stream error: stream ID 17; NO_ERROR
Repository or object not found: https://[SITE_ID].netlify.com/.netlify/large-media/verify
Check that it exists and that you have proper access to it
Post https://[SITE_ID].netlify.com/.netlify/large-media/verify: http2: server sent GOAWAY and closed the connection; LastStreamID=17, ErrCode=ENHANCE_YOUR_CALM, debug=""
Repository or object not found: https://[SITE_ID].netlify.com/.netlify/large-media/verify
Check that it exists and that you have proper access to it

batch response: Expected json type, got: "text/plain; charset=utf-8"
batch response: Expected json type, got: "text/plain; charset=utf-8"
...
batch response: Expected json type, got: "text/plain; charset=utf-8"
error: failed to push some refs to 'git@github.com:[ORG]/[REPO].git'

I was able to push all my lfs tracked files to Netlify Large Media servers by splitting my big commit into smaller ones with much lower payloads.

I would still get the same kind of errors but only once per single commit push.

@saw-mon_and_natalie, thank you for sharing this detailed explanation and solution.

I’m asking our developers about this rate limit issue and we’ll update this topic again as soon as we have more information about this.

1 Like

@saw-mon_and_natalie, would you please send us that site id? We’ll be able to research the rate limit with that information.

It is safe to post the site ids publicly.

If you prefer to send that privately, please feel free to private message (PM) any of us on the Netlify support team with that as well (me for example). PMs are enabled for your community login. I double checked to be sure.

1 Like

@luke, I just DMed you my site ids.

Thank you for that information, @saw-mon_and_natalie. I’ve shared the site ids with our Netlify Large Media developers and I’ll follow-up here when we know more.

1 Like

Hi @luke, thought I’d chime in as I’m facing the same issue I think. Even when trying to push a commit with just a single media asset.
The Netlify site ID is: bf362332-1bee-4e44-81e4-b6ebfb018ae6.netlify.app in case it helps.

Thanks in advance!

@ketch, I think you might be experiencing a different issue. Would you please test the solution in this post?

If you are not have the same issue as you see there, then would you please direct message me the output of the following command?

GIT_TRACE=true GIT_CURL_VERBOSE=true git lfs push --all origin

Again though, please DM that to me instead of posting that publicly - just in case. First please check the linked solution above though, as I strongly suspect that is the issue.

@saw-mon_and_natalie, I never followed up on this.

Our developers did oddities in the request pathing through our infrastructure at the time. However, my understanding is that you resolved this the rate limiting with smaller batches of commits.

Are there still unanswered questions or unresolved issues regarding Large Media for you, @saw-mon_and_natalie? I know there were several topics about different aspects of Large Media.

@luke, The errors that I got didn’t specifically include a rate limit error. Only the servers didn’t return the right response format.

Also, when I tried git push with smaller batches I would still sometimes get the same errors. But after trying once or twice more it would pass. I guess the probability of the server handling the large media with an error was less with a smaller batch.

I haven’t tried the #netlify-large-media-nlm feature since then. I was able to upload all my large media assets around the time I was seeking help here.

Thanks for the follow-up, @saw-mon_and_natalie. If there are future questions about Large Media or anything else Netlify related, I assure you that I’ll be faster to respond to those questions going forward.

@ketch, I’m also happy to assist if you are still having issues.

Hi, @ketch, I got your DM but I’m not able to provide assistance via DM. (We can take the conversation private if we need to share sensitive information but there is nothing sensitive/private in what follows.)

Are you seeing version 0.1.9 of the credential manager when you run netlify lm:info as shown below?

✔ Checking Netlify's Git Credentials version [0.1.9]

If not, the solution below works for most people:

npm i -g netlify-cli
netlify plugins:install netlify-lm-plugin
netlify lm:install
. ~/.netlify/helper/path.bash.inc
netlify lm:info

The final command should now show something like this:

$ netlify lm:info
  ✔ Checking Git version [2.26.2]
  ✔ Checking Git LFS version [2.10.0]
  ✔ Checking Git LFS filters
  ✔ Checking Netlify's Git Credentials version [0.1.9]

The key here is that the final line should say “0.1.9” and not some other version.

If this doesn’t resolve the issue, please let us know.

Thanks @Luke!
That was the problem, I had version 0.1.8 it seems despite re-installing the netlify-cli (I think).
I definitely know that I hadn’t run netlify plugins:install netlify-lm-plugin in a long while though, so could be that plugin was outdated?

Either way, it seems to be working much better now.
I do still get an error towards the end of a git push:

Authentication required: Authorization error: https://70a76375-6c9c-4aa6-b0cf-e19b4d3d0d16.netlify.app/.netlify/large-media/verify
Check that you have proper access to the repository
error: failed to push some refs to 'git@github.com:REPO'

But re-running git push shows that no files need to be uploaded to the LFS, and the command succeeds. Looking through the repo also suggests that the command worked as intended, with the media assets being pointers instead.

If there are still “Authentication required” messages, I would recommend checking the following next.

  1. Does the line . ~/.netlify/helper/path.bash.inc appear in .bash_profile (or equivalent file like . zshenv )? If not, try adding it there and opening a new terminal window.

  2. Have you logged out and back into Netlify CLI since updating? If not, try this:

netlify logout
netlify login

If git push still shows “Authentication required” after the two steps above, please let us know.

Hi @luke

Just following up with this one. I’m seeing similar issues when trying to push up images.

batch response: Repository or object not found: https://a34afbff-e858-4f3b-ba6d-3b25ef7db012.netlify.app/.netlify/large-media/objects/batch
Check that it exists and that you have proper access to it
Uploading LFS objects: 0% (0/29), 0 B | 0 B/s, done.
error: failed to push some refs to ‘git@github.com:teeerevor/quok.in.git’

I’ve followed this one and all looks fine

path :white_check_mark:
logout and In :white_check_mark:

Any other ideas?

updating to latest netlify-cli sorted my issue.

Thank you so much for feeding back, it’s always appreciated! Happy to hear you’ve resolved it.

1 Like