Deploys are getting stuck with just a "Creating deploy upload records" message

hi there, i have moved this post to this thread as it is the same error message.

Can you review the notes above and see what/if any apply to you? thanks!

We have a similar issue at the moment, however, we’re building through Gatsby Cloud and the issue resolves after about 4.5 minutes and deploys like normal. I’d imagine it’s quite likely they’re using the same library to make these calls, but it would have a different user-agent (presumably).

The builds aren’t big (1 minute for a full build), so for it to take 5 minutes to deploy is strange.

Gatsby cloud has been informed how to make their deploys more reliable; it’s up to them to implement the advice we gave. The advice was to undo the modifications in their use of our API client which should make things rather stable for their deploys.

Sadly, I don’t think the above is a help with our issue since we’re not using the netlify-cli, I’ve included our circleci config.yml file below:

   defaults: &defaults
  docker:
    - image: circleci/node:13
  working_directory: ~/repo

restore_cache: &restore_cache
  restore_cache:
    keys:
      - v1-dependencies-{{ checksum "package.json" }}
      - v1-dependencies-

version: 2.1
orbs:
  slack: circleci/slack@3.4.2

jobs:
  prepare:
    <<: *defaults
    steps:
      - checkout
      - <<: *restore_cache
      - run: yarn install
      - save_cache:
          paths:
            - node_modules
          key: v1-dependencies-{{ checksum "package.json" }}

  build-deploy-dev:
    <<: *defaults
    steps:
      - checkout
      - <<: *restore_cache
      - run: yarn build:dev
      - run: yarn deploy --site 91262ffb-3562-4e2e-994c-a59d3f399648 --auth $NETLIFY_ACCESS_TOKEN
      - slack/status

  build-deploy-prod:
    <<: *defaults
    steps:
      - checkout
      - <<: *restore_cache
      - run: yarn build:prod
      - run: yarn deploy --site e0f0c947-2260-439f-a0ba-da93edf2c786 --auth $NETLIFY_ACCESS_TOKEN
      - slack/status

workflows:
  version: 2
  build_and_deploy:
    jobs:
      - prepare:
          filters:
            branches:
              only: master
      - build-deploy-dev:
          context: NETLIFY
          requires:
            - prepare
      - hold:
          type: approval
          requires:
            - build-deploy-dev
      - build-deploy-prod:
          context: NETLIFY
          requires:
            - hold

Any help would be much appreciated. Interestingly, I just bumped the version down to 2 rather than 2.1 and then straight afterwards the 2 deploys I did straight after went through completely fine. But now its reverted back to failing again.

@fool i wrote the action, it’s pretty vanilla, it basically just calls client.deploy() with a bunch of options mostly passed in as environment variables from the github action workflow. you can see the source code here: friends-library/index.ts at master · friends-library-dev/friends-library · GitHub

I’m certainly not setting the user-agent myself. Maybe github automatically changes the user agent for all outbound http requests sent out of an action? I have no idea…

Can you tell me from looking at the code in my action if I’m doing anything wrong? Or what I could do to get my deploys to not hang in this manner?

Do you think it’s a rate-limiting issue? If i got several access tokens from Netlify and used one per site that deploys from my github actions, would that keep them from getting rate limited when more than one deploy is happening at once?

as always, really appreciate the help!

@fool @perry all my deploys from github actions (using the netlify node client) still failing today. is there any more you guys know about this issue? this same technique was working very reliably for months until yesterday, they all just keep hanging. i can do a manual deploy from my computer, but i’ve got my whole system built around ci/cd using github actions.

here’s another stuck deploy id: 5f5a5e5defce8f261a9e3941

Would love any help you guys could give. :pray:

(I also replied to @fool above with some more details about the deployment, and the a link to the source code).

I’m also getting these a LOT since some time. Was anything changed? I’m also uploading directly through the API and retrying all 429 (getting a lot of 429s). What should a well behaved client do other than retry?

And is it the correct behavior that the deploy just gets stuck at “uploading”?

Here is a thread with more info: API Uploads (files only): Deploys get stuck in state: "uploading"

Hey all! Just to let you know, today, this was discussed with our backend team to investigate further. Once we know more or have further Qs, we’ll get back!

Ok, please do keep us updated if you find out anything. The really odd thing is that for me it always runs successfully when we rerun the flow on Circle CI from the failed point. Feel free to check out my recent deploys to see if your BE team can spot any patterns.

Hi, @tech1. I noticed that the both the sites you mentioned are successfully deploying since about 8 am PDT on 2020-09-11.

Did you make some change do how you are deploying. We are investigating if this is being caused by a specific version of the netlify or netlify-cli NPM packages:

Did you update one of those to resolve this? If not, would you be willing to share what you are doing differently now?

Hi Folx,

We are also experiencing this hanging when using netlify-cli in our CI-CD across quite a few of our sites. We are not using Gatsby Cloud, just vanilla netlify-cli 2.54.0.

@luke still can’t get any deploy to go through from Github Actions, it’s been over a week since they started failing. Do you have any updates? Has your team found anything? Here’s another stuck deploy, if that helps… 5f6102ac6d3cd401468e63e0

Appreciate any help you can give!

To recap, I’m using the official node.js api client, deploying from a github action. This method had been working almost perfectly for a long time until early last week.

hi there, first up I’m sorry this has been a challenging thing to work through.

I’ve talked to a few different people here and we strongly believe this issue is caused by not having the newest js client or version of the CLI.

In the case of a github action, it would be the action that would need to have access to the newest version of the client and the cli. As we didn’t write or provide the action, we’re unable to really intervene here, but we’re happy to assist if this doesn’t help for some reason!

@perry I wrote the github action, so I bumped the netlify/js-client version from 4.3.0 to 4.3.13. Preliminary results are good, I did a test deploy and it went through – first one in quite a while. Hopefully that’s all it was.

fantastic news jared - we’re all pretty relieved we were able to figure out the problem, as we were collectively scratching our heads for a while. Kudos go mostly to @luke for his detective work.

thank you for your patience on this, and do let us know if something stops working.

1 Like

From yesterday, my drag and drop deploy in netlify is giving me the same error, it get stuck at Creating deploy upload records

Hey there, @sedghi :wave:

Thanks for reaching out about this! It looks like you have posted similar questions on two threads. We ask that all related questions stay on one thread. This helps us streamline support to you and keep relevant information in one place. Let’s continue this conversation on the following thread you posted on:

Any one finding this post later, I ended up using the netlify cli netlify depoy and it worked fine, for some reason the drag and drop did not work for me (was working before)

1 Like

Hi @sedghi - The issue impacting the Drag & Drop deployment method has been resolved. Please do attempt to deploy via Drag & Drop at your earliest convenience.

You can read more details and continue the conversation here: FIXED!: Drag & Drop Issue Status & Workaround