Automatic prod branch builds for monorepo nextjs app do not update code

Site name: priceless-volhard-14b73c

Automated builds from the prod branch for my site (main branch) do not update the code running on the branch endpoint. The build will successfully trigger, and will succeed & publish, however the code running is not the correct revision of the code.

Running a ntl deploy --build will successfully incorporate the latest changes.

How can I debug this issue? I’ve tried clearing the cache, updating the netlify.toml, neither of which have had an effect. At this point I’m considering disabling manual builds and just running my own build system using the ntl cli.

Hey @jlhasson,

At the moment, I can see that you’ve published an older deploy:

Is this expected to you?

Hey hrishikesh, yes, after being unable to get the code to update based on the automatic deploys I ran ntl deploy --build --prod from my branch which correctly updated the code. Previous branch deploys from main did not update with the latest code, therefore I could not publish them

Hey @jlhasson,

Could you please show us what are the “expected” changes vs how exactly it’s looking? It can help us see the problem.

Hey Hrishikesh, it’s difficult for me to describe the expected change without exposing parts of our code. Essentially in our main branch we updated some code, let’s say it was something like obj.some_attribute. In commits prior to this change we would see an error like Exception: obj has no attribute some_attribute. In commits after this change this error went away (because we changed the code to prevent it). I was able to build locally (i.e. yarn install && yarn dev) and see that the new changed worked correctly. Netlify would also trigger builds for the main branch. However, upon accessing the site at http://main--priceless-volhard-14b73c.netlify.com I would see the exception described above, thus indicating that although Netlify had triggered a build the code on the site did not reflect that in main. Does that make sense?

It’s probably too difficult to debug, and I’ve resolved it by manually pushing via ntl deploy --build --prod.

I’m not seeing the error that you mention in the console. In any case, I wonder why you would need to show us your code to show some changes. You could add some simple words or piece of text to make it obvious that something changed.

I’m now seeing the issue again, let me try to describe in detail what I see in a way that you can reproduce/understand:

Between deploys Netlify App (currently “published”) and Netlify App (most recent main branch deploy) I made a code change, the change is to update the expected response code from an API call:

-    if (res.status !== 200) {
+    if (res.status !== 201) {

When I run yarn dev locally and make the API call above, this condition succeeds, i.e. the condition correctly gets a 201 response & continues. In the branch deploy for this commit (found here: https://main--priceless-volhard-14b73c.netlify.app) the condition above fails, as it is still looking for a 200 response thus indicating that the code has not changed. This to me indicates that the deployed code does not contain the above change as expected indicating that the branch deploy did not include the change for some reason.

Is there a way to view the code that is deployed? If that were possible I could verify what I expect to be deployed vs what is actually deployed.

Hi @jlhasson,

We discussed this with the team and on the links you shared, we cannot see any asset being served with a 201 response code. I would assume, this is why the condition doesn’t get triggered.

I’d like to ask again, could you please make a visible change that is obvious without we having to dive into network tools and inspect? You could add something simple as a console.log() if you don’t want the site to have some changes.

Lastly, you can download your deploy from the deploy page or preview files using:

https://netlify-file-browser.netlify.app/

Hey @hrishikesh , I’ve followed your request. Can you PTAL at the following info and let me know if it helps you debug?

I’ve introduced a one line change in my code adding a console.log in the top level component (we are a React app on next.js). The log reads:

$ git diff HEAD~2 HEAD~1
+  console.log("main component rendered!");

The main branch deploy triggered and the build is here: Netlify App

The resulting site can be found here: Kailua Labs Sandbox

Please find that upon opening this site you do not see the console.log statement in the console.

I have created a dev deployment based on main by doing the following:

$ git pull origin main
$ cd web/content-enrichment # site base directory as configured in build settings https://app.netlify.com/sites/priceless-volhard-14b73c/settings/deploys#build-settings
$ ntl dev --live
...
◈ Functions server is listening on 43235                                                                                   [45/1652]
◈ Starting Netlify Dev with Next.js
yarn run v1.22.15
warning ../../package.json: No license field
$ next dev
ready - started server on 0.0.0.0:3000, url: http://localhost:3000
◈ Creating Live Tunnel for 5d3e0de5-a3ae-4830-aadc-b1067ddde7a3
event - compiled client and server successfully in 1793 ms (1866 modules)

   ┌────────────────────────────────────────────────────────────────────────────────┐
   │                                                                                │
   │   ◈ Server now ready on https://priceless-volhard-14b73c-8f9523.netlify.live   │
   │                                                                                │
   └────────────────────────────────────────────────────────────────────────────────┘

You can find the resulting site here: Kailua Labs Sandbox

Please note that upon the page loading you will see the console log “main component rendered!” in the console.

Is this info sufficient for you team to debug why the code is not updating? This one issue is making our team consider other platforms since our builds do not update deterministically.

Also happy to hop on a call to help debug if that’s possible/necessary.

@hrishikesh the dev server (with the console.log change) lost connection at some point, I restarted it, the link should still be the same: Kailua Labs Sandbox

Hey @hrishikesh , I had to update the deploy manually to overcome this issue, hopefully you can still access the build I’ve linked above & see the issue (url). As of now the production endpoint will no longer reproduce since I manually built and pushed via:

ntl deploy --build --prod

Hey @jlhasson,

I’m sorry to say, but after reading your latest messages, I think you’ve got a few things confused.

You’ve not created a dev deployment, you’re merely started a local development server. netlify dev --live doesn’t update your site, it creates a tunnel using which anyone over the internet can view your “local development”.

This is evident by:

It’s running the “dev” server

It’s creating a tunnel.

The server is running on netlify.live. Your production app runs on netlify.app.

That’s the only way using CLI to update your production site.

I’m not sure I understand why a development server running off main isn’t enough to reproduce this issue? I linked a build on Netlify from 1 commit behind main and a live link to the server running on main which showed the issue (one will have console.log statement and the other will not). Unfortunately since we’re actively developing the system no longer reflects this state :weary:

Do you have a way on your side to stand up two builds side by side? I.e. I can give you the build IDs for our system, which presumably contain all of the files needed (since it’s a static site) so you should be able to take those files & stand up both sites to debug? For example, between these two builds the issue can be reproduced as I described above (log will show in one but not the other):

Hi @jlhasson ,

We do need to see the deploys on Netlify and not on a development server to help debug further.