Netlify Remix site de-indexed early May 2024, many attempts made to re-index to no avail. PLEASE HELP!

Hello everyone, I’m at my wits end with an issue we’ve been dealing with since early May. Our entire site was de-indexed by google and hasn’t been re-indexed since. Our domain is ~12 years old, would get multiple hundred organic views a day and has never had an issue. It’s obviously a major issue because it’s a major driver for business.

Our Initial thought was it had to be something with the DNS or something along those lines; but we couldn’t find anything. Our strongest suspicion, still to this day is that we were de-index for performance issues. We had a significant layout shift issue and some performance issues that made us fail google’s PageSpeed Insights core web vitals assessment. We’ve since have done a substantial amount of work to fix issues and improve performance, doubling our scores in those assesments, but you can only make it so fast. We still fail the mobile assesment. Here’s a link of images showing the de-indexing and the performance issues: Google Performance Data - Album on Imgur

So, my question is multipart.

  1. Could our indexing issues be simply performance related? If so, how the hell can we get re-indexed?
  2. Is this a remix issue? I’ve done everything should of destroying our app to improve performance.
  3. Is this a netlify issue? I’ve reached out to support to see if there was anyway to improve things like main thread work, total blocking time and Javascript execution time and haven’t heard back yet.

It’s really really tough to diagnose an issue when there’s no clear indicatior of the problem; Google is so vague with this and there’s no real way to get answers. I do know google had new algorithm changes around that time and that’s where my performance based theory comes from. Any help here will be GREATLY appreciated.

Some updates because I have more info:

EDIT: Just adding a few more bits of info. We’ve turned on Netlify’s real user metrics and it shows us some conflicting and concerning results.

This is a chart for first contentful paint, there’s really nothing that could describe this data.
Imgur

This one shows our cummalative layout shift, which google says is 0
Imgur

And this is our interaction to next paint:
Imgur

Can anyone help me understand these and what the heck is with the labels p75, p95 and p99? If Netlify’s data is correct and my theory that performance is what’s keeping us de-indexed then I guess this makes sense but also makes fixing this almost impossible.

Lastly, here’s Netlify’s own lighthouse scores for our deploys. I know they aren’t real user metrics but it’s weird that lighthouse scores say very good but real user metrics show poor.

Imgur

@jon_sgr It’s going to be hard for anyone to self check and confirm anything you’ve provided in the screenshots without a link to the site, can you provide it?

It’s Google that have de-indexed the site, so I’d presume the most obvious direction of attack would be to determine from Google why that’s the case.

It seems a little backwards to be reaching out to Netlify, asking why Google’s systems have done something, but I presume you’ve done so because Netlify are easier to contact?

Thanks for the reply. Our website link is: https://goroostr.com

There’s no real avenue to get help from Google or even remediation advice. We’re posting on Netlify because we suspect this is a performance issue and wonder if there’s anything Netlify can do to help us.

@jon_sgr Netlify’s staff are primarily concerned with their own systems, and only really dedicate time to things deemed inside the scope of support.

Assisting with 3rd party systems and site optimizations is definitely outside of that.

I’m sure they can comment on the metrics seen on your account by their systems though.

I can’t really comment on the issue itself, but if you do believe it’s performance, and that the performance issue originates from Netlify’s systems (and not just your site), you could test this by hosting the same site elsewhere then checking the performance at that location.

thanks @nathanmartin, I want to clarify that I’m not saying this is a Netlify problem, more questioning if it could be a problem. I know I’m reaching for straws but I can’t find anything else that would cause this.

@jon_sgr Just a heads up I’m not saying it isn’t a Netlify problem either, it may very well be, I’ve not even looked at it.

Nor am I defending them, I’m just advising how I know they operate in regards to their scope of support, in the hope that you can source yourself an answer quicker (from wherever that may be - continuing to investigate yourself, asking elsewhere etc).

It makes sense to have asked on what is ostensibly a “community forum”, but despite the high number of posts there isn’t really an active community here.

I may even be the only active non-netlify community member at the moment, so if it’s the kind of question that would rely on volunteer support you’re unlikely to get an answer here unfortunately.

1 Like

We’ve responded to Google with a average time to last byte of 130.39 ms in the last 30 days for your site. 12,570 requests had status code less than 400 and 231 had more than 399. If I exclude 404s, this number drops to 37. Thus, I am inclined to agree with:

@jon_sgr, do you have any reason for this suspicion or anything to back up the following claim?

Hi!

I looked into this briefly and noticed that your site seems to use client side rendering for some pages. This is notoriously bad for SEO since the google crawl bot is only getting an empty page with no content.

I’d recommend doing a light SEO overhaul to the site, making sure that you set the necessary meta tags (e.g canonical tags seems to be missing) and that you aren’t rendering your pages on the client side. I also noticed that your robots file is referencing your sitemap through http, I’d change this to https. This should at least improve your SEO.

Performance alone shouldn’t de-index your pages (afaik).