Support Forums

SEO is not working after 200! redirect implemented

I have implemented 200! redirect in our site to render the page on hosted on https://children.wpengine.com/live/ as if it is in the same domain when a user visited https://www.parent.com/children/live

This is the redirect rule in _redirects file

/children/live/* https://children.wpengine.com/live/:splat 200! 

However, after 2 weeks the site has been published, the sites still not appearing on search engine (both Google and Bing). How is this 200! redirect affect SEO and search engine ability to crawl the webpages?

On Google search, I get something like this:

Other information:
The https://children.wpengine.com/live/ is hosted on WordPress and we have confirmed it should works fine. On Google Search Console, I am not able to request indexing for https://www.parent.com/children/live/ and their bot is not able to crawl the page and received 500 status error instead.

Indexing request reject on Google Search Console because the site returned 500 status error
Screen Shot 2021-07-08 at 10.31.38 AM

Let me know if you need the actual site URL addresses to help me with this

are sure you it’s 200 and not 301?

Yes, I purposely use 200 (rewrite) instead of 301 (redirect). The goal is to have the URL in the visitor’s address bar remains the same, while Netlify’s servers fetch the new location behind the scenes.

ref: Rewrites and proxies | Netlify Docs

When you assign an HTTP status code of 200 to a redirect rule, it becomes a rewrite . This means that the URL in the visitor’s address bar remains the same, while Netlify’s servers fetch the new location behind the scenes.

I did the Proxy to another service, Rewrites and proxies | Netlify Docs and it is working. However SEO is not working.

Hi @shafiemukhre

Rather than placeholder URLs, can you provide the real ones so the community can better test your issue?

Hi, @shafiemukhre. That site (www.parent.com) isn’t hosted at Netlify. That site is hosted at Shopify. Only sites hosted at Netlify can use our redirects feature.

@coelmay @luke

The parent website hosted on netlify is https://www.dremio.com. When a visitor go to https://www.dremio.com/subsurface/live/, the browser will will render the sites hosted on https://subsurface2.wpengine.com/live/

This is the redirect rule I am using:

/subsurface/live/* https://subsurface2.wpengine.com/live/:splat 200! 

Appreciate that you guys are looking into this!

Perhaps this has something to do with it

% curl -IL https://www.dremio.com/subsurface/live/
HTTP/2 200
cache-control: max-age=600, must-revalidate
content-type: text/html; charset=UTF-8
date: Fri, 09 Jul 2021 07:20:20 GMT
link: <https://subsurface2.wpengine.com/wp-json/>; rel="https://api.w.org/"
link: <https://subsurface2.wpengine.com/wp-json/wp/v2/pages/19>; rel="alternate"; type="application/json"
link: <https://www.dremio.com/subsurface/?p=19>; rel=shortlink
server: Netlify
x-nf-request-id: 01FA526Z9278VRWZKGA34WDMTF
x-cache: HIT: 7
x-cache-group: normal
x-cacheable: SHORT
age: 1
x-powered-by: WP Engine

There are both wpengine.com and dremio.com links in the headers.

link: <https://subsurface2.wpengine.com/wp-json/wp/v2/pages/19>; rel="alternate"; type="application/json"
link: <https://www.dremio.com/subsurface/?p=19>; rel=shortlink

Thanks @coelmay. I see, there’s multiple domain in the headers. How does that effect search engine ability to crawl?

If anyone can share articles/blogs or readings materials on how exactly redirect/rewrite affect SEO, that will be great. If it is specific to Netlify, that’s even better. I am quite loss at this point.

My current understanding is that the Search Engine crawler bot will open a page and crawl all available links available on that page equivalent to someone clicking it and go to next page. As long as the next link on that page is using the same domain, the bot will crawl that next page. If I remember correctly, I learned this from a video on youtube from an Engineer at Google that’s working on Search Engine.

Therefore, my understanding is that, it doesn’t matter if there’s redirect or multiple redirects, as long the final destination of the redirect happened to be on the same domain, the bot will keep crawling it and indexs it.

^ Please correct me if I’m wrong.

It probably doesn’t affect the ability to crawl. I am not a Google SEO expert (they likely exist on Google SEO specific forums) and have little understanding how Google “actually” works in this regard, but perhaps they are saying it is not available because of these headers. Possible?

There is also the fact that in your screenshot it shows a 5xx error but doesn’t specify which one.

You say Bing also doesn’t register the URL. What is the reasoning they give?

I got the screenshot of the 5xx error from Google Search console. I am not sure if Bing have something similar or not, will have a look and give a try.

I made an observation that Bing is not indexing the urls as well because of the same result I got when I did the search on Bing.

I previously asked the Netlify Support this question:

Question: For SEO purposes, will the proxied wordpress website on www.dremio.com/children will be indexed by search engines? These are some resources that I’m looking at for now - Rewrites and proxies | Netlify Docs - [Support Guide] Can I deploy multiple repositories in a single site?

Answer: That’ll depend on the headers that get set by the external source :slight_smile: This little tidbit might be handy for improving SEO for proxied content (consider setting a canonical header pointing to the proxied path on dremio.com).

From my personal experience in getting Google to index my website, I get 5xx errors from time to time even when not having any redirect. It’s a simple static page, no JS based rendering, nothing and yet I get the error. However, I haven’t got one in a few months now.

I tend to ignore them and they automatically get fixed when Google rechecks my website. But yes, I ignore them because my websites are more like personal stuff, so not highly SEO critical for me. But if you’re seeing the errors frequently, I think it’s worth asking in some SEO forums too, just to be sure of the possible causes.

1 Like

Thank you @hrishikesh for your input! It’s good to know that other people faced this issue as well.

When I created a new content, I noticed that it may take 7 - 10 days for Google to index it. So, I am expecting the same amount of time. Now after 2 weeks, still nothing happened. This is a new implementation though.

Yeah, this is a bit SEO critical to us. It’s fine to wait if it’s going to take longer time, we just need to confirm and know whether we are doing the right implementation or not.

Thanks for the suggestion, will ask some other SEO forums and SEO agency on this matter as well in parallel.

I’ve replied to the helpdesk ticket too, but adding a part of the reply here.

From the internal logs, I was not able to find Googlebot User Agent at least in the last 7 days. Furthermore, Bing bot was able to get a 200. If it’s still not indexing your website, I think it’s a tough case.

Thank you @hrishikesh! It seems that this issue was automatically fixed after about a month without us doing anything. I’m assuming that’s because it’s either they had to re-crawl the pages to notice it or it just takes more time.