Bug (and workaround): 404 page returns HTTP 200

[SmashManiac]

@maxbarbul

I did take into account the heuristic nature. The point here is that heuristics is a business logic of specific applications (search engine and other crawlers, content scrapers, hacker scripts, etc.).

My point was that if that heuristics are prone to failure, so they can’t be relied upon.

@maxbarbul

Second question is, Netlify is a convenient web hosting (pipeline, optimized engine, web server). So, they have to be application-agnostic and to have as few non-standard settings as possible at the same time provide best value for customers. So, I see a good decision in holding closer to web server standard behaviour.

I agree, and that’s exactly why I created this thread: so that Netfilty aligns with web standards while it currently does not. (Obviously some people in this thread disagrees with me about said standards, but I have yet to see authoritative evidence in support of their arguments.)

@maxbarbul

Third part is why do you care about those crawlers?

I care about by website having a good user experience, regardless of whether a human or a bot uses it.

@maxbarbul

But if you put your 404 in Disallow and don’t have it linked anywhere on your or other websites, you’re good.

You have no control over that beyond your own websites though, hence why it’s not a good solution to prevent search engine indexing despite a good probability of success. That’s one way search engines may find pages disallowed by robots.txt. And besides, it doesn’t resolve the core issue anyway.

@maxbarbul

If your page is already in the index and you want to remove it, you can add “noindex” tag, return 410 status or do whatever you want.

Returning HTTP 410 in Netlify for the 404 file requires the same workaround from my initial post to be effective, so that doesn’t help. As for the “noindex” solution, I would like to point out that it doesn’t work along with a robots.txt Disallow directive, and that it’s just a worse workaround for the soft 404 problem than the one I proposed in my initial post as it only applies to search engines and has other negative SEO consequences.

@maxbarbul

What you named a misconception was about people who thinks robots.txt would hide their pages from public.

I have no idea how anybody would ever believe such a thing, as it should be obvious from the name itself that robots.txt only applies to robots. This is not what I was referring to, and I’m pretty sure that’s not what the Google documentation I linked to in my previous post meant either. The big red warning box there clearly explains that the misconception is about hiding pages from search results, not from the public.

@maxbarbul

we’re discussing SEO here as you started with “soft 404” complaint.

That was never my intention. SEO is just one aspect of the soft 404 problem and the most obvious one to web administrators, hence why I gave it as an example. The Internet Archive relies on HTTP response codes for archival. API users rely on HTTP response codes in their business logic. Security assessment tools partially rely on HTTP response codes for analysis. Returning the wrong response may have far-fetched and unexpected undesired consequences to a bunch of systems. That’s what I want Netlify to fix.