Cannot access /sitemap.xml nor /robots.txt directly on my SPA

Support
,
#1

Hi, I have a single page application deployed on netlify and so far, everything is great. I’ve deployed it with a netlify.toml file with some redirects, one of them being:

[[redirects]]
from = "/*"
to = "/index.html"
status = 200

The thing is, that when I try to submit a /sitemap.xml url to google search console, google says it cannot find the file, and when I try the file url in my browser I get redirected to /index.html.

So the question is, how can I serve my single page application while maintaining direct access to existing static files on the root folder? I have the same problem, for example, with /robots.txt.

It would be nice to redirect to /index.html only if the requested static file is not found.

Hint: If I issue a curl command to /sitemap.xml it gets served correctly, but not on the browser.

#2

Hey @brielov,

I’ve just tested this and, as we would expect, existing files are not overwritten by redirect rules such as these.

Are you sure that your deployed site features the sitemap and robots.txt files?

Could you provide me with your site name and/or API ID, please (both of which are safe to share publicly) for me to take a further look?

If push comes to shove, given netlify.toml’s rule processing order, it’s possible to add a rule similar to this which would get processed before the overarching ‘all-to-index’ rule for a single page site.

#3

Hei @Pie! I have the same problem as @brielov, just with a static html page. I´ve right now a robots.txt file at root - but I get an 404. My redirect look like this in my netlify.toml
[[redirects]]
from = “/*/css/style.css”
to = “ladeklar.no/css/style.css”
status = 200

[[redirects]]
  from = "/*"
  to = "ladeklar.no/404"
  status = 404

I dont have an _redirects file. My site is https://www.ladeklar.no/.
Does not seem like the robots.txt or sitemap.xml are uploaded to netlify at all. Is it something I´m missing?

#4

That first redirect rule looks a little strange, you might wanna check that out :eyes:. If you’re using an asterisk in the ‘from’ part of the rule, the ‘to’ part will need a :splat :+1:.

You should download the site from the deploy page and make sure that they’re being deployed as expected, too. Can you do this and confirm that they’re present?

#5

Thanks for quick replay and an awesome product :smiley:

I know, sorry for first part. It´s weird. It´s because url from previous site ends up not displaying css when redirected to 404. It did not work though, and that´s another case.

I tried to download my deploy file now. Could not find robots.txt. Should I have the file in my ‘public’ folder to get it uploaded to Netlify?

#6

Okay, at least that shows us why it’s not appearing! robots.txt should live alongside the directory where index.html is served from, meaning it’s available at domain.com/robots.txt – just like domain.com/index.html :smiley:

#7

Ok, so I assume I just have to put it in my public folder and it will be solved :sunglasses:

I´ll come crying back if there´s something else. Thanks for great replies!

#8

Hi again @Pie,

So I got my robots.txt and sitemap.xml to work by putting them in my public folder. But I cant get old urls to be redirected properly. I was hoping to find a way to set all old urls to 404. I see in my deploy console that I have four redirect rule implemented, witch is in my Netlify.toml:

[redirects]]

from = “/”
to = “https://www.ladeklar.no/”
status = 404

[[redirects]]
from = “/*/”
to = “https://www.ladeklar.no/”
status = 404

[[redirects]]
from = “/author/*”
to = “https://www.ladeklar.no/”
status = 404

[[redirects]]
from = “/author/*/”
to = “https://www.ladeklar.no/”
status = 404

Here´s an example of an old url: https://www.ladeklar.no/010-bvp/. Is is something I´m missing for my redirect? None of the old urls are hosted anywhere.

#9

Hey @NikolaiLadeklar,

For your pages to 404, you’ll need to add force = true as the pages exist. The force means that the page won’t be served.

If you are redirecting to the homepage, I would also be tempted to create a dedicated 404 page. Or, redirect your users to the homepage with a 301 (‘permanently moved’ content). A 404 to the homepage is a little… strange :slight_smile:

Additionally, your rules are very vague (and incorrect) :frowning: which can cause problems down the line. Are there a lot of pages which you would like to 404?

For the example provided, you would need a rule such as:

[[redirects]]
from = "/010-bvp"
to = "/"
status = 301
force= "true"

If you want ANYTHING after the / to be redirected, try:

[[redirects]]
from = "/*"
to = "/:splat"
status = 301
force= "true"
#10

Thanks for a quick reply!

There´s a lot of pages I would like to redirect. Eiter by redirecting to homepage or to my own 404 page.

I tested your suggestion:
[[redirects]]
from = “/*”
to = “/:splat”
status = 301
force= true

But then the none of my urls worked. Is it something I´m missing here?

#11

Nope, that’s what I mean. You’ll have a tough time redirecting all of the old pages if there’s nothing to identify them (such as /path/path/here-are-all-of-the-files-I-dont-want-to-serve.html) :slight_smile:

You may be best off removing them entirely and then changing your rule to:

[[redirects]]
from = “/*”
to = “/:splat”
status = 301
#12

They´re all removed, but something strange happens. The urls with “/* /*” (often old urls) will be redirected to 404 without css. Even with this line:

[[redirects]]
from = “/*”
to = “/:splat”
status = 301

It seems like somehow I cant get my redirection to work for urls with trailing slashed.

#13

Ah, trailing slashes? Makes it a little tricky because everybody handles them differently :frowning:!

You may want to explore enabling pretty URLs in the UI, if you want the trailing slash.

Got an example to share, at all?

#14

Example of URL that redirects properly to 404: https://ladeklar.no/kfkkw

Example of URL that does not redirects properly to 404 (redirects, but without css): https://ladeklar.no/kfkkw/LFKs

Any non-URL with an slash «/» redirects to an 404 page without CSS.

Last ned Outlook for iOS

#15

Update your HTML from:
<link rel="stylesheet" href="css/style.css"> and <script src="js/main.min.js"></script>
to
<link rel="stylesheet" href="/css/style.css"> and <script src="/js/main.min.js"></script>

:wink:

#16

Awesome, thank you!! Been spending way to much time redirecting :smile:

1 Like