On Functions My axios GET is returning 403 , local 200

Support Edge Functions
, ,
1

Hello everyone,

I developed a crawler using Node.JS that will run trough a web page and return me a JSON with the desired data formatted.

It’s working well locally with node index.js and with netifly dev , but not after publishing. Checking for logs I saw this

Feb 9, 11:57:20 AM: 6d039de3 ERROR Error fetching data: AxiosError: Request failed with status code 403

The line is this:

const { data } = await axios.get(url);

The URL is https://www.tibiawiki.com.br/wiki/Aves

Checking for the Page Robots.txt , doesn’t seem to be a blocker for User Agent:

User-agent: *
Disallow:

I do have other crawlers like this for different URLS and they usually works, but not this one.

Any clue? If you do need for more , do let me know.

2

I myself also get a 403 when accessing https://www.tibiawiki.com.br/wiki/Aves:

image
image2489×1559 157 KB

Sounds like a problem with that website than Netlify.

3

That error screen looks like a Cloudflare error page, and the error makes me think they might have Geoblocking enabled?

I’m in the EU, and i’m able to access the site. Netlify Functions run in the US by default, so maybe that’s blocked? If you’re on a paying plan, you could try moving your Netlify Functions to a different datacenter, see Optional configuration for functions | Netlify Docs. Alternatively, you could try moving this scraping logic to Edge Functions - that way it would run close to the user-agent you’re requesting it with, which might be in a non-blocked geolocality as well!