User Agent Blocker generates incorrect robots.txt additions

Support
1

The User Agent Blocker is generating incorrect additions to robots.txt files based on selections found under Block user agents.

I did not enable the option The Actively block requests using an edge function in my testing.

I have not tested all the options fully, but it always appears to include GoogleBot and BingBot in the output even when those options are not selected.

If you select all the options (except the edge function), the output generated is:

User-agent: GPTBot
Disallow: /

User-agent: GoogleOther
Disallow: /
User-agent: GPTBot
Disallow: /

User-agent: GoogleOther
Disallow: /

User-agent: PetalBot
Disallow: /

User-agent: Factset_spyderbot
Disallow: /

User-agent: LINER Bot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Timpibot
Disallow: /

User-agent: GoogleBot
Disallow: /

User-agent: BingBot
Disallow: /

User-agent: YandexBot
Disallow: /

User-agent: DuckDuckBot
Disallow: /

User-agent: SemrushBot
Disallow: /

User-agent: AwarioBot
Disallow: /

Deselecting only the options under Search Engine Crawlers generates:

Screenshot 2026-03-23 112917
Screenshot 2026-03-23 112917858×971 30.5 KB

User-agent: GPTBot
Disallow: /

User-agent: GoogleOther
Disallow: /

User-agent: LINER Bot
Disallow: /

User-agent: ClaudeBot
Disallow: /

User-agent: Timpibot
Disallow: /

User-agent: GoogleBot
Disallow: /

User-agent: BingBot
Disallow: /

User-agent: SemrushBot
Disallow: /

User-agent: AwarioBot
Disallow: /

User-agent: DotBot
Disallow: /

User-agent: MJ12bot
Disallow: /

Note that GoogleBot and BingBot are still generated, but PetalBot and Factset_spyderbot are missing. The expectation should be that all the AI and SEO options would be included, and none of the Search Engine Crawlers are.

2

Could you let us know the site you tested this on so we can check further?

3

Tests were done on https://cites.cast.org, also on https://udlguidelines.cast.org

The udlguidelines site currently has a manually created robots.txt file.

4

Thank you. I can confirm the issue exists and a PR is up with the fix.

5

Just checking on the status to see if the fix been integrated?

There stills appears to be issues with the generated rules appended to robots.txt by the User Agent Blocker.

I checked with redeploys this morning on https://cites.cast.org and was still seeing the incorrectly generated results when the Search Engine Crawlers section is disabled.

I have resorted to a manually added list in the robots.txt for CITES since the incorrect entries for GoogleBot and BingBot were causing issues with the SEO for Google and Bing

6

I am having the same issues still, any fixes?

For example, I unselected Googlebot (as I want to allow it), but it still shows up as blocking it.

7

This should now be fixed.