Hi, not sure this is 'an issue', but trying out the new sitemap import feature.
Get this message:
Error importing sitemap
We weren't able to import this website because it
blocks requests from automated tools.
(pic attached)
--
Looked at logs, it's because of standard Cloudflare "Bot fight mode":
Service: Bot fight mode
Action taken: Managed Challenge
(pic attached)
--
Yes, I could disable bot fight mode, or add user agent/IP to allow list on Cloudflare side.
Possible improvement:
Maybe there could be way around via importing/uploading .xml, or pasting list of pages, or going direct to sitemap.xml.
Robots should be able to access /sitemap.xml page - but looking at CF event/request, it's going to hostname & maybe crawling from there?
