Trying to import your website and getting an error, possibly due to Cloudflare

@Jassi Bacha yeah if you have this on cloudflare its a good chance its them. I believe somewhere they might have a way to whitelist crawlers. If you can find/figure that out and add relume.io it "might" prevent this.

I'm wondering if I could just disable the cloudflare proxy for like 5 minutes and let it scan (I'm only really scanning once, right?)

I'm wondering if I could just disable the cloudflare proxy for like 5 minutes and let it scan (I'm only really scanning once, right?)

right yeah totally

it looks like we could potentially get relume.io on a known bot list with cloudflare too since this is the most common reason people can't import using the tool, I'll add that to the list when we come to do import v2

Trying to allow via WAF in Cloudflare as just doing the DNS Proxy off didn't seem to work, should I try a specific IP or aim for the relume.io domain ?

I would aim for the domain as I dont think I'd be able to get the IP

maybe try doing something like URI contains relume.io

instead of equals

Hmm yeah unfortunately I get no pings on this rule

standby, tagging an engineer

Oh, turned off super bot fight or whatever its called and I think the combo of that + the rule (or just the bot fight disable) worked?

But it looks like it's trying to crawl post categories and it's definitely going to hit limit of 50 before crawling the correct pages, oof.

yeah, I know I was going to mention that....

It still got the majority of what I wanted / needed, was hoping it would be able to crawl with more depth than just url and meta info but is what it is. I'll definitely keep my ear to the ground for when you guys have an Import V2 available.

yeah, definitely some quick wins here, just a matter of prioritizing it