Issues Importing Website with Cloudflare: Seeking Solutions

·

Trying to import my website https://evolvevr.ca/ and getting this error, I assume it's Cloudflare? Any way to get around this?

14 comments

· Sorted by Oldest

Matt J.
·
Jassi B. yeah if you have this on cloudflare its a good chance its them. I believe somewhere they might have a way to whitelist crawlers. If you can find/figure that out and add relume.io it "might" prevent this.
Jassi B.
·
I'm wondering if I could just disable the cloudflare proxy for like 5 minutes and let it scan (I'm only really scanning once, right?)
Matt J.
·
right yeah totally
Matt J.
·
it looks like we could potentially get relume.io on a known bot list with cloudflare too since this is the most common reason people can't import using the tool, I'll add that to the list when we come to do import v2
Jassi B.
·
Trying to allow via WAF in Cloudflare as just doing the DNS Proxy off didn't seem to work, should I try a specific IP or aim for the relume.io domain ?
Matt J.
·
I would aim for the domain as I dont think I'd be able to get the IP
Matt J.
·
maybe try doing something like URI contains relume.io instead of equals
Jassi B.
·
Hmm yeah unfortunately I get no pings on this rule
Matt J.
·
standby, tagging an engineer
Jassi B.
·
Oh, turned off super bot fight or whatever its called and I think the combo of that + the rule (or just the bot fight disable) worked?
Jassi B.
·
But it looks like it's trying to crawl post categories and it's definitely going to hit limit of 50 before crawling the correct pages, oof.
Matt J.
·
yeah, I know I was going to mention that....
Jassi B.
·
It still got the majority of what I wanted / needed, was hoping it would be able to crawl with more depth than just url and meta info but is what it is. I'll definitely keep my ear to the ground for when you guys have an Import V2 available.
Matt J.
·
yeah, definitely some quick wins here, just a matter of prioritizing it

Matt J.
·
Jassi B. yeah if you have this on cloudflare its a good chance its them. I believe somewhere they might have a way to whitelist crawlers. If you can find/figure that out and add relume.io it "might" prevent this.
Jassi B.
·
I'm wondering if I could just disable the cloudflare proxy for like 5 minutes and let it scan (I'm only really scanning once, right?)
Matt J.
·
right yeah totally
Matt J.
·
it looks like we could potentially get relume.io on a known bot list with cloudflare too since this is the most common reason people can't import using the tool, I'll add that to the list when we come to do import v2
Jassi B.
·
Trying to allow via WAF in Cloudflare as just doing the DNS Proxy off didn't seem to work, should I try a specific IP or aim for the relume.io domain ?
Matt J.
·
I would aim for the domain as I dont think I'd be able to get the IP
Matt J.
·
maybe try doing something like URI contains relume.io instead of equals
Jassi B.
·
Hmm yeah unfortunately I get no pings on this rule
Matt J.
·
standby, tagging an engineer
Jassi B.
·
Oh, turned off super bot fight or whatever its called and I think the combo of that + the rule (or just the bot fight disable) worked?
Jassi B.
·
But it looks like it's trying to crawl post categories and it's definitely going to hit limit of 50 before crawling the correct pages, oof.
Matt J.
·
yeah, I know I was going to mention that....
Jassi B.
·
It still got the majority of what I wanted / needed, was hoping it would be able to crawl with more depth than just url and meta info but is what it is. I'll definitely keep my ear to the ground for when you guys have an Import V2 available.
Matt J.
·
yeah, definitely some quick wins here, just a matter of prioritizing it