How to correctly crawl lots of pages without getting a bot checking screen from cloudflare?

1

Playwright

2) When you see "Attention Cloudflare" in your HTML title, or a "Please wait checking your browser" screen in your browser, it means Cloudflare wants to check if you're a human. If you're not using a browser, you automatically failed and you'll most likely get a Captcha in response. You can convince Cloudflare that you're a human by using Puppeteer or Playwright. Personally, I would use Playwright because it's more powerful than Puppeteer (or Selenium). You can use Playwright in headful mode by setting the headless: false launch option. This often convinces Cloudflare by itself. If it doesn't work, you'll need proper fingerprints, but that's quite a challenging task to pull out. I wrote a tutorial on How to scrape the web with Playwright so you might wanna check this out.

#Development #Tool #Browser Testing 231 social mentions

2

Apify

Convincing Cloudflare that you're a human and not a bot is not easy, but we do that every day at Apify so if you need more help with scraping, you can definitely get in touch with us.

#Web Scraping #Data Extraction #Data 21 social mentions