ProxyCrawl allows to crawl and scrape websites without the need to use proxies.
We have more than a million datacenter and residential proxies and we handle proxy rotation, captchas, blocks and infrastructure for you.
Start crawling and scraping the web with our API or use our crawler to asynchronously crawl websites. 1000 requests are free!
I love their support, and the service. The Crawling API and Crawler are very reliable, I use this service to scrape millions of pages from various websites like Zillow, Amazon, Yelp, Ebay, SimilarWeb. I've searched over 4 months to find a service to scrape Linkedin, I only found it with proxycrawl, I could not believe they can do it until I tried. Super awesome service and worth every cent, the service reduces my time on crawling and scraping the web to basically zero minute.
Scrapy - Scrapy | A Fast and Powerful Scraping and Web Crawling Framework
Apify - Apify is a web scraping and automation platform that can turn any website into an API.
StormCrawler - StormCrawler is an open source SDK for building distributed web crawlers with Apache Storm.
Apache Nutch - Apache Nutch is a highly extensible and scalable open source web crawler software project.
Scraper API - Easily build scalable web scrapers
Heritrix - Heritrix is the Internet Archive's open-source, extensible, web-scale, archival-quality web...