Top 12 Open-Source Alternatives to CommonCrawl

Product categories

APIs Blogging Data Data Extraction Internet Search Private Search Engine Search Engine Security & Privacy Web Crawling Web Scraping Web Scraping API Web Search

Summary

The top open-source alternatives to CommonCrawl are Scrapy, StormCrawler, and Apache Nutch. One of the criteria for ordering this list is the number of mentions that products have on reliable external sources. You can suggest additional sources through the form here.

Scrapy Landing Page
1

Scrapy

Scrapy | A Fast and Powerful Scraping and Web Crawling Framework
Pricing:
- Open Source
#Web Scraping #Data Extraction #Data 93 social mentions
StormCrawler Landing Page
2

StormCrawler

StormCrawler is an open source SDK for building distributed web crawlers with Apache Storm.
Pricing:
- Open Source
#Web Scraping #Data Extraction #Data
Apache Nutch Landing Page
3

Apache Nutch

Apache Nutch is a highly extensible and scalable open source web crawler software project.
Pricing:
- Open Source
#Web Scraping #Data Extraction #Utilities 2 social mentions
YaCy Landing Page
4

YaCy

YaCy is a free search engine that anyone can use to build a search portal for their intranet or to...
Pricing:
- Open Source
#Search Engine #Internet Search #Web Search 71 social mentions
Apache Solr Landing Page
5

Apache Solr

Solr is an open source enterprise search server based on Lucene search library, with XML/HTTP and...
Pricing:
- Open Source
#Custom Search Engine #Custom Search #Search Engine 17 social mentions
Mixnode Landing Page
6

Mixnode

Turn the web into a database!
Pricing:
- Open Source
#Web Scraping #Data Extraction #Data
DuckDuckGo Landing Page
7

DuckDuckGo

The Internet privacy company that empowers you to seamlessly take control of your personal information online, without any tradeoffs.
Pricing:
- Open Source
#Search Engine #Web Search #Internet Search 1666 social mentions
Hacker News Search Landing Page
8

Hacker News Search

a faster hnsearch
Pricing:
- Open Source
#Search Engine #Web Search #Internet Search 1905 social mentions
mwmbl.org
9

Mwmbl Search

An open source, non-profit search engine implemented in python
Pricing:
- Open Source
#Search Engine #Internet Search #Private Search Engine 4 social mentions
Kagi Landing Page
10

Kagi

Kagi is a privacy-focused, user-centric search engine. Great search experience starts with Kagi!
Pricing:
- Open Source
#Search Engine #Security & Privacy #Privacy Search Engine 120 social mentions
SerpApi Landing Page
11

SerpApi

Scrape Google search results from our fast, easy, and complete API.
Pricing:
- Open Source
#SEO #SEO Tools #APIs 69 social mentions
Colly Landing Page
12

Colly

Colly is a scraping framework to extract structured data from websites.
Pricing:
- Open Source
#Web Scraping #Data Extraction #Data 9 social mentions