ProxyCompass
CommonCrawl
Scrapy
StormCrawler
Apache Nutch
Heritrix
YaCy
Apache Solr
ACHE Crawler
Killed by Google
The Google Cemetery
Google Graveyard by SaaSHub
Pi-hole
Medium
Stadia
GrapheneOS
Startup Graveyard
CommonCrawl Killed by GoogleBased on our record, Killed by Google seems to be a lot more popular than CommonCrawl. While we know about 1163 links to Killed by Google, we've tracked only 90 mentions of CommonCrawl. We are tracking product recommendations and mentions on various public social media platforms and blogs. They can help you identify which product is more popular and what people think of it.
Https://commoncrawl.org/ is a non-profit which offers a pre-crawled dataset. The specifics of individual tools probably vary. I imagine most tools would be based on academic datasets. - Source: Hacker News / 4 months ago
Should the NYT not sue https://commoncrawl.org/ ? OpenAI just used the data from commoncrawl for training. - Source: Hacker News / 4 months ago
What you’re likely referring to is Common Crawl: https://commoncrawl.org. - Source: Hacker News / 4 months ago
> ... a project called "Nutch" would allow web users to crawl the web themselves. Perhaps that promise is similar to the promises being made about "AI" today. The project did not turn out to be used in the way it was predicted (marketed), or even used by web users at all. Actually Nutch is used to produce the Common Crawl[0] and 60% of GPT-3's training data was Common Crawl[1], so in a way it is being used... - Source: Hacker News / 5 months ago
> Let's share the index as public data Common crawl[1] data has been in AWS for over a decade. [1]: https://commoncrawl.org. - Source: Hacker News / 6 months ago
Https://killedbygoogle.com/ Their reputation is deserved. Google domains was killed only last year! - Source: Hacker News / 19 days ago
And this isn't even the first time I've been burned by Google's decisions. If you're familiar at all with the Google Graveyard, you'll know that Google has a long history of killing off products and services that people have come to rely on. This has happened to me a number of times, in both a personal and professional capacity, and frankly it's getting old. - Source: dev.to / 26 days ago
> Those who trust the Google’s app to listen to podcasts have almost three months to export their subscriptions to another app. Alternative take: Those who trust any Google app or service to do anything should reconsider, or at the very least keep a permanent and up-to-date contingency plan for when Google inevitably discontinues it. Fooled you once, shame on Google. Fooled you 294 times, shame on you.... - Source: Hacker News / 26 days ago
This will be in the Google Graveyard https://killedbygoogle.com/ by mid-2026. - Source: Hacker News / about 1 month ago
Just because Google has a couple of decent services that you're willing to pay for doesn't detract from the fact that most of their products have a worse life expectancy than a victorian child in the 1800s. https://killedbygoogle.com They ruined every single opportunity to be more than an advertising company since Orkut. With scrapped attempts, starts and lack of intention for most of the 2010s to even during the... - Source: Hacker News / about 2 months ago
Scrapy - Scrapy | A Fast and Powerful Scraping and Web Crawling Framework
The Google Cemetery - A list of dead Google products and why they died
StormCrawler - StormCrawler is an open source SDK for building distributed web crawlers with Apache Storm.
Google Graveyard by SaaSHub - The Google Graveyard is the complete list of discontinued products by Google. Also known as 'The Google Cemetery'
Apache Nutch - Apache Nutch is a highly extensible and scalable open source web crawler software project.
Pi-hole - Pi-hole is a multi-platform, network-wide ad blocker.