CommonCrawl [1] is the biggest and easiest crawling dataset around, collecting data since 2008. Pretty much everyone uses this as their base dataset for training foundation LLMs and since it's mostly English, all models perform well in English. [1] https://commoncrawl.org/. - Source: Hacker News / 3 days ago

Isn't this by problem solved by using commoncrawl data. I wonder what changed to AI companies to do mass crawling individually. https://commoncrawl.org/. - Source: Hacker News / about 1 month ago

There is project whose goal is to avoid this crawling-induced DDoS by maintaining a single web index: https://commoncrawl.org/. - Source: Hacker News / 3 months ago

In 1998, the Web was incomparably smaller. They could put their whole infra into a dozen boxes. By now, crawling and indexing is a herculean task, and also quite expensive, due to the sheer size. There is Common Crawl [1]; at 400 TiB it is huge, but it 60 days refresh interval it's far from being very comprehensive or very fresh. Good for research, but likely not good for a commercial search engine. [1]:... - Source: Hacker News / 6 months ago

Common Crawl Foundation | REMOTE | Full and part-time | https://commoncrawl.org/ | web datasets I'm the CTO at the Common Crawl Foundation, which has a 17 year old, 8. - Source: Hacker News / about 1 year ago

Hi, I have read few comments under the post, there are great suggestions also your questions regarding task are on the point. But I believe handling this with a script might be not easy. If I were you, I would use Apache Nutch or similar open source software/library.I have used Nutch for my thesis for similar task that I had to scrap a lot of blog pages and the other pages they were referencing. You can configure... Source: over 2 years ago

I've never used it, but I was on a project where we considered Apache Nutch: https://nutch.apache.org/. Source: over 2 years ago