Based on our record, DuckDuckGo: Bang should be more popular than CommonCrawl. It has been mentiond 196 times since March 2021. We are tracking product recommendations and mentions on various public social media platforms and blogs. They can help you identify which product is more popular and what people think of it.
Https://commoncrawl.org/ is a non-profit which offers a pre-crawled dataset. The specifics of individual tools probably vary. I imagine most tools would be based on academic datasets. - Source: Hacker News / 4 months ago
Should the NYT not sue https://commoncrawl.org/ ? OpenAI just used the data from commoncrawl for training. - Source: Hacker News / 4 months ago
What you’re likely referring to is Common Crawl: https://commoncrawl.org. - Source: Hacker News / 4 months ago
> ... a project called "Nutch" would allow web users to crawl the web themselves. Perhaps that promise is similar to the promises being made about "AI" today. The project did not turn out to be used in the way it was predicted (marketed), or even used by web users at all. Actually Nutch is used to produce the Common Crawl[0] and 60% of GPT-3's training data was Common Crawl[1], so in a way it is being used... - Source: Hacker News / 5 months ago
> Let's share the index as public data Common crawl[1] data has been in AWS for over a decade. [1]: https://commoncrawl.org. - Source: Hacker News / 6 months ago
Seriously, I'm surprised at how many devs haven't seen DuckDuckGo's bangs: https://duckduckgo.com/bangs I use them all the time for work. !mdn for MDN, !dnab for .NET, !npm for NPM, !py3 for Python3 docs, !debman for Debian Manpages, !w for Wikipedia, !a for Amazon, !g for Google when you really need it. I'm not affiliated with DDG; I just really, really love it. - Source: Hacker News / 3 months ago
I think everyone in the thread knows about "search engines" in Chrome and bookmark keywords in Firefox. The crux of the issue is that there are more than 10,000 bang commands in DDG. Setting up even a popular subset in any given browser is a significant investment. It's fine if it's the browser you use 99% of the time, but for those spanning multiple computers, phones, and other devices, simply using bang... - Source: Hacker News / 4 months ago
Where is the place to add personal bangs to DuckDuckGo? I can't find it in https://duckduckgo.com/bangs or https://duckduckgo.com/settings. - Source: Hacker News / 4 months ago
Kagi uses the same bangs as duckduckgo (and more), so there should already be 4 different bangs for it https://duckduckgo.com/bangs?q=protondb. - Source: Hacker News / 5 months ago
+Bangs!! https://duckduckgo.com/bangs eg type "jazz music!yt" and it will do a youtube search, !s for startpage, !g for google etc. - Source: Hacker News / 6 months ago
Scrapy - Scrapy | A Fast and Powerful Scraping and Web Crawling Framework
DuckDuckGo - The Internet privacy company that empowers you to seamlessly take control of your personal information online, without any tradeoffs.
StormCrawler - StormCrawler is an open source SDK for building distributed web crawlers with Apache Storm.
Brave Search - Private search that puts you first, not big tech
Apache Nutch - Apache Nutch is a highly extensible and scalable open source web crawler software project.
Whoogle Search - Self-hosted, ad-free, privacy-respecting Google meta-search engine