CommonCrawl VS Searx

CommonCrawl Reviews

Searx Reviews

CommonCrawl mentions (90)

• Ask HN: How does one implement web plagiarism?

  Https://commoncrawl.org/ is a non-profit which offers a pre-crawled dataset. The specifics of individual tools probably vary. I imagine most tools would be based on academic datasets. - Source: Hacker News / 4 months ago

• Things are about to get a lot worse for Generative AI

  Should the NYT not sue https://commoncrawl.org/ ? OpenAI just used the data from commoncrawl for training. - Source: Hacker News / 4 months ago

• Indexing a Billion Pages

  What you’re likely referring to is Common Crawl: https://commoncrawl.org. - Source: Hacker News / 4 months ago

• Interview with Viktor Lofgren from Marginalia Search

  > ... a project called "Nutch" would allow web users to crawl the web themselves. Perhaps that promise is similar to the promises being made about "AI" today. The project did not turn out to be used in the way it was predicted (marketed), or even used by web users at all. Actually Nutch is used to produce the Common Crawl[0] and 60% of GPT-3's training data was Common Crawl[1], so in a way it is being used... - Source: Hacker News / 5 months ago

• Google's Plan to Stop Apple from Getting Serious About Search

  > Let's share the index as public data Common crawl[1] data has been in AWS for over a decade. [1]: https://commoncrawl.org. - Source: Hacker News / 6 months ago

Searx mentions (40)

• Just a reminder that WhatsApp is also owned by Facebook

  Meaning, you can go to public instances like searx.me,. Here's the documentation on how to start it up. But , you dont have to trust Searx that they are good people nor do you have to trust their data habits like DDG. Source: about 2 years ago

• Instead of lashing out at duckduckgo for doing what they think is best, ask the deeper question of why we’re all still using centralized services and being disappointed when they behave in a predictably centralized way.

  Consider a future where something like https://searx.me/ is as ubiquitous as Tor. Source: about 2 years ago

• DDG, once a hero has now fallen.

  For those looking for a replacement for Duckduckgo; I would highly recommend using Searx. It's an open source privacy respecting search engine with many decentralized private instances you can swap between. The link I sent is the primary instance, but here is a link with dozens more, and my own private instance. Source: about 2 years ago

• DuckDuckGo is out. I guess I'll try Brave Search.

  The most based solution: Https://searx.me. Source: about 2 years ago

• Uh oh.. What will the vaccinated think of this news?

  Searx.me and Startpage.com are the best search engines right now that are anti-censorship and anti-bias. Source: about 2 years ago