YaCy VS Apache Nutch

YaCy might fit your bill: https://yacy.net/. - Source: Hacker News / 2 months ago

It turns out you can make it all the way to become president of Harvard [1] while ignoring this rule so it is questionable whether it is as set in stone as you make it out to be, at least in certain disciplines. In a way these models are a perfect mirror of the current academic climate. They plagiarise without remorse, they follow the latest identity-politics diktat to a point and make up 'facts' when needed to... - Source: Hacker News / 3 months ago

Here you go: distributed, peer-to-peer, FLOSS search engine: https://yacy.net. - Source: Hacker News / 4 months ago

> Now I just need some kind of open source search engine to run on it ... Here you go: https://yacy.net. - Source: Hacker News / 4 months ago

I remember https://yacy.net/ but the big problem of this project was java and had not implementations in others languages. I mean it as imagine torrent was only in perl. - Source: Hacker News / 8 months ago

Hi, I have read few comments under the post, there are great suggestions also your questions regarding task are on the point. But I believe handling this with a script might be not easy. If I were you, I would use Apache Nutch or similar open source software/library.I have used Nutch for my thesis for similar task that I had to scrap a lot of blog pages and the other pages they were referencing. You can configure... Source: over 1 year ago

I've never used it, but I was on a project where we considered Apache Nutch: https://nutch.apache.org/. Source: over 1 year ago