Hi, I have read few comments under the post, there are great suggestions also your questions regarding task are on the point. But I believe handling this with a script might be not easy. If I were you, I would use Apache Nutch or similar open source software/library.I have used Nutch for my thesis for similar task that I had to scrap a lot of blog pages and the other pages they were referencing. You can configure... Source: over 1 year ago

I've never used it, but I was on a project where we considered Apache Nutch: https://nutch.apache.org/. Source: over 1 year ago

I deployed whoogle search at home. For laptops it is rather easy to define it as the default search for alfred and the respective browser, but for iOS devices I have not found a way (except some rather clunky and intrusive extensions) to change the default search engine beyond what Apple tells you. Is there a way to to this via pfSense or pihole? So whenever someone searches in the address bar in safari on iOS... Source: about 1 year ago

I host an instance of Whoogle Search that I use and never looked back. It's like someone brought Google back to the golden age. No (direct) tracking, no ads, complete control over the results. Source: about 1 year ago

You know what this community needs? Our own community whoogle server. Source: over 1 year ago

Whoogle for the win; https://github.com/benbusby/whoogle-search. Source: over 1 year ago

I use whoogle with still uses Google under the hood but strips any PII out of the requests, I also run mine on a publicly hosted server. Source: over 1 year ago