Hi, I have read few comments under the post, there are great suggestions also your questions regarding task are on the point. But I believe handling this with a script might be not easy. If I were you, I would use Apache Nutch or similar open source software/library.I have used Nutch for my thesis for similar task that I had to scrap a lot of blog pages and the other pages they were referencing. You can configure... Source: over 1 year ago

I've never used it, but I was on a project where we considered Apache Nutch: https://nutch.apache.org/. Source: over 1 year ago

Ironically I had to use a search engine to discover what "Mwmbl" was. It's apparently a search engine. But, visiting the front page, I see something akin to a git commit log?! I'm not sure I'd have guessed that this was a SE if Brave Search did not tell me it was (even then I'm not convinced yet). https://mwmbl.org/. - Source: Hacker News / 5 months ago

How does the homepage of https://mwmbl.org/ not have a single sentence explaining what it is or even an "About" link? From Github:. - Source: Hacker News / 5 months ago

I wondered if this approach would be feasible for a distributed crawler: https://github.com/mwmbl/mwmbl#crawling (and, yes, another vote for changing the domain name; you can have a quirky project name, but if I can't remember the cat-walking-on-keyboard domain, I'm not going to use it). - Source: Hacker News / 8 months ago

This is what we're building at https://mwmbl.org. - Source: Hacker News / about 1 year ago