Stealth Web Scraping in Python: Avoid Blocking Like a Ninja

1

Instagram

Import sys Import requests Session = requests.session() Response = session.get('http://instagram.com', allow_redirects=False) Print(response.status_code, response.headers.get('location')) For redirect in session.resolve_redirects(response, response.request): location = redirect.headers.get('location') print(redirect.status_code, location) if location and "accounts/login" in location: sys.exit() # no need to exit, return would be enough # 301 https://instagram.com/ # 301 https://www.instagram.com/ # 302 https://www.instagram.com/accounts/login/.

#Social Media Apps #Social Network #Photos 66 social mentions

2

puppeteer

While avoiding them - for performance reasons - would be preferable, sometimes there is no other choice. Selenium, Puppeteer, and Playwright are the most used and known libraries. The snippet below shows only the User-Agent, but since it is a real browser, the headers will include the entire set (Accept, Accept-Encoding, etcetera).

#Automated Testing #Browser Testing #Software Development 102 social mentions