Scrapfly.io Reviews and details

Social recommendations and mentions

• Real challenge for web scrapers- Need Help!

  Try with https://scrapfly.io with JavaScript rendering enabled, and see if it works. Then means you can use proxies to scrape the site. But just to let you know, their proxies are expensive. But really fast. You have 1000 free credit to try. Source: 10 months ago

• Help progressing AWS lambda and playwright vs pivoting approach

  The question I have is am I going to face an issue once I have deployed the lambda and all its required dependencies? Along the line of ip blocking etc. At this point with all the moving parts would it be easier and maybe even cheaper to use something like https://scrapfly.io/? Source: 11 months ago

• Getting detected by Cloudflare for no apparent reason.

  As for solutions, you are on point. Running a headless browser or using a web scraping API that does that for you (I work at one: https://scrapfly.io hi) is the easiest way to do it. Note that because of javascript fingerprinting you still need to fortify your headless browsers with various scripts like puppeteer-stealth. Source: over 1 year ago

• Best Configuration/Infrastructure for...

  Alternatively, you can spend 30$ or something on a web scraping API (like Scrapfly, I work here) that runs cloud browsers for you and save you a significant headache :). Source: over 1 year ago

• Rotating Proxy Service for Many, Many Requests

  If you're only interested in getting the job done, then I'd recommend skipping all of this magic and using a web scraping API that manages the connection for you. I work at scrapfly.io and the cheapest plan should easily handle your use case :). Source: over 1 year ago

• cloudscraper module bypassing cloudfare protection running the program from linux but not working when running from windows

  Now, what could you do to fix this? Well first, you're quite lucky it works on your Linux machine which might not last that long - if you scale up your scraper a bit it's likely it'll start responding with 403s just like your Windows version. Then, getting around cloudflare anti-scraper protection is pretty difficult, it's possible but requires a lot of research: you probably need real, fortified browsers with... Source: over 1 year ago

• Need help scraping indeed for a project

  To get around that you need browser automation (via Selenium, Playwright or Puppeteer) with loads of patches and high-quality proxies or a web scraping API that does that for you like ScrapFly (disclaimer: I work here). Source: over 1 year ago

• Any updated methods of getting Instagram follower counts?

  Though take note that you might be requested to login if Instagram doesn't like you for whatever reason (they usually give you X no-login views per a certain amount of time). For scraping this at scale you'll need some proxies or web scraping APIs (like ScrapFly - I work here :). Source: almost 2 years ago

• Free cloud service for Web scraping.

  That being said, give it a shot - it's better than no proxies at all, usually. For small projects, I'm really fond of Webshare's free 10 proxies which are surprisingly stable and easy to work with (though have a low bandwidth limit). Other than that, another free resource you can take advantage of is ScrapFly's free plan which will get you up to 1000 requests/month (I work here :). Source: almost 2 years ago

• Deploying an indeed.com scraper to aws lambda/ec2

  Though blocking is such a huge subject that web scraping API such as ScrapFly (I work here) can save you a lot of time :). Source: almost 2 years ago

• Instagram doesn't show any content without login

  I'm not a fan of self advertising but that's the reason ScrapFly web scraping API was founded - because abstracting away proxy pool logic out of your web scraper just makes everything so much easier! Was this proxy used by someone or by your own web scraper? How to distribute proxies across your whole scraping stack etc becomes such an exhausting problem so it's much easier to separate it from your scraper - let... Source: almost 2 years ago

• Newbie question: why my first scraping encounters CAPTCHA, but it opens fine in browser?

  There are many ways to approach this. First, you can try httpx with http2 protocol enabled instead of requests. Otherwise you are most likely looking at either using a web scraping API (like ScrapFly I work here :P) or automating a web browser yourself. I have a short intro blog here on Scraping Using Browsers but to summarize you can use Selenium, Playwright or Puppeteer to take control of a real browser which... Source: almost 2 years ago

• Seeking Alpha Web scraping

  Solution like scrapfly.io performs very well, you can also try https://github.com/berstend/puppeteer-extra/tree/master/packages/puppeteer-extra-plugin-stealth on your own but not really maintained anymore and detected. Source: almost 2 years ago

• Im building we scraper and need proxies , How to check if proxy company is reliable ?

  Though I've used all sort of proxy scenarios from free proxies, to TOR and even recently I tried out cool new approach of using wireguard VPNs as proxies etc. There are a lot of interesting options out there but generally it's a _very time intensive subject_, so if you just want to scrape some data be wary of this and go with scraping API (like ScrapFly 💙) or at least good big name proxies. Source: almost 2 years ago

• Is it legal to make an open source GitHub repo showing how to scrape data from Realtor/Zillow.com?

  I've been using https://scrapfly.io/ to bypass security but it is a paid service. I'm open to other suggestions, and trying to convert my repo to let someone swap in their own web scraper security bypassers. Source: almost 2 years ago

• Avoid Response [412] using python requests package

  I'd definitely recommend either investing into 2captcha (1 captcha solve can get you quite far if you're doing things right) or take a look into web scraping APIs that solve this stuff for you like ScrapFly. Disclaimer: I work here, it's pretty awesome, you can try 1000 pages/month for free! :D. Source: about 2 years ago

• Large-scale web scraping?

  Hello 👋 Scrapfly co-founder here. If you are looking for a reliable service with a high success rate, you could try our API. :). Source: about 2 years ago

• Looking for webscraping services suggestions

  Hello, You can integrate our API (Scrapfly). We have built a Python SDK that will simplify your experience. If you have any questions, don't hesitate. Source: about 2 years ago

• Problem with Cloudflare Api shield

  Hello 👋 Scrapfly co-founder here. Bhphotovideo.com is working well with our API; You can test it for free 🙂. Source: about 2 years ago

• Can anyone suggest a afffordable webscraper. For my project?

  Hello 👋 Scrapfly co-founder here, With Scrapfly you can start with only $15 :) but you will need to code a bit. Source: about 2 years ago

• What are the best 5 Web Scrapping API/Tool to scrape data?

  - Scrapfly (Probably the best web scraping api on the market) - If you liked ScraperAPI you should test them - you will see the difference - Scrapy (Web Scraping Framework from Zyte) - Browserless (Automation). Source: over 2 years ago