Playwright VS Scrapy

Playwright Reviews

Scrapy Reviews

Playwright mentions (231)

• Typed E2E test IDs

  We start with a project that was bootstrapped with npx create-next-app. For the E2E test we use Playwright and set it up as described in the testing guide provided by Next.js. - Source: dev.to / 1 day ago

• Playwright Scraping infinite loading & pagination

  Playwright is a powerful tool developed by Microsoft, it allows developers to write reliable end-to-end tests and perform browser automation tasks with ease. What sets Playwright apart is its ability to work seamlessly across multiple browsers (Chrome, Firefox, and WebKit), it provides a consistent and efficient way to interact with web pages, extract data, and automate repetitive tasks. Moreover, it supports... - Source: dev.to / 2 days ago

• Sometimes things simply don't work

  The consensus I could gather is either use playwright or use a workaround to solve it in the puppeteer layer. The root cause of the bug is a websocket size limitation on the CDP protocol for chromium. - Source: dev.to / 10 days ago

• The best testing strategies for frontends

  With the advent of tools like Puppeteer and now Playwright, end-to-end testing has become much easier and more reliable. For anyone who's used Selenium in the past, you know what I'm talking about. Puppeteer has opened the way in terms of E2E tooling, but Playwright has taken it to the next level and made it easier to await for certain selectors or conditions to be fulfilled (via locators), thus making tests... - Source: dev.to / 11 days ago

• Playwright Web Scraping 2024 - Tutorial

  In this tutorial, our main focus will be on Playwright web scraping. So what is Playwright? It’s a handy framework created by Microsoft. It's known for making web interactions more streamlined and works reliably with all the latest browsers like WebKit, Chromium, and Firefox. You can also run tests in headless or headed mode and emulate native mobile environments like Google Chrome for Android and Mobile Safari. - Source: dev.to / 14 days ago

Scrapy mentions (93)

• What is SERP? Meaning, Use Cases and Approaches

  While there is no specific library for SERP, there are some web scraping libraries that can do the Google Search Page Ranking. One of them which is quite famous is Scrapy - It is a fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pages. It offers rich developer community support and has been used by more than 50+ projects. - Source: dev.to / 5 months ago

• Creating an advanced search engine with PostgreSQL

  If you're looking for a turn-key solution, I'd have to dig a little. I generally write a scraper in python that dumps into a database or flat file (depending on number of records I'm hunting). Scraping is a separate subject, but once you write one you can generally reuse relevant portions for many others. If you can get adept at a scraping framework like Scrapy you can do it fairly quickly, but there aren't many... - Source: Hacker News / 10 months ago

• What do .NET devs use for web scraping these days?

  I know this might not be a good answer, as it's not .NET, but we use https://scrapy.org/ (Python). Source: 11 months ago

• BeutifulSoup and getting URLs

  Take a look at Scrapy. It has a fairly advanced throttling mechanism for you to not get banned. Source: 11 months ago

• Looking for a Python (or R) program or package to save only images from any plain vanilla website

  Not only Windows, you can also use it on Mac and Linux too. But for Python and CLI, you can use scrapy. Source: 12 months ago