Scrapy Reviews and details

Social recommendations and mentions

• What is SERP? Meaning, Use Cases and Approaches

  While there is no specific library for SERP, there are some web scraping libraries that can do the Google Search Page Ranking. One of them which is quite famous is Scrapy - It is a fast high-level web crawling and web scraping framework, used to crawl websites and extract structured data from their pages. It offers rich developer community support and has been used by more than 50+ projects. - Source: dev.to / 5 months ago

• Creating an advanced search engine with PostgreSQL

  If you're looking for a turn-key solution, I'd have to dig a little. I generally write a scraper in python that dumps into a database or flat file (depending on number of records I'm hunting). Scraping is a separate subject, but once you write one you can generally reuse relevant portions for many others. If you can get adept at a scraping framework like Scrapy you can do it fairly quickly, but there aren't many... - Source: Hacker News / 10 months ago

• What do .NET devs use for web scraping these days?

  I know this might not be a good answer, as it's not .NET, but we use https://scrapy.org/ (Python). Source: 11 months ago

• BeutifulSoup and getting URLs

  Take a look at Scrapy. It has a fairly advanced throttling mechanism for you to not get banned. Source: 11 months ago

• Looking for a Python (or R) program or package to save only images from any plain vanilla website

  Not only Windows, you can also use it on Mac and Linux too. But for Python and CLI, you can use scrapy. Source: 12 months ago

• Automating Amazon Price Tracking with Python

  The first step in automating Amazon price tracking with Python is to scrape the product pages of Amazon.com for the desired product. To do this, you can use a web scraping library like BeautifulSoup or Scrapy. In the following example, we will use BeautifulSoup to scrape the product page for a MacBook Pro on Amazon.com:. - Source: dev.to / about 1 year ago

• WebScraping

  Lots of good suggestions here -- wanted to suggest the python tool, https://scrapy.org. Source: about 1 year ago

• What are the most underrated python libraries?

  Scrapy – one of the most comprehensive web scraping frameworks available for Python developers. Scrapy was designed around speed, reliability and extensibility – allowing users to quickly extract data from websites with minimal effort thanks to its powerful spiders that automatically traverse through whole sites from page-to-page until all relevant information has been scraped off them. Source: about 1 year ago

• Show HN: SiteGPT – Create ChatGPT-like chatbots trained on your website content

  Not to go full "Dropbox in a weekend", but if you're technical enough to self-host, this is something you can build for yourself Everyone is going straight to embeddings, but it'd be easy enough to use old school NLP summarization from NLTK (https://www.nltk.org/) Hook that up a web scraping library like https://scrapy.org/ and get a summary of each page. Then embed a site map in your system prompt and use... - Source: Hacker News / about 1 year ago

• Celery lock a variable between two processes

  In general celery tasks should be idempotent if possible, for scraping consider if Scrapy might not be more appropriate, it already implements a lot of the rate limiting/retrying you have to replicate in celery yourself. But regarding locking you are right to consider databases/redis since celery workers might run on entirely different machines even. In the case of a paginated scrape with celery, you could... Source: about 1 year ago

• fastest web scraping options

  You can use automation tools like Selenium or Playwright. You can work with a full-fledged framework such as Scrapy. I also recently discovered a Python tool like selectolax Lexbor, which allows you to extract data very quickly. Source: about 1 year ago

• Scrapy extension blocking login to AVer PTZ Camera (CAM520 Pro)

  This is not related to https://scrapy.org/ and so not related to this subreddit either. Source: about 1 year ago

• What steps do you apply in the "L" when doing ELT?

  The sha256 is there establish the uniqueness of the file. It isn’t great for capturing whether or not you have already seen the file before, tho, because it is rather expensive to calculate (imagine your csv file were gigabytes on size — you would have to stream in whole file down in order to see if it had changed!). In the past I have used a sha256 of information that the server hosting the file gives me about... Source: about 1 year ago

• How to run webs scraping script every 15 minutes

  You may want to check out [estela](https://estela.bitmaker.la/docs/), which is a spider management solution, developed by [Bitmaker](https://bitmaker.la) that allows you to run [Scrapy](https://scrapy.org) spiders. Source: about 1 year ago

• Extracting JSON data

  Hi, in this case the data is in the html itself (no data.json). You can use this xpath to get the data: //div[@id="vue-match-centre"]/@q-data There are many ways to get this info, the one my company uses is the scrapy framework. Here is some code that uses scrapy to get this data into a json file:. Source: about 1 year ago

• What Python library is the best to scrap from OpenCritic?

  I recommend using Scrapy as that is what we use at my place of work, Bitmaker (bitmaker.la). An example spider would look like this:. Source: about 1 year ago

• Is there a program available for bulk image reverse searching?

  In the past I used stuff like beautifulsoup for webscraping but I’ve heard good things about https://scrapy.org/. Source: about 1 year ago

• How I used Scrapy for my ML Project

  I wanted to invest my time and energy in learning the fastest, most efficient one, that can scale with my as my projects get more and more complex scrapy. After all, I want my projects to shine so bright in my cv it blinds the recruiter's eyes. - Source: dev.to / about 1 year ago

• scrapy.Request(url, callback) vs response.follow(url, callback)

  The fist question I am asking is merely for general understanding. And for the record, the second sentence on the 'Requests and Response' section of scrapy.org is:. Source: over 1 year ago

• Looking for a good open source web scraping tool

  Scrapy was a good option when I had to use it, built with Python and built-in framework to scrape lots of pages (crawling) https://scrapy.org. Source: over 1 year ago

• Anybody here versed in AI, machine learning, deep learning, etc.?

  You can use 'web scraping' if it's legal, to then scrape all the data from the web, into text, then get all of that data. How you do this, is if you're using chrome, right-click on a particular value in the table, then click 'inspect'. Then it will direct you to the HTML for that element in 'developer tools'. Right-click that HTML responsible for it, then do 'copy', then you will see a list of options, I like to... Source: over 1 year ago