Running a scrapy spider in the background in a Flask app

1

Scrapy

I'm building an app that uses Flask and Scrapy. When the root URL of my app is accessed, it processes some data and displays it. In addition, I also want to (re)start my spider if it is not already running. Since my spider takes about 1.5 hrs to finish running, I run it as a background process using threading. Here is a minimal example (you'll also need testspiders):.

#Web Scraping #Data Extraction #Data 93 social mentions

2

Flask

I'm building an app that uses Flask and Scrapy. When the root URL of my app is accessed, it processes some data and displays it. In addition, I also want to (re)start my spider if it is not already running. Since my spider takes about 1.5 hrs to finish running, I run it as a background process using threading. Here is a minimal example (you'll also need testspiders):.

#Python Web Framework #Web Frameworks #Developer Tools 42 social mentions