RabbitMQ dude! Spin up a free instance on cloudamqp.com (not affiliated, but I use them in prod - free instances already are enough for many use cases), throw a bunch of consumers in the queue, push as many jobs as you want and you'll have an awesome indestructible working process. Source: about 2 years ago

We can prefer to install RabbitMQ on our local machine but in that case, the installation steps would be different from the operating system to the operating system and we would need to mess with some network settings. Therefore, we make this step with http://cloudamqp.com. To do that let's create https://cloudamqp.com. Later on, click on the button Create New Instance and create a new message broker instance. We... - Source: dev.to / about 3 years ago

I have also a spark cluster created with google cloud dataproc. Source: about 1 year ago

Specifically, we heavily rely on managed services from our cloud provider, Google Cloud Platform (GCP), for hosting our data in managed databases like BigTable and Spanner. For data transformations, we initially heavily relied on DataProc - a managed service from Google to manage a Spark cluster. - Source: dev.to / about 2 years ago

With that, the best way to maximize processing and minimize time is to use Dataflow or Dataproc depending on your needs. These systems are highly parallel and clustered, which allows for much larger processing pipelines that execute quickly. Source: over 2 years ago