RabbitMQ dude! Spin up a free instance on cloudamqp.com (not affiliated, but I use them in prod - free instances already are enough for many use cases), throw a bunch of consumers in the queue, push as many jobs as you want and you'll have an awesome indestructible working process. Source: about 2 years ago

We can prefer to install RabbitMQ on our local machine but in that case, the installation steps would be different from the operating system to the operating system and we would need to mess with some network settings. Therefore, we make this step with http://cloudamqp.com. To do that let's create https://cloudamqp.com. Later on, click on the button Create New Instance and create a new message broker instance. We... - Source: dev.to / about 3 years ago

The information can be stored in a database or as files, serialized in a standard format and with a schema agreed with your Data Engineering team. Depending on your information and requirements, it can be as simple as CSV, XML or JSON, or Big Data formats such as Parquet, Avro, ORC, Arrow, or message serialization formats like Protocol Buffers, FlatBuffers, MessagePack, Thrift, or Cap'n Proto. - Source: dev.to / over 1 year ago

Data formatting is another place to make gains. When dealing with huge amounts of data, finding the data you need can take up a significant amount of your compute time. Apache Parquet and Apache ORC are columnar data formats optimized for analytics that pre-aggregate metadata about columns. If your EMR queries column intensive data like sum, max, or count, you can see significant speed improvements by reformatting... - Source: dev.to / over 2 years ago

The following stack captures layers of software components that make up Hudi, with each layer depending on and drawing strength from the layer below. Typically, data lake users write data out once using an open file format like Apache Parquet/ORC stored on top of extremely scalable cloud storage or distributed file systems. Hudi provides a self-managing data plane to ingest, transform and manage this data, in a... - Source: dev.to / almost 3 years ago