What is the separation of storage and compute in data platforms and why does it matter?

1

Apache Spark

However, once your data reaches a certain size or you reach the limits of vertical scaling, it may be necessary to distribute your queries across a cluster, or scale horizontally. This is where distributed query engines like Trino and Spark come in. Distributed query engines make use of a coordinator to plan the query and multiple worker nodes to execute them in parallel.

#Databases #Big Data #Big Data Analytics 56 social mentions

2

Apache Parquet

Apache Parquet is a file format that is used to store data in a columnar format. This allows for faster reads and reduced disk space. Not only is the Parquet format open source, but it also has an entire ecosystem of tools to help you create, read and transform data.

#Databases #Big Data #Relational Databases 19 social mentions

3

Amazon S3

Object Storage: Services like AWS S3 can be used to store files on-demand with "infinite storage". There is no need to provision or manage capacity and it is completely decoupled from the instance or serverless that is doing the processing.

#Cloud Hosting #Object Storage #Cloud Storage 172 social mentions