Can someone help me understand why data batch processing and data streaming processing pose such different challenges in data management?

1

Google Cloud Pub/Sub

Because of these and other differences, the tools used are also different. With batch processing, data might be read from large files, processed, and stored in an OLTP (Online Transaction Processing) database (like MySQL) or OLAP (Online Analytical Processing) system (like Google BigQuery). But these would not be good solutions for streaming applications, because they are not optimized for high throughput on a lot of small transactions. In that case, a good choice of tools might be Pub/Sub (based on Apache Kafka) as a messaging queue, with BigTable (which is optimized for high throughput at low latency) for storage.

#Stream Processing #Data Integration #Web Service Automation 15 social mentions

2

Google Cloud Bigtable

Because of these and other differences, the tools used are also different. With batch processing, data might be read from large files, processed, and stored in an OLTP (Online Transaction Processing) database (like MySQL) or OLAP (Online Analytical Processing) system (like Google BigQuery). But these would not be good solutions for streaming applications, because they are not optimized for high throughput on a lot of small transactions. In that case, a good choice of tools might be Pub/Sub (based on Apache Kafka) as a messaging queue, with BigTable (which is optimized for high throughput at low latency) for storage.

#Databases #NoSQL Databases #Key-Value Database 6 social mentions