Apache Kafka Use Cases: When To Use It & When Not To

1

Apache Kylin

A Kafka-based data integration platform will be a good fit here. The services can add events to different topics in a broker whenever there is a data update. Kafka consumers corresponding to each of the services can monitor these topics and make updates to the data in real-time. It is also possible to create a unified data store through the same integration platform. Developers can implement a unified store either using an open source data warehouse like Apache Kylin or use a cloud-based one like Redshift or Snowflake. In this instance, the organization uses BigQuery. Data to this warehouse can be loaded through a separate Kafka topic. The below diagram summarizes the complete architecture.

#Databases #Big Data #Data Dashboard 1 social mentions

2

Grafana

Clickstream events are also stored in a searchable form for further analysis later. Elastic search is a common candidate for such a data store because of its quick retrieval features. Grafana, an open-source visualization analytics framework is a good fit for implementing the dashboard since there are no complex requirements here to justify developing a custom dashboard. Grafana integrates well with Elastic search and is a defacto framework for visualization for complex requirements. Developers can use the KSQL DB queries in Kafka to implement real-time features like the number of page views over the last ten minutes or the number of errors over a time period. The complete architecture will look as shown below.

#Data Dashboard #Data Visualization #Data Analytics 197 social mentions

3

Google BigQuery

A Kafka-based data integration platform will be a good fit here. The services can add events to different topics in a broker whenever there is a data update. Kafka consumers corresponding to each of the services can monitor these topics and make updates to the data in real-time. It is also possible to create a unified data store through the same integration platform. Developers can implement a unified store either using an open source data warehouse like Apache Kylin or use a cloud-based one like Redshift or Snowflake. In this instance, the organization uses BigQuery. Data to this warehouse can be loaded through a separate Kafka topic. The below diagram summarizes the complete architecture.

#Data Management #Data Warehousing #Data Dashboard 35 social mentions

4

Amazon Redshift

A Kafka-based data integration platform will be a good fit here. The services can add events to different topics in a broker whenever there is a data update. Kafka consumers corresponding to each of the services can monitor these topics and make updates to the data in real-time. It is also possible to create a unified data store through the same integration platform. Developers can implement a unified store either using an open source data warehouse like Apache Kylin or use a cloud-based one like Redshift or Snowflake. In this instance, the organization uses BigQuery. Data to this warehouse can be loaded through a separate Kafka topic. The below diagram summarizes the complete architecture.

#Big Data #Databases #Relational Databases 26 social mentions